Damien Mannion

Most programming languages have tools that can help to identify errors and to warn about potentially problematic code—often called ‘linters’. They will always be imperfect, but routinely using such tools is a great and inexpensive (in time and effort) way to pick up on potential issues. A few linting tools for different languages are pylint (Python), mlint (Matlab), eslint (JavaScript), and lintr (R).

Most programming languages also have tools that will automatically format code to follow a particular set of conventions. This is particularly useful in spotting problems because of how it facilitates visual pattern recognition—by standardising the formatting and presentation of the code, it makes the content more visible and helps to highlight any unusual elements. It is also very useful for code review by removing any distracting debates about formatting preferences.

I particularly like black for Python because it doesn’t really have any options—you just have to accept its conventions. This took a bit of getting used to at first as I learned to get over some of my formatting preferences, but the benefits of consistent formatting strongly outweigh such concerns. For other languages, there is prettier (JavaScript), MBeautifier (Matlab), and styler (R).

Notebooks (e.g., jupyter) have become a common way to write code and allow it to be interleaved with prose and graphical output. The use of such notebooks have many appealing qualities, particularly for tutorials and demonstrations. However, the way that they are typically utilised can introduce potential issues. For example, Pimentel, Murta, Braganholo, & Freire (2021) highlight “hidden states, unexpected execution order with fragmented code, and bad practices in naming, versioning, testing, and modularizing code” (p. 2).

An awareness of the strengths and weaknesses of your chosen programming language will allow you to capitalise on the former and exercise caution around the latter. For example, the ability to use named function arguments is a great feature of Python. It allows me to call a function via something like np.std(a=data, axis=0), rather than just np.std(data, 0). Capitalising on this strength of Python would mean using these named arguments wherever possible. Conversely, if I am using a language that doesn’t have this feature, then I would need to be particularly careful that the order in which I supply the function arguments matches the order of function parameters.

Part of this is also choosing your data representations and abstractions carefully. For example, I typically find multi-dimensional arrays to be a great way to represent the data that are part of my typical workflows. For a long time, I based this representation around the ndarray of numpy. I was always concerned, though, that my code would often include statements like:

mean = data.mean(axis=2)

That is, I would need to keep track of the axis indices to be able to specify which axis to operate along. This was always worrying, and I would often include many checking statements to make sure that the shape of the array was as I had expected.

I have since moved to mostly using the wrapper over numpy that is provided by xarray. This allows the use of named dimensions, which makes such operations much clearer and less error-prone:

mean = data.mean(dim="ppant")

Most programming languages have something equivalent to an assert statement, in which a logical expression is stated and an error emitted if it evaluates to being false. This is a great way to document and enforce your expectations about the state of something in your code at a particular point (such as those described above in Known constraints and consequences).

For example, let’s say that I use a convention where each participant in an experiment is assigned an identifier that begins with the character ‘p’—such as "p1001". At some point in my Python code, I am working with such an identifier in a variable called ppant_id. To express my expectation about the structure of this identifier, I could include something like:

assert ppant_id.startswith("p")

This will raise an error if the contents of the variable does not match this requirement.

assert (0.1 + 0.2) == 0.3

assert np.isclose(a=(0.1 + 0.2), b=0.3)

Including comments in code does not necessarily make the code more understandable or reliable. Comments can sometimes just re-describe what is being implemented in the code, but with less precision and with the vulnerability to becoming outdated if the code changes. Instead, aim to make the code as clear as possible without requiring comments, by using informative variable names and expression structures, and reserve comments for explaining why the code has its particular form.

For example, say if you are running a study in which participants respond to a prompt about whether they completed the study with appropriate diligence or not. A participant accidentally selected ‘no’ when they meant to select ‘yes’, and they immediately email you to report their mis-click. You decide to modify the data when it is being loaded to reflect their diligence as reported in the email (it is perhaps debatable about whether this is an appropriate response), and you include something in the function that loads the data like:

if participant_id == "p1001":
    diligent = True

You might decide to add a comment to this code:

# set p1001's diligence to true
if participant_id == "p1001":
    diligent = True

However, this comment doesn’t really add anything beyond what is already being expressed in the code. Instead, a better use of comments here would be to give some context and explain why the code is there:

# participant p1001 emailed on 2022/1/6 to say that
# they mis-clicked on the diligence question
if participant_id == "p1001":
    diligent = True

# participant p1001 emailed on 2022/1/6 to say that
# they mis-clicked on the diligence question
if participant_id == "p1001":
    assert not diligent
    diligent = True

A useful approach for preparing comments is to take the perspective of somebody (perhaps your future self) who has access to the code but does not have any additional information or context. If something in the code is surprising or has an unclear rationale under these conditions, it is likely to be a good candidate for a comment explaining why your current self, with your additional contextual knowledge, wrote the code in this way.

A common and productive way to develop research code is to combine writing code into a file (e.g., analysis.py) with interactive exploration via a read-eval-print loop (REPL) shell such as ipython. This approach is facilitated by structuring the code so that it is able to be easily explored in the interactive shell.

For example, say you are writing code to implement a data analysis workflow. You might structure your code as a single monolithic function:

def run_analysis():

    # load the data
    # ...

    # compute descriptive statistics
    # ...

    # compute inferential statistics
    # ...

This structure makes it difficult to isolate the different components of the analysis so that they can be explored individually. A good indicator of this difficulty is if you find yourself inserting temporary return statements into the function so that you can explore variables that are defined within it.

Instead, it is often useful to break it up into a set of individual functions with much more limited responsibilities; e.g., load_data, calc_descriptives, and calc_inferential—each of which may themselves be decomposed into separate functions. This increase in modularity makes exploration easier, and is also likely to improve the overall understandability of the code.

There will inevitably be times in which you recognise a failure of your workflow to reproduce your intention. It is very useful to not ‘waste’ such situations and instead use them as an opportunity to reflect on how the mistakes happened and how they can be avoided in the future. Without this mental footprint, it is likely that you will either directly repeat the mistake in the future or else infer some incorrect causal mechanism. It is particularly important to avoid being satisfied by being able to resolve the mistake but without knowing why or how.

I have made many programming mistakes that stand out in my mind, but here are a few specific examples:

• I was coding an experiment that involved interacting with an external audio device. For the initial version of the experiment, I was communicating with the device over the parallel port and this imposed a limitation on the available number of audio tracks that could be played (about 250). Mindful of this restriction, I stored the track numbers in an array with an 8-bit unsigned integer data type (i.e., a data type that could store values between 0 and 255). This limit later became overly restrictive, so we moved to a serial communication method that allowed many more tracks. But, unfortunately, I didn’t update the data type of the array. This had the consequence that when I would do something like track_nums[2] = 300, the value stored inside track_nums[2] would be 44 rather than 300 (an integer overflow). Luckily, I caught this problem during piloting—but I now avoid using such specific data types unless necessary (e.g., for images).
• I was selecting some random image indices for display in a vision experiment and hard-coding them into the script. There were quite a few of them, and I had each index on a different line in the code. To make them line up nicely, I zero-padded the numbers so that they always had three digits (e.g., 32 was written as 032). Unbeknownst to me, these leading zeros actually caused Python to interpret the numbers as being in an octal numbering system—so my 32 was being stored as a 26. Thankfully, this behaviour was changed in Python 3 to instead raise an error. This taught me that programming languages can have some surprising quirks, and the importance of interactive exploration of data structures.
• I had written some code to present visual stimuli in a functional magnetic resonance imaging (fMRI) scanner. I was participating in a pilot version of the experiment inside the scanner when I noticed that there was a timing bug in the code that had the effect of the visual presentation becoming subtly but increasingly out of sync with the scanner acquisition over time. Recognising this bug made me very annoyed with my mistake—this annoyance then had physical effects on me that caused the goggles that I was wearing to fog up so I couldn’t see the screen! This bug thus had both direct and indirect consequences for the validity of the data. It was a good lesson in the subtleties of timing, particularly across different display systems, and the importance of experiencing experiments yourself where possible.

Although we are likely to have a preferred language, we can sometimes need to write, edit, or review code in other languages. For example, I prefer Python but need to be able to write in JavaScript for web situations, Matlab to collaborate with colleagues, and R to cross-check analyses under different approaches.

Unfortunately, switching between languages can introduce the potential for errors by incorrectly applying different language conventions. For example, I recently ruined a web experiment that I had written in JavaScript because I forgot to increment a counter variable. A likely contributor to this mistake was that I was still thinking in a Python-based approach, in which loops are typically constructed in a way that don’t require counters. I also spent a long time in R recently trying to work out why I was receiving an obscure (to me) error message—until I remembered that R uses one-based indexing, unlike Python and JavaScript.

The above example about the mistake in the web experiment also exemplifies the role of Intuition described earlier. The data from this experiment had conformed to reasonable expectations, except for one aspect that just didn’t seem quite right. We had noticed it earlier but could not come up with any explanation for what sort of error in the code could have caused it. But the nagging feeling persisted and prompted me to do a thorough code audit, when the mistake was discovered.

Other than simply being mindful that you are working in a different programming language than usual, it can also be helpful to include other contextual cues when switching. For example, you might consider using different text editors for different languages. Additionally, when switching between languages that have elements with options, you might make your selections so that they are different between languages. For example, when I began switching my dominant language from Matlab to Python, I made the conscious decision to use the " symbol rather than the ' symbol for strings (both of which are acceptable in Python) because I was using ' in Matlab. For a similar reason, I name my variables using camelCase in JavaScript and snake_case in Python.

It is often interesting to come back to code that you wrote a while ago—it can be surprising how much that was seemingly clear at the time is no longer readily understandable. In addition to being a good opportunity to reflect on the choice of coding structures and the use of comments so as to improve future intelligibility, this is also an indicator that caution is needed when making any changes to the code.

Responding to peer review of a manuscript can be a particularly dangerous time for introducing mistakes into a code base. A few months or more have typically passed since you last looked at the code, and now it needs to be changed to be able to address issues that are raised in peer review. By their nature, there is a good chance that the code wasn’t initially structured to be able to readily implement such changes. They need to be handled with particular care.