Damien Mannion

I have been thinking more about usefulness of interactive online activities in learning and understanding, inspired by the great examples given by:

• The wonderful articles by Bartosz Ciechanowski.
• The demonstrations of retinotopic mapping and study results from the Gallant lab.
• The articles in the Distil journal.
• The general concept of explorable explanations.

Recently, a colleague mentioned that an interactive way of exploring the Fourier analysis of images could be a useful teaching tool. Implementing such an activity on a website sounded interesting, so I thought I would have a go at putting one together.

The result is the “Image frequency spectrum analysis” site. The site and its code available in a public Github repository. The main features of the site are:

• All calculations performed entirely within the browser.
• Includes example natural and synthetic images.
• Custom images can be uploaded or grabbed from a webcam.
• Windowing can be applied to the input image.
• Amplitude spectrum displayed on either log-polar or (zoomable) Cartesian axes.
• Optionally shows the spatial frequency slope.
• Can set low-pass, high-pass, or band-pass filters and see the reconstructed output.
• Output can be saved to the local computer.

Here is a screenshot of the site:

A few things I have learned, recognised, or appreciated while putting this together:

• Coding these sorts of interactive activities requires a different approach than I would apply to the tasks that I am more familiar with. The sort of code I have typically written is to perform a single task, such as running a data analysis, creating a stimulus, or handling an experiment. While there is sometimes the need to provide options that affect execution at runtime, the code can mostly proceed fairly sequentially once those options have been parsed. However, interactive activities require the code to be able to respond to a diverse set of user events. The code structure and planning is more about managing state and understanding what is and isn’t affected by a particular action.
• Constraints are an important part of learning from interactive activities. I have been an advocate of learning a general-purpose programming language as a way of unlocking a means of deeply exploring topics of interest. However, lately I have been thinking about how constraints can be important during learning. For an example from the current context, someone who knows Python could be provided with sample code that loads an image, runs a Fourier transform, etc.—from which they can then design their own exploratory path. While I still think that such an approach would ultimately allow a deeper understanding (along with the many generalisable benefits of knowing a programming language), it comes with the downsides that they would need to keep details of the computation and implementation in mind and that the dimensions of the topic that are likely to be fruitfully explored can be less obvious. A simple, constrained interactive activity can be a very useful adjunct.
• Debugging sites on mobile devices is difficult. The site was developed for non-mobile use (I am not a frequent user of mobile devices so they tend to be an afterthought with me—something I will need to amend), but I wanted it to at least work as much as possible on mobile devices. When I first tried, I found that the input image would not display on Firefox and Chrome on mobile (oddly, Safari on mobile worked fine). I was a bit stuck on how to go about debugging this issue. There didn’t seem to be a way of accessing the console to see if there were any error messages, and most methods unearthed from an internet search weren’t very Linux friendly. I ended up learning that the console can be accessed (sort of) in mobile Chrome by opening a tab to chrome://inspect. This allowed me to narrow it down to the createImageBitmap function as being the problem, but I couldn’t work out precisely why it was failing—so I did some rewriting to avoid using this function. I also received a permissions error when trying to capture from the camera, despite having allowed camera access—that one remains unfixed (luckily, uploading a file works just as well on mobile devices).
• Good multidimensional array functionality is very useful. I am very familiar with the multidimensional array support in Python that is provided by numpy (see Harris et al., 2020, for an overview), and have often wished for something similar in JavaScript. Happily, I came across the scijs set of packages—while very different from numpy in a lot of ways, it has a similar-enough foundation to be very useful. It did, however, require me to go beyond the vanilla JavaScript approach that I am familiar with and instead use the node.js ecosystem. After a bit of getting used to, it wasn’t too bad and I am glad to have learned at least the basics of it.
• I’m not sure about the best way to separate HTML and JavaScript for sites like this. I am a big fan of the HTML format and enjoy using it, and I like and try to follow the separation of concerns philosophy. However, the best way to do this for sites like this is a bit unclear to me. Things like the canvas objects and the form widgets seem to be more naturally created within the JavaScript code, but it also seems more prudent to define them in the HTML. I find it a bit awkward to move between the two concerns.
• I still struggle with CSS. For whatever reason, I don’t find CSS to be very intuitive and always struggle to use it properly. That struggle continued with this project, but I feel like I at least made some progress. An important realisation is that the more clearly and logically the structure of the site is expressed in the HTML, the easier it is to apply CSS.
• Working with a webcam is surprisingly easy on the modern web. I thought that allowing users to capture an image directly from their webcam would be great functionality for the site to have—but I also thought that implementing such a feature would be very difficult. I was pleasantly surprised to find that it is actually pretty straightforward in modern web browsers.
• The MDN web docs are a fantastic resource. It is rare to find such a comprehensive, up-to-date, well-written, and well-structured set of documentation. I find it to be tremendously useful.

Overall, I really enjoyed and feel like I learned a lot from this project—and I hope that it may be a useful teaching resource.

References

1. Harris, C.R., Millman, K.J., van der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., del Río, J.F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., & Oliphant, T.E. (2020) Array programming with NumPy. Nature, 585, 357–362.