The answers have been edited for length and clarity.
What papers have changed your life?
Surrogate gradient learning in spike neural networks. Neftci Eo, Mostafa H, Zenke F. IEEE Signal Processing Magazine (2019) and Spytorch Tutorial
This paper shows how to use modern machine learning in a spike neural network, a network of model neurons that communicate through discrete action potentials or spikes, like real neurons. When it came to it, spiked neural networks were obsolete because they were computationally easy to work with, but unlike artificial neural networks that don't spike like real neurons, they were not compatible with machine learning. The problem was that the spikes are separate events and therefore the key step of gradient descent, a critical step of gradient descent, an important algorithm used in machine learning. In this paper, we presented some clever tricks the author calls “surrogate gradients.” This method bypasses the issue of differentiability and allows you to use the full power of modern machine learning in spiked neural networks. A similar advance over previous work was that it was simply made out of the box. The author also provided very simple example code (available in the Spytorch tutorial) that allows you to install in 10 minutes and explore other research questions in a few hours. This paper rekindled the field of spike neural networks. This is progressing faster than it was decades ago.
When was your first encounter with this paper?
I tried to solve it accurately when I first read it, but thanks to Outlook's terrible search feature, I was unable to determine the exact date. I know I met author Friedemann Senke and talked about it in November 2017, but Spytorch wasn't released until January 2019. I think I read it for the first time between June and September of that year.
Why does this paper mean to you?
I began a career in neuroscience working on a surge in neural networks. I wrote the “Brian” spike neural network simulator software package and then created a spike model for sound localization. Over time, I moved away from the spikes, especially as the power of machine learning became clear. Spytorch combines two main strands of my research, spike neural networks and machine learning, to eat and eat my cake. This is one of the main tools I've been using since it was announced (though there are now more advanced software packages).
How did this study change how we think about neuroscience and how we challenge previous assumptions?
I hypothesized that we must first understand neuroplasticity in order to make progress in understanding how neural networks work. In this paper, I felt that it was the first time I could approach neural computational problems with spikes simply by training spikes to recognize and speak to visual gestures, make multimodal decisions in noisy environments, and look at the types of solutions found. Of course, we still need to think about it a little, but it's much easier than before.
How did this study affect your career path?
I didn't have the right tools, so I was floating away from spikes through neural networks. I think it's safe to say that the day I passed the Spytorch tutorial completely changed the direction of my career path. I don't think it's just me. For me, this is a model of impactful science, and I find it a bit sad that contributions like the tutorials are not formally recognized by the academia on a more personal level. that's nice.
Are there any underrated aspects of this paper that other neuroscientists should know?
I don't think many people in neuroscience understand how easy it is to do this kind of research. I also think we may have to answer old-fashioned questions about relating neural mechanisms to computational roles or functions. In the first paper I wrote using this tool, I had to change two lines of code in the Spytorch tutorial. This will answer the question of why neurons are so heterogeneous. Ok, I'm exaggerating a bit. In the end, we, of course, created much more code than that. The point is that these tools are so good that this type of research is now very accessible and that more people should do it.
