In this special episode of the Brains and Machines podcast, Dr. Sunny Bains and Dr. Giulia D’Angelo talk to four early career researchers: Dr. Kenneth Stewart, a computer scientist at the U.S. Naval Research Laboratory in Washington DC; Dr. Laura Kriener, a postdoctoral researcher at The University of Bern in Switzerland; Jens Pedersen, a Ph.D. student at The Royal Institute of Technology (KTH) in Stockholm, Sweden; and Dr. Fabrizio Ottati, an AI/ML computer architect at NXP Semiconductors in Hamburg, Germany. They discuss learning rules for spiking neural networks, primitives for computations on neuromorphic hardware, and the benefits and drawbacks of neuromorphic engineering.
[FULL TRANSCRIPT BELOW]
Taylor Marvin (TV): You are listening to EE Times On Air, and this is EE Times Current. For Eric Singer, I’m Taylor Marvin. Welcome to Brains and Machines, a deep dive into neuromorphic engineering and biologically inspired technology. In this special episode, Dr. Sunny Bains of the University College London and Dr. Giulia of the Czech Technical University in Prague talk to four early-career researchers about their work.
Giulia D’Angelo (GDA): Welcome to Brains and Machines. I am Giulia D’Angelo…
Sunny Bains (SB): … and I’m Sunny Bains.
GDA: In today’s episode, Sunny and I will be talking to Dr. Kenneth Stewart, Dr. Laura Kreiner, Jens Pedersen, and Dr. Fabrizio Ottati about learning rules for spiking neural networks, using primitives to represent computations to be implemented on neuromorphic hardware, and the advantages and disadvantages of neuromorphic engineering.
The four approach the problem from different points of view. Kenneth and Laura strive to understand the important roles involved in learning networks, while Fabrizio and Jens are addressing problems and providing solutions at deeper layers in the technology stack.
SB: Our first guest is Dr. Kenneth Stewart, a computer scientist at the U.S. Naval Research Laboratory in Washington, DC. He’s been working on one-shot learning, few-shot learning, and learning to learn. I caught up with him at the CapoCaccia Workshop towards Neuromorphic Intelligence last year.
Kenneth Stewart, welcome to Brains and Machines. Can you tell me a little bit about your academic background and how you got interested in machine learning and, specifically, neuromorphic approaches to machine learning?
Kenneth Stewart (KS): So I was always interested in robots and artificial intelligence and this idea of bringing intelligence and cognition to computers. And so I did an undergrad—I did a minor in cognitive science, I took courses in artificial intelligence and machine learning, and got involved in undergraduate research in projects related to machine learning as a senior. I saw where the gaps were, right? At the time (it was like 2016–2017, when deep learning was starting to take off), people were looking at these convolutional neural network models. And those were heavily inspired by our own visual systems, but they could only learn from what data you gave it. What it was missing was that sort of dynamic, or interactive, aspect of learning.
And that’s where I stumbled upon neuromorphic computing—when I was trying to think about where to go for grad school. And so I found UCI and I found Emre Neftci, and he’s been a long-time researcher in neuromorphic, so I thought, “Oh, it’d be really cool to work for him.” And, fortunately, it worked out where I got to do that!
SB: So, talk about that project and how it eliminated these gaps.
KS: So I did this really interesting undergraduate research project with a professor in computer science. Her goal was to make a robot that could learn tasks from a human—the target was these household tasks, right? Like folding a t-shirt or, say, making tea.
I was looking a lot, at the time, at computer vision; I would make a computer vision model that the robot could then understand, visually, the world. And then I worked with a couple undergraduates who were looking at natural language processing. And so the idea was to combine our expertise to create a system where a human could, by interacting with certain objects on a table, be able to teach a robot how to do some sort of recipe, like making tea. So the human would explain, “When I pick up this kettle, I pour water into a cup.” And so, by doing those actions, the robot could then learn. But the problem at the time was that, with deep learning, they can’t really do this sort of interactive online learning that is required for a task like this. It only knows what it knows at the time of deployment. And so that’s the gap that I was looking at trying to fill in my PhD.
SB: So you went to work at UC Irvine with Emre Neftci, and he was working on surrogate gradient descent and spiking networks. Explain now how surrogate gradient descent was a really important idea and made things possible in spiking networks that weren’t possible before.
KS: One of the successful things in deep learning is you have to do a gradient descent to learn. Like, it’s basically solving this optimization problem. You have these multiple layers in a neural network. There’s an output of an error, and then you have to basically have a way to communicate to each of the different neurons and synapses in that network how to change or how to adjust their plasticity based on that error. And so if you have this multilayer network, you need to be able to go back through the layers and update them appropriately.
SB: And people will know that as backpropagation, right? It’s a commonly understood concept in neural networks generally, but it’s not—or it wasn’t previously—applicable to spikes.
KS: Yeah, that’s correct. And the reason for that is, a spike is not normally differentiable, right? It’s just this event in time. And so you can’t differentiate that. If you do, it just goes to zero and there’s no information. So the idea behind surrogate gradient descent is that, rather than just directly taking the gradient of a spike, you basically create what’s called a surrogate function that is differentiable. It still captures the dynamics of the neurons and the spiking networks, so that you can still use the same principles behind gradient descent for training, but on this non-differentiable framework.
SB: Right. And so that has been important in neuromorphic engineering in that it’s made these backpropagation techniques—that have already been very successful—it’s made them sort of accessible, now, to those who are working with spiking networks, which are potentially lower power and work with the underlying physics and the electronics rather than treating everything as a symbol and as a number.
So one of your contributions has been related to one-shot and many-shot learning. Can you talk a bit about what that is, in the context of both this surrogate gradient descent and just in machine learning generally?
KS: As we’re interacting with the world, we are constantly taking in, essentially, data. And we can really quickly and rapidly adapt to that. But in a traditional machine-learning training framework, you basically just feed in potentially thousands, millions, now even billions of examples of different data. And so that takes tons of time to train it, right? You can’t deploy that. If you want to learn something new in a real, interactive environment, you can’t just have a model sit there for hours to learn.
And the idea behind one-shot is you learn from just one example of the data. For example, you have some neural network trained to detect certain species of dog. But then there’s a new species of dog you want to detect. You don’t want to have to retrain your whole network on all the species of dogs, and give tons and tons of examples of the new species in order for it to learn. So, ideally, you just take what the model already knows—just from one representation or as few as possible of that new species of dog—learn what that species of dog is, and be able to infer for future examples.
SB: Right. So one-shot learning, essentially, it’s a way of leveraging existing learning on the chip, or existing knowledge-base on the chip—or on the system—and extending it to new classes and new examples. Is that right?
KS: Yeah, that’s generally what the process is. You do pre-training—you train on many examples—and then the idea is that, then, you leverage that knowledge to then learn new examples rapidly (or in one shot).
SB: So I understand you’ve also written about meta-learning. I wonder if you could say a little bit about what that is, and where it comes from?
KS: So the idea behind meta-learning is… meta-learning means learning to learn. Rather than just having a very fixed set of tasks (or a fixed goal) from the beginning, to instead learn a more generalized representation of the domain (or the space) so that you can then very rapidly adapt to new samples. Because you’ve learned to learn these different representations in this domain (or space). And that relates to one-shot learning because if you have this model that has a very generalized representation of features and things across a domain, and then you see something that’s related but still completely different, you can basically immediately infer from all these known generalized representations—because you’ve learned to learn this type of data, or task, or domain.
SB: And meta-learning is an already existing, known quantity in machine learning generally, right? This is already a sort of field of study.
KS: Yes, it is. The first paper on it, at least that I know of, is ‘Model-Agnostic Meta-Learning,’ which is the algorithm I’ve been working with. That came out in 2017 and, since then, people have been building a lot upon that.
SB: Can you say a little bit about how this one-shot learning, and spiking-based meta-learning, how they work in practice? What are the mechanisms that you’re using to implement that?
KS: I took the general idea behind meta-learning and this model-agnostic meta-learning learning algorithm. In normal deep learning, you do the gradient descent. But in meta learning, the idea is to do a gradient descent of the gradient descent. That’s why it’s called learning to learn. And so it’s this idea of what’s called bilevel learning. And the mechanism, you can describe it as sort of like an outer and an inner loop.
One test that I’ve done that’s pretty simple is on double digits. So, there’s this very famous machine learning dataset called MNIST, which is basically digits 0–9. But let’s say you want a network to learn double digits—using that for meta-learning and one-shot learning. So maybe you have a network that can’t necessarily know every single double digit at the same time, but you still want it to be able to recognize double digits. So what you can do is you can get it to learn the general representation of what a double digit is, so that it can just in one-shot learn whatever that double digit number is you need it to know at that time, rather than just knowing at the same time all double digits and then just being fixed on that set number of digits.
SB: Now, you and I talked about this before, and my understanding was that the way that some of this ‘fine tuning,’ if you like, of the knowledge of the system, that some of that happens by focusing on the final layer of the network and allowing that to change while allowing the rest of the network to stay fixed. Can you talk a little bit more about that and the advantages of that approach?
KS: Yeah. So I think you’re referring to error-triggered learning, which is an algorithm that I’ve been exploring for this sort of fine tuning. So the idea is rather than just always updating the network, you only update it when there’s a sufficient error. So that you can selectively tune the model as appropriate to help avoid this problem of catastrophic forgetting.
SB: How is that error generated? Is it about the size of the error? So if the error gets beyond a certain point then you start looking at more than just the last layer? How does it select when it wants to go a bit deeper and when it doesn’t?
KS: Yeah, I mean, I haven’t exactly studied this yet. But that would probably be the most simple thing to try, is just to have a certain threshold. And when the error maybe reaches a certain threshold, then you can check in previous layers.
SB: So the advantage of your technique is that it allows people to build small, power-efficient chips—probably for the edge market—and to be able to have them learn on the fly, and not be absolutely rigidly fixed, so that if new things become important as you go along then you can cope with that.
KS: Yeah. And just this idea of using this fine tuning throughout the world. One project I’ve been looking at is in prosthetics, where you want to adjust the prosthetics and the movements to the particular person, so that they can use it easier and more naturally. Another avenue I’ve been interested in is this idea of zero-shot learning, where you can learn without changing the weights at all, right? That’s something that these GPT-type models are capable of doing.
SB: How does that work?
KS: Basically, rather than just learning a representation, they’re really learning contrasts or associations between examples of data. Let’s say you have different fruits, right? You have orange, apple, banana, and then you want to learn kiwi. I guess the model already knows, like, looks at apple, banana, orange, and then tries to see how similar kiwi is to those other fruits. And then, rather than updating weights, it basically just uses what it already knows to infer what this new fruit is.
SB: When you look at your long-term career and all of the various things you could do, what are the kinds of things that really inspire you to think that you could make a contribution in, or that you might want to get involved with, or explore?
KS: So, in the long term, just being involved in this community for several years now during my PhD, I really want to see this promise of neuromorphic succeed. I always tell myself that I’ve been in the machine learning space adjacent so, worst case, I could just go and do traditional deep learning and machine learning. But I really think that the next step is to make computing more energy efficient and smarter, in terms of how you can learn more effectively and learn in more interactive, dynamic environments. These are the promises that neuromorphic has.
SB: Kenneth Stewart, thanks for coming on to Brains and Machines.
Next up, Giulia talks to Dr. Laura Kriener, currently postdoc researcher at the University of Bern, Switzerland. She’s been working on a different kind of backpropagation learning rule for spiking neural networks.
GDA: Laura Kriener, welcome to Brains and Machines.
Laura Kreiner (LK): Thanks so much for having me.
GDA: I’d like to know a little bit more about you. So you’re currently a postdoctoral researcher at the University of Bern, and I want to know about your background. So could you please tell us about your journey and why you joined the neuromorphic community?
LK: Sure. So, I studied physics in Heidelberg. And at the time, there was Professor Karlheinz Meier’s research group, Electronic Vision(s), in the Physics Institute in Heidelberg. And they developed the BrainScaleS-2 neuromorphic system. So, when I was looking for a place to do my bachelor thesis, there was this event where many professors from different institutes talked about their research groups and introduced them, and I happened to be in the session where Professor Meier was giving his talk. And, after this 15 minute talk, I guess you could say I was hooked. The decision was made!
I attended his lecture on brain-inspired computing, I did my bachelor thesis in the Electronic Vision(s) group, and later I came back for my master’s thesis with the same group. And then, for my PhD, I moved to Mihai Petrovici’s group at the University of Bern. And that group, they don’t—we don’t—build neuromorphic chips ourselves, but rather it’s more focused on the algorithmic and the network part. But in that group, I’ve always been one of the people very much focused on getting the algorithms that we developed there, and getting the networks actually running on neuromorphic hardware.
GDA: Let’s talk about your work now. In your paper, ‘Fast and energy-efficient neuromorphic deep learning with first-spike times,’ could you briefly explain to the audience what is time-to-first-spike coding? And why we did not have such a rule before?
LK: Yeah, so I actually would like to answer the second part of the question first, because I think that’s easier to explain what’s new about this.
Julian Göltz and I worked together on this project. We wanted to get inspired by machine learning and, in particular, by the very, very fundamental and incredibly powerful error backpropagation algorithm that you have in machine learning. That algorithm has been around since the ’80s and it’s super powerful, but people have said all the time that you cannot use it for spiking neural networks. And the reason for that is because the way error backpropagation functions, it requires that if you do a small change in your network then also the output of the network only changes a little bit—so that it’s differentiable, mathematically speaking. Everything in your network is differentiable. But if you have a spiking neural network then the output of a neuron is either it spikes or it doesn’t. So it’s a discrete output, it’s an all or nothing response—there are no three quarters of a spike or something, so it’s not differentiable. And therefore people said, “Well, you can’t use error backpropagation.”
And what we did in this project is we found a way around this. And essentially what we said is, “Well, if you don’t say the output of a neuron is ‘it spiked or it didn’t,’ but rather you look at when the neuron spiked, then this output can change in little amounts—the neuron can spike a little bit earlier, or a little bit later, if you change something in your network.” And suddenly everything is differentiable again. And, voila, you can do error backpropagation. This is what we worked on.
And now I can come back to the first part of your question—what this time-to-first-spike encoding is. So the thing is, if you’re now working with spiking networks and you want to do some task, then you now need to somehow encode the information in the spikes, right? And we have said before that we use the timing of the spikes. And time-to-first-spike encoding is one way of encoding information in spikes. I think I’ll explain it the easiest with a machine learning example. So, let’s say we want to distinguish different images of handwritten digits—so the classic MNIST dataset. And then, if I have given my network this image, how do I find out which class it has decided this image belongs to? Let’s say we have 10 different kinds of digits that we want to distinguish, so we give 10 output neurons into our last layer of the network. Then we say, “Well, each of these codes for one of the different classes.” And then we simply read out the network’s response by saying, “Whichever neuron fires first, that’s the class that the network chose.” And it turns out that, for our mechanism of doing error backprop on spikes, this is super convenient.
GDA: This work is based on the BrainScaleS neuromorphic platform. And now what I’d like to understand, and what I’d like you to explain to the audience, is what’s the difference between BrainScaleS and another analog counterpart? For example, the DYNAPs, developed by the University of Zurich?
LK: So first of all, BrainScaleS—or BrainScaleS-2 in this case, which we developed in Heidelberg—is a mixed-signal neuromorphic system. So that means you have analog neuron circuits, where the electrical quantities within that circuit (so the voltages and the currents) follow the same dynamics and the same differential equations as a neuron model. So you don’t simulate a neuron like you would normally on a computer, but rather you have something that behaves like a neuron, that emulates a neuron, and you just watch the dynamics unfold. So this is the analog part, but then there’s also a couple of digital parts on the chip, which is why it’s called mixed-signal. And that’s, for example, if a neuron spikes then you want to transport the information that this neuron has spiked to all the postsynaptic partners, and this is done in digital circuits. This is why it’s called mixed-signal.
The special part about BrainScaleS compared to, for example, the DYNAPs in Zurich is that it’s accelerated. That means even though the neuron circuits emulate or mimic the dynamics of a neuron, they simply run faster. So normally a neuron operates on the time scales of, like, milliseconds—a couple of milliseconds. The BrainScaleS neurons do the same things, but within microseconds. So they are just faster.
And typically—like DYNAPs, for example—they run in real time, which means the timescales on which the circuits operate are approximately the same as the ones that biological neurons operate on. And both systems have advantages and disadvantages. It depends on what you want to do. In particular, for our approach now, the speed of BrainScaleS played very nicely into our hands.
GDA: Now that we understand the ingredients of your work—so we understand the rule that you developed, and then we understand BrainScaleS a bit—I’d like to ask you to give us an overview of your work, and the results that you and your team achieved.
LK: So one of the key results I kind of already snuck into the answer to your earlier question, because just the fact that you can do error backpropagation on spiking neural networks and how to do that is already one of the key results of our work. It’s kind of a theoretical result. But then, of course, we wanted to apply this and run this on neuromorphic hardware and, in this particular case, BrainScaleS-2. Turns out, it worked! So we were able to train our spiking neural network that was running on BrainScales with our method for error backpropagation in spiking neural networks (with what is called in-the-loop training) and on the MNIST dataset. And it does very well. It classifies the data with good accuracy and it does so very quickly, and also quite energy efficiently. We measured how long it takes to classify the whole MNIST test data set—so 10,000 images, one after the other. And it turned out that you can classify, with our system, the whole MNIST test data set within one second. And really, we started the watch when we sent off the images to the chip and we stopped the watch once we had, for every single image, the classification back on our host computer.
And then also we measured the energy consumption while classifying the images, and it turns out that you only need eight microjoules per image. Eight microjoules… it’s kind of hard to judge how much or how little this is, but Julian has a very nice way of explaining this and I’m just going to steal it. He says that if you have the energy that is within one single banana, an average banana, then you can classify with this, I think, 30 billion MNIST images in our setup with the energy of one banana. So I think this gives a nice impression on how little it is.
GDA: It is a nice impression! And to be fair, I still remember when you were telling me at CapoCaccia how many nights you spent looking at the results to see if it was working. So that’s amazing!
Your ongoing research, titled “Gradient-based methods for spiking physical systems,” is trying to compare the results you obtained with BrainScaleS with several different approaches. Can you talk a little bit about that?
LK: Yeah. Our method of doing error backpropagation with SNNs isn’t the only one. Around the same time, a couple of others cropped up. And one very famous one is the surrogate gradients by Emre Neftci and Friedemann Zenke. That one was actually also implemented and realized on BrainScaleS-2 by Benjamin Cramer and Sebastian Billaudelle.
So, we are now in the position that we have fundamentally different algorithms of realizing error backprop in SNNs that have been deployed and are running on the same neuromorphic chip. And what we are now very much interested in is: what are the differences we want to compare? Like, how do they fundamentally, in their inner workings, get around the difficulty of non-differentiability? And what are the network structures that arise? What are the dynamics that happen in those networks? So what we want to compare is, fundamentally, how do they get around this non-differentiability of the spikes, what are the network structures that arise from training, how do the networks evolve during training, and also what does this mean for the applicability on the neuromorphic chip. What are the advantages and disadvantages of each? And maybe you can also combine the two methods and then get a ‘best of both worlds’ approach.
GDA: And it feels incredibly interesting and actually useful for all of us.
Now, the last question: I would like to know a little bit more about your future, where your research will go and what you are planning to do next.
LK: For the very near future, I’m very much interested in the project that I just described. Because even though we now have these multiple methods for training spiking neural networks, they are still way more finicky to train and harder to control than normal ANNs that are trained with backprop. And I think if we can understand in a deeper way how our methods for training the SNNs actually function, how they compare to each other, maybe we can learn a thing or two about why it is that the SNNs are harder to train and harder to control. And maybe, from this understanding, this allows us to actually solve these problems and make them easier to train and make them easier to control.
But also, from a bigger picture, I think the results that I have described by now are very much on fairly simple datasets and also datasets that are on short timescales, or even static images. And the reason for that is that these are the easiest setups. And if you develop new algorithms and methods, you for sure want to start with the easier setup and not make your life harder than necessary by immediately going to the complex task. But generally, it’s accepted that we think that the SNNs have their advantages when they operate on more temporally rich and more complex tasks. And this is where we expect them to be actually better than ANNs. But in that area, it is very much still unclear. So for example, what kind of data is that really, where the SNNs are better? If I have that kind of data, how do I encode it into spikes? If the data changes, do I need to change the encoding? What kind of network structures are good for training on this kind of data? And also, maybe I can give my SNN even more temporal computational power by, for example, changing the neuron dynamics or also changing, for example, synaptic transmission delays and things like that. And I think this is very much still out in the open and people start exploring this. But I think it’s very necessary that we start exploring because this will show us the way where, or if, SNNs actually have a proper computational advantage on something.
GDA: Laura Kriener, thanks for taking part in the Brains and Machines podcast.
LK: Thank you for having me.
SB: Thanks, Giulia. It’s really interesting to hear about these two different approaches back to back, and to hear that they’re both running successfully on the same hardware.
In the next interview, Giulia talks to Jens Pedersen, a PhD student at the Royal Institute of Technology—that’s the KTH in Stockholm, Sweden. He’s working on a way to provide an intermediate representation between the software and hardware layers to make it easier to use and design neuromorphic systems.
GDA: Jens Pedersen, welcome to Brains and Machines.
Jens Pedersen (JP): Thank you very much for the invitation, Giulia.
GDA: So, tell me a bit about you, about your background. I know you have a peculiar academic path, but most importantly, why did you join the neuromorphic community?
JP: So, I actually started with a degree in political science. And I realized that was a mistake! I think political science is extremely fascinating, but I decided to shift gear to computer science, which was a tremendous opportunity to understand the notion of computation and really getting a better idea of how we compute and how we make intelligence work for us. And that’s a very broad notion, but I think the term ‘intelligence,’ it’s something I’ve been interested in since childhood. I was curious about this term. What is it? How do you define it? And I realized not a lot of people have great answers. So I realized that I think there’s a need for people to really dive into the physics of the world and understand how brains and machines—speaking of this beautiful podcast title—are doing it. Right? How do these nervous systems create this beautiful, intelligent organization? And so that really drove my search from computer science.
My master’s thesis was on neural systems. And I discovered these spiking neural systems and found out, okay, they’re pretty close to biology. And that led me to neuromorphic engineering. And then I met my PI, Jörg Conradt, at a conference in Munich, in the Human Brain Project program he was a part of, and he did a wonderful lecture where he talked about, “Okay, we can take these spiking networks and we can fuse them with machines. We can build robots with this intelligence.” That spoke to me. It’s fundamentally different, I think, than a lot of what’s happening in the deep learning field right now, where you have all these very big large language models. Because the neuromorphic intelligence that I’m interested in is very practical, it’s very low level in the physical sense. You’re really trying to engage with the real physical world as it is. Not as it’s curated in some kind of high-dimensional embedded vector space in large language systems.
GDA: So you are doing a PhD now? Where?
JP: I’m a PhD student at the KTH Royal Institute of Technology in Sweden, in the Neurocomputing Systems Lab led by Jörg Conradt. It’s a very fascinating place. It’s a little cold and dark at times, but we have a wonderful lab with a lot of neuromorphic gadgets and electronics tinkering that you might want to do to study these systems. And I’m close to finishing, so I’m handing in in half a year or so.
GDA: Let’s start talking about your work. So your paper is mentioning a problem with which I resonate a lot; in our community, we have too many software and hardware solutions with their unique technologies, which make it difficult to have reproducible results. Can you expand a bit for the listeners? What is this problem, and what is causing this problem for us as researchers?
JP: I actually want to go back in time a little bit, because there’s a wonderful theoretician called von Neumann—many listeners are probably familiar with him—who actually wrote a fantastic book called ‘Brain and Computers.’ He was already, then, looking at the differences between analog and digital computing. And I’m mentioning this because this is a very, very, very old problem, but for various historical reasons, the digital platforms really gained momentum. They’ve pretty much taken over the world. And now we are left almost with the archaeological kind of efforts to dig out the old insights that happened in analog compute almost 100 years ago. Essentially, we’re trying to make nature get physics to do the compute for us. And so even von Neumann had many thoughts and notions on how to approach this, but especially here for neuromorphic hardware and software, it’s an extremely complicated technology stack to align.
So I think it’s very natural that we see this very diverse set of technologies. And I think it’s a good idea to have that diversity. The problem comes when I, as a PhD, need to sit down and finish some work, right? Because I don’t have infinite time to study all these many different systems. And so, just like we had to do with the digital computers early on, 90 years ago, we have to do the same with neuromorphic computers now. Namely, being able to abstract that in some high-level way. Find, like, a language, a way to express, a way to model these neuromorphic systems so that we can work with them across boundaries. Otherwise I think we’re in a problem, right? The neuromorphic field will stagnate. We will not be able to develop anything, because the insights are so isolated to each particular platform.
GDA: Now that we understand the problem, we can definitely introduce your work, titled ‘Neuromorphic Intermediate Representation.’ So, what is the purpose and the concept behind this work?
JP: So the idea of neuromorphic intermediate representation… you’re basically wedging in an abstraction—in this case, a few computational primitives (we can come back to that in a second)—you’re wedging that in between the physical hardware layer and the abstract software layer. What that specifically means for us is actually taking a step back to the old days of analog computing, if you will, and then reformulating what we mean by computational primitives.
So, for digital computers, you have bits and gates that can represent the flow of information. But that’s not how nature works fundamentally. Most neural systems, particularly in computational neuroscience, are represented with equations as change in time. So some sort of system that, when you apply some current to it, you permute it somehow. And that’s really, I think, a much more reasonable fundamental primitive for computing with neural systems.
And that’s what neuromorphic intermediate representation is about. Essentially, it’s nothing new. People have been making these equations for hundreds of years. But I think what’s new is that I think we found a level of abstraction which is meaningful both for computational models and for analog systems. That doesn’t mean that this will cover all possible cases of computational models and analog systems—that’s far from the case, there’s much more research to be done—but I think this is a first step at something which is declarative, meaning that we have this idealized normative representation that more and more people can adhere to, can understand, can subscribe to. And that means both in the software stack and in the hardware stack.
GDA: So, with NIR, you propose to bridge neuromorphic simulators—Lava, Nengo, Norse, Rockpool, SyNAPSE, snnTorch, Spyx—and digital neuromorphic hardware platforms such as Loihi via Lava, Speck, SyNAPSE, and SpiNNaker 2. So I have two questions, and one is, how does this really translate into reality? And, why these specific frameworks and not some others?
JP: In practical terms, what we hope this gives to the community is a way to use and describe the same models on different platforms. And I should really point out, by the way, we’ve only looked at digital hardware for the moment, but I think there’s a beautiful point to be made here, because we just talked about how difficult and complicated analog and digital hardware is to build, and that means they will be different. So some hardware platforms will have leaky integrate-and-fire neurons and some others might not. So even though we have the same language, a way of describing models, they will not work exactly the same on all different platforms. And in some sense, I think that’s a wonderful analogy to nature. Even though we might have the same substrate or rough structure in our nervous system, our brains are wildly different. There is room for discrepancies and divergent behaviors.
When I say that we want to translate these models into specific platforms, what I’m hoping is actually not to get the exact same behavior, but to get the same topology, if you will. The same structure. The same kind of computation. And the hope, of course, then is that with better and better algorithms that can adapt, that can fuse, that can transform into whatever is needed at a given instance, just like biological systems.
And coming to your second question—why this, and not others—I think because we’ve reduced to these kinds of physical primitives, these naturalistic primitives that are declaring some computation. We’re not implementing it, we’re not stating how the world should look like in concrete terms at every single point in the execution of the model. We’re putting the normative frame around the computation and then, essentially, leaving the choice of implementation up to the individual platforms. And I think that’s very important. Because previous efforts, I think, for instance, the PyNN neural network framework has been a wonderful effort to, for instance, bridge SpiNNaker that you mentioned with the NEST simulator—something like that, right?—that has been very focused on the computational aspect from a digital point of view. It’s been extremely important to reproduce the models exactly like they were on all systems. And I think that’s a misconception, because that’s not how these models will operate when we eventually get the holy grail of analog neuromorphic hardware to work.
GDA: So if you could imagine having one software and one hardware of your dreams, which kind of characteristic would you love to have?
JP: It’s a wonderful question. So, honestly, I think the hardware of my dreams is already here. We just don’t know how to use it. And that’s exactly the problem. We can look at nature and we can see exactly what it’s doing, we just don’t understand it in a way that we can rephrase it into our own problems and solve our own tasks.
I think, honestly, we have to somehow create models that are more appropriately fitting the human mind, and maybe less appropriately fitting physics. And I know that’s a controversial statement, but I don’t think the problem here is the physical hardware. The hardware exists and it works. The problem is us not understanding what’s happening.
I think, honestly, the software of my dream will be able to take something like NIR, describe all the wonderful developments that’s been happening in deep learning over the past years, and then add to it so we can make it plastic and adaptive. Because then, the hardware in principle doesn’t really matter.
GDA: Now I’d like to ask you the results of this work, and what you’re working on now, and what’s your future?
JP: This is actually just one branch of what I’ve been doing in my PhD. So, having a background in computer science, I’ve been focusing a lot on generating reproducible software environments. I think open sourcing tools is as much a scientific endeavor as generating and researching good ideas. So I spent some time developing a neuron simulator, together with Christian Pehle, called Norse. It’s been a tremendous help in my research. I’ve also developed software to stream and work with address-event representation in event-based cameras and files—it’s called AEStream—which has also been a lot of fun.
And my research at the moment, actually, is to try to understand vision from a geometric point of view in neuromorphic networks; to take some of the ideas from group theory and geometry and look into the spatiotemporal domain of neuromorphic computing. Because I think there’s a lot more work that needs to be done in understanding not just how we compute with space—I think that’s what a lot of vision models are doing—but also compute in time, because that’s essentially what neuromorphic systems have to do. We have to incorporate these temporal dimensions.
And next, I’m lucky to go and visit a professor at Stanford, Sadas Shankar, who’s taken an interest in this neuromorphic intermediate representation. And I think we’re more focused on moving that closer towards physics and understanding what exactly does this mean when we relate this to the notion of computation. So we want to expand that and see, okay, how can we bring some of these guarantees from the digital world into the analog world, and how can we expand that.
GDA: Jens Pedersen, thanks a lot for being part of the Brains and Machines podcast.
JP: It was a pleasure, Giulia.
SB: Thanks, Giulia. They can be controversial, but I’m very interested in this issue of intermediate representations. So I’ll definitely read up about this work. I think I see some travel to Sweden in my future also.
In the last interview, Giulia talks to Dr. Fabrizio Ottati, an AI machine learning computer architect at NXP Semiconductors in Hamburg, Germany. He is marking neuromorphics’ homework by comparing artificial neural networks and spiking networks in both static and dynamic scenarios.
GDA: Fabrizio Ottati, welcome to Brains and Machines.
Fabrizio Ottati (FO): Ciao, Giulia, thank you for having me.
GDA: Let’s start from scratch. Tell me about you—who you are, your background, but most importantly, why you got into this field.
FO: Right now, I am an electronic engineer. In particular, I work on designing a processor for AI at NXP. And my background is fundamentally in computer architecture. I did my PhD in Politecnico di Torino. I worked on acceleration of the spiking neural networks on digital hardware platforms, in particular PGAs.
Actually, who got me into this field is a man that some among the audience may know, Gregor Lenz, with whom I got in touch on an open-source project called Tonic. From there, we started playing together with software for event cameras and eventually SNNs. And then I got to know this amazing community made of amazing people, and so on and so forth.
GDA: Let’s delve into your work a little bit more. So, one of the first sentences of the abstract of your work—which, by the way, has a cool title, ‘To Spike or Not to Spike’—caught my attention, and it’s “The power efficiency of the biological brain outperforms any large-scale deep learning model; thus, neuromorphic computing tries to mimic the brain operations.” And, in reality, nobody really measured this difference, but everyone would agree with you, right? So my question is, what do you think is the most interesting thing about biological communication through spikes?
FO: Most people claim that the brain is more efficient. And that is actually true. But we also have to take into account the fact that we have been trained throughout evolution, so we have a really strong bias in our model, which is our brain. And throughout our life what we do is not learn, but simply fine tune. Just like larger language models fine tune on different tasks, we fine tune on different tasks of life, with the strong difference being that we do multiple things at once and we do it on a 20 watts budget.
The advantage of the brain, most probably, is due to the incredible amount of interconnections that are present in it. Which, right now, we are not able to implement in an efficient way in silicon—mostly because the brain is in 3D space and, right now, we are only able to work in 2D space when we try to build our computers.
Regarding spikes, I’m kind of convinced that they are not the most efficient way of doing the computing. For the way our computers are built, at least. But yeah, we have plenty to learn from the biological brain.
GDA: I agree with you. You also say that the biological substrate has severely constrained the evolution of the biological brain. And I’d argue that embodiment helped throughout evolution to create the best (or the ‘good enough’) structure for a specific environment. Could you expand your point of view a little bit more?
FO: Sure. I am completely ignorant of evolutionism, but what I would try to guess is that humans tried to do the best they could with what we’ve got. We have this mass in our head that has been selected throughout evolution in order to do the best things in the best manner, consuming as little energy as possible. That’s what we have been trained to do, and that’s why our brain—the way we interact, the way we learn—has been constrained by our biological substrate.
Unlike in our silicon labs, we couldn’t cancel our brain and start with a new design from scratch. And that’s why computers are way better than us at math, for instance.
And that’s why we are severely constrained by the biological substrate. And that’s also a reason why I think we should not try to emulate the brain too closely; just getting inspired is enough.
GDA: This is an interesting point of view.
So, you did a smart thing: you asked yourself when and where it’s really needed to take inspiration from nature. And you compared it with an ANN, an artificial neural network. Could you expand a bit and talk about your work, and explain and summarize it for the listeners, please?
FO: So what we tried to do in ‘To Spike or Not to Spike’ is to look at a subset of the current tasks that are being investigated in machine learning, and we focused on two: the first one was a spatial task, not time related, which consists of object recognition; and another task was time related, a spatiotemporal task, which in our case was keyword spotting—so audio processing. And what we understood is that SNNs are a disadvantage with respect to artificial neural networks when it comes to purely spatial tasks. And that’s due to several things, but the most important ones are that, first, a spiking neural network needs time to make a prediction—it needs to go through a series of timesteps to make a prediction. This involves several computations that are done over and over (over and over means the number of timesteps that you need to perform a computation). And even if you are really efficient in a single timestep, you can’t make up for it. An ANN will always outperform you because, in an ANN, everything is static and you don’t need multiple timesteps to perform a simple inference (which consists in recognizing where the cat is in the photo, basically).
When it comes to temporal tasks, it’s a different story. In there, we are playing a fair game—SNNs and what we consider recurrent neural networks, that’s a fair game. Because both models need a notion of time to process data. But still, SNNs are lacking performance in terms of accuracy. The main reason behind that, in my opinion, is that we still try to reason with these extremely discrete spikes that cannot encode information in a good way, in my opinion—at least not as well as I get in the recurrent unit or a long short-term memory, which are some state of the art recurrent cells in the artificial domain. But we also noticed that, at the current moment, our learning solutions in this temporal feed are pretty advanced with respect to an ANN. In ANNs, as far as I know, on-chip learning is performed on general platforms, such as microcontrollers. While there are chips out there that do the full learning, maybe they are not too complicated tasks, but still they perform the whole training process directly on-chip, from Charlotte Frenkel.
GDA: So, I see you mentioned that SpikeGPT, the first spiking LLM, is estimated to use 22 times fewer operations in its execution when compared to its conventional non-spiking counterpart. And I think it’s an important bit of information. Can you explain it to everyone?
FO: What Jason Eshraghian has been doing is to perform lightweight transformers, based on spikes, that allows a much less complex computational pipeline to perform much, much fewer operations. This is an important thing, because right now transformers are really prohibitive to be executed in something that is not a GPU. We are making progress on that, but still they have a complexity that makes it not feasible for edge devices to run it. In fact, there are plenty of works that are starting to come out from linear transformers that translate from a quadratic (in terms of tokens complexity) to a linear one. And then, recently, Microsoft came up with a sort of mixed transformer, a sort of recurrent transformer, called the Retentive Network. So the community is really trying to overcome this huge computational problem, because it’s not sustainable at the current moment. I read an article some time ago where it was written that a single prompt fed to a GPT-3 model from OpenAI costs them four cents. That’s not sustainable.
GDA: And could you briefly explain what a transformer is, for the listeners that do not know?
FO: So a transformer gets a sequence in input, which could be a sequence of words, and tries to look at it from a different perspective—which are called the different heads of attention—and tries to find connections between the words that give a meaning to the word, to understand the concept in the phrase. And this is translated to plenty of applications, from natural language processing (for instance, code generation) to everything nowadays… even recognizing cats.
GDA: Thank you. And since you are one of the founders of Open Neuromorphic, I’d like you to explain a little bit about what Open Neuromorphic is, what it claims to do, and why people should join this community. And, at the end, I would like you to tell us a bit more about your future, or where you think that this community should go in your specific point of view.
FO: Open Neuromorphic was born out of a reunion among a lot of researchers, which were me, Gregor, and Jason. And what this community tries to do is to simply gather people and make them talk about anything that’s worth considering in this AI domain—from computer hardware in my field, to Jason’s SNN models, to Gregor’s space application and event-based vision pipelines, and so on and so forth. And the nice thing about this community is that everyone can bring something, discuss something, and start something. And that’s what I like the most about the community. Everyone can do whatever they want. And we are growing at a steady pace. And I do think that this is the direction that we should keep; to allow everyone to be in this space, enjoy research, enjoy the open-source technology out there, and maybe, if we are lucky, make it go faster.
GDA: Fabrizio Ottati, thanks for being with us on Brains and Machines.
FO: Thank you, Giulia.
SB: This was the last interview of this episode, and the last episode in the season. Brains and Machines will start up again in the autumn, and one of the upcoming interviews will be with Carver Mead, father of neuromorphic engineering. We hope you will join us then.
TM: That brings another episode of EE Times Current to its end. Thank you for listening, and thanks to our guests, Dr. Kenneth Stewart, Dr. Laura Kriener, Jens Pedersen, and Dr. Fabrizio Ottati. EE Times Current is available through the major podcast platforms, but if you get to us at our website at eetimes.com, you’ll find the transcript along with direct links to the other stories we’ve mentioned and other resources. Thanks for listening.
