Caltech’s Leading-Edge AI Curriculum
While the course curriculum for EE/CS 148 is at the graduate level, the class is open to undergraduates, although Perona and Gkioxari tell those students that prior experience with neural networks and programming would be beneficial.
“This class is open to all Caltech students in so far as they are curious and they want to learn something really useful,” Perona says. “We did our best to make it not too difficult to qualify for the class, although the class is fairly advanced.”
That was the experience of Chase Blagden (BS ’23), who studied information and data science while at the Institute. He says he found the pace of the class brisk and the material challenging but totally fascinating.
“I never personally skipped a lecture because it was actually one of the classes I was most looking forward to going to,” Blagden says. “It’s one of the best courses I’ve taken at Caltech.”
The course is organized into roughly four sections. It begins with a review of the basics of neural networks and the programming frameworks necessary to work with them. Students then dive into a type of neural network architecture called a transformer. When applied to large language models, which are deep neural networks trained to recognize patterns in natural language, transformers can power programs like ChatGPT.
In the next section of the class, students use a transformer to build their own small version of a ChatGPT-like program. The final section of the class then pivots to focus on diffusion models, the technology behind the DALL-E2 and Midjourney applications.
During the term, Perona also leads a discussion on ethics in AI, considering questions such as human racial biases becoming expressed in algorithms and who is responsible for the behavior of AI systems, as well as a more general discussion on how to think about new and disruptive technologies.
“Agriculture is an extreme example. The farmers took over the world and the hunter-gatherers basically got wiped out,” Perona says. “Something that is very good for some may not be good for everyone, so you have to think about it.”
Perona remains an optimist and sees a proper understanding of both social issues and AI technology as an opportunity to improve areas in which purely human institutions have fallen short. “It’s much easier to fix algorithms than people,” he says. “The principle is: think ahead about what could go right and what could go wrong, what to watch for. Know how to measure effects and then be proactive and be involved, because it can have a big positive impact in society.”
The course’s mix of theory and hands-on experimentation is not intended to simply teach students how to implement a ChatGPT-type program but to understand how it works and develop new ideas based on it, according to Suzanne Stathatos, a second-year PhD student studying with Perona, who served as a teaching assistant for the class.
“There’s a value in practical application, and then there’s a value in teaching how to think, and I think that this class does both of those,” Stathatos says. “Students start with a vague understanding, and they end in a place where they can read something in a paper on machine learning and implement it themselves.”
Beyond the technical skills, Stathatos says, students can learn practical lessons based on Gkioxari’s industry experience.
“One thing that I really liked about the class is Georgia feeds in little quizzes throughout the lecture,” Stathatos says. “She said those are the kinds of questions that appear in interviews for machine learning engineering jobs, so you’re also getting prepared to have this information at the tip of your fingers to do well for an interview for that kind of field.”
Gkioxari and Perona also invited guest lecturers such as Ross Girshick, a research scientist at Facebook AI Research, who led a team that recently developed a large language model called Segment Anything that can be used to identify objects from images.
“[Segment Anything] is state of the art, and it came out, I think, in the first two weeks of class,” Blagden says. “It was super exciting that they could get someone like Girshick to lecture to the class about this essentially brand-new, weeks-old thing.”
Ultimately, Gkioxari and Perona hope the class will help to disseminate knowledge of machine learning and generative AI techniques throughout the sciences as students go on to pursue different fields of research in their careers.
“It will help our colleagues in science and engineering be more effective, more efficient,” Perona says. “Slowly but surely, AI, machine learning, is becoming a foundational topic in the sciences.”
And that’s a foundation Perona expects graduating students will begin to build upon right away as they go on to academic posts, pursue further studies in the sciences, or enter industry, as Blagden plans to.
“I’m going to be working as a machine learning research engineer at Aurora Flight Sciences, which is a Boeing subsidiary,” he says. “But this class has inspired me. I kind of want to go back to grad school.”
The class culminates in final project proposals, a paper where they describe an AI research project they would like to undertake themselves. Perona and Gkioxari plan to help the students with the most interesting proposals as they begin work on their proposed projects over the summer.
“These students will be able to use what they learned in class to change the world yet again,” Perona says.
