Already 10 years ago.
At the time, OpenAI felt like one (well-established) startup among others. DeepMind already existed, but it wasn't yet fully integrated into Google. And at that time, the “deep learning triad”, LeCun, Hinton, and Bengio, was published. deep learning in nature*.
Today, AI is like a public good. At the time, the people who knew about it and were interested in it were mainly academics and technology geeks. Today, even children know what AI is and interact with it (or worse, flat bad).
This is a fast-paced field and I've been lucky to have been a part of it for just a little while 'back in the day'. Eight years ago, classic ML such as clustering, K-means, and SVM was still being taught in universities, although it was gaining momentum. This was also the year the community started to understand that all it needed was attention (and linear demographics). In other words, it was the perfect time to start learning about machine learning.
As the year comes to a close, now feels like the perfect time to zoom out. Every month, we review and publish small practical lessons. About every six months, I look for underlying larger themes, patterns that recur from project to project.
This time, you'll see four threads all over your notes.
- Deep Work (my all-time favorite)
- over-identify with one's work
- Sports (and exercise in general)
- blog
deep work
Deep work seems to be my favorite subject. It shows up everywhere in machine learning.
Machine learning work may have several focuses, but most days revolve around a combination of:
- Theory (mathematics, proofs, careful reasoning),
- Coding (pipelines, training loops, debugging),
- Writing (project reports, papers, documents).
Both require sustained concentration over long periods of time.
The proof of the theorem does not emerge from a 5-minute snippet. Needless to say, coding punishes interruptions. If you're deep in a bug and someone pulls you out, you have to rebuild, not just “restart”, and that just wastes time**.
Writing is also fragile. Good writing requires attentiveness, and the first thing you lose is when you keep sending small messages.
I am fortunate to work in an environment that allows for several hours of focused work several times a week. This is not standard. To be honest, it might be an exception. But it's incredibly fulfilling. I I can work on a problem for hours and then feel exhausted..
Tired but satisfied.
For me, deep work has always meant two things, and I already emphasized this six months ago.
- skill: Being able to concentrate deeply for long periods of time.
- environment: Having conditions that enable and protect that concentration.
It's usually easier to acquire (or reacquire) a skill if you don't have it. What is difficult to change is the environment. can be trained concentrationBut you can't single-handedly remove meetings from your calendar or change your company's culture overnight.
Still, it helps to name the two parts. If you are struggling with deep work, it may not be because you lack discipline. As I know from my experience, sometimes the environment simply does not allow what you are trying to do.
over-identify with one's work
Do you like your job?
Let's hope so, since this will take up most of your waking hours. But even if you like your job overall, there will be times when you love it more and times when you love it less.
Like everyone, I have experienced both.
There was a time when I felt energized just by the fact that I was doing something with ML.
oh!
And there were times when I got really depressed due to lack of progress or setbacks because the idea just didn't work.
No, that's amazing.
Over the years, I've come to believe that extracting too much identity from work is generally not a wise strategy. There is a lot of variation in approaches to and work with ML. Experiments fail, baselines outperform fanciful ideas, reviewers get it wrong, deadlines loom, data breaks, and priorities shift. If your latest workout raises or lowers your self-consciousness, you may be visiting Disneyland for a roller coaster ride.
As a simple analogy, imagine you are a gymnast. It takes years of training. You are flexible, strong, and in control of your movements. Then I break my ankle. Suddenly you can't even do the simplest jumps. You can't train the same way you did years ago. If you only Athletes — If that's what identity is all about —You'll feel like you're losing yourself.
Thankfully, most people are into more than just their area of expertise. Even if I forget sometimes.
The same applies to ML. You can be an ML engineer, researcher, or “theorist.” They can also be friends, partners, siblings, teammates, readers, runners, and writers. Even if one part of you is sluggish, other parts will keep you stable.
This does not mean that work doesn't matter. it's about be considerate without being depressed.
Sports or general exercise
Admittedly, this is a no-brainer.
ML jobs aren't known for involving a lot of movement. The miles you earn are the miles you tap your fingers on the keyboard. Meanwhile, the rest of the body remains stationary.
I don't need to explain what will happen if I leave it alone that happen.
The good news is that it's easier than ever to counter them. Nowadays, there are many boring but effective options.
- height adjustable desk
- Meetings took place while walking (especially when cameras were off)
- walking pad under desk
- Short movement routines (ideally during deep work blocks)
Over the years, physical activity has become an integral part of my job. This allows you to start your day on a smoother, less stiff, less slouchy, already “compacted” state. And it also helps to relieve fatigue after deep work. Concentrating deeply can be mentally exhausting, but it can also have physical effects, such as raising your shoulders, lowering your neck, and making your breathing shallower.
It will be reset when you move it.
I don't consider it “fitness”. I handle it like this Insurance that allows you to continue working for many years to come.
blog
Daniel Burke***
If you've been reading Towards Data Science's ML content for a long time (at least 5 or 6 years), the name may sound familiar. He published many ML articles (back when TDS was still hosted on Medium) and brought ML to a wider audience with his unique writing style.
His example inspired me to start a blog for TDS as well. It started from the end of 2019 to the beginning of 2020.
At first, writing these articles was easy. Just write, publish, and move on. But over time, it became something else – practice. Writing requires precision in putting your thoughts down on paper. If you have trouble summarizing something, you probably don't understand it as much as you think you do.
Over the years, I continued to cover machine learning roadmaps, write tutorials (like how to work with TFRecords), and, of course, return to deeper work. That's because it continues to prove important for machine learning practitioners.
And blogging has two benefits.
It was financially rewarding (so much so that it helped fund the computer I use to write this over the years). But more importantly, it's a rewarding writing exercise. I see blogging as a way to train my translation skills. This means taking technical content and turning it into words that other readers can actually communicate.
In a field that is fast-moving and loves new things, these translation skills are strangely stable. The model will change. The framework changes (Theano, anyone?). But the ability to think clearly and write clearly becomes more complex.
lastly
Looking back after eight years of “ML in action”, none of these topics were about specific machine learning models or specific tricks to speed up training.
Rather, the lesson is:
- deep workenable progress
- don't be overly specifichelp you overcome setbacks
- movementPrevents your body from silently deteriorating
- blogtrain clarity by sharing experiences
None of these lessons are strongly tied to machine learning.
But they keep showing up and have stayed with me over the past few years of machine learning.
References
* Deep learning Nature article by LeCun, Bengio and Hinton: https://www.nature.com/articles/nature14539;The annotated reference section is also worth a read in itself.
** Check out the highly accessible digest from the American Psychological Association at https://www.apa.org/topics/research/multitasking.
*** Daniel Bourke's homepage for posts about machine learning: https://www.mrdbourke.com/tag/machine-learning.
