Sometimes it’s twenty nine o’clock. It’s February, a short month.
The standard period is approximately 4 weeks. Approximately 20 business days. On a larger scale, not much progress has been made in the last 4 x 5 days. Still, as always, there’s quite a lot going on each day. Several experiments are performed. Some ideas are rejected. Some discussion moves things forward. Several code changes turned out to be more important than expected.
As I reflect on the past month, there are three lessons from the world of ML research and engineering that stand out to me.
- Interacting with others is important, but
- Although documentation is often underestimated until it’s too late,
- MLOps only make sense if they actually fit the environment in which they are intended to be used.
1. Interaction with others
If you read ML papers regularly, you know the pattern. Citations usually only show the first author’s name. Other names will only appear in the references section. Does that mean the first author did it all by himself?
Rarely. Only in special cases where one author wrote the paper alone.
Most research is based on exchange. It lives from discussions with co-authors, comments from colleagues, questions that force you to sharpen your thinking, and adjacent disciplines that bring in ideas that you couldn’t generate within your own discipline. Good research often feels like stepping into someone else’s territory to learn. just Their language is good enough to bring back something useful.
But this doesn’t just apply to academic papers. This applies equally to everyday engineering work.
A simple interaction with a colleague can save you hours of wandering down the wrong path. Five minutes of conversation at your coffee machine will give you the one missing piece you need to get set up. It’s important even in casual conversations. Not every useful discussion begins with a scheduled meeting with polished slide decks. It might start with, “By the way, I noticed something strange in the logs.”
This month I was reminded of that again. A few small interactions made things clear much faster than thinking it through alone. Nothing dramatic, nothing worthy of a keynote. There’s just the normal quiet value of talking to other people who are thinking the same thing.
2. Documentation
Have you made any changes to the code?
That’s certainly true.
And do you remember the next day? why did you make those changes? I hope so, but after all, it’s only one day. But what about after a week? 1 month later? Six months later?
Things become less obvious there.
Most changes to the codebase are minor and benign. Not every small bug fix deserves a long explanation. Renaming a variable, fixing a typo, or fixing a harmless logging issue usually doesn’t require special documentation. The same is often true for bug fixes that do not change relevant conclusions from previous results.
However, some changes are different.
Changes may change the prerequisites. Some change the way data is preprocessed. There are also training characteristics, evaluation logic, and even influences. Meaning of output. It’s worth writing down these changes, as you’ll forget them later when you return to the project.
This month I was reminded once again that documentation is not primarily for abstract future collaborators. It’s for my future self. Now, when you have a deep understanding of the code, everything feels obvious. It will go away in 3 months. Then you’ll look at a row, configuration, or mysterious data transformation and ask yourself, “Why on earth did I do this?”
That’s an easily avoidable question.
3. MLOps in action
The goal of most ML research is to create some form of trained model.
However, I believe that only a small number of these models will actually be used.
Many models remain where they were born: in notebooks, research servers, internal presentations, and papers. Beyond that, productive use of a model requires more than the model itself. You need the tools and principles of infrastructure, processes, monitoring, repeatability, and deployment strategies: MLOps.
Reading job ads in that direction, MLOps appears to be closely tied to cloud providers such as AWS, GCP, Azure, cloud-native pipelines, managed services, and distributed deployment environments. And yes, those tools are important. These are important and in many settings are exactly the right choice.
But it’s worth asking a simple question: Is the target environment actually a cloud environment?
Achieve automated quality control in industrial environments. Assume that the model is used directly in production, near the machines that create the product. Do we really believe that all relevant data is simply streamed from the enterprise to the cloud? Especially when that data reflects the core processes of the enterprise and is part of its competitive advantage? I doubt that many companies would be completely comfortable exposing their production-critical environments like that.
This is where a more grounded view of MLOps becomes important.
MLOps are certainly useful. However, this is not a specific set of tools, but rather a collection of ways to replicate tools under changing conditions. And it needs to fit the environment in which it will be used, not the other way around. The goal is not to shoehorn every implementation problem into the mold of a trendy tool. The goal is to make the model useful under real-world constraints and create the necessary tools for the problem at hand.
In some cases, that means cloud pipelines. In some cases, that means an on-premises deployment. In some cases, that means a restricted environment with limited connectivity, strict access controls, or hardware constraints at the edge. In all these cases, the principles such as version control, reproducibility, monitoring, safe rollout, and robust operations are similar. However, implementations can vary widely.
Thoughts as a conclusion
February was short, but not empty. As with every other month of the year, there are many lessons to be learned.
- Progress in ML often relies on interaction with others, not just solitary thinking.
- Documentation is most important when you think you don’t need it.
- MLOps only become valuable when adapted to real-world environments.
The same lesson will probably be held again next month. It’s not necessarily the flashy lessons, but the quiet “oh, yeah, that might be a good idea” lessons that influence our daily actions.
