Embrace uncertainty, hire the right people, and learn from data
This blog post is an updated version of part of a conference talk I gave at GOTO Amsterdam last year. The talk was: Watch Online.
Delivering value and positive impact through machine learning product efforts is not a trivial task. One of the main reasons for this complexity is the fact that two sources of uncertainty intersect in ML efforts developed for digital products. On the one hand, there is uncertainty related to the ML solution itself (Can we predict with high enough quality what we need to predict?On the other hand, there is uncertainty about what impact the system as a whole will have (Will users love this new feature? Will it actually help solve the problem we're trying to solve?).
Due to these uncertainties, failures in ML product efforts are relatively common. However, there are strategies to manage and improve the odds of success.Or at least survive with dignity!It is important to start any ML initiative in the right direction. In a previous post, I described my main learnings in this field: start with the problem (and define from the beginning how the predictions will be used), start small (and keep it small if possible), and prioritize the right data (quality, quantity, history).
However, initiating a project is just the beginning. The challenge of successfully managing an ML initiative and having a positive impact continues throughout the project's lifecycle. In this article, we share three key lessons learned to help your ML initiative survive and thrive.
- Embrace uncertainty: Innovation, stopping, pivoting, and failure.
- Surround yourself with the right people: Roles, skills, diversity, and network.
- Learning from the data: Able to identify the right direction, make improvements, detect failures and make plans.
It is very difficult (even impossible) to plan an ML initiative in advance and develop it according to that initial plan.
The most common project plan for ML initiatives is ML Lifecyclebreaks down the phases of an ML project into business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Although these phases are depicted as sequential steps, many representations of this lifecycle have arrows pointing backwards. At any point in the project, you might find something that forces you to go back to a previous phase.
This leads to projects where it is very difficult to know when to finish: for example, during the evaluation step, you may realize that certain features are not properly encoded thanks to model explainability techniques, forcing you to go back to the data preparation phase, or your model may not be able to predict with the required quality, forcing you to go back to the beginning of the business understanding phase and redefine your project and business logic.
Whatever your role in an ML initiative or project, recognition is important. Things don't go as plannedThe key is to embrace all this uncertainty from the very beginning and use it to your advantage. This is key both for stakeholder management (expectations, trust) and for yourself and the rest of the team (motivation, frustration). How?
- Avoid overly ambitious time and deadline constraints. Ensure that ML initiatives are truly perceived as innovations that require exploring the unknown and are high risk, but also high reward and potential.
- Knowing When to Stop By balancing the value of each improvement (ML models can always be improved, but this comes at a cost in terms of time, effort, and opportunity cost.
- Be prepared to pivot and fail. You can continually leverage the learnings and insights gained from your projects and even change the scope of the project or even abandon it based on new learnings.
Every project starts with people. The right combination of talent, skills, perspectives and networks empowers you.
Gone are the days when machine learning (ML) models were confined to a data scientist's laptop. Today, the true potential of ML is realized when models are deployed and integrated into enterprise processes. This means that more talent and skills need to work together to make it possible (data scientists, machine learning engineers, back-end developers, data engineers, etc.).
The first step is to Skills and roles The skills needed to successfully build end-to-end ML solutions, but it takes more than a role that covers a set of skills. Diverse team Tools that bring different perspectives and empathy to different user segments have been proven to help teams improve the way they work and build better solutions (see “Why having a diverse team makes a better product().
Although it is not often talked about, there are people who are important to the execution of a project that go beyond the team itself. I call these people “networkYour network is the people who are really good at something specific and who you can trust to ask for help or advice when you need it, to unblock, accelerate or empower you or your team. Your network could be business associates, managers, staff engineers, user researchers, data scientists from other teams, customer support teams, etc. Build your own network and identify who your go-to allies are for your specific situation and needs.
Projects are continuous learning opportunities, and learnings and insights often come from checking the right data and monitors.
There are three big groups of metrics and measurements in any ML initiative that provide great value in terms of learning and insight: monitoring model performance, service performance, and bottom-line impact. I've covered this topic in detail in previous posts.
Having the right data and monitors when developing or deploying an ML solution is important for:
- Make sure you're heading in the right direction. This includes everything from properly designing a solution and choosing the right features to understanding whether a project needs to pivot or stop.
- Know what to improve and how: Understand whether your outcome goals were met (through experiments, A/B testing, etc.) and dig deep into what worked, what didn't, and how you can continue to deliver value.
- Timely detection and planning of failures: It allows you to address issues quickly, before they impact your business. And even if they do, with the right metrics you can understand what went wrong, take control of the situation, and make a plan to move forward (while maintaining stakeholder trust).
Effectively managing a ML initiative from start to finish is a multifaceted and complex task. In this blog post, based on my experience first as a data scientist and more recently as an ML product manager, I'd like to discuss the elements I believe are key when approaching ML projects: embracing uncertainty, surrounding yourself with the right people, and learning from the data.
I hope these insights help you to successfully manage your ML initiatives and drive positive impact through them. Stay tuned for future posts on the intersection of machine learning and product management 🙂
