Top 3 lessons learned: Problems, Size, and Data
This blog post is an updated version of some of the conference talks I gave at GOTO Amsterdam last year.This talk is also available to watch online.
As a Machine Learning Product Manager, I'm interested in the intersection of machine learning and product management, specifically creating solutions that add value and positive impact to products, companies, and users. However, delivering this value and positive impact is no easy task. One of the main reasons for this complexity is the fact that two sources of uncertainty intersect in machine learning efforts developed for digital products.
From a product management perspective, this field is uncertain by definition. It's difficult to know how a solution will impact the product, how users will react to it, whether it will improve product and business metrics… Having to work with this uncertainty , is what makes a product manager potentially different from other roles. project managers, product owners, etc. Strategies to overcome this uncertainty include product strategy, product discovery, opportunity sizing, prioritization, and agile and rapid experimentation.
The field of machine learning also has a strong connection to uncertainty.i always like to say “The goal of predictive models is to predict things that may or may not be predictable.”. This means projects are difficult to scope and manage, you can't commit upfront to high quality deliverables (good model performance), and many initiatives end up remaining permanently offline as a POC. means. Clearly defining the problem to be solved, doing initial data analysis and research, and starting small and getting closer to the product and business are actions that can help you deal with ML uncertainty in your project.
Mitigating this risk of uncertainty from the beginning is key to developing initiatives that ultimately deliver value to the product, the company, and the users.In this blog post, we will detail: Top 3 lessons learned when starting an ML product initiative to manage this uncertainty from the beginning. These learnings are primarily based on my experience, first as a data scientist and now as an ML product manager, and can help increase the likelihood that ML solutions are deployed in production and achieve positive impact. Masu. Get ready to explore:
- Start with a problem and define how you want to use predictions from the beginning.
- Start small and stay small if possible.
- Data, data, data: quality, quantity, and history.
I'll be honest, I learned this the hard way. After a model is developed and the predictive performance is determined to be “good enough,” I find that the model's predictions are not actually usable for a particular use case or are not useful for solving a problem. I have been involved in a project to
There are many reasons why this happens, but the ones I've commonly found are:
- Solution-driven initiatives: Even before GenAI, machine learning, and predictive models were “cool” solutions, some efforts started with ML solutions.Let's predict churn“(user or customer who left the company)”, “Let's predict user segments“…the current GenAI hype further exacerbates this trend, putting pressure on enterprises to properly integrate GenAI solutions “everywhere.”
- Lack of end-to-end design of the solution: In very rare cases, predictive models are standalone solutions. However, models and their predictions are typically integrated into larger systems to solve specific use cases or enable new functionality. If this end-to-end solution is not defined from the beginning, the model may prove useless once implemented.
To get your ML initiative off to a good start, it's important to: Start with a good problem to solve. This is the foundation of product management and iteratively strengthens product leaders to: marty kagan and melissa perry. This includes product discovery (through user interviews, market research, data analysis, etc.), opportunity sizing and prioritization (by considering quantitative and qualitative data).
Once an opportunity is identified, The second step is to consider potential solutions Machine learning and GenAI techniques should be included if they help solve the problem.
If you decide to try a solution that involves the use of predictive models, The third step is to define and design the solution or system end-to-end.. In this way, you can ensure requirements for how the system uses predictions and influence the design and implementation of the prediction part (what to predict, what data to use, real-time or batch, technical feasibility checks, etc.).
However, I would like to add that it is possible. Notable exceptions to this topic. If this technology is ultimately going to truly revolutionize your field and the world as we know it, it makes sense to start with the GenAI solution rather than the problem. There's a lot of talk about this, but it's not yet clear whether it will happen. Until now, we have seen this revolution in very specific fields (customer support, marketing, design, etc.) and related to people's efficiency in performing specific tasks (coding, writing, creating, etc.). I did. But for most companies, unless it is considered an R&D effort, delivering value in the short/medium term means focusing on the problem and using GenAI among other potential solutions to the problem. It should mean considering it as well.
Harsh experiences also lead to this learning. These experiences had in common large ML projects defined in a waterfall manner. This type of project is expected to take 6 months and will follow the ML lifecycle step by step.
What could be the problem? Let me remind you of my previous quote “The goal of predictive models is to predict things that may or may not be predictable.”! In this situation, you might arrive at the fifth month of the project and, while evaluating the model, realize that the model cannot predict what you need it to predict with sufficient quality. To make matters worse, he arrives at month 6 with the supermodel deployed to production and realizes that it doesn't add any value.
This risk, combined with the uncertainty associated with the product, makes it imperative to avoid large-scale waterfall initiatives where possible. This is not new, nor is it only relevant to ML initiatives. As such, there is much to be learned from traditional software development, Agile, Lean, and other methodologies and ways of thinking. By starting small, quickly and continuously testing hypotheses, experimenting and scaling, you can effectively mitigate this risk, adapt to insights, and become more cost-effective.
While these principles are well-established in traditional software and product development, applying them to ML initiatives is a little more complicated because defining “small scale” for ML models and deployments is not easy. However, there are some approaches that can help you start your ML efforts small.
Rule-based approach, simplifying predictive models through decision trees. In this way, there is no need to deploy a model, and “predictions” can be easily implemented as “if-else statements” in production as part of a feature or system.
proof of concept (POC) is used as a method to offline validate the predictive feasibility of an ML solution and suggest the feasibility (or not) of a predictive step in a production environment.
minimum viable product (MVP) focuses on key features, capabilities, or user segments first and expands the solution only when its value is proven. For ML models, this means, for example, predicting only the simplest preferred input features, or only segments of data points.
buy instead of build, Leverage your existing ML solution or platform to reduce development time and initial costs. Only if it proves to be worthwhile and costs increase significantly, may it be the right time to decide to develop an ML solution in-house.
Using GenAI as MVP, For some use cases, especially when text or images are involved, you can use the genAI API as a first approach to solving the system's prediction step. For tasks like text classification, sentiment analysis, and image detection, GenAI models deliver impressive results. Once the value is validated and the cost increases too much, the team can decide to build certain “traditional” ML models in-house.
Although it is possible and fast to use GenAI models for image and text classification, they require complex models that are too large (expensive, uncontrollable, hallucinatory, etc.) for what simpler, controllable models can predict. Note that this means using . Some interesting analogies include the following ideas: Deliver pizza by truck: It's possible, but why use a bicycle?
Data is a recurring problem that data scientists and ML teams encounter when starting ML initiatives. How many times have you been surprised by data with duplicates, errors, missing batches, strange values… and how different it is from the toy datasets you find in online courses?
It is also possible that the required data simply does not exist. Tracking of specific events was not implemented, collection and appropriate ETL were recently implemented. I experienced this having to wait several months before I could start the data. Projects with sufficient historical and volumetric data.
This all relates to the following maxim:Garbage goes in, garbage goes out.”: An ML model is only as good as the data used to train it. In many cases, you have a greater chance of improving your solution by improving the data than by improving the model (Data Centric AI ) The data must be of sufficient quantity, historicity (data generated over several years may have more value than the same amount generated in just one week), and of sufficient quality. To achieve this, mature data governance, collection, cleaning, and preprocessing are critical.
from ethical AI From a perspective, data is also a major source of bias and discrimination, so it is of paramount importance to be aware of it and take action to reduce these risks. Data governance principles, privacy and regulatory compliance (e.g. EU GDPR) is also key to ensuring responsible use of data (especially when dealing with personal data).
and GenAI model This is turning around. Huge amounts of data are already used for training. When using these types of models, you may not need the quantity and quality of data for training, but rather fine-tuning (see Good Data = Good GenAI) and building prompts (nurturing context, few shots). learning) may be required. , search extension generation… — all these concepts were discussed in a previous post!).
It is important to note that when using these models, you lose control over the data used to train the model and can suffer from a lack of quality or variety of data used therein. There are many known examples of bias and discrimination in GenAI's output. That could have a negative impact on our solution. A good example is the Bloomberg article, “How ChatGPT is a recruiter’s dream tool — tests show it’s racially biased.” In this sense, LLM leaderboards that test for biases, or LLMs specifically trained to avoid these biases, are useful.
We understand what makes ML product efforts particularly challenging: the combination of uncertainties associated with developing solutions for digital products and the uncertainties associated with trying to predict things through the use of ML models. I started this blog post to discuss.
It's reassuring to know that there are actionable steps and strategies available to reduce these risks. But perhaps the best one has to do with getting your efforts off the ground. To that end, starting with the right problem and end-to-end design of the solution, reducing the initial scope and prioritizing data quality, quantity, and historical accuracy can be very helpful.
I hope you found this post helpful and helpful as you take on new endeavors related to ML products.
