Originally published on Forbes
When Henry Castellanos first presented his machine learning model to company executives, he found himself struggling with the kind of self-doubt that is so common and almost universal among data professionals.
On the other hand, his models looked great. It did a powerful job of predicting which dental patients would not show up for their appointments, allowing medium to large dental clinic chains to strategically double-book high-risk times, similar to overbooking airline flights. The project promised healthy profits. If Henry’s model predicts accurately enough, companies could dramatically reduce the high cost of empty dental chairs, while also largely avoiding the impact that occurs when two patients show up for the same appointment.
But on the other hand, compared to the magic crystal ball, Henry’s model was the worst. This was about twice as good as a random guess, but if we had a magic crystal ball we could have easily outperformed this prediction by flagging no-shows and only no-shows without error. Specifically, if we use Henry’s model to flag the top 10% of patients at highest risk, those most likely to be no-shows, we find that about half never actually show up. This is about twice as good as guessing at random (as about a quarter of all appointments were no-shows).
For Henry, it didn’t necessarily bother him that a virtual clairvoyant model would defeat his abilities. Like almost every qualified data scientist, he knew that: Uncool models are valuable – The magic crystal ball is just an illusion, and the best we can hope for from ML is to predict more accurately than to guess. Still, predicting rather than guessing is more than enough to improve your operational “numbers game.” Big win in terms of revenue.
However, Henry still had to actively use his model to sell the business.
Everyday failures in the machine learning industry
As the meeting began, Henry felt: The numbers are healthy, but I had no intention of persuading the executives..
“Ultimately, I didn’t have confidence in myself.” he told me during a video interview. “There was no direct answer to the question of whether my model was actually worth it, and I was wondering how I could answer it. Really Communicate the operational and financial implications of using the model. ”
Henry technically verified his model. Although this is an industry standard and generally considered good enough, it is incorrect. The internal tug of war he experienced at the time is unique to this country. Predictive AIOccupation. In this field of technology, we are taught to create sound models, but then we are taught to screen those models only with respect to that sound model. relativeAnalyze predictive performance, or technical performance, without making data-based predictions. absolute The business value you get from using it.
This standard but flawed practice pays no attention to the obvious universal maxim. In other words, you can’t sell anything without first understanding the customer’s problem and seeing things from the customer’s perspective. When it comes to predictive AI projects, the selling point is the use of models. And the business side of the data scientist doesn’t care at all that the model makes predictions that are “2x better than guessing.”
Instead, they care about money and other KPIs.
Q: Is the model better? A: Who knows?
To be clear, as a geek myself, I’m certainly interested in that kind of technological means. comparatively good performance. This means that the model works exactly as it was trained. ML discovered a pattern that generally holds true. It is now encoded as a model that can be used to change probabilities in the numbers game known as “doing business.” 2x better than predicted means the model has the following characteristics: lift Out of two. Lift is one of the few standard metrics used by the ML industry to evaluate models. Other metrics include precision, recall, F-score, and AUC.
But none of these high-level metrics can serve the customer, or the business, on their own. They all accomplish variations on the same theme. That is, we see that the model performs relatively well, but reveals little about its potential absolute value. They are helpful, but not sufficient.
Therefore, by sticking to these standard metrics, data scientists are unable to answer the most obvious questions about the model they are trying to sell.How good is it?” Without it Anchor model performance to valuethe answer to this question about the goodness of the model is still subjective. Without a business value estimate, you can just as easily claim that your model is “bad” as it is “good.”
Ironically, the most formal and technical indicators obscure things. Without further insight, they leave buyers’ decisions at the mercy of their whims and whims. Rationality usually prevails and ill-informed decision makers ignore launches. The model is never used and the project realizes no value.
This tragic accident continues to this day. After decades of progress and numerous waves of hype, predictive AI still routinely follows a process that is doomed to stall and fail.
- We train our models using “rocket science” known as ML algorithms (good!).
- Evaluate models only in terms of technical metrics that cannot assess potential value (bad!).
- Unable to convince business stakeholders to use the model – therefore Most ML models fail to deploy.
Instead, communicate financial performance to stakeholders
After a typical data scientist experience of feeling that something was missing in their sales pitch, and an initially disillusioned but ultimately lukewarm response from stakeholders (aka customers), Henry made a decisive and fundamental shift. He moved on to showing executives what was important: profits. The model is Expected to generate an additional $500,000 in revenue. In terms of annual income.
Henry made sure the models knew exactly what they were expected to do. By double-booking flagged appointments, businesses can avoid a constant number of empty dental chairs and save hundreds of dollars each time. This process sometimes resulted in inadvertent double bookkeeping, each time causing inconvenience and financial loss (such as the loss of dissatisfied patients), but the ultimate payoff was likely great.
Attending this conference was a completely different experience. Henry felt confident as an expert with the fundamentals to win sales. “I felt like this gave me the validation that I needed to be able to go into a meeting and confidently say, “This model is going to make money.” Henry’s boss, and his boss’ boss, were excited.
The lesson is clear. Data scientists, that nagging feeling, that certain lack of confidence, is telling you something. Solutions to business problems are more than just predictions comparativelygood. The solution is to predict it well enough to prove it. absolutely Precious. When data experts adopt the old and popular method and do not provide data Visualization of potential valuethey are very unlikely to do so Sell your business using ML models.
For more information about Henry’s project, Watch this video webinar, demo and interview.
About the author
Eric Siegel is a leading consultant helping companies implement machine learning and a former professor at Columbia University. He is the founder of the long-running Machine Learning Week conference series, the instructor of the highly acclaimed online course Machine Learning Leadership and Practice – End-to-End Mastery, editor-in-chief of The Machine Learning Times, and frequent keynote speaker. He wrote the best-selling book Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, and Die, which is used in hundreds of college courses. He also wrote The AI Handbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary efforts bridge the stubborn technology-business gap. At Columbia University, he received an Outstanding Faculty Award for teaching a graduate computer science course in ML and AI. He later served as a business school professor at UVA Darden. Eric has also published analytical and social justice editorials. You can follow him on LinkedIn
