What frustrates me is the countless people online, in person, and even in my comments section saying, “How will AI replace data scientists?”
This is frustrating because opinions like this often come from people who don’t work in the field, and it discourages people who want to become great data scientists from pursuing this career path.
Needless to say, I categorically disagree with this view and believe that AI will not replace data scientists at least not within the next 10 years.
This is coming from someone who has worked in the space for five years at various companies and has seen what the industry looks like before and after AI.
I’m not at all worried about AI taking my job. In this article, I’ll explain exactly why I think this way and hope to put an end to this threatening behavior.
need to learn AI
Before I get into the actual “nitty-gritty” of the article, let me first say that I’m not a total AI hater.
I use AI every day and am consistently improving my skills in AI as it is a great productivity tool for:
- Creating boilerplate code
- Brainstorming technical ideas
- Preparation and drafting of documents
- Create data visualizations and charts quickly
- Overall a great and intelligent sparring partner
This technology is here to stay, so we need to learn how to use it. Otherwise, you will be left behind.
The ability to use AI tools will become the norm, just as everyone is expected to use email or know Microsoft Word today.
AI will not replace data scientists, but it may be replaced by individuals with less technical skills but greater AI proficiency.
Data scientists must be familiar with tools such as:
There are many others.
They will become a staple in our industry, just as Python has become the lingua franca for machine learning.
It is inevitable, so you must get on board as soon as possible.
there will be bigger problems
Let’s take a closer look at the skills AI will need to develop in order to fully replace data scientists.
- Break down vague business problems into framed mathematical systems or algorithms.
- Communicate with non-technical stakeholders and explain specific results with live questions.
- Always write error-free production code so that all business-critical decisions run smoothly.
- Consider both logical and human trade-offs between complexity, architectural design, and development processes.
- Build relationships and trust across your team, company, and industry.
If AI were to master all these skills to a level superior to that of today’s data scientists, what jobs would remain the same?
Most of them will become extinct as well.
If this happens, we’ll have much bigger problems to worry about, almost singularity-level problems, and your concerns about whether you should get a data science job will pale in comparison.
The AI singularity is a theoretical future point where artificial intelligence exceeds human intelligence, leading to rapid uncontrollable and irreversible technological growth.
If data scientists are replaced, we will have bigger fish to fish in our lives than just worrying about our careers.
lack of mathematical reasoning
One thing that is largely lacking in AI is mathematical reasoning.
I’m not talking about the amateur math that most people ask of AI:
- Please help me find the gradient of this function.
- Compute the determinant of this matrix.
- What is the formula for Fibonacci numbers?
“Mathematical reasoning” here refers to the ability to solve unsolved mathematical problems.
For example, AI is currently unable to solve the Riemann Hypothesis because it lacks the creativity and conceptual reasoning to make significant advances in pure mathematics.
The Riemann Hypothesis is a famous unsolved prediction that suggests that there is an underlying order hidden in the seemingly random distribution of prime numbers. It is centered around the “zero” of a complex mathematical tool called the Riemann zeta function, and proposes that all non-trivial zeros lie on a single vertical line (the “critical line”).
The Riemann Hypothesis is an extreme example because it is probably the most difficult problem to date.
However, it shows that AI does not surpass humans in mathematical ability, which is the basis of data science.
Most people forget that these AI models are actually a type of model called large-scale language models (LLMs), which are specifically designed to predict the next word from a pre-computed probability distribution.
These models can either output based on the data they see, or run only based on that output. They can only break away from what already exists, not necessarily create something “completely new.”
Data science jobs require you to develop new solutions to invisible problems. In fact, it takes data scientists and machine learning practitioners to initially build and maintain these AI models.
AI still makes mistakes
As someone who uses these tools every day in a variety of applications, AI makes a ridiculous number of mistakes.
These LLMs often “hallucinate.” This is a term you’ve probably heard before, and it occurs when these AI models produce plausible outputs, but are actually highly inaccurate.
This stems from the fact that they are probabilistic models in nature and can potentially “stitch together” meaningless words to meet user demands and expectations.
Humans also make mistakes, but the difference is that most humans only realize their mistakes after they have corrected them. Depending on the scenario, they are not very confident in their initial response.
AI, on the other hand, is so stubborn and smart and so sure of the answers it gives that it psychologically tricks us humans into thinking it’s right.
Imagine how uncomfortable this would be at work.
AI data scientists are unable to set expectations when implementing a given solution because they are unable to determine exactly how outrageous or ridiculous the output will be.
What we miss about many data science and machine learning projects is the lack of nuance and intangibles that we humans have.
performance limits
What’s interesting to me is that these AI models haven’t actually improved significantly over time.
There are two reasons.
- The underlying algorithm is still the same. All of these LLMs are transformer So each “new” model isn’t really all that “new”.
- There is a finite amount of information in the world, so there is a limit to the amount of data you can train on.
For example, OpenAI’s GPT model has essentially been trained to some extent on the entire internet, and there isn’t much “new” data to work with.
There is literally an upper limit to how good they can perform.
This data also comes from humans, so it cannot exceed human intelligence. That’s the ceiling.
These AI models will only improve further if there are large-scale scientific advances in the underlying algorithms.
And no further improvements mean the status quo remains the same: AI has not yet replaced data scientists.
unable to build relationships
Sadly, despite how many people empathize with these robots, AI is incapable of forming human relationships.
Humans are social creatures, and most of the world’s business interactions occur through relationships.
People do business with, employ, and work with people they like, even if that person doesn’t have the most “technical” qualifications.
It’s just how we are wired to behave from a biological standpoint.
Stakeholders will trust you as a data scientist if you deliver consistent results to them.
Even if AI finds a “better” solution to your problem, your stakeholders will likely prioritize you because of the invisible relationships you’ve built.
Every job is built on connections with people. Some parts will be automated, but much will not.
For data scientists, it is very difficult to automate:
- Data storytelling of technical issues to specific stakeholders
- Gather requirements from business leaders about the problem you want to solve
- Communicate and influence members of other teams and departments
All active parts of a human being are impossible to replace.
Has anything really changed?
One of my old line managers once asked me:
Has anything really changed since AI was introduced?
Sure, we now have better tools to solve certain problems and are more productive in certain aspects of our jobs, but honestly, the role of a data scientist hasn’t changed all that much.
Take a moment and think about how AI has substantially changed your daily life.
Even if there were, I wouldn’t be able to name many.
AI has been around in its current form for over four years now, and from where I stand, society as a whole has not been significantly affected.
That’s all there is to say here.
If you want to really dig deep into learning AI after reading this, I recommend my previous post which provides a complete and detailed roadmap of everything you need to master AI.
You can check it out below!
One more thing!
Join our free newsletter. Every week I share tips, insights, and advice from my experience as a data scientist and machine learning engineer. Additionally, as a subscriber, my Free resume template!
dishing the data
Weekly emails to help you land your first job in data science or machine learningnewsletter.egorhowell.com
