Mid-sized companies have successfully built data and ML platforms, and building AI platforms is now extremely challenging. This post explains three important reasons why building an AI platform should be careful, and instead proposes my thoughts on promising directions.
Disclaimer: It is based on personal views and does not apply to cloud providers or data/ML SaaS companies. Instead, research on AI platforms should be doubled.
Where am I coming from?
In my previous article From data platforms to ML platforms We shared how data platforms evolve into ML platforms towards data science. This journey applies to most small businesses. However, there was no clear path for small businesses to continue developing their platforms into AI platforms. still. With leveling up to the AI platform, the path branched out in two directions.
- AI infrastructure: “New electricity” (AI inference) is more efficient when generated centrally. This is a game for major engineers and large model providers.
- AI Application Platform: You cannot build a “beach house” (AI platform) on constantly moving ground. Evolving AI capabilities and new, new development paradigms make persistent standardization challenging.
However, despite the continued evolution of AI models, there are still directions that are likely to be important. It is covered at the end of this post.
High barriers to AI infrastructure
Databricks is only a few times better than your own Spark jobs, but Deepseek can be 100 times more efficient than LLM inference. Training and services for the LLM model requires significant investment in infrastructure, and, importantly, control over the structure of the LLM model.
In this series, we briefly shared the infrastructure of LLM training. This includes parallel training strategies, topology design and training acceleration. On the hardware side, in addition to high-performance GPUs and TPUs, a significant portion of the cost was spent on networking setups and high-performance storage services. The cluster requires an additional RDMA network to enable non-blocking, point-to-point connections for data exchange between instances. Orchestration services should support complex job scheduling, failover strategies, hardware issuance detection, and GPU resource abstraction and pooling. The training SDK should promote asynchronous checkpoints, data processing, and model quantization.
When it comes to model services, model providers often incorporate inference efficiency during the model development stage. Model providers are likely to have better model quantification strategies and produce the same model quality that will significantly reduce model size. Model providers may develop better model parallelism strategies to control the model structure. It can increase batch size during LLM inference, effectively increasing GPU usage. Additionally, large LLM players have the advantage of logistics that allow them to access cheaper routers, mainframes and GPU chips. More importantly, average model providers with stronger model structure control and improved model parallelism capabilities can take advantage of cheaper GPU devices. Deprecation of GPUs can be a greater concern for model consumers who rely on open source models.
Take DeepSeek R1 as an example. Let's say you're using a P5E.48XLARGE AWS instance. It costs $35 per hour. Assume you're doing it the same as NVIDIA and achieve 151 tokens/second performance. Generating 1 million output tokens costs $64 (1 million /(151 * 3600) * $35). How much does Deepseek sell tokens for 1 million people? 2 $ only! DeepSeek can achieve 60 times the efficiency of cloud deployments (assuming a 50% margin from DeepSeek).
Therefore, the inference power of LLM is certainly like electricity. This reflects the diversity of applications that LLM can power. It also means that it is most efficient when generated centrally. Nevertheless, as hospitals have generators for emergency situations, self-hosted LLM services are required for privacy-sensitive use cases.
Continuously moving ground
Investing in AI infrastructure is a bold game, and there are hidden pitfalls in building lightweight platforms for AI applications. The rapid evolution of AI model capabilities makes the paradigm of AI applications unaligned. Therefore, there is a lack of a solid foundation for building AI applications.
The simple answer to that is: be patient.
Taking the overall view of the data and the ML platform, the development paradigm only emerges when the functionality of the algorithm converges.
| domain | The algorithm appears | A solution appears | Big platforms are emerging |
| Data Platform | 2004 – MapReduce (Google) | 2010–2015 – Spark, Flink, Presto, Kafka | 2020 – Now – Databricks, Snowflake |
| ML Platform | 2012 – Imagenet (Alexnet, CNN breakthrough) | 2015–2017 – Tensorflow, Pytorch, Scikit-Learn | 2018 – Now – Sagemaker, Mlflow, Kubeflow, Databricks ML |
| AI Platform | 2017 – Trans (The only thing you need to be careful is caution) | 2020–2022 —Chatgpt, Claude, Gemini, Deepseek | 2023 – Now – ?? |
After years of intense competition, several large model players are standing in the arena. However, the evolution of AI functions has not yet been converged. With advances in the capabilities of AI models, existing development paradigms will soon become obsolete. Large players are just beginning to stab them on agent development platforms, and new solutions are appearing like popcorn and ovens. I believe the winner will eventually appear. For now, standardizing agents itself is a difficult call for small and medium-sized businesses.
Path dependencies for old success
Another challenge in building an AI platform is rather nuanced. This is to reflect the thinking of platform builders, whether they have path dependencies or not from the previous success of building data and ML platforms.
As previously shared, since 2017, the data and ML development paradigms have been well aligned, with the most important tasks for the ML platform being standardization and abstraction. However, the development paradigm for AI applications has not yet been established. If your team follows the previous success story of building data and ML platforms, you could end up prioritizing standardization at the wrong time. The possible directions are as follows:
- Building an AI Model Gateway: Provides centralized auditing and logging of requests to LLM models.
- Building an AI Agent Framework: Develop a self-built SDK for creating AI agents with enhanced connectivity to the internal ecosystem.
- Standardize RAG Practices: Build standard data indexing flows to lower the standard of engineer build knowledge services.
These initiatives are certainly important. But ROI really depends on the size of your company. Anyway, you have the following challenges:
- About the latest AI development.
- Customer adoption rate when customers can easily bypass abstractions.
Assuming that data builders and ML platforms are like “closet organizers”, AI builders need to act like “fashion designers”. It is necessary to embrace new ideas, carry out rapid experiments, and even accept levels of imperfection.
My thoughts on promising directions
There are so many challenges ahead, but remember that working on an AI platform is still happy right now, with a considerable amount of leverage that has never been before.
- The conversion ability of AI is more important than the conversion ability of data and machine learning.
- The motivation to adopt AI is stronger than ever.
Choosing the right direction and strategy is important to the transformation that can be brought to your organization. Below are some of my thoughts on the directions that could lead to less confusion as the AI model scales further. I think it's equally important to make AI platforms a reality.
- High Quality, Rich Semann Data Products: Data products with high accuracy and accountability, rich explanations, and reliable metrics will have more impact on the growth of AI models.
- Serving OLTP, OLAP, NOSQL, and ElasticSearch, the scalable knowledge services behind an MCP server: MACTP, OLAP, NOSQL, and ElasticSearch may require multiple types of databases to support the delivery of high-performance data. It is difficult to maintain a single source of truth and performance with constant inverse ETL jobs.
- AI DevOps: AI-centric software development, maintenance and analysis. Over the past 12 months, the accuracy of code generation has increased significantly.
- Experiment and Monitoring: Given the increased uncertainty of AI applications, evaluation and monitoring of these applications is even more important.
These are my thoughts on building AI platforms. Please let us know what you think about that. cheers!
