Despite all the hype surrounding generative AI, there’s a clear bias in the infrastructure that powers it: it’s built for text. While large-scale language models (LLMs) have benefited from years of intensive optimization and are now faster and cheaper to run, image and video generation (diffusion models) are largely built into existing systems.
result? AI video generation remains slow and prohibitively expensive.
Last week, an AI infrastructure startup machgen We emerged from stealth to change that. Founded by Kismat Singh and Manoj Krishnan, MachGen is building a high-performance inference and fine-tuning stack from the ground up designed specifically for diffusion models.
Their proposal is simple but ambitious. The goal is to reduce diffusion latency and computational cost by an order of magnitude, enabling interactive real-time video generation.Here’s what they put together blog post:
Genealogy of the founders of NVIDIA, Google, Intel, and Meta
MachGen’s team has spent their careers running AI models faster at some of the world’s largest technology companies.
The founders’ joint resumes include co-building the TRT high-performance inference team at Nvidia, leading data center AI software at Intel, driving the vLLM inference effort at Google, and building the large-scale model training infrastructure behind PyTorch at Meta.
As Singh and Krishnan watched the LLM inference stack mature from a clunky research prototype to a highly optimized production-grade system processing trillions of tokens, they noticed a familiar pattern emerging with adoption.
“We didn’t start MachGen to reduce existing bills by a few percent,” the founders said in the launch announcement. “If videos that take minutes can now be produced in seconds and at a fraction of the cost, we not only have a cheaper version of today’s products, but we also enable interactive production, real-time personalization, and tailored advertising at scale.”
Why the LLM Handbook Fails with Video
The central technical problem that MachGen seeks to solve stems from the fact that diffusion models are fundamentally different beasts than LLMs.
The optimizations that made LLM cheap (such as paged KV caching and continuous batch processing) are built on autoregressive patterns, predicting one token at a time. However, the diffusion model is non-autoregressive and relies on calculations throughout the “denoising” loop.
The industry is over-indexing LLMs, and existing service delivery frameworks are slowing adoption. To fix this, MachGen redesigned the stack across four key pillars.
- Note: MachGen uses low-precision recipes and sparse attention techniques to reduce the computing budget that typically has a large impact on video generation.
- caching: Since the standard LLM KV cache is not forwarded, MachGen built a unique cache recipe that takes advantage of the spatial and temporal redundancy inherent in the diffusion model.
- kernel: Although open source GPU kernels for LLM are abundant, diffusion kernels are lacking. MachGen has developed highly optimized kernels specifically for diffuse architectures.
- Parallel processing: To minimize the communication overhead between multiple GPUs, we implemented a customized kernel that overlaps communication with data-independent computations.
numbers
MachGen claims that these fundamental changes significantly improve performance without sacrificing quality or relying on extracted models.
The company’s internal benchmarks show a 4-6x reduction in latency and a 2-3x reduction in cost compared to standard inference deployments.
- Image generation: HiDream runs in 1 second (baseline is 6 seconds), Flux runs in 1.5 seconds (vs. 6.2 seconds).
- Video generation: Wan 2.2 runs in 16.5 seconds (vs. 98 seconds) and LTX 2.3 completes in 10.7 seconds (vs. 67 seconds).
MachGen meets the needs of developers of all sizes and brings this optimized stack to market in several ways. Products include a direct model API for instant access, MachGen cloud (a fully managed service), a self-hosted option for enterprise VPCs, and a serverless platform where customers can bring their own models and MachGen handles GPU provisioning and autoscaling.
North Star: World Model
While democratizing video generation is MachGen’s immediate go-to-market strategy, the company is playing the long game.
Their ultimate goal is a world model, an AI system that can understand, simulate, and reason about the physical world. This is the next frontier in AI, heavily backed by billions of dollars in funding and spearheaded by research institutions associated with Fei-Fei Li, Yann LeCun, and xAI.
Importantly, today’s world model is computationally similar to the diffusion model.
“Currently, there are essentially no production-grade inference solutions for world models, and we intend to be the first to build one,” said the founders. “What we do for a living right now is adoption. The global models are our North Star. When they come out, performance isn’t a nice-to-have. It’s going to be what determines whether they actually work.”
