Luma Labs, the AI company known for developing generative 3D model “Genie,” has entered the AI-powered video tech space with its new product “Dream Machine,” which has garnered rave reviews among social media users. The product competes directly with OpenAI's upcoming Sora tool, among other new AI video makers.
What is Luma AI?
Luma AI is a platform that allows you to create 3D models using text, images, and video inputs. The tool helps professionals in architecture, product design, content creation, research and development, and more, come up with abstract concepts. The company streamlines the process of exporting your work across different platforms.
3D models are also driving meta-activities such as gaming, virtual reality, and e-commerce by simplifying the creation of 3D online spaces.
Developed by LumaLabs in San Francisco, California, Luma AI leverages knowledge of machine learning, augmented reality, and 3D photography to merge the fields of AI and 3D modeling.
Using NeRF (Neural Radiance Fields) technology, images and videos taken with your smartphone are transformed into highly realistic 3D models, NeRF alters ordinary scenes and objects by adding an element of realism.
Co-founder and CEO Amit Jain was previously with Apple Vision Pro and Apple's computer vision team, which he now leads.
Among other gadgets, the company's latest contribution is the Dream Machine, an AI model designed to create high-quality, realistic videos from text and images. So far, it has produced accurate, consistent, and dynamic visuals. The tool is on its way to creating a universal imagination engine, now available to everyone.
“We built Dream Machine on a scalable, efficient, and multi-modal transformer architecture and trained it directly on video,” Luma Labs said in a statement on its blog page.
They describe the tool as an “imagination engine,” saying, “We find joy and purpose in making things. At Luma, we're building general-purpose AI systems that help people make beautiful, powerful, and creative things that are unavailable or simply impossible.”
How have people responded to Dream Machine?
The launch of Dream Machine caused a surge in demand, which overloaded Luma Labs' servers, so they had to implement a queuing system.
Introducing Dream Machine – a next-gen video model that uses AI to create high-quality, realistic shots from text instructions and images. Available to everyone today. Try it for free here https://t.co/rBVWU50kTc #LumaDreamMachine pic.twitter.com/Ypmacd8E9z
— Luma AI (@LumaLabsAI) June 12, 2024
She's alive 🤯 pic.twitter.com/dnr3nPC1Aw
— fofr (@fofrAI) June 13, 2024
B.P.O. and Barber – My kids' precious stuffed animals came back to life tonight translator And a little effort from Dad. 🐦⬛ I'll show you tomorrow morning. pic.twitter.com/ELm4bNtokW
— Stormcrow (@Stormcrow_JE) June 13, 2024
The early videos posted on social media by previous Dream Machine users were incredibly polished and stunning, as if only the best results were selected.
First #Luna The AI and the results were amazing. I switched careers from film and animation to development a few years ago and am happy to hang on for a few more years. pic.twitter.com/Yn6JegVA8P
— Amir | Ξ𝕏𝕏ΞO (@Amir_EXXEO) June 13, 2024
The text-to-video generation field is fiercely competitive, with major players such as OpenAI's Sora and Lightricks' LTX Studio both offering excellent video results.
Other startups doing similar things include Pika Labs Inc. and Runway Inc., both of which focus on generating video content from text.
How to use Luma AI's Dream Machine
To get the most out of the tool, users must enter more detailed prompts than they would with a tool like ChatGPT. The company says that by entering, for example, “A grazing cow moves slowly across a tranquil pasture, with the camera following the cow with smooth lateral movements,” it can create a highly realistic five-second video clip showing that exact scenario.
The company says the Dream Machine should be able to create such a video in just two minutes or less.
I tested it with ReadWrite and got some interesting results. I used the following prompt: “A dream scene in which a woman climbing a mountain transforms into an eagle and flies away”
It's pretty accurate at first, but then does some weird stuff, like cloning itself and showing two women, one standing and the other flying away.
The second prompt was even less successful. “A scene where a sports bike traveling along Route 66 in the US transforms into an eagle”
Instead of the motorbike driver transforming into an eagle, he speeds sideways and has a minor accident while the bird appears to fly away randomly. What we discovered is that for best results, you need to be pretty specific with your text prompts.
For the final test, we used prompts suggested by Dream Machine itself. “An elderly woman smiling underwater in a scuba diving suit. Her expression conveys calm and happiness.”
The unique recommendations worked pretty well: You can't zoom in and find the distorted eyes that you see in a lot of AI images, but they do seem very realistic.
Is Luma AI's Dream Machine free?
As mentioned before, it's open to anyone interested, but you'll need to register an account with your email address. Once you've done so, you'll get a free plan with 30 generations per month, while more expensive plans cost $499 and allow up to 2,000 generations per month.
Currently, the service is in high demand and access may be limited; Dream Machine's website notes that “generation will take 120 seconds, but due to high demand your request will be queued.” The ability to create 30 videos for free is useful, especially since most people probably won't need more than that for personal use.
One of Dream Machine's biggest advantages is its open-source framework with Luma AI. In contrast, OpenAI's Sora, while impressive, is only available to a select number of test users, and the company has hinted that key features will be reserved for paid subscribers, similar to the ChatGPT model.
Meanwhile, Dream Machine is currently available to anyone who wants to try out its features, and in the future Luma AI plans to release various APIs and plugins that will allow integration with creative software tools such as Adobe.
Featured Image: Luma Labs / Canva
