As many have pointed out, the last Apple WWDC was heavily focused on machine learning and took place right after Apple launched the new iPad Pro and iPad Air. To everyone's surprise, the iPad Pro came equipped with a new M4 chip, leading some users to ask if they would upgrade: does the iPad Pro need an M4 chip?
The answer depends on your perspective on applications, privacy and latency in the world of AI. If you want to run cool apps like Luma AI to create NeRF in the cloud, then obviously the computations won't be done locally so the perceived difference will be smaller.
Features you'll love
There are a lot of key things about the iPad Pro: Firstly, it's incredibly thin – just 5.1mm – and it also has a great P3 screen with 1600 nits peak brightness and an adaptive 10-120Hz refresh rate.
The new pen is great with improved performance, orientation and clickability, but that alone may not be enough to buy you a new M4 iPad. The most notable and impressive addition is the updated Magic Keyboard, which features an aluminum palm rest, a row of function keys and a large glass trackpad with haptic feedback and multi-touch gestures. do (That alone is well worth the upgrade.) Oddly, the regular travel case no longer covers the pen when attached to the iPad Pro, which means the pen can easily fall out of its charging position in your bag, a strange step backwards.
But the main benefit for M&E professionals is the processing on the iPad. There's no need to upload footage to the cloud when using FCP or DaVinci's Resolve. These packages are doing more and more machine learning processing and that's only expected to increase. From roto to expert color correction, it's clear that localized processing is key and the M4 delivers.
A great example of what we're going to see more and more of is the ability to work with spatial video in these editing programs and immersive content. When you're working with spatial video, you're essentially working with high-resolution stereoscopic dual images, which is very high bandwidth. In the case of immersive content that wraps around you with Apple Vision Pro, it's very high-resolution content, it's super high dynamic range.
More and more great apps are letting you work with your data locally, improving responsiveness and user experience. Even the new calculator on your iPad offers great new machine learning features you never knew you needed. Apple has publicly stated that it aims to introduce new features that can be used on older devices. However, in the case of the Apple Intelligence feature introduced at WWDC '24 this year, the limitation is the hardware requirements.
Using large language models and other machine learning approaches can be very computationally expensive, so the size of the Apple Neural Engine determines whether these features will be fast enough on device without the cloud. In theory, some of these features could run on older systems, but it would be slow and cumbersome. Naturally, the M4 chip has the most capable neural engine, and the Core Machine Learning update allows advanced generative machine learning and AI models to be optimized and run faster and more efficiently on device. This will allow the iPad to use machine learning and other mobile devices to perform many more everyday functions.
In addition to the traditional apps we've been using, there are new apps made possible by the powerful Apple Neural Engine: Many will point out that this is most relevant for games, but for M&E artists it will more likely involve animation tools and apps like Procreate Dreams.
These animation tools offer an incredible level of control on a device such as a mobile tablet that can easily be used on a train, a plane, or even while sitting in the park. Naturally, Mac Studio or other Mac models with professional digitizing tablets will be the first tool used by experienced animators and environment artists. Still, you have to admit that the power now available on the iPad is astonishing.
While fxguide doesn't tend to work with audio products, this ML applies to audio editing as well: the ability to remove background noise, refine a mix, and isolate instruments to work individually are all valid and powerful uses of machine learning that audio teams are already leveraging.
Going forward, as more ML audio tools are added to products like FCP and Resolve, the demand for editing software for more advanced local ML processing will only grow. Audio will likely become a major ML tool, requiring inference locally on mobile devices. Google Deepmind has already released video-to-audio research that uses video pixels and text prompts to generate rich soundtracks. “Video generation models are advancing at an incredible pace, but many current systems can only generate silent outputs. One of the next big steps to bring generated movies to life is creating soundtracks for these silent videos,” Google announced last week. In addition to creating scores and soundtracks for generated videos, these new approaches enable the creation of sound effects and automatic synchronization with the video. For example, they can generate Foley footsteps or, as Google has already shown, generate video-to-audio (V2A) drumming to accompany a video of a drummer.
These multimodal applications are expected to continue to grow in popularity, and local ML processing eliminates the need to upload dense video files to the cloud, making these innovations more responsive and with lower latency, resulting in faster processing.
Apple encourages the use of more AI on devices for both performance and security reasons, and clients and many artists prefer not to have their material uploaded, regardless of the development company involved, as this ensures that their material is not aggregated for use as training data without their consent.
Generally, the more processing that can be done locally, the better the user experience. Adobe Lightroom offers AI tools such as automatic object removal that runs in the cloud. For still images, latency isn't a big issue, but for more complex tasks or those requiring multiple frames, local processing is ideal.
The iPad Pro is a big step forward from the original iPad and iPad mini. Advances in processing power, screen resolution and size, and the Neural Engine all transform the iPad from a passive reading and viewing device into a creation tool.
