Nvidia announces the creation of an artificial intelligence that can create 3D models based on text

Nvidia, the world’s leading manufacturer of GPUs, has announced the creation of Magic3D, a generative AI that can create 3D models based on a text description. Magic3D creates a 3D mesh model with a colored texture in approximately 40 minutes.

In the presented video, Nvidia specialists generated a 3D model of a frog after entering the phrase: “blue frog with poison dart sitting on a water lily”. The obtained object, after applying the necessary improvements, can be used in CGI graphic scenes or video games. Nvidia treats Magic3D as a response to DreamFusion – a system for converting text into 3D objects presented by Google researchers in September 2022.

Nvidia’s research and development team argues that their technology will allow anyone to create 3D models without the need for special training. “When refined, the resulting technology could accelerate the development of video (and VR) games and perhaps eventually find application in special effects for movies and television. We hope that with Magic3D we can democratize 3D synthesis and open up everyone’s creativity in creating 3D content.”

Magic3D uses a two-step method that takes a rough model created at low resolution and optimizes it to a higher resolution, using a text-to-image model to produce a 2D image. This is then optimized to a volumetric NeRF (Neural Radiation Field). According to Nvidia, the Magic3D modeling technology allows you to generate them twice as fast as DreamFusion.

Magic3D can also perform quick 3D mesh editing. Given a basic prompt and a low resolution 3D model, the text can be modified to alter the resulting model. Additionally, the authors of Magic3D illustrated keeping the same output design through multiple iterations and implementing a 2D image style (such as Cubist painting) into a 3D model.

Source: www.research.nvidia.com