Nonetheless from “A teddy bear washing dishes,” as generated by Google Imagen Video.
At the moment, Google introduced the event of Imagen Video, a text-to-video AI mode able to producing 1280×768 movies at 24 frames per second from a written immediate. Presently, it is in a analysis part, however its look 5 months after Google Imagen factors to the speedy growth of video synthesis fashions.
Solely six months after the launch of OpenAI’s DALLE-2 text-to-image generator, progress within the area of AI diffusion fashions has been heating up quickly. Google’s Imagen Video announcement comes lower than every week after Meta unveiled its text-to-video AI instrument, Make-A-Video.
In keeping with Google’s research paper, Imagen Video contains a number of notable stylistic talents, reminiscent of producing movies primarily based on the work of well-known painters (the work of Vincent van Gogh, for instance), producing 3D rotating objects whereas preserving object construction, and rendering textual content in a wide range of animation types. Google is hopeful that general-purpose video synthesis fashions can “considerably lower the issue of high-quality content material era.”
The important thing to Imagen Video’s talents is a “cascade” of seven diffusion fashions that rework the preliminary textual content immediate (reminiscent of “a bear washing the dishes”) right into a low-resolution video (16 frames, 24×48 pixels, at 3 fps), then upscales it into progressively greater resolutions with greater body charges with every step. The ultimate output video is 5.3 seconds lengthy.
Video examples offered on the Imagen Video web site vary from the mundane (“Melting ice cream dripping down the cone”) to the extra unbelievable (“Flying via an intense battle between pirate ships on a stormy ocean.”) They comprise apparent artifacts, however present extra fluidity and element than earlier text-to-image fashions reminiscent of CogVideo that debuted 5 months in the past.
One other Google-adjacent text-to-video mannequin additionally formally debuted at this time. Known as Phenaki, it may possibly create longer movies from detailed prompts. That, together with DreamFusion, which may create 3D fashions from textual content prompts, reveals that aggressive growth on diffusion fashions continues quickly, with the variety of AI papers on arXiv growing exponentially at a price that makes it tough for some researchers to keep up with the most recent developments.
Coaching knowledge for Google Imagen Video comes from the publicly obtainable LAION-400M image-text dataset and “14 million video-text pairs and 60 million image-text pairs,” in line with Google. Consequently, it has been skilled on “problematic knowledge” filtered by Google however nonetheless can comprise sexually express and violent content material —in addition to social stereotypes and cultural biases. The agency can be involved its instrument could also be used “to generate pretend, hateful, express or dangerous content material.”
Consequently, it is unlikely we’ll see a public launch any time quickly: “We’ve determined to not launch the Imagen Video mannequin or its supply code till these considerations are mitigated,” says Google.
