A few months ago, OpenAI showed off “Sora,” a product that can generate videos based on a short prompt, much like ChatGPT does for text or DALL-E does for images,
and I asked myself a pretty simple question:
"...how can someone actually make something useful out of this?" and "how do I get this model to do the same thing every time without fail?" While an error in a 30-second-long clip might be something you might miss, once you see one of these strange visual hallucinations it's impossible to ignore them.
A month later, OpenAI would debut a series of short films, including one called “Air Head," a minute-and-twenty-second-long piece about a man with a balloon for a head, one that changes sizes 23, 24, 26, 27, 29, 32, 34, 39, 41, 42, 43 and 45 seconds into the piece, at which point I stopped counting because it got boring.
The very nature of filmmaking is taking different shots of the same thing, something that I anticipated Sora would be incapable of doing as each shot is generated fresh, as Sora itself (much like all generative AI) does not “know” anything. When one asks for a man with a yellow balloon as his head, Sora must then look over the parameters spawned during its training process and create an output, guessing what a man looks like, what a balloon looks like, what color yellow is, and so on.
It repeats this process for each shot, with each “man with balloon as his head” character subtly (or not-so-subtly) different with each repetition, forcing users to pick the outputs that are the most consistent. Achieving a perfect like-for-like copy isn’t guaranteed, but rather, filmmakers must pick the ones that are “good enough.”
This becomes extremely problematic when you’re working in film or television, where viewers are far more likely to see when something doesn’t look right — a problem exacerbated by moving images, high-resolution footage, and big TV screens.
www.wheresyoured.at
