Researchers have developed a novel AI model, dubbed “MarioVGG,” that can simulate gameplay from Super Mario Bros. based solely on video footage. Although the results exhibit glitches and are too slow for real-time gameplay, this development showcases how a limited model can infer impressive physics and gameplay dynamics through video analysis and input data.
The team behind MarioVGG aims to create a reliable and controllable video game generator, possibly even substituting game development and game engines entirely with video generation models in the future. The researchers initiated the project by training their model on 737,000 frames from Super Mario Bros., excluding levels 1-1, to reserve it for evaluation purposes. These frames were preprocessed into 35-frame chunks, enabling the model to learn the immediate results of various inputs.
To simplify the scenarios, the researchers focused on two inputs: “Run right” and “Run right and jump.” This limited set of movements posed challenges, as the system had to look back a few frames to determine the start of a “run.” Any jumps that required mid-air adjustments were omitted to avoid introducing noise into the training dataset.
ai-generated gameplay dynamics
After preprocessing and around 48 hours of training with an RTX 4090 graphics card, the team used a standard process to generate new frames from a static game image and a text input, either “run” or “jump.” Although only a few frames long, these sequences can be linked to create extended gameplay that maintains coherent and consistent dynamics.
Despite the promising results, some still question whether the AI has grasped the fundamental principles of Mario’s universe. Skepticism exists about the AI’s ability to apply learned concepts to new or slightly varied situations within the game. The MarioVGG project represents a significant step towards the future of video game development, where AI models could enhance or even replace traditional game engines.
However, the current limitations, such as occasional failure to follow user commands, visual glitches, and other anomalies like Mario transforming into other characters or disappearing, underscore the need for longer training periods and more diverse gameplay data to refine the model. As research continues, models like MarioVGG could evolve into sophisticated tools, potentially transforming video game development and interactive media. While AI models like MarioVGG are not going to replace video game developers or the standard engine any time soon, the idea of just explaining to an AI how you want a game’s physics and environments to function instead of coding it all manually is far too tempting for technical obstacles to discourage.
Noah Nguyen is a multi-talented developer who brings a unique perspective to his craft. Initially a creative writing professor, he turned to Dev work for the ability to work remotely. He now lives in Seattle, spending time hiking and drinking craft beer with his fiancee.
