Game Engine Learning from Video

Intelligent agents need to be able to make predictions about their environment. In this work we present a novel approach to learn a forward simulation model via simple search over pixel input. We make use of a video game, Super Mario Bros., as an initial test of our approach as it represents a physics system that is significantly less complex than reality. We demonstrate the significant improvement of our approach in predicting future states compared with a baseline CNN and apply the learned model to train a game playing agent. Thus we evaluate the algorithm in terms of the accuracy and value of its output model.

Acknowledgments
We gratefully acknowledge the NSF for supporting this research under NSF award 1525967. Special thanks to Brent Harrison for advice and draft comments. This paper contains images generated by the Infinite Mario Bros. game.