At this year's Upper Bound conference, AI researchers, engineers, and entrepreneurs gathered to share the latest advancements and opportunities that artificial intelligence offers in their fields. Many sessions explored how machine learning is transforming creative pursuits, offering new tools for storytellers and video game designers, and creating more immersive and engaging experiences for audiences.

Creative AI: Lensing the Imagination

Max Kreminski, Director of Midjourney's Storytelling Lab, used his session “Creative AI: Lensing the Imagination” to argue that better interfaces could lead to deeper collaborative creativity between humans and artificial intelligence. 

Kreminski says that current large language models are pretty good at responding directly to the prompts that people put into the system. But they struggle with creating divergent content that goes off in creative new directions. Additionally, people tend towards short initial prompts, which end up producing generic responses from the LLM.

“When you’re faced with a blank text box, you think of a few things by default. And so people tend to go with things like ‘cat pirate.’ You see a lot of cat pirate worlds,” he says

This can lead to more homogeneity in the ideas produced by people using LLMs like ChatGPT as a creative tool, while scoring higher on other aspects of creativity, like flexibility and elaboration. 

Kreminski suggests that it is an information theory issue: by using short prompts, users are providing a small amount of creative input into the system, so most of the ideas have to come from the data that the LLM is trained on. This also makes the user feel like they don’t have creative ownership over the idea. When people put similar inputs into similar models, they get similar results back. 

There are several ways to tackle the issue, Kreminski says. One is to design the tool’s interface to get more creative input from the person using the tool, to prompt the user to think differently and deepen their own creative intent. 

This could involve breaking down the generative process into smaller units, allowing the user to make more frequent, smaller decisions to refine the work.  Other techniques include having the model explicitly ask the user open-ended questions to encourage more diverse inputs and generating small variations for the user to choose from.  

He also pointed out that while most LLM prompts are written out, that isn’t the only interface that is possible. He showed off some experimental interfaces they have played with, such as having the user draw visual representations of story and character arcs or moving objects around in a space to represent relationships between characters, which provides new ways of adding ideas to the creative system.

“We need to build interactive systems that elicit intent,” he says. “To achieve this, we need a novel user interface,  novel AI on the back end, and novel evaluation models.”

Infinite Ripples: Showcasing Alpha-Stage Generative Behaviour in Games

Brian Tanner and Alex Kearney, co-founders of Artificial Agency, showcased their work in using agentic AI to make video games more immersive and alive. 

In their session, “Infinite Ripples: Showcasing Alpha-Stage Generative Behaviour in Games”, the pair demoed the technology they hope will shape the future of how people will interact with video games.

"We imagine that tomorrow's games won't be scripted, they'll be alive —shaped by agentic AI and personalized to you."  

Their efforts have been mostly focused on two applications of agentic AI: building more in-depth characters, as well as "game directors" that can dynamically alter variables in the game world to heighten tension and enjoyment for the player. 

Tanner and Kearney emphasized the need to combine "game thinking" with "AI thinking."  While AI research often strives for general, simplified algorithms, game design is built on layers of interconnected systems and often requires planning for countless specific scenarios.  Their approach gives AI agents access to the game's systems in the same way a designer would, allowing for more abstract reasoning.

While many uses of AI rely on starting from scratch to learn their environment through trial-and-error, that approach doesn’t always work with building immersive games. Those early mistakes made by an agent learning its environment can take a player out of the experience — instead, the pair noted that starting agents off with some context about the world around them and their ability to affect offered much better results. 

Tanner also recommended offloading already-solved problems like agent pathfinding, which are already built into the game engines that video games are built on. This allows Artificial Agency to focus on building the complex, agent-driven behaviours that will power the next generation of games.

Tanner and Kearney wrapped up the session with several demos showing off the technology’s ability to understand the game world around it, as well as abstract thinking. The demos included agents who could do tasks based on prompts written in natural language, and a game director who could create a basic game level based on requests from the user.

The final demo showed many of these concepts combined into one system - a game director that controlled a simulated office of coworker characters. The director could create dynamic problems for the characters, like a blown breaker that shut down the power in the office, that the characters could reason out and solve - none of it being pre-scripted.

They noted they are now collaborating with game companies interested in the technology, with the aim of making new genres of games that aren’t possible with current methods.

“We are building tomorrow’s games with today’s technology,” Tanner says.

Iterative Generators for Procedural Content Generation

In his session, Ahmed Khalifa, a lecturer at the University of Malta's Institute of Digital Games, detailed how different types of AI can be used in creative fields.  He drew a distinction between "one-shot" generators, which create a complete work from a single prompt, and "iterative" generators, which are designed to make small, specific changes to an existing piece.  

For example, a one-shot generator might create a full image of an "avocado chair," while an iterative generator could take a photo and change the colour of a person's shirt without altering anything else. This allows for a series of small adjustments, giving the creator more fine-tuned control. 

While procedural generation has been used in games for a long time, Khalifa noted it presents unique challenges.  Video games have a very low tolerance for mistakes; a small error in a generated image might be ignored, but a flaw in a generated game level could make it unplayable. Because iterative generators make small changes and work with existing content, they are effective tools for augmenting the work of human designers rather than replacing them.  

Khalifa sees these generators as a collaborative tool.

"You don't want your AI to take over everything...having an iterative generator, it can work very well with humans."  

Furthermore, because they train on the changes between versions, they can be effectively trained with the small datasets typical of game design. A handful of levels could be used to create millions of data points, allowing the models to be made far more robust.

Khalifa wrapped up by emphasizing that iterative generators work very well in collaborating with humans in the creative process, and offered a very powerful tool for game designers.

Khalifa ended by urging designers to try iterative generators in their processes.

“They are really fun, there are a lot of really hard problems that need to be solved, and you should really try to work with them.”