A new paper by Amii researchers is unlocking the power of options in reinforcement learning. This method offers the promise of agents that are better at long-term decision-making and handling more complex tasks efficiently without needing constant intervention from humans.

The paper, Mastering Atari 2600 Games with Discovered Options, was co-authored by Erik M. Lintunen, recruited under the Canada Impact+ Research Training Award, and his supervisor, Amii Fellow and Canada CIFAR AI Chair Marlos C. Machado. In it, they introduce Wayfarer, an autonomous decision-making architecture that can master complicated behaviours using only the agent’s data stream, all without human data, intervention or hardcoded domain knowledge.

Reinforcement learning agents learn behaviours by trial and error, but they can struggle with complex tasks that require many small steps. Think of an agent that is built to navigate its way through a large house. When moving through and around obstacles, each step is a decision that the agent has to make: which direction to move, at what speed, and even the decision on whether it should move at all. This can lead to decisions that make sense in the short term but are inefficient for the long-term goal, accompanied by time-consuming dithering at every decision point. Agents might lack the purposeful behaviour necessary for trying something different (like going to another floor); they might struggle to distinguish, among millions of decisions, which one was responsible for its success, and none of this knowledge is necessarily easily portable to a new situation.

Options are a method to help with all these issues by reducing the complexity of the problem. They are a collection of those smaller steps treated like a single decision. Instead of an agent making a hundred small decisions to navigate through a room of obstacles, the agent only has to make one strategic choice, like “get across the room.” Think about it as a skill. The smaller steps are all part of this larger action, reducing the number of strategic decisions the agent has to make, meaning more attention can be spent on long-term goals.

Options have long been a promising tool in reinforcement learning, but up until now, they largely had to be crafted by humans, or at least refined and built by agents using human feedback to guide the process. Wayfarer is a large step forward in unlocking the true power of options, finding a way to let agents discover their own effective options, learn when to use them, and then reuse them for similar tasks.

Wayfarer is a domain-agnostic algorithm that works by combining three networks. The first is a representation network that takes in the environment’s observation and builds a representation of it, allowing the agent to distinguish what varies slowly versus quickly, what seems likely to happen or not, and what it can or cannot control. Second, a skill network builds a collection of options based on that representation and from the agent’s experience. Finally, the third network is the one responsible for actually solving the task; it evaluates the challenge in front of it and decides which actions and options would provide the maximum rewards.

These networks work together to create what the researchers call a "virtuous loop”: the agent’s representation of the environment defines internal goals to build options, the agent gains experience based on the option it uses, and then learns from that experience to build a better representation and stronger options. The agent learns by doing, actively building knowledge and meaning from their experiences in a constructivist way. Then it uses those lessons to do more.

Marlos C. Machado speaks at Upper Bound 2026

Play

Wayfarer’s approach yielded some impressive results. The researchers tested it using the Atari 2600 challenging set, a collection of ten difficult Atari games used to benchmark the performance of RL agents. Many of the games, including Montezuma’s Revenge and Private Eye, require long-term planning, exploration, and strategy, making them an excellent benchmark for testing the limits of autonomous agents.

They found that Wayfarer had a median performance boost of 2.5 times over learning without options, while outperforming established baselines (like IQN and Rainbow) and matching heavy model-based systems. And it did so without human intervention or designed options, instead discovering its own.

Machado, who has spent a decade working on the potential options in RL agents, says that Wayfarer fulfills the promise of options, perhaps for the first time. It is work that could lead to better autonomous agents that are able to do complex tasks in situations where human guidance isn’t possible or feasible, including anything from industrial robotics to autonomous vehicles, using fewer resources and less computing power.