How Rainbow Six Siege Developed AI That Acts Like Real Players

Rainbow Six Siege consistently adds new content for its players, whether in the form of new Operators, game-changing gadgets, or maps. During the 2023 Six Invitational, the team behind Rainbow Six Siege unveiled the Year 8 Roadmap. In addition to teasing new Operators, Battle Passes, and improved anti-cheat measures, the roadmap hinted at a new feature coming down the pipeline: the Defender AI playlist. Designed to help players practice playing against Operators, test different strategies, and learn the intricacies of Rainbow Six Siege maps, the Defender AI playlist will consist of AI-generated bots that behave as real players do. At the 14^th annual Ubisoft Developers Conference (UDC), Technical Architect Alex Busby and AI Programmer Martin Perreault gave a presentation entitled Operation: Bots, in which they detailed how the new system would benefit all players and why the Rainbow Six Siege team chose to develop two simultaneous frameworks using traditional AI and machine learning. Ubisoft News spoke with Busby for an in-depth look at the topic.

State of Bots in Rainbow Six Siege

Rainbow Six Siege has a massive player base, with 85 million players, many of whom have been playing the game since its 2015 launch. However, the Rainbow Six Siege team noticed new players had a difficult time onboarding, as they found the steep learning curve and experienced player base a tough learning environment. These players tend to spend most of their time eliminated from gameplay, unable to learn or enjoy the mechanics of the game - despite the already robust system of AI-generated bots in practice arenas.

In their current iteration, bots in Rainbow Six Siege function as procedurally generated enemies that players encounter in Training Grounds and Situations, but not in live PvP matches. While they make good target practice for players learning the game or practicing a map, they're missing two key elements that would prepare players for live gameplay: the bots are nameless adversaries with basic tactical gear, not Operators with specific gadgets that change the game, and they aren't able to emulate the behaviors of a real player. It's something the Rainbow Six Siege team wants to change by developing dual AI frameworks, which use both machine learning and traditional AI to produce bots that behave more like actual players.

"Our goal is for bots to be completely indistinguishable from players, and we want them to emulate the strategies and movements of a real player," says Busby. "The initial goal of the bots is to prepare newcomers for PvP and help them get accustomed to the environment that is playing Rainbow Six Siege."

AI and Machine Learning

To create the new system of bots, the Rainbow Six Siege team, in tandem with researchers and AI innovators at Ubisoft La Forge, decided to build parallel frameworks: one that uses a more traditional AI framework, and another that uses machine learning. The long-term goal is for the new bot feature to exclusively use machine-learning AI, but the traditional AI method was needed to get the feature released quickly and without game disruption. According to Busby, to produce the traditional AI framework, the team used the existing AI architecture in Rainbow Six Siege, but turbocharged it with match data from the game, essentially teaching the AI to use the data to behave like a player, but without reducing code complexity.

Busby emphasized that any data used to build this system or inform the AI's learning process is solely information players have already authorized Ubisoft to have via terms of use, and does not include any personal information. "The data we're using is everything that's already being sent to our servers in order to have the game behave like it does as a live network experience," he explains. "So there's no data we're gathering that they haven't already sent to our game server to use within the replication of the game."

As the team built this new system, they made sure to structure it in a way that would allow them to swap out the traditional AI with machine learning without disrupting the data pipeline. However, machine learning takes much more time to develop. At its core, machine learning frameworks take a large data set, such as a series of match replays in Rainbow Six Siege, and analyze it. The program then creates different scenarios that produce predictable outcomes for different in-game situations.

"One of the ways we're doing this is, for all cases where a player has thrown a grenade, we look at all the data around the 'world state' when it was thrown," explains Busby. "Then, we produce a module within machine learning to predict, in a given situation, if it's a good time to throw a grenade."

Armed with that information, the machine-learning model will not only tell bots with high confidence that it's a good time to throw a grenade, it will also reduce thousands of lines of code. This, according to Busby, positively impacts Rainbow Six Siege's ongoing production, reducing developer workload and allowing them to focus on other priorities.

Machine learning teaches bots optimal strategies and player behaviors, and reinforced learning, a specific type of machine learning, helps them evolve and adapt to the ever-changing strategies of Rainbow Six Siege. According to Busby, the primary difference between the two AI programs is that machine learning uses a single input of a data set to extrapolate conclusions, while reinforced learning uses a continuous loop of data to produce desired behaviors. Using the same example of throwing a grenade, reinforced learning will be able to analyze bots' performance in different sessions and improve their approach. As Busby explains it, "A reinforced learning system will be able to produce much more complex behavior, like strategizing on the full round of a game rather than small modules."

Emulating Player Behavior

To teach bots how to play like players, both the traditional AI and machine learning frameworks need copious amounts of one thing: Data. The team had two options to collect the information. The first is telemetry, which would send a burst of information from live games in Rainbow Six Siege that the developers would analyze, then extract and aggregate desired behaviors. However, this method showed a very sparse data set, which meant extra work for the team.

Instead, they opted for option two, match replay. Match replays represent information that players have access to in Rainbow Six Siege and have authorized Ubisoft to use. With this method, all the team had to do was collect a series of match replays from the Rainbow Six Siege servers, feed it to the frameworks, and then the AI would reconstruct a round and begin extrapolating data and desired behaviors.

"It's a treasure trove of information, and exactly what we need," says Busby. "It's a very dense data set of critical gameplay information and timelines to reconstruct player behavior."

To create a proof of concept for the match replay method, the Rainbow Six Siege team had their testers get together and play competitively to provide a normalized data set for the AI to analyze. This initial group represented an average player at home, not a pro or a new player, which is the initial behavior the new bot feature will emulate. As the AI network grows and aggregates more data, Busby estimates the framework will gather information from roughly one percent of rounds played, and will be focused on maps and Operators the team wants the bots to learn.

Launch and Future Plans

When the new feature launches, bots will be able to play as five Defenders, with plans to add new Operators and maps to their repertoire with every new season release. With new Operators, the team at will use internal Ubisoft studios to collect two weeks of data from progress testing, enabling bots to play as those new characters and use their gadgets when they release. The AI framework will then collect match replay data from live players, feeding into the reinforced learning loop, allowing the bots to improve. Through this process, bots will also be able to adjust their strategy to accommodate new abilities and gadgets, which often have the potential to drastically change tactical possibilities in Rainbow Six Siege.

"We will, of course, have to do a quality pass to make sure that this system makes sense from a difficulty point of view," Busby clarifies. "If our initial intention is a forgiving environment for new players, and seasoned players find a way to drive the bot data to make that entry point for new players more difficult, that's a problem. We want to be able to siphon or split the strategies so that we can support different levels of difficulty."

Though reducing the barrier of entry for new players was the inspiration for creating the new bot system in Rainbow Six Siege, Busby says there will be different levels of difficulty, allowing average players to level up their skills, and pros to hone their skills. To help the AI frameworks identify difficulty levels for bots to emulate, the development team will contextualize the input data by player rank, putting data for elite players into one bucket, silver players in another, and copper players in yet another.

In the long term, the team wants to have offline training for the bots to determine whether Attackers or Defenders have a higher probability of winning. Then, the team will have a way to analyze the data of how bot behavior impacts rounds and adjust the framework to maintain an appropriate balance.

"We want to set a stage for elite players to hone their skills before a big match," says Busby," and my personal, deep, deep ambition is, at the end of an esport event like the Six Invitational, to have the winners play against our elite bots and see who wins."

For more on Rainbow Six Siege, check out the Year 8 Season 1 Operator and gadget guide, read about its recent WWE cosmetic crossover, and stay tuned to Ubisoft News for more updates.