Listen to this article

While it becomes second nature for those who have been doing it for years, driving is a complex task that requires those behind the wheel to always be at attention. Your brain is constantly making decisions about the road conditions, your speed and position, the speed and position of the cars around you, observing traffic laws, road marking, and more. 

Autonomous vehicles need to be able to pay attention to all of these things, without eyes or human reasoning to help them do it. For Zoox, a subsidiary of Amazon, this is even more of a challenge because its purpose-built robotaxis needs to learn almost everything about driving from simulation.

Robotaxi companies that have started rolling out autonomous taxi services in recent years, like Cruise and Waymo, do a lot of training in simulation as well, but they also conduct extensive real-world training with safety drivers behind the wheels of their robotaxis to step in when the system might make a mistake.

The company has a test fleet of vehicles that it uses to validate its technology that has a geometrically identical sensor architecture and configuration to its purpose-built robotaxis to translate the learnings from miles driven in its test fleet to its ground-up robotaxi. However, its purpose-built robotaxis don’t have steering wheels or pedals, meaning they can’t actually drive out onto the road with a safety driver behind the wheel.

Now that the company has deployed robotaxis in Foster City and Las Vegas, it is gathering on-road data that it can learn from as well. To prepare for these deployments, Zoox has used extensive simulation testing to ensure its robotaxis are safe enough for the roads. Additionally, there are a number of scenarios that its robotaxis haven’t encountered in the real world, and simulation ensures that it’s prepared for as many scenarios as possible. 

By integrating safety and simulation, Zoox has built a robust simulation framework that allows the company to test millions of driving scenarios and learn from them.

---

Learn from Agility Robotics, Amazon, Disney, Teradyne and many more.

---

Preparing for all the things the road brings

Even though you might take the same route to work every day, at around the same time, it’s likely the drive isn’t the same each time you take it. There could be a biker on the road or an emergency vehicle speeding towards its destination. These unusual occurrences are called edge cases, and they’re one of the most difficult things for autonomous vehicles to plan for simply because they rarely happen. 

To try to prepare for as many of these strange cases as they can, Zoox’s team uses a few different methods to generate use cases for their system to test in simulation. 

“One is obviously through our test vehicle logged miles. We drive our test vehicles with safety drivers quite a bit in our launch intent areas,” Qi Hommes, the Senior Director of System Design and Mission Assurance at Zoox, said. “And anytime we encounter something unexpected those are inputs into the development of those simulation scenarios.” 

When Zoox’s team runs into these unexpected situations, it puts that situation into simulation and tests it over and over. The team also uses these situations to generate countless similar situations for its system to test. 

“We want to just extensively vary that one example case and then run our development software through to see how we performed, where we might be lacking, and further inform the software team to make changes and improvements,” Hommes said.

Additionally, Zoox can procedurally generate challenging or potentially dangerous scenarios, according to Yongjoon Lee, Zoox’s Director of Simulation. 

Translating simulation to the real world

“The key challenge is simulation is always just an approximation of the real world,” Lee said. “So there’s always a gap, and the gap could manifest in, you know, shortcomings to validation and training in unexpected ways.” 

Zoox’s team works hard to try to discover these gaps between simulation and the real world and fix them. But it’s a tough issue, and, according to Lee, one of the biggest ones facing the industry as a whole right now. 

One of the other big challenges with simulation is dealing with the sheer amount of data that simulations can generate. Zoox’s engineers need to examine any scenario where the system failed and if the scenario is relevant, and this can be a very manual process. 

“For example, it will suddenly generate a pedestrian as you’re driving by a place because for some reason the simulation pops up a pedestrian, and that just doesn’t happen in the real world,” Hommes said. “So you get one of these cases where in simulation it looks like a collision.” 

These kinds of cases need to be weeded out an ignored, but not all of these scenarios are irrelevant. 

“We should worry about realistic scenarios, and making sure we don’t have collisions. So that triaging process is pretty intense. Given how much simulation we do, it’s a challenge,” Hommes said. 

Recent advances in AI mean that now Zoox can speed up this triaging process, according to Lee. The company is able to use AI to determine which scenarios are relevant, giving Zoox engineers time to focus on more challenging work. 

Zoox is also using AI to improve simulation realism and, in particular, the behaviors of humans in simulations. 

“I think we’re collectively learning how important it is to make sure the simulator is correct and realistic,” Hommes said. “And that the entire pipeline is configured and run in a way that produces results.” 

Zoox’s safety benchmarks

Zoox has a comprehensive list of metrics that the company sets internally to ensure that its technology is safe enough for the roads, according to Hommes. These metrics are divided into what the team calls safety cases. 

“So a safety case is basically an argument you want to make,” Hommes said. “You say, hey, if A B C and D are true, then in conclusion, E must be true, which means we are confidently safe enough. To us, that means to be able to drive safer than a human driver.” 

The company’s entire approach to safety is data-driven by a number of engineering metrics. It’s a quantitative approach, that doesn’t leave room for anyone to decide a vehicle is safe enough for the roads without it hitting certain benchmarks. 

“Zoox has never put any autonomous technology anywhere without it having passed our safety bar that we set internally,” Hommes said. “And we don’t lower that bar just because we want it to go out faster or because other companies are out on the road.” 

These benchmarks include industry safety standards and the company’s own standards where industry ones don’t yet exist. The team also spends time validating every piece of software and hardware in the vehicle and running simulations to determine what would happen if any of these elements malfunctions, according to Hommes.

One important theme in Zoox’s approach to safety is redundancy. The autonomous vehicle industry is still in the early stages, so it can be difficult to find hardware components that have been tested to the extent that they need to be to ensure they’ll be safe on the road. To combat this, Zoox has backups of important hardware components that can take over if one fails. 

In all, Zoox is pushing the bounds of the role that simulation plays in the development of autonomous vehicles by using it for safety validation as well as training. 

“I think as the scale of deployment becomes larger, and development and release of software becomes more frequent, simulation has to play a bigger role in validating the autonomous driving software at a higher bar more comprehensively,” Lee said.