That’s ODD! Or, Applying Operational Design Domains to Robotics

Bringing robots out from behind fences and into less structured environments presents an interesting problem – with infinite possible variability in those environments, how do you communicate to customers and users that this kind of environment will work, but this kind of environment won’t?

Or how do you quantify how a robot can do a lot of things, but not everything?

Borrowing from Automotive

Some years ago the most excellent Mahmood Hikmet¹ introduced me to the concept of operational design domains, a construct used in SAE standards in the self-driving vehicle industry. An ODD is the set of conditions under which an autonomous driving system can be expected to work well and safely. Companies that want to field robotaxis and similar which need local approvals have to document and submit their ODDs and explain how they will ensure they’re only operating within them.

Let’s think through this for cars for a moment. It used to be that Waymos and their ilk could only work in daytime and when it was not raining, snowing, or foggy. So the original ODDs might have said something like

• Daytime hours
• Dry weather conditions
• Visibility of at least n miles

Over time, that expanded, and they could do nighttime, and in rain. ODD expansion represented advances in system capability that allowed the cars to safely and successfully handle a broader set of environments. But there are other parts of their ODDs that we can extrapolate from their current deployments, like:

• On local roads, not highways, and not country dirt roads (though some, like Waymo, have expanded to some highway routes in some areas)
• Well-marked roads where road markings are reliably visible at nighttime
• In countries where you drive on the right, not left, side of the road
• Certain predictability about pedestrian behaviors²

This means that Waymos, Zooxes (Zooxi?), and their ilk can do well in places like San Francisco, LA, Vegas, Phoenix, or Austin. Those are all places with fairly well maintained grid pattern streets, mild weather conditions, and a certain amount of confidence that pedestrians – where they exist at all – will adhere to certain behavioral expectations. These places are all within the current robotaxi ODD.

But consider Boston. Our roads were designed by drunk meandering cows. Our road markings become invisible at night in a light rain — our road paint loses its reflectivity quickly with repeated salting during winter. And that’s if they’re there at all; frequent resurfacing work due to damage from ice every winter means lane markings can often be temporary spray paint in arbitrary colors and shoulders are defined by “where the gravel is” versus “where the gravel is less”. For up to six months of the year, you may get accumulating snow, meaning you know where the lane markings are by looking at where the cars are not and figuring that as long as you stay in the tire tracks of the car in front of you, you are probably in a lane. Frequent sprays of slush onto your own car from the cars in front of you mean your sensors are frequently blocked. You’re allowed to drive in the breakdown lane on certain highways during rush hour, which means regularly crossing a whole bunch of solid lines. I can confirm from personal experience that my L2 ADAS systems in my cars do very poorly with all of these scenarios and usually notify me that they’re shutting off, if they could even be enabled to begin with. I can also confirm that my GPS navigation occasionally gives up the ghost on a certain exit into the Seaport district from the Pike, where a tunnel under a surface road under an overpass means my car can’t tell which of the three it’s on and thus struggle to tell me which lane to be in for my next action.³

Oh, and our pedestrians are feral and regularly cross against lights (because the walk signals often don’t work at all, so we’re trained to ignore them) and where there are no crosswalks (because there aren’t enough and no one’s walking a full block for no reason).

Boston, in other words, is well outside the current ODD of self-driving systems, and I’d argue is maybe even the final US-based boss for them.

So ODDs are a useful way to talk about where self-driving tech works and where it doesn’t. This system’s ODD is highways only and it can’t do stoplights; this system’s is local roads only and it can’t do highways. This system can do Austin, not Boston. It can do light rain but not fog. And so on. This gives us a way to articulate “it works, but not everywhere all the time.”

Mapping to Robots

This is why I started porting the ODD framework over to robots. In my last job leading the AMR program at a large tech company you’ve heard of, we spent some time struggling to communicate with leaders and with partner teams throughout the company that “just because we have it working at this one site doesn’t mean we’re ready to go everywhere else.” We were trying to explain that the design was “done”, in the sense of we had successfully completed all of the typical design, test, and build milestones that the company usually used for the product development lifecycle and it has been validated in its original target application, but also, it would need a bunch of new development work to operate successfully in certain applications that had narrower aisles, different payload designs, or different warehouse management system software. This conversation became much easier once we could talk in terms of ODDs.

The key to using ODDs is not to describe what a system cannot handle, but instead to only describe what you know for sure it can handle. There may be infinite things it can’t handle and you’ll exhaust yourself trying to establish the right boundaries that way. Going at it as “what do I know for sure it can do” is a far smaller set of things, and much easier to define.

The first step I like to take is figuring out the categories of ODD parameters that I care about. For example, for an autonomous mobile robot, I might care about:

• Environmental conditions
• Payload characteristics
• Integration interfaces

So you could do something like – I have this robot, and it will work for any application in which the following conditions are met:

• Environmental conditions
  • Indoors
  • Flat concrete floors without steps or ramps
  • Climate-controlled ambient temperature between 0 and 30 deg C, up to 80% relative humidity
  • Artificial lighting
  • All travel aisles at least 6′ in width
• Payload characteristics
  • Pallets of GSA standard dimensions or less
  • Wood or plastic pallets
  • Pallet load less than or equal to 60″ in height, including the pallet base
  • Pallet weight less than or equal to 2500 lbs, including the pallet base
  • Pallet load is securely wrapped, strapped, or contained
  • Single unstacked pallets only
• Integration interfaces
  • Running OurChosenWMS version 2026 or later
  • Pallet fill and staging locations conform to published patterns with specified clearance on all sides
  • Only interacted with by personnel who have taken the HowToAMR safety training

If I document this clearly, this greatly simplifies the following types of conversations:

“Hey, I’ve got an application for your robot. I need to move around 1500 lbs pallets that are 48″ high and I’m happy to reorganize my dock layout to accommodate it.” –> Great! Your application is within the published ODD; let’s talk.

“Hey, I’ve got an application for your robot. I need to move around 1500 lbs pallets that are 48″ high and I’m happy to reorganize my dock layout, but my pallets are made of metal, not wood or plastic.” –> Huh. This might not be a dealbreaker; we’d just never considered metal pallets before. Let us test it out and see if we can expand our ODD to include metal pallets without any new work!

“Hey, I’ve got an application for your robot. I need to move around double-stacked pallets that are 60″ wide and 72″ high and 3000 lbs. Also, they need to be moved between buildings, which means transiting outside.” –> Godspeed, friend, that is well outside of our ODD, our three year roadmap doesn’t get us anywhere close to it, and you’re probably not going to fund the dedicated multi-year development effort it would take to make it happen.

Getting Manipulative

AMRs are a pretty straightforward mapping for ODDs given how similar they are, conceptually, to self-driving cars, but what about manipulation robots?

Let’s start with the categories we might care about:

• Environmental conditions – where will it be operating?
• Workpiece characteristics – what is the system going to be picking up and doing something with?
• Pick and place scene characteristics – what is it picking things up from? What is it placing them on or into?
• Task characteristics – what task success criteria might make things easier or harder?
• Integration characteristics – same questions as with AMRs. What physical and virtual interfaces are we expecting to be in place?

Then we can get into specific parameters:

• Environmental conditions: not just temperature and humidity but also environment type that may define the types of tasks, objects, and challenges it may encounter. A home? A factory? A hospital?
• Workpiece characteristics: how big, small, heavy, or light can we handle? Do we have two coordinated arms to handle large objects? Is our end effector sensitive enough to manipulate fragile objects? Can it detect and react to slip and thus handle slippery objects? Is it dextrous enough to handle non-rigid objects like cables or clothing? Is it sophisticated enough to handle conformable objects? Designed to handle aesthetically important objects that cannot be scratched?
• Scene characteristics: How much structure is needed in a pick or place scene? How precise can we be in placement? Can we guarantee placement pose? Picking up from and placing on a counter is different from picking up a part from a box of many items and placing it with precision onto an assembly jig, or placing a plate in a specific pose within a structured dishwasher rack.
• Task characteristics: between picking up and placing an object, what can we do with that object – only move it? Reorient it? Perform some operation like folding or connecting? Can we only guarantee successful completion of a single task within a certain cycle time? Can we only guarantee a certain % confidence in first-try success, meaning tasks that require a higher incidence of successful first-tries are not within the current ODD? Is our current performance (success rate, ability to detect exceptions or anomalies, etc) compatible with applications where a mistake may have safety consequences?
• Integration characteristics – does it need home wifi and a connection to a remote teleoperation service to take over when the autonomy fails? Does it need to be integrated with a factory MES or warehouse WMS, or a custom call/dispatch system at a hospital?

By thinking through these you could define why your manipulation system might work to stack boxes onto a pallet but not fill prescriptions at a pharmacy, or might be fine to fold laundry but not to handle dangerous chemicals.

Roadmapping with ODDs

ODDs become an interesting axis for improvement to think about, because expanding an ODD – adding features or capabilities that expand the set of conditions under which your robot will work – is distinct from performance improvements. “My localization accuracy improved by 10%” or “I use 15% less energy in a shift” may be quantifiably valuable performance improvements, but don’t necessarily expand the set of applications you can address. On the other hand, “We upgraded the payload lift motor to handle 3000 lb pallets instead of hitting our limit at 2500 lbs” does. So when you start roadmapping, you can quantify the value of ODD expansion — and thus the expanded market you can address — alongside quantifying the value of performance improvements.

It can also be a useful tool to think through your first pilot or your first customer. Who has a use case that fits firmly within your current ODD? You may want the customers whose use cases are beyond it, but they’re not the right ones to prove yourself on. And matchmaking potential use cases with different phases of ODD results in a deployment roadmap over time, allowing you to communicate to your customers when you expect to be able to address their needs.

Think of it as the social model of robot performance. A “good” robot is good within the conditions it was built for. In an environment with different conditions, it’s not the robot’s fault it can’t perform – it’s just the wrong environment for it. Your job is just to understand and define those conditions for success to ensure that everywhere your robot goes, it can be a good robot.

1. Moody is a delightful human being and is one of the few people out there who can match me with nerdy spite. You may enjoy “How To Win An Argument By Changing An International Standard.” ↩︎
2. Reading about the debate around making jaywalking illegal in a country that doesn’t even think about jaywalking as a concept, let alone something you should not be allowed to do, was fascinating and hilarious to me ↩︎
3. Once, at this exit, instead of the next turn instruction, the Google Maps app on my phone simply said “0”. ↩︎