Don’t worry, we’re not going to be carrying flower-strewn portraits of Steve Jobs around in the streets today. It’s a slow burn but it’s worth it.
At Amazon Robotics, our products have many many stakeholders but the main everyday users boil down to three main categories: the onsite Technicians who maintain the physical portions of our products, the Workers who directly interact with our products to make customer orders happen, and the Operators who manage the overall super system of people and technology to get the right stuff out of the building on time.
Back at the barn, we’ve got Support Engineers, Design Engineers, and Product Managers watching from afar. And every one of us has a story of when a Technician, a Worker, or an Operator has done something apparently insane in a good faith effort to get better behavior out of their system.
In an AR Storage system, happy little drive units bring “pods” (those yellow shelves) over to a workstation on the fence line where a Worker can interact with them by stowing, picking, or counting items. Sometimes those pods just don’t show up for a while, leaving a Worker with nothing to do. At one site, an Operator observed that it seemed like the system was getting “stuck” and the best thing they could do was to log out of the workstation and log back in again, similar to how rebooting one’s computer solves most ills. It seemed to work, as that workstation would then have pods show up. And, as Operators talk and share tips and tricks just like anyone else, this behavior rapidly spread throughout the network.
It drove us nuts because that’s not how the system works at all and it was actually hurting their overall system performance. When you don’t get pods at a workstation, it’s usually because they’re either stuck in traffic or traveling from far away and, just like driving to Grandma’s house, the Google Maps arrival time estimate gets a little fuzzy the further away you start. All that logging out does is tell everybody who’s already en route to stop and turn around. Logging back in just asks for new work that’s probably coming on totally different pods from different locations. Strobing through logout/login cycles just increases traffic inside of the system and slows down everyone else around you.
But Operators don’t know any of that without doing a lot of homework to research it and figure it out. It’s not how any other kind of material handling system works, so they couldn’t infer it from experience, and the system provides no other hints that it might be the case. We provide training on how to manage our systems but don’t do a great job at the why it behaves the way it does.
So, the Operators are doing what human beings do best: they experimented and built up an empirical model of how the world works with what information they had. It’s how we’ve worked out how to understand the world around us since we became human. It’s how we developed mythology in the first place[1].
Bret Devereaux, a classicist who blogs prolifically at A Collection of Unmitigated Pedantry, makes this argument brilliantly in his series Practical Polytheism. Greco-Roman religious practices were a rational, functional system for understanding and attempting to influence a world that operated on complex, invisible mechanisms. If it seems like every time you see four red birds together that it rains the next day, the next time your fields are dry you’re going to try to collect four red birds to see if the rain shows up. Or, in a more modern perspective, if you need it to rain you wash your car.
And so it is with our own infinitely complex and invisible systems. Devereaux’s framework applies directly to how we fill in those gaps to make sense of things and find the levers:
- We observe the system exhibiting a behavior. Could be good or bad or just weird.
- We try to figure out what preceded that behavior and try to replicate it to see if there’s a relationship.
- Neato, it appears to have worked!
- Even neater, it worked again! Even if it doesn’t always work, you still share it because hey, at least try it, right?
- The thing that sometimes works becomes received wisdom, handed off between people over and over.
- Suddenly, operationalization occurs. You always try the thing that sometimes works as part of your standard procedure.
Congratulations, you’ve built a ritual. Two more generations and you’ll be interpreting robot entrails to foretell the future. (Usually, that future is “very soon you will need to mop”).
Technical training doesn’t fully immunize against this, either. Our drive units have batteries, motors, some sensors, and a motherboard-like brain driving all of it. A backplane ties it all together. The backplane is dirt simple and has an insanely high reliability rate. When they do get returned to us as defective, we almost always turn them back around with no fault found. And yet, we see a large percentage of our Technician population advising each other to replace backplanes whenever they cannot immediately diagnose a problem on a drive unit[2].
Our Technicians are generally smart, practical people keeping a whole lot of stuff working hard in a 24/7 environment. They’re not stupid; they’ve just found that when they replace a backplane, a whole lot of issues tend to clear up. But replacing the backplane also incurs a whole lot of “invisible” work such as resetting clocks, reseating connectors, or updating firmware. It’s that psychological reinforcement of the physicality of replacing the backplane that keeps the ritual alive.
So, how do we break the wheel?
Requiring everyone to sit in a sauna and doubt everything they believe to be true is impractical; only René Descartes really had the time and dedication to sit in a sauna[3] and redevelop literally everything from first principles. So we need to give people a more useful, more accessible, more legible framework for the world around them. Otherwise, as natural empiricists, we default to mythology.
To be clear, I’m not trying to kill God over here. Empiricism still serves us well, and faith systems give us comfort and meaning and understanding beyond the edges of science. There’s still plenty we can’t explain with math and logic. Like, you can’t explain the 2026 White Sox without a Pope and the magic of friendship. We don’t know why it works, it just does… in mysterious ways.
That ability to explain why is what gets us from Practical Polytheism to the Enlightenment. And to bring it back to our AR system, it really boils down to being able to answer four simple questions.
- How do I know it’s working?
- What can I do to affect the system?
- When can I not do anything to help?
- When I can’t help, what’s making the system behave the way it is?
Let’s take them each in turn.
1: How do I know it’s working?
It’s all about believing that the system has got this, and will be able to tell you when it doesn’t. You need a system that can provide the positive confirmation of performance. In other words, your system must be capable of proving the hypothesis that it is, in fact, performant[4].
And no, that’s not a dashboard[5]. Dashboards only provide the negative confirmation: “this particular piece isn’t broken”.
Dashboards proliferate inversely to the level of trust one has in a system to Just Work. You don’t lie awake at night worried about sewer infrastructure because you’re very confident that your poo will be taken care of once it goes down the drain. If you drive an automatic, you’re not typically paying attention to what gear you’re in because it sounds right.
Constant bombardment with solely negative confirmations doesn’t produce confidence in humans; it produces anxiety in a never-ending doom loop of trying to salve that anxiety with… more negative confirmations. This is why your program review full of otherwise rational people keeps going sideways, and this is why your Operators keep looking to empirical means besides the dashboard to gain certainty.
2: What can I do to affect the system?
A while back I had mapped out what factors actually impact an AR system’s performance. Of the 28 factors, the only ones actually controllable from the field were floor cleanliness[6] and having trained, disciplined staff available[7]. They’re victims of everything else, from customer whims to the weather.
But neither floor cleanliness nor staff excellence have immediate levers to pull for immediate gain. You need to either be great at or suck at either one for weeks for them to have an impact, and there’s no magic wand that can turn those around overnight.
In a world with only structural levers and no acute levers, good design makes or breaks you. Making the clear link between performance and those structural factors, and designing good incentives for consistency over heroics, gives your operators the real levers they need.
Give them the visibility to accept the things they cannot change, the levers and incentives to change the things they can, and the system fluency to know the difference.
Speaking of that…
3: When can I not do anything to help?
This question requires the most careful handling because it’s all about helplessness. You’re at the mercy of factors larger than you that do not care about your opinion, whether they be weather, Gauls, or the Ever Given wedged in the Suez Canal[8]. How do you help your operators get through these situations without them sending an intern to look for four red birds?
First, acknowledge that there’s nothing that your user can actually do to affect the situation. Explain the situation in plain language so that they can understand not just that they cannot do anything to help, but why they cannot action it. And finally, give some sort of estimate as to when the situation will be over or when they’ll be able to take an action to help.
Anything else, including silence, puts you directly into that anxiety doom loop of “I need to be doing something.” Because…
4: When I can’t help, what’s making the system behave the way it is?
Which is the other accelerator in the doom loop. Shit’s bad, you don’t know why, it’s not your fault, but you better believe you’re going to catch hell for it anyway.
Our Operators deal with this constantly. A thunderstorm shuts down the yard of a sort center 500 miles away, which means we’re not going to send trucks to them. Since we’re not going to send trucks to them, we’re not going to pick any of the customer orders destined to them because we don’t have anywhere to put them. An Operator running the picking operation has no idea that there’s a thunderstorm, they just know that they aren’t getting work to do but they have a volume forecast they’re expected to meet and won’t.
In 2012, Italian seismologists were convicted of multiple manslaughter for failing to predict the 2009 L’Aquila earthquake after attempting to assess a cluster of swarm tremors that had been affecting the town in the months prior[9]. The Italian Supreme Court eventually overturned the convictions, but not before the damage of accountability without agency had been done. Getting yelled at for a thunderstorm 500mi away is the milder end of that same spectrum.
The fix here is both organizational and technical in nature. First, you have to give your Operators visibility into what’s actually affecting their system, even to two or three degrees removed. They have to know why they’re getting hosed. And then, leadership has to dispel their own mythology about how their systems are supposed to work and believe their operators that they’ve entrusted with that system. Not doing so leads to Potemkin villages: ritualistic performative orchestrations designed to provide cover rather than value.
So now we’ve come full circle: from understanding how mythology works, to why it worked then, and how we still develop it every day even (or perhaps especially) in deeply technical disciplines. Human beings are about as smart today as we were three thousand years ago, doing the best we can to understand stuff way more complex than we can easily perceive.
The difference here in the technical world is that all of our systems were designed by human beings. That means that we can change them to be more legible and more intuitive. Because it’s 2026 and an underslept Operator on Back Half Nights should never be in the position of having to search out four red birds.
And they say that the classics are merely decorative in the modern age.
[1] – SURPRISE, IT’S A LIBERAL ARTS POST!
[2] – I am in the Slack channels and I can see you. You cannot hide.
[3] – Yeah, I know it was technically a poêle, not a sauna. Descartes can find me and fight me should his dead ass wish to pursue it.
[4] – You cannot prove a negative, and you cannot disprove a hypothesis. Further surprise, you’re also getting statistics today!
[5] – And it’s DEFINITELY not freakin’ Page 0 Metrics.
[6] – Told you, mopping is in your future and the future is now.
[7] – Dear Operators: I am absolutely serious when I say that motivating people to maintain bin etiquette will be your biggest performance differentiator.
[8] – Which was directly responsible for Supply Chain draining my desk rum.
[9] – For those of you who missed it, yes really.