,

Bellum Longum et Desertum

Story time! Long ‘un, so grab a cozy and/or refreshing beverage.

So what does it mean to truly be in it to win it for the long term1? Today I’m going to tell you a story about the unloved drive unit that somehow became the centerpiece of my mobile robotics work and a focal point for many of my frustrations with how the robotics industry do. In an industry that only cares for the latest cool demo video, what happens when you actually manage to ship a durable product that fulfills its entire lifecycle promise?

This is my long war.

Bradford intensifies. Bradford always intensifies.

In 2007, Kiva Systems marketed solutions that used two mobile-robotic drive units2: the medium-duty lifter F-DU (marketed as the DU 1000) and the heavy-duty lifter R-CE (marketed as the DU 3000). We retired the F-DU when we wound down the last of our third-party contracts years after acquisition, but we kept the R-CE as our first drive unit to run in production Amazon. R-CE first saw Amazon service in 2013 in our Nonsort business line that handles items that don’t fit well into a tote but can be handled ergonomically by a single human being without special tools. Think dog food, Instant Pots3, that sort of thing. R-CE also later saw service in other business lines as well. We stopped producing the R-CE in 2015 but kept supporting its operation in the field.

In 2015 we added the Atlas-S heavy lifter drive unit, joining the Atlas-G in the Atlas family and completing our hardware portfolio refresh. Atlas-S was R-CE’s intended successor, and in many sites the Amazon business lines elected to swap in the Atlas-S. But Nonsort kept soldiering on with R-CE, choosing not to fix what wasn’t broken4.

So we’ve got two heavy lifter drive units bopping along, being awesome, doing their jobs. However, time passes, and we design our drive units to a 10-year design life5. And the problem with equipment that just does its job… it’s boring. There’s no measurable cost savings in a process path for just keeping things going. There’s no flashiness in stuff just working. It’s routine. No one cares about it. It becomes invisible.

Well, things weren’t boring, exactly. Remember that Kiva Systems led the vanguard on producing mobile robots at scale. Sure there’d been AGVs in production, some extremely cool exquisite low-scale military and space exploration stuff, and laboratories full of postdoc work desperately marketing themselves as the next revolution, but we really were the first. So in the mid-00’s we were still learning how to really make a mobile robot6 and R-CE had some of those warts. Some were due to the state of technology at the time: lead-acid batteries were industry standard, and our running lights were incandescent bulbs because 2007 was riiiight before the LED revolution really kicked off7. Others, like chassis design, were empirical evolutions because no one else had figured it out yet, either. We had some painfully scary, though ultimately harmless, things happen in the field as a result.

The anti-model for whether someone should be involved in decisions for a product that can maim or kill.

Those kinds of events tend to stay with you, if you’re a fully formed human being. It’s traumatic, knowing what could have gone wrong, and you carry a deep sense of shame. So, even after we faced the aftermath, and even after we pulled an emergency retrofit that pulled drive units back to the barn from across the country while the network was going through Peak, and even after we had to retrofit the retrofit when we discovered months afterward that we’d overcorrected8, and even after many years of successful and safe operation post-retrofit, that shame stayed with our crew. Folks didn’t even want to talk about R-CE, much less pay attention to it. I think most were just kind of hoping that the Nonsort business line would shut down their AR buildings and we’d be absolved of having to deal with it at all.

I owned the Prime Pantry portfolio from 2015-2017, which still operated some R-CE drive units at the time. People would flee from me in the hallways. I might have been the only one left at AR who cared about the R-CE and I couldn’t do much, if anything, to help it. I wasn’t even sure how old R-CE was; at the time I’d thought they were introduced in 2009. My direction at the time was “just keep them running”.

Them running. You can tell because all of their lights are on!

So when R-CE turned 10 in 2017, the milestone came and went without fanfare. The organizational focus, my own included, was all about our new Sortation offering. We’d always been able to just take care of things because we only had one offering, the mobile-robotic storage system, so we had no mechanisms for regularly reviewing our existing portfolios. Somehow we were operating on three continents and still in growth mode9.

And then, in early 2020, I carped the diem and declared that I would create a Mobile Robotics product portfolio because we needed to act like adults. We had all these ‘bots out there and someone needed to love them. I became the Mother of Drive Units.

And I meant it. I laid out the full portfolio, of our capabilities and overlaps in the product space, of our opportunities and weaknesses, all that good Product Management stuff that the paid Substack newsletter influencer crew loves to preach. AR finally, for the first time, had a holistic view of our bread and butter product line including a definition of what a product lifecycle looks like and where all of our ‘bots fell on it.

Leadership received it well and took it seriously, but we collectively kicked the can down the road. This was the depths of the COVID lockdowns, and we were in the Forever Peak of the entire planet trying to order toilet paper all at once. An entire year of Black Fridays gives tunnel vision to everyone.

Time passed, virologists developed some truly amazing vaccines10, and we came out of lockdown to a somewhat normal business cadence. But every second, those R-CE were still doing their jobs and ticking off the seconds of wear and entropy, and I knew it. I wrote Product Requirements for a new drive unit, Titan, that could be dropped in as a replacement for either R-CE or Atlas-S and could support new storage modalities. The due diligence made sense and the math all worked. It was the no-brainer decision, and yet no one took me up on it.

Doing everything right didn’t move the needle, so it was time for something different. I became Cato and began my one-woman campaign against all threats to my fleets: time itself. And R-CE would be my Carthage11.

Whoops, wrong Time War.

First, I had to figure out why I wasn’t getting traction even though everyone seemed to agree that R-CE retirement had to happen. My leadership chain were intelligent, sensible people that I’d worked with for years, so I knew the problems weren’t malice or stupidity. I settled on four factors:

  1. Sustaining isn’t sexy. Classic tech industry problems that are utterly alien to other capital-heavy industries. You don’t get promoted for a clean-running fleet, you only get yelled at when something breaks.
  2. Bad KPIs as perverse incentives. Focusing on metrics like entitlement12 that emphasize short-term gains makes long-term investments impossible. Entitlement has no mechanism to price in multi-year probability-weighted risk, which means business continuity has no value in this scheme.
  3. Out of sight, out of mind. We weren’t reviewing our generally available portfolio performance regularly. As long as something wasn’t figuratively on fire it flew below the radar, sometimes to the point of leaders forgetting we still had a product in the field.
  4. Trauma avoidance. When you feel you’ve got a plate already full of problems, you’re never going to invite one that just makes you feel bad about yourself13.

I’d won their minds, now I needed to win their hearts. I tarted up our sustaining work with memes and catchy slogans; the phrases “largest mobile-robotic fleet known to humankind” and “our ‘bots work 24/7/365/10” came from this effort. I started working out with Finance how to translate business continuity into something the business could understand. I booked regular 1:1’s with leadership and made sure that no less often than once a quarter I touched on the R-CE lifecycle situation and how nice it would feel to no longer have them on the books. I even added “R-CE delenda est” to my email signature with links to a one-page memo explaining the rectitude of my stance. The pressure would be gentle and constant.

And then frickin’ Hasdrubal shows up in Numidia.

Turns out, suppliers don’t make parts indefinitely, and you can only do so many “last-time buys” before it really is the last time14. A TPM colleague in Hardware Support Engineering began waving the flag in 2021 on spare parts projections and suddenly, I had a new best friend. We opened a new front: mild psychological terrorism in the form of spares shortages reports in our bi-weekly operational reviews of the network. We made the ticking clock real, and we got enough attention for a director to sideload a few hardware engineers to prototype out the Titan.

Every other year our customer fulfillment organization hosts OpsLive, a huge internal conference and expo that showcases technologies that Amazon plans to deploy into our buildings. In 2022’s edition I scored a table in the AR booth where I placed the one Titan prototype that we’d produced, with a clear 3D-printed fairing to emphasize all of the inherited technologies inside. I buttonholed every single operator that had ever had to deal with the R-CE and showed them that there was a better way to live. I gave them marching orders on how to campaign for getting Titan into their buildings with anyone else who would listen. Afterward, my leadership advised me that I would not get a sales commission but that the message had been clearly received.

So, unofficially at least, game on! Events sped up from here. We got another TPM bestie from our business vertical who chased down the value of business continuity of the remaining R-CE buildings and prepared the capital request. Hardware Engineering continued to fund the development work on the side, and Software signed up at a discount because moving off of R-CE meant they could deprecate the last of the lead acid battery charge management code. We figured out how to do the drive unit and charger swaps in 72hr downtime periods.

At which point I was told that Titan would probably be below the line for 2023 budgeting after all.

I played my last card: full-blown psychological terrorism. I told the decision-makers that when— not if, but when— critical parts began failing in the field with no replacement, that I would pull them in on every high severity ticket and every escalation. They have the responsibility and I will not eat the accountability for them. I will ensure that they receive full ownership for their life choices.

A few days later, we were quietly informed that Titan funding would go forward as originally planned15.

“We thought you’d forgotten about us.”

SAT1 Maintenance Lead

The retrofit itself went like a whirlwind. Everyone who had fought and shouted for the R-CE retirement and the Titan launch over the years came in as part of the team: HSE TPM, Business Vertical TPM, Customer Success Managers, Product Reliability Engineers, Design Engineers… we were on a mission and our focus did not waver. As sensible people, we started with one zone and ran it through 2023 Peak to prove it out with plans to finish the fleet retirement in 2024. Within weeks of our critical spares shortage tip-over point, we started pulling R-CE out of the field in a warehouse in Schertz, TX16.

We had the usual NPI launch war stories: losing pallets in the warehouse, frantic Home Depot runs, the firmware tweak miss that required getting a firmware engineer FedExed to the site, eating sad warm salads at a plastic table with a TPM who literally lived on white Monsters for the duration, and finally that golden, cathartic moment when everything finally worked and we could leave site at 2am confident that the zone will launch and fly the next day.

Titan rocked it. Our big blue boi made us all proud. Even the industry liked it, and they don’t like anything associated with Amazon.

We did have a stutter-step time gap when we had to move the Titan assembly line between our Boston-area buildings, pushing the last building conversions to early 2025. But we’d built contingency for that by holding back some R-CE from the industrial chipper-shredder so that we could harvest them for parts in case we needed to keep things running for a few more months than Plan A allowed for.

We completed the first ever17 mobile-robotic lifecycle flip on my birthday week. As the last R-CE rolled off of the floor and towards a farm upstate where it had lots of space and could frolic with all of its friends, I gave a eulogy to our team and the folks who had joined us from the site and the business line. There were tears and hugging and handshakes.

We’d done it. We managed to land R-CE, the ‘bot that no one cared about, cleanly and respectfully18. Atlas-S has a drop-in replacement in the catalog ready to go whenever it’s ready to go home and rest. We are fully covered in our heavy lifter capability for another decade. Carthage has, in fact, been defeated.

So, is this the right way to end-of-life and decommission a fleet? Oh HELLS to the no.

This was luck, persistence, and one stubborn Product Manager waging a years-long war that never should have had to be fought19. By 2023, everything was critical path. We had no idea what we were doing with managing the tail-end of a product portfolio and had no right to have pulled it off as smoothly as we did.

I am the first. Now it’s my job to tell you about how to do the second, third, and nth. Guess where we’re going next in this blog stream. 🙂

  1. Note that this differs from “longtermism”, which is just a bullshittist-fucknut way to handwave away externalities to gullible audiences. More on these shitbags in Move Fast and Break Accountability. ↩︎
  2. Fun fact: words mean things to people. Originally, Distrobot made systems that used robots, which led investors and potential customers to ask how many robots they needed to buy. However, the storage system is non-linear in its work capacity, which means that you need to size the entire system correctly rather than the usual linear MHE mental model of 1 robot = 1 work unit. Distrobot was then renamed to Kiva Systems to emphasize that you’re buying a holistic system, and the mobile robots were called “drive units” to mimic MHE vocabulary and take the buyer out of that linear context. ↩︎
  3. A certain group of people just heard Fortunate Son start playing in the background and don’t know why. ↩︎
  4. I drive a car born in 2010 (and he’s SUCH A GOOD BOI!!) and can empathize. ↩︎
  5. “Design life” is the length of time that we design the equipment for and verify with testing. I need to talk about the different types of lives that equipment can have at some point. ↩︎
  6. “How did you solve the grounding issue?” Is probably the best shorthand litmus test for whether a mobile robot is legit. ↩︎
  7. Not exactly secret; a guy bought an F-DU off of eBay and did a public teardown a decade ago. ↩︎
  8. I was in the maintenance cage at a site when we discovered this and threw my phone into the robotic field. Fortunately it landed on the floor where my colleague, who was safety trained to enter, retrieved it without judgment. ↩︎
  9. There’s only one thing in nature that pursues growth indefinitely: cancer. ↩︎
  10. FIGHT ME, PLAGUE RATS. Oh right, you can’t, you’re too busy bringing consumption back. ↩︎
  11. Wikipedia admin: “Why are we getting a traffic spike on the Third Punic War traffic from the techbro IPs?” ↩︎
  12. Our fulfillment organization uses a metric called “entitlement” to determine the value of a feature or product, which is calculated simply as the variable cost savings of the improvement less its initial capital cost. This works great for process path improvements because it can be easily calculated by an operator, but poorly for just about anything else. ↩︎
  13. As an aside, a significant portion of my de facto role has been armchair psychiatrist. Organizational trauma shows up in your architectural decisions, and my proudest achievements have come from helping this place process that trauma so we can unfuck our self-inflicted design constraints. ↩︎
  14. Something I should have anticipated but only learned with R-CE: it’s the little fiddly bits that go permanently out of stock first. Washers, nubbins, and connectors. And those suppliers won’t stand their production lines back up for any price offered. ↩︎
  15. Brinksmanship sucks, and your target must be convinced that there’s a strong chance you’ll actually follow through. You can’t pull this one very often. ↩︎
  16. We even did a PR piece about it! ↩︎
  17. To my knowledge, the first known to humankind. Come at me. ↩︎
  18. And, importantly, completely unlike whatever the cinnamon toast fuck that Figure did. Who fucking signed that check. I am mad, and maybe I just want to talk… maybe. ↩︎
  19. As an American, I have a preternatural urge to pick unnecessary fights that drain me of health and wealth. ↩︎