Moral crumple zones

Madeleine Claire Elish’s concept of the “moral crumple zone” — a human operator who absorbs legal and moral responsibility for a system they had limited control over. Like a car’s crumple zone, the human is designed to take the impact, protecting the technological system’s integrity at the operator’s expense.

Elish illustrates it through Three Mile Island, Air France Flight 447, and Uber’s 2018 self-driving fatality. In each case, operators bore the blame for failures that were baked into system design, management decisions, or automation handoff problems.

The term is going to matter more as AI advances. The more we hand control to automated systems, the more the remaining humans become not decision-makers but liability sponges — kept in the loop to absorb responsibility for outcomes they didn’t really control. As Elish puts it: “While control has been effectively distributed, responsibility has not scaled accordingly.”

Via Fred Hébert’s notes on the paper.

Moral Crumple Zones — original paper in Engaging Science, Technology, and Society