Tesla, 467 Crashes, and the Failure Mode That Works Exactly as Designed

In May 2016, a Tesla Model S drove under a semi-truck on a Florida highway. The driver was killed. Autopilot was engaged.

The car's forward-facing camera couldn't distinguish the white side of the truck against a bright sky. The radar system, designed to filter out overhead structures to avoid false braking, classified the truck as an overhead sign. Neither system braked. The car drove full speed into the trailer.

This was not a software bug. This was not a hardware malfunction. Every system worked exactly as designed. The failure was in what the design didn't account for.

In the years since, NHTSA has documented 467+ crashes with Autopilot engaged. 54+ fatalities. A December 2023 recall covered over 2 million vehicles. And the central engineering question remains: How do you analyze a failure mode where the system works correctly and the outcome is still catastrophic?

The Pattern That Caught NHTSA's Attention

The individual crashes tell different stories — edge cases in perception, unusual road geometries, unexpected objects. But one pattern kept repeating: Teslas crashing into stationary emergency vehicles on highways. Firetrucks. Police cars. Ambulances. With lights flashing.

Between 2018 and 2023, multiple incidents followed the same sequence:

1. Autopilot has been engaged for an extended period of highway driving
2. The system performs flawlessly — lane keeping, speed control, smooth operation
3. Traffic ahead slows or stops due to an incident
4. A fire truck or police car is parked on the shoulder or partially in the lane
5. Autopilot does not detect or react to the stationary vehicle in time
6. The driver does not intervene
7. The car hits the emergency vehicle at highway speed

How do you not see a firetruck with flashing lights? That's the wrong question. The right question is: Why was nobody looking?

Automation Complacency Is Predictable

Tesla's Autopilot is classified as SAE Level 2 — the driver must remain engaged and ready to take over at all times. The driver is the backup system. The driver is the safety net.

But here's the engineering reality that SAE Level 2 doesn't account for: if you build a system that handles 99.9% of highway driving flawlessly, the human behind the wheel will eventually stop paying attention. This isn't a character flaw. It's a well-documented cognitive phenomenon.

Research on automation complacency dates back decades — air traffic control in the 1970s, nuclear plant monitoring in the 1980s, aviation autopilot studies throughout the 1990s. The finding is consistent across every domain: when automation is reliable, human vigilance degrades. The better the automation works, the faster the degradation.

For Autopilot, the math is brutal. The system handles routine highway driving — lane holding, speed management, basic following — with high reliability. A typical highway commute might involve 30 minutes of monotonous driving where Autopilot performs flawlessly. The driver's attention naturally drifts. Not because they're irresponsible, but because their brain has learned through repeated experience that the system handles this task.

When the system encounters something outside its operational design domain — a stationary vehicle in a lane where traffic normally flows, a crossing pedestrian at an unexpected location, a construction zone with unusual lane markings — neither the automation nor the human is ready.

The automation wasn't designed for this edge case. The human isn't paying attention because the automation has been handling everything else. Both fail simultaneously, not because either broke, but because the system-level design created this failure mode.

The FMEA Gap: Human Behavior as a Failure Domain

Traditional automotive Design FMEA is built around hardware and software failure modes. What happens when the sensor fails? When the processor hangs? When the actuator sticks? When the software produces an incorrect output?

These are critical analyses and they should be done rigorously. But they miss an entire failure domain: how human behavior changes in response to the system's design.

Consider how a thorough FMEA for an ADAS system might catalog failure modes:

• Camera obscured by dirt or snow
• Radar returns false positive from bridge structure
• Processing latency exceeds safety threshold
• Actuator fails to execute braking command
• Software misclassifies object

Each of these is a legitimate failure mode. But none of them capture: "System works correctly for an extended period, causing the driver to reduce attention to a level where they cannot respond to a situation the system doesn't handle."

This isn't a hardware failure. It isn't a software bug. It's a failure mode that emerges from the interaction between a well-functioning system and a normal human brain. And it's as predictable as corrosion in salt spray.

The Monitoring Problem

Tesla's response to the attention problem was driver monitoring. Initially, Autopilot checked for hands on the steering wheel — apply slight torque, and the system is satisfied that you're "engaged."

The problem is immediately obvious to anyone who's driven with the system: you can rest your hand on the wheel with minimal pressure and the system reads it as attentive driving. You can look at your phone, turn to talk to a passenger, stare out the side window — the system doesn't know. It only knows about torque on the steering wheel.

This is a detection control problem in FMEA terms. The severity of "driver disengaged while automation active" is high (potential for fatal collision). The occurrence is high (predictable cognitive phenomenon, every long drive). The detection should be rated against the monitoring system's ability to actually detect the failure mode.

Hands-on-wheel monitoring detects whether your hands are on the wheel. It does not detect whether you are cognitively engaged with the driving task. These are completely different things, and the FMEA — if it analyzed this failure mode at all — should have rated the detection capability accordingly.

Later Tesla vehicles added cabin cameras for driver monitoring. This improves detection but doesn't change the fundamental issue: the system design creates the complacency that the monitoring then tries to detect. You're generating the hazard and then trying to detect it, rather than preventing it.

SOTIF: The Standard That Exists For This

The automotive industry actually has a standard that addresses this class of problem. ISO 21448, Safety of the Intended Functionality (SOTIF), specifically covers hazardous behaviors that arise from the system working as intended but encountering conditions outside its design assumptions.

Traditional functional safety (ISO 26262) asks: "What happens when the system malfunctions?" SOTIF asks: "What happens when the system functions correctly but the situation exceeds its capability?"

The Tesla Autopilot crashes sit squarely in SOTIF territory. The perception system correctly processed the sensor data available to it. The planning system made reasonable decisions given its understanding of the scene. The failure was that the scene exceeded the system's operational design domain — and the driver wasn't available as a backup because the system's normal operation had degraded their vigilance.

For engineering teams building ADAS features, SOTIF analysis should cover:

• Known limitations of the perception system — what objects, configurations, and lighting conditions can the system NOT reliably detect?
• Triggering conditions — what real-world scenarios bring the vehicle into these limitation zones?
• Human response degradation — how does extended use of the automation affect the driver's ability to respond when needed?
• Monitoring effectiveness — does the driver monitoring system actually detect cognitive disengagement, or just physical presence?

What Changes When You Take Human Factors Seriously

If your FMEA for an ADAS system includes human factors failure modes, the analysis changes substantially:

Failure Mode: Driver cognitive disengagement during extended automated driving

Severity: 10 (potential for fatal collision if automation encounters unhandled scenario)

Occurrence: 8 (predictable cognitive phenomenon, occurs with high reliability after extended automation use — this is not a "maybe," this is established human factors science)

Detection: Variable — hands-on-wheel monitoring scores poorly (3-4), cabin camera monitoring scores better (5-6), but no current production system reliably detects cognitive disengagement in real time

This analysis produces a high risk priority number that demands mitigation. And the mitigations that fall out of the analysis are different from what you get if you only analyze hardware and software failures:

• Design mitigations: Limit continuous automation time. Introduce deliberate engagement tasks. Vary the driving task to maintain cognitive activation.
• Interface mitigations: Escalating alerts with increasing urgency. Avoid "all clear" steady-state indicators that encourage disengagement.
• System mitigations: Degrade automation capability gradually rather than maintaining full performance until a cliff edge. A system that occasionally requires minor driver input keeps the driver in the loop more effectively than one that runs perfectly until it doesn't.

These are design decisions, not afterthought patches. They need to be made during system architecture, informed by human factors analysis, and traced to safety requirements.

The Industry Inflection Point

The Tesla Autopilot story matters beyond Tesla because the entire automotive industry is moving toward increased automation. Every OEM and Tier 1 building ADAS features — adaptive cruise control, lane centering, traffic jam assist, highway pilot — faces the same fundamental question:

If you build a system that works well enough that humans trust it, what happens when it encounters something it can't handle?

The answer isn't to build a perfect system — that's not achievable in the near term for general driving. The answer isn't to blame the driver — that ignores predictable human cognitive behavior. The answer is to design the system with full awareness that human behavior in response to automation is itself a failure domain that must be analyzed, mitigated, and verified.

This means expanding your FMEA practice beyond hardware and software to include human factors. It means SOTIF analysis alongside functional safety analysis. It means tracing HMI design decisions to safety requirements with the same rigor you trace braking algorithms to stopping distance requirements.

The Failure Mode That Works as Designed

467 crashes. 54+ fatalities. Not because anything broke. Because a system worked exactly as designed, and nobody formally analyzed what would happen when humans responded to that design exactly as human cognition predicts they would.

The most dangerous failure mode in automated driving isn't a sensor glitch or a software crash. It's the quiet, predictable erosion of human attention that happens when automation is just good enough to trust — but not good enough to deserve that trust unconditionally.

If your FMEA doesn't have a line item for that, it has a gap. And the gap isn't theoretical. It's on a highway right now, in every vehicle with a Level 2 system and a driver who hasn't touched the steering wheel in six minutes.