At the beginning of this month, there was an unfortunate accident in San Francisco where a normal, human-driven vehicle struck a pedestrian. This is already bad, but it gets worse: the human-driven car sped off, but the impact from that car upon the woman who was walking threw her into the path of a Cruise autonomous vehicle, which made an emergency stop, but not before running over the woman and trapping her under the rear axle. Cruise and Waymo are currently the only two licensed driverless taxi services in America, and now that list is down to one because the California DMV has just revoked Cruise’s license to operate driverless vehicles. A major reason for this decision was that Cruise withheld crucial video of the October 2 trapped pedestrian incident, according to reporting conducted by Aaron Gordon, writing for Vice. 

The text of the DMV’s suspension news release gives some important details:

The California DMV today notified Cruise that the department is suspending Cruise’s autonomous vehicle deployment and driverless testing permits, effective immediately. The DMV has provided Cruise with the steps needed to apply to reinstate its suspended permits, which the DMV will not approve until the company has fulfilled the requirements to the department’s satisfaction. This decision does not impact the company’s permit for testing with a safety driver.

So, it looks like Cruise is still permitted to operate their vehicles if they have a human safety driver ready to take over if needed.

The specific reasons for the suspension are listed as well:

Today’s suspensions are based on the following:

• 13 CCR §228.20 (b) (6) – Based upon the performance of the vehicles, the Department determines the manufacturer’s vehicles are not safe for the public’s operation.

• 13 CCR §228.20 (b) (3) – The manufacturer has misrepresented any information related to safety of the autonomous technology of its vehicles.

• 13 CCR §227.42 (b)(5) – Any act or omission of the manufacturer or one of its agents, employees, contractors, or designees which the department finds makes the conduct of autonomous vehicle testing on public roads by the manufacturer an unreasonable risk to the public.

• 13 CCR §227.42 (c)- The department shall immediately suspend or revoke the Manufacturer’s Testing Permit or a Manufacturer’s Testing Permit – Driverless Vehicles if a manufacturer is engaging in a practice in such a manner that immediate suspension is required for the safety of persons on a public road.

These are pretty severe: “the Department determines the manufacturer’s vehicles are not safe for the public’s operation,” is a pretty clear statement, but more disturbing is this: “The manufacturer has misrepresented any information related to safety of the autonomous technology of its vehicle,” referring to the withholding of the video.

Person trapped under cruise vehicle https://t.co/IsNvQgLQlF

— FriscoLive415 (@friscolive415) October 3, 2023

The missing video in question has to do with what happened after the Cruise AV came to the first emergency stop. The video of the incident provided by Cruise to the investigation apparently only showed up to the point of the emergency stop, where the pedestrian was first pinned under the car. After the emergency stop, the Cruise vehicle attempted a “pullover maneuver while the pedestrian was underneath the vehicle,” which refers to the vehicle pulling off active traffic lanes. Normally, this is a very good idea, but in this case that involved dragging the person under the car for 20 feet at seven mph, which is a significantly worse idea.

According to Vice, the DMV only found out that footage of the pullover maneuver existed via learning about it from “another government agency,” prompting the DMV to ask Cruise for the footage, which was then provided.

Cruise spokesperson Hannah Lindow was reached out to by Vice, but has yet to respond regarding the video footage that was not initially provided.

Cruise did provide a statement on ex-Twitter, which I’ll quote here:

“We learned today at 10:30 am PT of the California DMV’s suspension of our driverless permits. As a result, we will be pausing operations of our driverless AVs in San Francisco. Ultimately, we develop and deploy autonomous vehicles in an effort to save lives.

In the incident being reviewed by the DMV, a human hit and run driver tragically struck and propelled the pedestrian into the path of the AV. The AV braked aggressively before impact and because it detected a collision, it attempted to pull over to avoid further safety issues. When the AV tried to pull over, it continued before coming to a final stop, pulling the pedestrian forward.

Our thoughts continue to be with the victim as we hope for a rapid and complete recovery.

Shortly after the incident, our team proactively shared information with the CA DMV, CPUC, and NHTSA, including the full video. We’ve stayed in close contact with regulators to answer questions and assisted the police with identifying the vehicle of the hit and run driver.

On October 3, 2023. representatives of the Department of Motor Vehicles and the California Highway Patrol met with representatives from Cruise to discuss the accident. During the meeting. the department was shown video footage of the accident captured by the AV’s onboard cameras. The video footage presented to the department ended with the AV initial stop following the hard-braking maneuver. Footage of the subsequent movement of the AV to perform a pullover maneuver was not shown to the department and Cruise did not disclose that any additional movement of the vehicle had occurred after the initial stop of the vehicle. The department only learned of the AV’s subsequent movement via discussion with another government agency. The department requested Cruise provide a copy of the video with the additional footage, which was received by the department on October 13. 2023.

The reasons why Cruise may have decided to not show that particular part of the footage aren’t known exactly, as Cruise isn’t talking about that in detail yet, but from an outside observer, it’s pretty telling. The initial part, where the AV made an emergency stop to avoid hitting the person, that’s good footage to show from Cruise’s perspective, because it shows the AV behaving as it should, taking whatever steps necessary to avoid harming a person; the fact that the person was hit can’t be blamed on the car – that’s just physics.

What happened next, though, is extremely important because it reveals one of the biggest unsolved problems about automated driving, which is that driving isn’t just a lot of sensory inputs and mechanical/physical reactions, it’s also something that requires a lot of general awareness of surroundings and situations, and that often can blur into understanding social contexts and a huge variety of human communication and behaviors. Even things as brutally simple and basic as “if there is a person stuck under the car, do not drive,” a nuance of human society that this Cruise vehicle failed to realize.

Automated driving means a car integrates into a mass of messy and wildly varied human behavior because at this moment, cars are still extensions of us and our bodies, magnifiers of actions we take and decisions we make. An AV that obeys the rules of the road and the mechanics of driving is a good way to accomplish the driving task, but, especially in a crowded urban environment like San Francisco, that’s not all the way there.

The Cruise’s problem here was pretty simple – it’s a machine that has no self-awareness, no knowledge of what it’s actually doing. That means when things aren’t happening as expected, it lacks the ability to reason through the new circumstances and come to a reasonable decision, so it relies on what it knows, which, in this case, ends up meaning that a person got dragged 20 feet across the pavement.

As it stands, Cruise is suspended from driverless operations, and as far as the future holds, the “DMV has provided Cruise with the steps needed to apply to reinstate its suspended permits, which the DMV will not approve until the company has fulfilled the requirements to the department’s satisfaction. “

I’m curious to see if this will affect how developers approach the scope of what AVs need to do in the future. As strange as it sounds, knowing not to drag someone trapped under the car is a pretty nuanced task for artificial intelligence. I just hope these blurrier, less clearly defined requirements of a functional automated vehicle are taken as seriously as the more obvious challenges.

A Video Showing A Police Officer Yelling At An Autonomous Car Has Me Worried About Robocar Emergency Overrides

This 1960 Issue of Popular Science Has Front-Wheel Drive Car Owners Talking Smack And Early-Autonomy Experiments That Still Make Sense Today

We’re Entering The Wising-Up Phase Of Self-Driving Cars And It’s About Damn Time