I don't think the police would let an ordinary driver have their car back 4 hours after they ran over a pedestrian! I think they'd keep it for some time to aid their investigation. Is Cruise treated differently here, or am I just wrong?
I'm also surprised they seem to be sharing video by starting a video call and then screen-sharing while it plays?? I'm struggling to come up with a better strategy for degrading the quality.
Presumably the reasons why police want to retain a car in a investigation don't apply here. They know who (what) the driver was, and with the telemetry from the vehicle, they can know exactly what happened. They don't need to collect evidence to prove which human was behind the wheel. There's no point searching for drugs or alcohol. They don't need to dust for fingerprints. They don't need to confirm that damage to the vehicle corroborates eyewitness accounts about what collided where.
Still doesn't make any sense. NTSB doesn't let airline take custody of plane immediately.
You want time to recover data, investigate the physical condition of the aircraft/vehicle, etc.
How thorough of the job can the police do in the dark from 9:30pm-1:30am?
This completely looks like a special snowflake exception that Cruise, given its behavior afterwards, should not have had... and at this point, should not be trusted to have again.
I think you hinted at the real reason but didn’t spell it out.
There are no protocols to follow, lists of checkboxes to check, or “perpetrators” to harass in a self-driving car collision (I have to remind myself to avoid the word accident).
At least that is my guess.
It is important to remember
I think that to the police officers this is just a day in their job they are clocking in and out.
They are just following their instructions.
It might be life changing for someone who got hit but the world just keeps spinning.
How do you check the basic mechanicals of the car at that hour?
So that the driver (automated or human) cannot later claim "oh there was a mechanical failure" ?
What's to stop Cruise or other bad actors to take possession of the car and then clip the brake lines to claim it was actually an OEM vanilla car mechanical fault and not their AV hardware/software?
It's not that outlandish. You already have them doing a whole bunch of lies of omission and smoke screening their video evidence, enough so to trigger a wave of resignation/firings.
No police department in the world has the resource or staff to properly do an investigation of these incidents.
Unrealistic take: The police should have kept the vehicle and demanded the source code be released so that it may be investigated if proper safety mechanisms have been implemented and if any errors in the systems may have caused it to hit the pedestrian. In the meantime the vehicles running on the same hardware and software should be put on probation.
Realistically: Police cannot investigate this and as long as these vehicles are classified more like computers and less as an aircraft these incidents are "glitches".
> NTSB doesn't let airline take custody of plane immediately.
Presumably that happens because the NTSB want to examine the airplane.
In this case there was no equivalent organization to the NTSB run by the Police and nothing about the car that needed to be examined by the Police to establish what had happened.
It's not like Chase the company is going to run away, or Chase the company will get away with destroying their data logs if the police asks for it later. Keeping the car serves no benefit.
>... with the telemetry from the vehicle, they can know exactly what happened.
The telemetry would be coming from Cruise, who is not a trustworthy source. This incident is a perfect example of why we shouldn't rely solely on telemetry - whether you believe they were just an innocent victim of bad internet connections or actively trying to downplay the dragging, it's clear they were not a wholly reliable source of information.
There is rarely any doubt about who was behind the wheel and police is not collecting evidence about that. Also, alcohol and drug use is determined by breath or blood test.
Car crashes never lead to dusting for fingerprints.
> I'm also surprised they seem to be sharing video by starting a video call and then screen-sharing while it plays?? I'm struggling to come up with a better strategy for degrading the quality.
It's a very effective way to prevent the video from being spread through the entire Internet faster than anyone can reasonably act.
Investigations should happen blameless, behind closed doors, not with stuff being spread all over the internet and armchair experts negatively impacting the investigation with completely unfounded speculations.
That's at least the way aeronautic and European railway investigations happen, and for good reasons - you don't want people holding back evidence for fear of repercussions.
> while a human might react to the collision at -2.9s and slow down or stop, "scene understanding" as a human might do it is non-existent in most or perhaps all AVs...
That's a good point. If you see a pedestrian hit by a car anywhere near your car, you stop, right? If you see a pedestrian walk out into the road against the lights you at least slow down and/or cover the brake pedal because who knows what they're going to do?
I think this is a critical part of cracking self-driving cars - "scene understanding", or more accurately, identification AND MODELLING of all traffic participants. A tiger chasing a deer will identify and model what the deer will do, what the terrain is, and take appropriate decisions on how to best pursue; a good driver will do the same - e.g. understanding that a round object bouncing in the street of a residential neighborhood is a ball, and where a ball comes, a child will follow.
I also think that this feedback loop of "Model the world, take action, observe, update model" is a key component of consciousness (whereas we model ourselves!) and key to AGI. I would not be surprised that AGI comes from a self-driving car development model.
OTOH, this is not what people do in emergency situation; the modelling loop is just too slow. In emergencies people need to rely on reactions which are built through training and practice. A good modeller can often avoid the emergency situation (i.e. your ball example), but once in an emergency situation it's a reaction situation.
The difficult part of the UK driving test is doing exactly that modelling loop. I agree with you. It's not about responding to an emergency - yes the loop would be too slow for that. It's about foreseeing the emergency before it happens. Looking ahead further down the road for things that could turn into hazards that should be prepared for ahead of time. It's credited with reducing the accident rate, though I'm not entirely sure how they worked that out.
Imo our reflexes differ based on context. But I'm not sure if it's the reflexive system chosing one of N options that are always available or the planner that "activates" just a few options based on the current context.
Maybe it's not reliable enough as-is, but seems to me that this "something odd is going on, better slow down or stop" requirement might be entirely solvable with anomaly detection. When the data is not as it normally is, take extra care. It's not that complicated and AGI is hardly required.
The problem is “something odd is going on” requires precisely the “scene understanding” that is currently absent from all AVs.
The reality is that nearly all sensory input is novel in some way and a large part of the work the human brain does is quietly filtering out a massive amount of irrelevant input in a way that is computationally infeasible with current technology.
I'm guessing this is one of the things that Tesla's Dojo supercomputer combined with the billions of miles collected from their fleet is trying to solve
I'm guessing not; what NNs do is pattern matching. Basically they try to "remember" what should be done at in a given situation, but does not have continuous representation and modelling of other objects; not even "object permanence" - when a car disappears from their world, it's gone.
You might stop, yes. But many humans don't. That channel is mild and tends to be relatively work-safe; it doesn't take much searching youtube to find the rest.
It's hard to avoid thinking that maybe humans should never be allowed to drive automobiles.
Properly trained, well rested, and sober humans are very, very good. If you can create an AV with a single open-source software version, entirely fixed with zero changes during the whole duration, and get it to drive 200 million miles, I'll reconsider.
I'm not sure that's actually true, but even if I posit it... "properly trained, well rested, and sober" does not actually describe humanity. You can revel in your moral superiority to the average driver, or you can make the roads safer.
AVs do not need to be better than the best human drivers. AVs need to be better than the average human driver. The average human driver is shit.
Indeed, and you'd hope that AVs would do this. Notably the car which caused the initial incident didn't do this, they hit the pedestrian then drove off!
Yeah, it's not like the humans in this story were showing their best side, but AVs have to tak that kind of stupidity into account I think.
Like the fact that a cyclist on the other side of the road is a hazard to you, not because of the cyclist, but because an impatient driver stuck behind them is likely to pull out suddenly round the cyclist, forcing you to brake.
>Wood shows full video, "reportedly with internet connectivity issues from his home computer"
There's one for the back to office lot.
>wheels were moving at different speeds (because one wheel was spinning on pedestrian's leg)
Eurgh.
But in general, this is Cruise learning what traditional car companies already knew - you need to act with integrity. I don't see how senior executives at cruise could reasonably be allowed to continue to work in any safety critical industry after this.
> "But in general, this is Cruise learning what traditional car companies already knew - you need to act with integrity"
Agreed - but also a key mantra in political and crisis communications: the coverup is what kills you. The coverup (or just the appearance of it) is far more damaging than the acute crisis itself.
Well, they need to act with integrity towards safety regulators, anyway. I wouldn’t say the auto industry, broadly, has the greatest reputation for integrity.
This article takes quite a long time to get to the accident itself:
> 10-02, 9:29pm: human-driven Nissan Sentra strikes pedestrian in crosswalk of 4-way intersection at 5th & Market in SF
>Pedestrian entered crosswalk against a red light and "Do Not Walk" signal and then paused in Nissan's lane. Police report cited both driver and pedestrian for code violations and concluded that the driver was "most at fault"
>The impact launched the pedestrian into the path of the Crusie AV
>Cruise AV braked but still hit pedestrian
>After coming to a complete stop, the AV moved to find a safe place to stop, known as a "'minimal risk condition' pullover maneuver (pullover maneuver) or 'secondary movement.'"
>AV drove up to 7.7mph for 20 feet, dragging pedestrian with it
This reminds me of the case discussed in the movie Bridge of Spies. There were multiple impacts. One side sees one accident, starting and the first impact and ending at the second. The other sees two distinct accidents, one impact and a second separate impact rather disconnected from the first. Crashes are messy. They are always up for interpretation. That is the reality of driving. Only in hindsight can we apply the arbitrary legal rules to allocate blame. The reality is almost always that everyone is at least partially at fault.
If not “at fault”, everyone is probably doing something they could/should have done better or more safely. In this case, the pedestrian could have avoided crossing against the light, and could have continued walking rather than stopping; the Nissan could have avoided the incident by noticing the pedestrian and stopping; the Cruise could have avoided its part of the incident by having more situational awareness, and knowing that when a pedestrian is acting strangely it should probably slow down, and when a pedestrian gets hit in the next lane, it should immediately stop.
All complex systems are in “failure mode” of some sort at all times. But complex systems that work have enough wiggle room and safety mechanisms that true failure doesn’t happen until there’s some criticality of overlapping errors.
Yes, there are some failure modes that are so devastating that everyone else could have been behaving perfectly and it would have made no difference, but likely many if not most traffic accidents are the confluence of multiple failures, even if most are minor and would never cause an accident on their own.
But that is how it works out. In this situation the pedestrian apparently didnt follow one rule. The first car didnt follow another. And the second did something so odd that, if by a human driver, probably qualifies as negligence. Nobody has clean hands. The legal debate is about who has more or less blame, who has enough to be adjudicated reaponsable for which resulting injuries.
From legal standpoint, situation in which one driver is entirely at fault and another not at all is completely standard.
Likewise ethically, morally or responsibility wise. The "everyone at fault" sometimes happen, but it is not clear at all whether it is majority of cases.
Legal determinations come after our accepted rules are applied to the situation. Those rules ebb and flow dependent on the nature of the inquiry and the legal standard applied. Even seemingly irrelevant factors such as insurance coverage can be pivotal. Where both divers are covered by the same insurer, there is a higher likelihood that both will be found somewhat at fault. An insurer on the hook for both sides isn't going to spend money arguing forcefully for an all-or-nothing determination. But such things are totally irrelevant to anyone seeking to prevent accidents from happening in the first place.
Definitely does not work that way where I live. The "guilt" is determined independently of insurance situation. The insurance companies decides whose insurance will be used to pay which does not influence fault determination at all.
> Where both divers are covered by the same insurer, there is a higher likelihood that both will be found somewhat at fault.
Not true at all.
And in particular, beyond being fined for not having mandatory insurance, you insurance does not influences at all who goes to jail or who will be fined and for what.
> The legal debate is about who has more or less blame, who has enough to be adjudicated responsible for which resulting injuries.
The legal debate fairly often ends with one driver being ruled as the cause of the accident with no blame spread.
Fault is not just loaded term, it literally means that you are to blame.
But even "almost always everyone is contributing to the crash / making the accident worse than it could have been" is pretty tall claim and would require a lot of evidence. Unless you count "leaving the house in the morning instead of staying home" or something like that as a contribution toward accident.
A typical large inquest will come up with a list of several dozen items that could have prevented or mitigated a situation, and I don't think common accidents are any different from unusual ones. Perhaps you wouldn't call all several dozen "to blame", but certainly all contributed.
Thought experiment - vehicle A disregards a stop sign and pulls out directly in front of vehicle B* and there is a collision.
I don't see how vehicle B "could have prevented or mitigated a situation" other than by not being on that road at that moment. The driver of vehicle A is completely at fault, B is blameless.
* spoiler, it's actually not a thought experiment: my mother was in Vehicle B when this happened to her when she was coming to pick me up from school when I was about 10 years old. Having your mother arrive at your school in the back of a police car was a unique experience....
I have been in almost exactly the scenario you described as vehicle A. I came up to an intersection and my view was occluded by a car parked to my right in a no parking zone. There also was a gradual bend in the road a few hundred feet to my right. I looked right, then left and started creeping out. I then saw that there was a car barreling towards me that I couldn't see (because they weren't visible when I first looked right). By the time I saw them I couldn't stop fast enough to prevent being hit. But he barely hit the passenger quarter panel and the skid marks were completely straight which is to say that I was barely in his lane and had come to a complete stop when hit.
I did mess up by not creeping slightly slower but the driver who hit me had far better visibility (I checked from that perspective). I am almost certain they were going too fast for the roadway and may not have been fully paying attention. I didn't just blast through. I was creeping out at very low speed. An attentive driver would have seen me and taken evasive action and/or honked.
The third party that had illegally parked obstructing my view can and drove off while we were sorting out the wreckage. In my shock, it took weeks before I realized that they had also contributed significantly but I had no proof since they'd just driven away.
I was considered fully liable for the crash since I had entered the roadway without the right of way. I accept that I made a small mistake. I understand why the insurance people made that determination. But it still burned to be considered fully at fault considering all the other mistakes (including bad road design) that contributed.
I will bet though that driver B in my crash felt that I was completely at fault and might have described it exactly as your mother did her accident. In his case, I think he likely would have been leaving out his role in not paying attention and driving too fast. There can be more nuance even to seemingly cut and dried cases.
If Vehicle B was paying attention they can see vehicle A not slowing down (assuming there is sight lines). The options are then to stop despite having the right away, or take a sharp turn. Vehicle B probably could have been going slower to allow more time for the above (this is one reason why round-abouts are so safe: they force everyone to go slow - which is what the other response was getting at)
I have almost been in many accidents, but one of the two parties (sometimes me, something the other person - in proportion to suggest we are all average drivers) was paying enough attention to see the mistake and take evasive action to prevent it. When investigators look at accidents, often even when they can assign 100% blame to one party doing something stupid: there is still evidence that the other parties could have safely avoided the accident if they had taken the correct action.
> 10:05am: Wood and VP of Global Government Affairs Prashanthi Raman have virtual meeting with Mayor of SF's transpo advisor. Wood shows full video, "reportedly with internet connectivity issues from his home computer"; neither Wood nor Raman brings up or discusses pullover or dragging
> 10:30am: Virtual meeting with NHTSA. Wood shows full video, "again having internet connectivity issues causing video to freeze and/or black-out in key places including after initial impact" and again not bringing up or discussing pullover or dragging
Convenient “internet connectivity issues” indeed. Looks a lot like students willingly corrupting zip files when sending assignments in order to gain time when they couldn't finish…
I really like the admiralcloudberg blog, which provides extensive analysis of plane crashes and the NTSB investigation that follows.
I hadn’t thought of it, but I think the idea of similar investigations for self-driving car accidents is a really great idea.
Though, one recurrent theme of that blog is that blame and persecution are generally counterproductive to understanding the accident and improving safety.
Blame and persecution are not counterproductive, they are crucial to improving safety.
It is called accountability and if nobody is accountable for anything then there is no actual motivation to get businesses and financial institutions to do anything about it.
What is counterproductive is blaming and persecuting wrong things.
The goal is to get companies to be accountable for being committed to taking this process seriously, providing honest information whenever it is needed and then to honestly implement the recommendations once the process is concluded.
We know that technical bugs happen regardless of good intentions. So we should err on the side of being lenient.
Where we should not be lenient about is simple negligence where all of the information was available and the decision was made regardless. For example, if the manufacturer knew that bolts were lying around and decided to save on the costs by being willingly blind to the fact.
And where real persecution should strictly follow is hiding information and evidence that is obviously needed to improve safety.
Drivers can't be, but road designers and car manufacturers can and should be.
It should be unacceptable to design roads or cars in a way that increases the likelihood of injury or death and every serious injury or death should be investigated to determine if the road or car contributed to the accident.
> When the light turned green, the Nissan Sentra and the AV entered the intersection. Against a red light, a pedestrian entered the crosswalk on the opposite side of Market Street across from the vehicles, passed completely through the AV’s lane of travel, then stopped mid-crosswalk in front of the Nissan Sentra.
I wonder whether this flagged alarms in Cruise's systems when the pedestrian crossed in a potentially dangerous situation. These kind of 2nd and 3rd order conditions seems particularly hard to train for.
This sort of thing happens all the time in SF. There are a lot of crazy people in some areas of the city. If you stopped for every one of them (even the ones a lane or two away) you’d never get anywhere. Heck, I’ve had someone run in front of my vehicle, then pull out a collapsible baton and try to rob me. I pinned the throttle and they jumped out of the way.
Is there anything here to train for? The initial collision wasn't in any way Cruise vehicle's fault; but the subsequent repositioning shows their procedures and sensor coverage are lacking. Oh, and their integrity.
Yes, if a pedestrian is crossing against the light in front of you, you slow down even if you don’t think you’ll hit them, even if they’re already past your lane. If the pedestrian stops on the street they are crossing against the light, you slow down, even if they aren’t in your path, especially if you are aware of the vehicles around you and know the pedestrian is in the path of another nearby vehicle. If you see the vehicle in the lane next to you speeding up when you’ve noticed the pedestrian is stopped in their lane of traffic, you slow down and maybe veer away if possible. If you see that the vehicle next to you has hit a pedestrian, you immediately hit the brakes rather than wait 2.4 seconds until you’ve determined you’re going to hit that pedestrian before you decide to hit your brakes.
Those are all things an alert human driver should be ready and able to do. I would expect a quality AV system to do them better than a human, but in this case, none of those things appeared to happen. This AV decided “pedestrian is out of my lane, continue accelerating!” Only stopping when the pedestrian was right in front of the vehicle.
> This AV decided “pedestrian is out of my lane, continue accelerating!”
That's what human drivers do too. If you slowed down anticipating that a pedestrian's collision with some other car might fling their body in front of your car, you'd never make it to your destination. There's just too many crazy people in SF
Cruise isn't at fault for the collision with pedestrian, since it was a second order impact. The primary cause being Nissan Sentra and pedestrian. But could cruise avoided the impact and is this how humans react when we drive? Here is the timeline from the report.
- AV starts moving at -9.2s, after light changes
- Prediction output shows pedestrian path crossing AV travel lane: -7.7s
- Pedestrian leaves AV's travel lane: -5.3s
- Pedestrian pauses at crosswalk : -4.7s
- Contact b/w Nissan and Pedestrian; -2.7s
Did the driver of the Sentra ever get caught? And what are the odds that the ped who was hit got an extremely nice settlement from this? I can only imagine the flooding of emails and calls they got from injury lawyers after Forbes broke the story
Also, based on how the Cruise AVs we saw in Houston perform, this was all but inevitable. They drove extremely erratically and casually drove down wrong way lanes several times.
Thanks for doing this danluu.com. I hope our journalists go back to being useful about sharing facts again and not just marketing like they are doing now.
> Cruise incident response playbook outlines roles of Incident Commander, SLT, CMT as well as how to respond in the weeks after incident. Playbook was not followed, said to be "aborted" because "too manually intensive"
What? Who ran risk management for this company? If you're facing program halt you keep running the plan until you run out of money to pay your employees, and then keep running it after if you can get volunteers. If these people actually cared about AVs and safety, or even the continued existence of their company, they would work to extract every single detail that could possibly help any AV or robotics/AI company in the future, up to a very high cost for themselves. Deeply unserious.
In a hypothetical situation where these cars are deployed widely and are actually as close to perfect as we'd hope then that would be a much safer environment for you and everyone else than one in which drivers can be drunk, under the influence of drugs, tired or in too much of a rush.
I just don't believe the "close to perfect" story. And I know that those who died while they were not close to perfect yet really don't care that there is this hypotetical future.
In this specific case, it might be worth noting that, had the pedestrian not been dragged... They almost certainly would have died still, having been double-collided by multi-ton vehicles.
Human drivers are still, mile-for-mile, the most dangerous threat to all pedestrians, blind and sighted.
I know people who have died from human driven cars, where the driver was sober. Autonomous cars only need to beat that, which is a much lower bar than perfect. Of course the closer to perfect they get the better, but a lot of people die on the roads, if autonomous cars can improve on that then we need them now, not waiting for some perfect. (I don't know if they really improve - all data I've seen is confusing and may be an attempt to hide the truth in some way)
I know that was the official conclusion but I can't understand why -- the pedestrian walk out into the street (disrespecting the do-not-walk sign) and then paused in front of the car that hit her.
and on top of that, the pedestrian was crossing against a red (do not walk) light. Lots of human factors issues in this incident from all sides (driver, pedestrian, Cruise executives, regulators, media, policy makers, etc.)… The technology issues don't seem particularly profound.
Why? You'll get much cheaper taxi services + the cars around you will be much safer (and the laws will be changed so far fewer bad drivers will be allowed to drive).
I would happily pay extra money for a human driver. Especially because I rely on their cooperation when getting in / out. A human taxi driver will call out to me when he stops, knowing that it isn't easy to destinguish him from other cars. And when getting out, I often ask the driver for the favour of showing me the door, when I drive to a place I dont know well. These are all things a robotaxi can not do for me. Which means the service quality will go down. I dont care about prices going down if the service is also diminished.
To ameliorate your fear, a practical and potentially effective solution may be wearing a retroreflector, or some sort of light-based signalling device that would make it much more clear that you should not be run over.
This remedy was already valid for normal human drivers. I am more concerned about automatic drivers not honoring the "Vertrauensgrundsatz" (principle of trust) which is a traffic law from my area whic, to my knowledge, doesnt even have a counterpart in the US. Basically, drivers learn that there are two classes of pedestrians: "normal" pedestrians, and those which basically cant be trusted. IOW, drunk or blind people. If you run over one of those, the likelihood of it being your fault rises.
To put it in more direct terms, I wnder if AI training does and/or can even distinguish between different classes of pedestrians. I am suspecting the current approach is to treat them all the same. Which would result in a driver that doesnt know and therefore violates the "Vertrauensgrundsatz", someone who wouldnt even get a drivers license where I dwell...
Are you aware of the recent developments in AI at all? This isn't an unsolved challenge, video multimodals would likely crush this problem.
However, the advanced, LLM based systems with tons of world knowledge (and therefore can easily tell if a pedestrian is abnormal) take tons of processing power and latency, so cannot be deployed in the current AI driving systems. So it'll take 5-10 years for this to be solved.
Also hope you know that Germany isn't the center of the world. Its not even the center in terms of cars anymore. German regulators don't get to dictate the progress of self driving.
When it comes to regulations, I hope my country is actually "the center of the world". Actually, the attitude you display in your last paragraph is more confirmation of my suspiicion then I actually want. Tech innovation people like you easily take a few dea people as long as innovation is not hindered. I am afraid of you guys.
Wow, letting German regulation be the center of the world. Have you forgotten dieselgate? And how German regulators were in bed with the car companies?
You are not afraid of corrupt bureaucrats, but of idealistic innovators trying to change the world?
By the way, there is no way to tame safety via checklists or 'policies'. Boeing sure had checklists and layers of policies trying to control risk, can't save them when they outsource everything to workers who don't give a crap about their job.
But people in the AI world care, they absolutely do care about what they build. Pride and passion in their work beats any checklists, and I'm confident self driving cars will be a massive safety boon for all.
you realize this whole thing was caused by a human driver hit and run, right? You have humans to fear. The less human drivers there are the safer you are
Really? It seems to me it was entirely caused by the pedestrian doing not one but two stupid things: 1) crossing the street when it wasn't her turn and 2) pausing in front of the car that then hit her.
The "run" part happened afterwards and should of course be punished harshly but the "hit" part seems to be her own fault.
I'm confident that once driverless cars are good enough, blind pedestrians will be far safer than they ever were with drivered cars. But we're not there yet. Right now driverless cars should be banned from our roads.
Cruise and Waymo are inviting catastrophe by having their embryonic systems practising on public roads without a human in the driver's seat. None of these systems are anywhere near good enough to be let loose unmonitored. I know it's unpopular to praise Tesla here, but in my view they deserve some credit for sitting pretty at SAE L2 while everyone else seems obsessed with quickly impressing investors with a half-baked rush to L3 or L4.
It seems that Waymo has a command center where a human is monitoring the cars and can immediately issue commands.
Having taken Waymo a number of times in SF now, I have experienced this once. The car stopped and an unsolicited call from customer support came into the car to check on things.
While subjective my experience in Waymo has been positive and feels surprisingly safe.
The car drives conservatively but naturally. It shows a great understanding of other cars and people and cones from LIDAR on its HUD. It takes safe routes and picks safe side streets and alleys for pick up and drop off. There is a real-time support team watching things.
I doubt it’s ready for every city — SF is well equipped for public transportation with lots of well mapped local taxi routes and loading zones in the city.
And there will be surprises and accidents.
But it’s absolutely wild that we’ve reached this level of self driving, and it will be transformative for human kind as it continues to improve.
“Rush” is subjective. Waymo have been driving around with human drivers for years before let loose and they are still highly restricted to city limits and local routes. And I was taking design courses in self driving control systems taught by General Motors back in uni in 2005.
> I'm confident that once driverless cars are good enough, blind pedestrians will be far safer
I can’t reconcile the industry’s assertion that safety is the penultimate objective with our testing these things, uncontrolled, in densely populated urban areas.
At some basic level it will always just be about profit. Their safety “just” needs to be good enough to not get shut down. Cynically, talk of drunk drivers etc., is marketing to sell the story to regulators and the public.
More optimistically, at least it’s possible to diagnose these mistakes and work to mitigate them. The fact that we could now start treating auto collisions as seriously as plane incidents might be reason to hope.
Money may be the medium of business— it’s how resources get distributed and how businesses gauge their success— but the primary structure of our society is humanity, not money. The way businesses or any other entity affects people is the only measure we should use to judge their usefulness and appropriateness of their behavior. That might be complicated to assess, but “oh well, you know, money” strikes me as the wrong place to land. While it may seem quaint to many, i won’t accept that some powerful group’s pursuit of wealth is more important than any non-wealthy person’s right to not get mangled by a pseudo-sentient taxi driver robot. If that’s the way it works out, as it obviously has, I’m pretty sure we should concentrate on making things closer to the way they should be rather than bemoaning the way things are.
Development at a glacial pace. Their product timeline and finances are not my concern, and it’s not everybody else’s responsibility to mitigate the risks of their product development. Lots of things in industry would develop much more quickly if we decided to set the product safety bar low enough— I’m glad we at least recognize when we screwed up by taking away the operating licenses of companies like this. What happened here was exactly what I envisioned happening when the slick, CYA-first, blame-everyone-else, ethics-are-negotiable attitude endemic to the software (and other) industries entered this safety-critical field. The Therac-25 fried multiple people before the industry acknowledged enough culpability to trash a design that would never have been released if they had used proper design and engineering discipline in the first place. It was just a whole lot cheaper to let a developer check his own work and blame everyone else.
> I know it's unpopular to praise Tesla here, but in my view they deserve some credit for sitting pretty at SAE L2 while everyone else seems obsessed with quickly impressing investors with a half-baked rush to L3 or L4.
My concerns with Tesla are primarily about the safety implications of how it's automation capabilities get marketed, rather than the hardware itself… I also don't "get" the obsession over the "vision only" approach, which seems like an unwarranted compromise to me.
"Tech culture" intersecting with safety critical products and services indeed needs to be reckoned with.
There's the issue of whether people get complacent with sufficiently good assistive driving systems. (Many will.) Up to and including driving in circumstances they would not have otherwise trusted themselves to drive.
Of course, not that long ago, a fair number of people were looking forward to a near-future when they'd have a personal robo-taxi at their beck and call. And it's pretty clear that's not happening anytime soon.
You can't ever fully stop wilful abuse of technology by end users (humans keep looking down at their smartphones) but you can mitigate the problem of complacency with increasingly better human monitoring. IMHO Tesla should be restricting FSD to vehicles where the car can visually monitor the driver's attention.
Predictions of the future have always been poor, but ML/AI has thoroughly fucked up any prior intuitions we had around the rate of advancement. So many things progressing more rapidly than we thought possible, while others being an endless slog of local maxima after local maxima.
The local maxima associated with machine learning/deep learning (plus hardware acceleration) led a lot of people--even before generative AI--caused a lot of people to assume broader tasks couldn't be that hard. Especially given early successes that seemed pretty impressive. At one level, the state of self-driving cars is pretty impressive. On the other hand, it's nowhere ready to take little Sally to soccer practice by itself which is what a lot of people really wanted.
At this point, even some relative skeptics like Rodney Brooks are probably looking overly optimistic in terms of timelines.
I suspect that current approaches only take us so far and some other things from cognitive science and so forth will need to be pulled in.
Agreed, and it doesn't matter how far away it is. If it takes ten years, it takes ten years. There should be no cars without humans in the driver's seat until the statistics bear out a level of safety at least one order of magnitude greater than human drivers.
Cruise could already be an order of magnitude safer than a human driver. We have enough milage data on Cruise to guess that it's comparable to a human driver, but to be assured that it's at least an order of magnitude safer requires a lot more data.
And you can't get that data with a safety driver in the car. A safety driver should always override in emergency situations, whether necessary or not.
We do not know that. We have some data about how they perform in idealized situations rather then general ones. And we have next to no independent data.
Also, we could lower amount of people who die to human drivers too and I am all for it. Speed limits, regulations about car design, higher driving age, better public transport, there is a lot that could be done.
If it was the same guy, why wouldn't he have constant connectivity problems if he has any problems? I know with my home Internet, when it's flaky, it's often flaky for days or weeks at a time. (The real question of course is why the video conferencing software was willing to drop video, why it wasn't uploaded as a file to a cloud server, why he didn't confirm they'd seen everything, and so on and so forth - but glitchy home connections are, and always have been, a fact of life.)
Yeah I don't buy that at all, sounds like made up excuse that is hard to disprove.
But even if true, clearly the Cruise folks must have expected questions about the dragging and if none came should have wondered why. They witheld evidence, intentionally or not, but knowingly.
That occurred to me as well. Also, it seemed notable that the one meeting that was not marked by connectivity problems seemed like the one with the audience least likely to have regulatory implications (SFPD/SFFD).
Somewhere it was mentioned that all these meetings were being conducted by different sets of Cruise team members… could have just been that the person presenting at those meetings didn't have the same connectivity issues.
> Although, as noted above, the instructions for the narrative section of the SGO report require submitters to “[p]rovide a written description of the pre-crash, crash, and post-crash details,” some Cruise interviewees expressed their belief that the requirements were narrower. In their view, the narrative called for a description of the collision itself, and thus did not include post-collision actions like the pullover maneuver and pedestrian dragging.
* 10:05am: Wood and VP of Global Government Affairs Prashanthi Raman have virtual meeting with Mayor of SF's transpo advisor. Wood shows full video, "reportedly with internet connectivity issues from his home computer"; neither Wood nor Raman brings up or discusses pullover or dragging
* 10:30am: Virtual meeting with NHTSA. Wood shows full video, "again having internet connectivity issues causing video to freeze and/or black-out in key places including after initial impact" and again not bringing up or discussing pullover or dragging
* 11:05am: Cruise regulatory, legal, and systems integrity employees have pre-meeting for DMV and California Highway Patrol (CHP) briefing; Cruise team doesn't discuss pullover and dragging
* 11:30am: hybrid in-person and virtual meeting with DMV and CHP. Wood shows full video, again with internet connectivity issues and again not bringing up or discussing pullover or dragging
Oh indeed, at least the presenter wasn’t stuck as a cat avatar. That would make the coincidental recurrence of connectivity issues even funnier, but shouldn’t the authorities have received a copy of the video, devoid of “internet connection” issues? Or was it not in legal position to demand the artifact and just have it shown over Zoom?
They claim the NHTSA got one. However, if, ah, connectivity issues obscure the video of the Bad Thing in the meeting, and the Bad Thing is not mentioned in the meeting, then the chances of it being noticed even if a video was sent do decline.
I'm sure someone did. It may not necessarily have been someone who had attended the meeting and thus was aware that the key fact had not been disclosed in the meeting, tho.
With stuff like this you want to be absolutely crystal clear what happened in your written and verbal reports. You can't just rely on a video that you may or may not have emailed over after previously showing it in such a manner as to obscure the key info.
> 10-02, 9:29pm: accident occurs
> 10-03, 1:30am: AV back at Cruise facility
I don't think the police would let an ordinary driver have their car back 4 hours after they ran over a pedestrian! I think they'd keep it for some time to aid their investigation. Is Cruise treated differently here, or am I just wrong?
I'm also surprised they seem to be sharing video by starting a video call and then screen-sharing while it plays?? I'm struggling to come up with a better strategy for degrading the quality.