Pretty confident if there were a 1989-level earthquake, most startups and tech companies would just decamp to somewhere else for the recovery period. The easiest is when you already have an office somewhere else (we have London and soon Asia, although SF is still far larger); I'd probably pick Hawaii or Seattle personally.
It's easy enough to find an off-season vacation destination and move everyone there; essentially anywhere is cheaper than SF, so that's easy. Not every company has to move to the same place.
It'd suck to be a service-industry worker in SFBA after an earthquake, but tech employers can just move, and hire service-industry people to fix up their offices for a few months before returning (or not).
(this might not be necessary after a 7, but after something insane like a 9, would definitely happen)
I've noticed a lack of of human planning in most DR plans. In some industries it's called business continuance. Our company has focused mainly on the technical side of the business: backups, DR, offsite access.
Depending on the situation, earthquakes can bring fires and tsunamis. So what do you tell the network engineer that had their house burn to the ground? "Sorry for your lose; we're decamping for Las Vegas. Wheels up in 8 hours"?
If the big one does hit there is a lot more at stake than just your precious data.
Google has a program called DiRT (Disaster Recovery Testing)[0], when in a certain week of the year, some major exercises that simulating the loss of headquarters are carried out, including the (simulated) loss of some teams, some network connections in the headquarter and/or the physical access to the headquarters buildings. That makes a lot of sense when you place your headquarters at a place that can be demolished by an earthquake.
That is actually an excellent idea. Loss of manpower is something that is overlooked in a lot of companies. I believe part of it is job security (make yourself invaluable). And part of it is HR optimisation. Why pay for 2 people to do the same job when 1 will do? The same reason you pay for 2 servers, 2 backup processes, etc.
I imagine for a large tech company, it would go something like this:
1. Immediately call extra hands in other locations (datacenters, customer service, etc); begin over-time scheduling to alleviate need for workers in disaster zone. Doesn't require any coordination of disaster zone, just remaining executives located outside.
2. Offer two special payments to workers in the impacted zone: 1. Disaster recovery assistance, to help them cushion emergency needs and get their family/belongings/etc recovered faster; 2. Relocation bonus, to create an incentive for quick relocation to temporary (or new) office.
3. Spend as much as you to have to get extra emergency services in to the area, and generally assist in government/regional recovery area. This will free up the last of your staff for temporary relocation.
I doubt that a large company (we're talking Google, Microsoft, Facebook, etc) would take more than a couple days to have other sites working overtime to cover the load, less than a week to have already transported some of their staff to a new site, and less than a month to have brought almost all of the office back online.
This timeline, of course, assumes that the problem wasn't that their office fell on all their employees, killing them or something of that nature.
Likely, we'd see a hiccup in the development cycle, but in terms of keeping infrastructure online, emergency bug fixes, etc, we'd probably see relatively little damage.
Disasters tend to have a knock on effect that can ripple for years. 2 that come to mind were the floods in Thailand and the 9-11 terrorist act. The floods temporarily knocked out the supply chain of HDD components, which had the knock on of system builders not having enough components. And eventually, technicians not having enough spare replacements for current systems.
1. In most businesses these other locations are supplementary to HQ. Typically there is no one at the datacenter and CS centers rely on guidance from HQ. They are not in the position to make decisions and drive the company.
2. Growing up and working in Florida you natural disasters are like clockwork. I can say for sure that 1 is covered by insurance and the gov't and 2 is going to be hard to pull off without looking like a complete dick. Also, you have other constraints. Airports may be closed and driving can be dangerous. Plus cars need fuel which is typically in short supply and expensive.
3. If you have a company of a few thousand I would say a few hundred are essential. Where are you going to find a place to house a few hundred people at once? Remember that other companies are thinking the same as well.
I also believe the most essential employees are that way for a reason. If they have the same preparedness in the company they also have the same mentality in their community. It would not be illogical to imagine the person in charge of DR for the company is also a member of the volunteer fire department.
Microsoft would be able to recover, then they find out the caterers don't have food to serve the cafeteria, employees can't come to work because there is no fuel, etc. These are the soft issues that most DR plans don't account for.
Used to work in Florida, and it's a pretty noticeable difference in DR/Continuity strategy to what I work with now out here (SF). For the first 24 hours after Hurricane Ivan it was completely focused on: Are our employees safe; do they have water/food; what medical aid is needed; how can we get blankets and cots into our office so our employees have a place to sleep tonight? All the roads are washed out, there was no real in/out of town for a week, no power to our main DC for 3. Having redundancy in Chicago and Virginia didn't really mean shit when we couldn't get key people to those data centers to run them full steam.
The end result was a high reliance on contract employees at the fail-overs, and a real appreciation for the human element.
In 1989, most of SF was back up and working by the next Monday. Of course, that was a different era without much telecommuting, and many people suffered tragic losses, but it wasn't bad enough to "decamp" anywhere. The oldtimers seem to complain more about the world series than the quake.
Yea. In real life it doesn't quite work that way. You would be pulling families apart ("Honey, you stay at home in Palo Alto, fix the house, make sure everything is okay with the kids while I go to Hawaii for 2 months") isn't gonna fly.
But then again, I keep forgetting that the 'real' valley is just for early 20s-fresh-out-of-school-startup-single-guy-whizz-kids apparently.
Do you run periodic disaster recovery tests including simulating one of your location being offline (including employees being unreachable) for e.g. one day? This is a good test to check whether the know how is well balanced geographically.
All of the companies that don't already have away-from-the-Bay data centers will be begging for hardware. And if your name isn't Microsoft or Oracle or some other big ams, you'll be on the end of a product delivery pipeline measured in months. Hardware is going to be fought over.
If all your data is in your own building, you might be okay (assuming you can get at it, stuff it into a truck and move it out). Otherwise you're going to be waiting for a while.
No one really has servers In the Bay Area anymore, in my experience. A few startups with legitimately special security needs and competence (Stripe), but mostly people either use the cloud (not in the bay) or colo/managed hosting (also not in the bay, usually).
There might be a run on laptops, monitors, chairs, etc which overwhelm the Hawaii and Las Vegas retailers for a while, but basically California's high energy costs, low power circuit availability, high sales/use tax on servers, etc have solved this problem already.
There are plenty of servers in the Bay Area. There are several large colos in the South Bay (Equinix has several facilities in San Jose), San Francisco has a large colo at 200 Paul Ave, there's a bunch of stuff in the East Bay as well. Amazon has a nothern california region, not sure exactly where those servers are, etc.
Sure, I have equipment in several of those, but it isn't a sole hosting location for most large tech companies. Really big companies tend to be multi site. Smaller ones use cheaper clouds or managed hosting generally outside the area. A lot of the bay area gear is enterprise for local companies which would already be screwed by an earthquake, or network, etc to support people who live in the Bay Area.
SFBA is critical for personnel, not so much manufacturing or hosting.
There is about 3mm square feet of datacenter space here. There are very few large companies which have servers only here and not somewhere outside the area but which would be reasonably expected to continue operating if their servers were somehow unaffected.
Okay, but you started by saying "no one really has datacenters in the Bay Area anymore", and now you're talking about sole hosting, which is a completely different assertion.
True. (You may also note the time of my initial post; I was still awake at 0320, and this was shortly after the quake...); I wasn't being particularly precise.
However, SFBA isn't even the first location for most companies I see. They go either into the cheaper AWS cloud regions (us east or the Oregon), or managed hosting somewhere (rarely SFBA).
Bay Area companies which directly get colo early on are fairly rare: stripe, square, etc. I fully support it as a strategy, but it is statistically insignificant.
Aside from price, east coast single location also gives you a lot better latency to Europe. Asia is usually screwed anyway, but the extra 80ms makes a big difference.
It's easy enough to find an off-season vacation destination and move everyone there; essentially anywhere is cheaper than SF, so that's easy. Not every company has to move to the same place.
It'd suck to be a service-industry worker in SFBA after an earthquake, but tech employers can just move, and hire service-industry people to fix up their offices for a few months before returning (or not).
(this might not be necessary after a 7, but after something insane like a 9, would definitely happen)