The UK (infrastructure) has a tendency to fall apart when it rains, is too windy, too cold, or too warm.
We as a people seem to cope best with a moderately cool climate and without any surprises.
In the last couple of weeks we’ve been unfortunate to receive a lot of inclement weather; cold, wind, and rain.
Whilst this has presented some unique challenges (for us) that are more common in countries where the climate is more extreme. It has raised the topic of disaster avoidance through the migration of workloads away from locations where an impending disaster could occur, to ensure that business operations continue.
In areas with extreme climate this is pretty common, I’ve worked with organisations who will at notice of a hurricane warning flick operations from one location to another just in case. I am in complete awe of these folks at how they as human beings cope, let alone as business leaders.
Here in the UK we don’t tend to be subject to these kinds of weather alerts on a frequent enough basis for it to factor into most organisations Business Continuity planning (though it should) above other more prevalent issues like Aircon failure, plague of locust, etc.
So… Disaster Avoidance (albeit a bit late) for one customer this week was quite interesting and made me ponder a bit.
They run a well known public facing webservice out of their physical presence hosted on a hyper-converged infrastructure and a leased line. This service is used by thousands of people every day.
When the weather came, so did the power brownouts, thankfully handled by UPS; but we observed short periods of internet connection loss that undoubtedly meant users couldn’t get to the hosted services.
What to do?
A plan was hatched to reduce dependency upon the physical presence and migrate the presence to Azure and rely on built in protections between availability zones and regions that are out of reach of organisations without multiple physical presences.
Backup data was being restored to Azure to stand up test systems when the weather changed (it’s sunny now) for the better and things calmed down.
I am simple believer in the 7 Ps, “Prior planning and preparation prevent p**s poor performance.” With the problem now known, preparations to de-risk it are in motion! (replacing that particular problem with some of the risks a hyperscaler brings!)