Worldwide Tech Outage Started with Defective Crowdstrike Update to Microsoft Windows: Nothing to See Here.

photorandy2023

2 years ago

I have written several articles on postings related to Big Tech, Social Media and Corporations. A list of links have been provided at bottom of this article for your convenience. This article will, however address different aspects on these Industries.

An issue with a commonly used security software called Crowdstrike shuttered large technology systems around the globe, including airlines, transit systems and stock exchanges.

A major IT outage has hit businesses across the world, grounding planes as well as affecting banks and the healthcare sector.

George Kurtz, CEO of IT security firm Crowdstrike, said it had traced the issue to a “defect found in a single content update” for the security software it provides for the Microsoft Windows operating system on computers.

Microsoft said the issue was caused by an “update from a third-party software platform” and that the “underlying cause”had now been fixed.

The Conversation spoke to Professor Alan Woodward, an expert in cybersecurity at the University of Surrey, about what went wrong and how the problem could be resolved.

CAN YOU EXPLAIN WHAT’S HAPPENED HERE?

I think there are two things. First, Microsoft seems to have had a problem with its Azure cloud computing platform. It’s a bit unclear, but there was a degree of degradation in that service starting in the evening of 18 July. However, it didn’t fail altogether.

But by far the bigger problem seems to be an update that appears to have been done in the late evening of July 18 for [IT security company] Crowdstrike’s Falcon product – a computer threat checker. Falcon works by having some “agent” software deeply embedded in the operating system of every PC, which monitors that computer and “calls home” if there’s a problem. It also receives updates on what to look out for if there’s a threat. It’s used a lot by large organisations throughout the world, which have a huge number of PCs to police.

I’m sure Crowdstrike are urgently investigating what happened. This piece of software is designed to protect people from ransomware attacks and the like. From the latest information I’ve seen, it looks like the update system file was somehow released in an incorrect format.

The Windows operating system gets to this update and it doesn’t know how to cope, so it crashes. That’s why people have been getting the “blue screen of death” [a computer screen with an error message indicating a system crash].

And the big problem is, you can’t fix this issue remotely. You have to go into every machine separately and put it into “safe” or “recovery” mode to isolate the software. From there, you should be able to reboot the machine and get it up and running again. But if you’re a big global company with a large distributed IT estate, that’s going to take a long time.

WHY HAS THIS OUTAGE HAD SUCH WIDE-RANGING EFFECTS?

Crowdstrike has been a great success – its security software is used by hundreds of thousands of major clients around the world. So airlines, airports, railways, hospitals, stock exchanges … they’re all going down.

It started in Australia when they got up for business on Friday. The update had clearly been sent out last night UK time, and it has just rippled around the world.

With deliberate ransomware attacks, they’ll typically take out one or two targets at a time. But in this case, it’s happened to thousands of organisations at once. We’ve not had anything like this before.

How Crowdstrike will fix the software is yet to be determined. As I’ve explained, it’s clear how companies can work around the issue. But for some very large organisations, this could affect their critical infrastructure and business for a long time yet – it’s going to take them days to physically work round all those machines.

CAN SECURITY COMPANIES ENSURE THIS DOESN’T HAPPEN AGAIN?

Security software is very intertwined with a computer’s operating system – it’s buried deep in there. There has to be a way that if something is found to be corrupted, it doesn’t just keep crashing the system – this may have to be done in cooperation with Microsoft, which owns the Windows operating system.

There’s got to be some way of backing out of it, and there is. However, most people trying to log into their blank PCs don’t know how to put their PCs into safe mode and revert to a previous state.

At the moment, it looks like it’s one corrupted file that’s producing a global problem. Computers download updates all the time, so how Microsoft prevents that from happening with this update, I don’t know. It’s not immediately obvious. And the million dollar question is: how did this corrupted file get released in the first place?

HOW LONG BEFORE THIS PROBLEM IS FULLY RESOLVED?

It’s certainly going to take days, if not weeks. It’s like those hospitals in London that got attacked with ransomware. They’re still suffering – there’s a very long tail on these things.

And in this case, it’s not just a long tail but a very broad swathe of global organisations in transport, health and everywhere else. I don’t think we’ve seen anything like this before.

Massive CrowdStrike Tech Outage Highlights Global Vulnerabilities

Companies and governments alike need to step up cybersecurity practices in the wake of massive technology failures associated with a CrowdStrike update.

The global information technology outageon July 19, 2024, that paralyzed organizations ranging from airlines to hospitals and even the delivery of uniforms for the Olympic Games represents a growing concern for cybersecurity professionals, businesses and governments.

The outage is emblematic of the way organizational networks, cloud computing services and the internet are interdependent, and the vulnerabilities this creates. In this case, a faulty automatic update to the widely used Falcon cybersecurity software from CrowdStrike caused PCs running Microsoft’s Windows operating system to crash. Unfortunately, many servers and PCs need to be fixed manually, and many of the affected organizations have thousands of them spread around the world.

For Microsoft, the problem was made worse because the company released an update to its Azure cloud computing platform at roughly the same time as the CrowdStrike update. Microsoft, CrowdStrike and other companies like Amazon have issued technical work-arounds for customers willing to take matters into their own hands. But for the vast majority of global users, especially companies, this isn’t going to be a quick fix.

Modern technology incidents, whether cyberattacks or technical problems, continue to paralyze the world in new and interesting ways. Massive incidents like the CrowdStrike update fault not only create chaos in the business world but disrupt global society itself. The economic losses resulting from such incidents – lost productivity, recovery, disruption to business and individual activities – are likely to be extremely high.

As a former cybersecurity professional and current security researcher, I believe that the world may finally be realizing that modern information-based society is based on a very fragile foundation.

THE BIGGER PICTURE

Interestingly, on June 11, 2024, a post on CrowdStrike’s own blog seemed to predict this very situation– the global computing ecosystem compromised by one vendor’s faulty technology – though they probably didn’t expect that their product would be the cause.

Software supply chains have long been a serious cybersecurity concern and potential single point of failure. Companies like CrowdStrike, Microsoft, Apple and others have direct, trusted access into organizations’ and individuals’ computers. As a result, people have to trust that the companies are not only secure themselves, but that the products and updates they push out are well-tested and robust before they’re applied to customers’ systems. The SolarWinds incident of 2019, which involved hacking the software supply chain, may well be considered a preview of today’s CrowdStrike incident.

CrowdStrike CEO George Kurtz said “this is not a security incident or cyberattack” and that “the issue has been identified, isolated and a fix has been deployed.” While perhaps true from CrowdStrike’s perspective – they were not hacked – it doesn’t mean the effects of this incident won’t create security problems for customers. It’s quite possible that in the short term, organizations may disable some of their internet security devices to try and get ahead of the problem, but in doing so they may have opened themselves up to criminals penetrating their networks.

It’s also likely that people will be targeted by various scams preying on user panic or ignorance regarding the issue. Overwhelmed users might either take offers of faux assistance that lead to identity theft, or throw away money on bogus solutions to this problem.

WHAT TO DO

Organizations and users will need to wait until a fix is available or try to recover on their own if they have the technical ability. After that, I believe there are several things to do and consider as the world recovers from this incident.

Companies will need to ensure that the products and services they use are trustworthy. This means doing due diligence on the vendors of such products for security and resilience. Large organizations typically test any product upgrades and updates before allowing them to be released to their internal users, but for some routine products like security tools, that may not happen.

Governments and companies alike will need to emphasize resilience in designing networks and systems. This means taking steps to avoid creating single points of failure in infrastructure, software and workflows that an adversary could target or a disaster could make worse. It also means knowing whether any of the products organizations depend on are themselves dependent on certain other products or infrastructures to function.

Organizations will need to renew their commitment to best practices in cybersecurity and general IT management. For example, having a robust backup system in place can make recovery from such incidents easier and minimize data loss. Ensuring appropriate policies, procedures, staffing and technical resources is essential.

Problems in the software supply chain like this make it difficult to follow the standard IT recommendation to always keep your systems patched and current. Unfortunately, the costs of not keeping systems regularly updated now have to be weighed against the risks of a situation like this happening again.

Conclusion

When I first looked into this subject, I have to be frank, I thought it was caused by malicious intent. After just a brief investigation of the matter I am confident thatit was nothing but poor coding. I have been involved with computers since the mid 1980s. What people fail to realize is that thse major software platforms are composed of millions of lnes of codes, and it only takes a few lines of code in critical places to bring down major systems. So it is not surprising from time to time mistakes will be made. Well let me tell you that this error was a real humdinger.

Resources

scientificamerican.com, “Worldwide Tech Outage Started with Defective Crowdstrike Update to Microsoft Windows.” By Allan Woodward & The Conversation US; scientificamerican.com, “Massive CrowdStrike Tech Outage Highlights Global Vulnerabilities.” By Richard Forino & The Conversation US;