It has been a rough two weeks for many of us who support technology for our school districts.
We have experienced the worldwide meltdown of CrowdStrike due to a faulty update. That event caused a worldwide meltdown of many Microsoft online resources, shutting down many banks, airlines, hospitals, and even traffic lights worldwide.
This felt like what we were told Y2K was supposed to be like.
Worldwide losses from the CrowdStrike/Microsoft event are estimated to be 15 billion dollars. Delta alone estimated that the event cost them $500 million.
Many of you were directly or indirectly caught in that event.
Then, we had a few other serious Microsoft and VMware events back-to-back-to-back!
The pace of the meltdowns and security issues to remediate has been insane.
There has been a lot of finger-pointing as to who was at fault for the CrowdStrike update issue. Some articles suggested this was not the first instance of major issues in recent months. However, this was the first "you are dead" incident.
I am not going to get into the food fight that is now going on in the press and with lawyers over who is at fault.
I just want to talk about the philosophy of how updates are done.
Many know we predominantly use SentinelOne as part of our CSEDR service.
We have received several questions as to whether what happened to CrowdStrike could happen to SentinelOne.
The short answer is no. It cannot happen with SentinelOne because they operate under a different philosophy.
If you have been around as long as I have, you know that Windows was the Wild West back in the day, where anyone could reach into C:\WINDOWS and do almost anything they wanted, at times that caused severe stability issues.
To Microsoft's credit, they took their stability issues seriously and, in future versions of Windows, started forcing separation between the operating system and software applications. That resulted in "protected files" and the core Windows processes being segmented into a Kernel section and a User section. The kernel is the "holy of holies" of Windows. The user comes after (hopefully) a stable base operating system is loaded.
Microsoft has some pretty stringent rules about applications getting access at the Kernel level. CrowdStrike interacts with Windows at that level. To be fair, Microsoft has a documented way to operate at that level, and CrowdStrike says they followed the rules. However, in light of the carnage, Microsoft has announced that they are re-evaluating everyone's access privileges at the Kernel level.
CrowdStrike has said that because they interact at this lower level of Windows, they are dramatically faster and can see and react to threats faster than other products.
SentinelOne and many other products follow a less aggressive user-level interaction. They wait for Microsoft to present a stable OS, and then they do what they do. This difference in philosophy means that, for the most part, a failed SentinelOne update would mean that the agent failed to load properly on a single machine and shouldn't take the entire system offline.
Furthermore, in the CSI CSEDR service the SentinelOne application version updates are pushed out intentionally by our upstream SOC partner and are not pushed out in an automated, unsupervised way.
The MITRE ATT&CK organization periodically invites CrowdStrike, SentinelOne, and other major EDR products to test their offerings against "real-world" attack scenarios. Both products score well in those tests. SentinelOne has consistently scored highest in those tests with the fewest delays and changes.
While speed is admirable, I do believe core stability and the ultimate results are the most important evaluation criteria.
You can read SentinelOne's official response to the CrowdStrike incident here.
Regardless of what EDR product you have implemented in your district, I believe the single most important investment you can make in enhancing your network security defenses is adding an "eyes on glass" Security Operations Center (SOC) with 24x7x365 security oversight to your EDR endpoints and, ultimately, your entire network.
If you'd like to outsource most of the work maintaining your EDR clients and have security oversight from a SOC watching your EDR endpoints, please reach out to Lisa to discuss our EDR, MDR, and XDR service offerings.
-Scott Quimby, CISSP
You must be logged in to post a comment.