Tech Tidbit: When the “Bat Phone” goes off on a weekend, it is never a good thing

January 15th, 2022
Tech Tidbit: When the “Bat Phone” goes off on a weekend, it is never a good thing


"I think someone shot a torpedo at us"

-The Hunt for Red October

It started out as an ordinary weekend full of kid's schedules and errands. I was over at the Boy Scout Camp in the woods picking up my son from the annual Klondike event.

The bat phone goes off.

One of our Security Analyst teams believed a district was under active attack.

The security analyst commentary was quite specific:

Windows Defender says it blocked the events, but they believed that Windows Defender had been bypassed. They listed a specific account they believed was compromised and specific servers that were compromised. Defensively they put a number of those servers into network isolation to freeze the attack in place while we all figured out what was going on.

The good news is that the "attack" turned out to be an unannounced Penetration Test.

However, the analysis of what was going on was spot on.

The defense was real-time while I was off doing other things.

In my discussions with the security analysts during this event I learned a few things:

1 - Windows Defender actually saw the launch of this attack sequence. I again heard the commentary that as a basic antivirus product, Windows Defender is actually very good.

2 - I was also told that in this case the fact that there was a companion product installed next to Windows Defender actually created some unexpected synergy that enhanced what Windows Defender was able to see. Also, the analyst commented that it was interesting that Windows Defender was seeing some things that the more advanced companion product didn't appear to see.

This "live trial" and subsequent commentary from those far smarter than me re-enforced a few points that I have been saying for quite a while:

1 - Windows Defender should be installed and known to be working on every server and workstation - at least as a second opinion tool. It is free. It plays nice with anything else you might have installed. However, your present security stack came to be, please make a point to survey your network and make sure that Windows Defender is installed everywhere it can be installed - regardless of what else you are doing.

2 - Reading your antivirus logs is important. However, we know that most of us don't read all those logs like we should. We are all overwhelmed, and it becomes white noise. Someone has to be looking at those logs. You simply need to figure that out.

3 - Having meaningful alerts of actionable events is very important. Just this morning I was on a different district's server. Cisco AMP which is a highly rated EDR product with a number of advanced protection capabilities was installed. Plain as day there was a big message saying that this endpoint was completely unprotected until that server was rebooted. I know Cisco AMP automatically has that specific alert on the console. I know an email alert can be setup to push that alert in front of the appropriate eyes for remediation. Allowing gaps whatever your protection you have decided to deploy is a dangerous game. "All" is a hard thing to implement, but the stakes here of being wrong are very, very high. You need to be vigilant. People tend to get out of sorts on re-imaging where they "forget" the last step and reinstall the endpoint clients that provide your protection.

4 - Invest your money not in basic antivirus but more advanced companion tools that provide additional layers of protection that create improved synergy an enhance the effectiveness of your overall security stack. Different products and layers look at things from different perspectives. While nothing absolutely sees everything, hopefully one of the layers sees the bad thing and either blocks it or at the very least pulls the fire alarm to alert us that bad things are going on.

It is like peanut butter and chocolate. Better together.

5 - And once again, as scary as it is to have the bat phone ring on the weekend, there is tremendous piece of mind in having a full Security Operations Center and/or live Security Analysts playing centerfield looking at and correlating all those alerts and, if necessary, intervening pro-actively while I am off doing other things.

Knowing that they always doing this and are also available real-time to have a discussion 24x7x365 is quite frankly awesome.

"What's in your security stack?"

If you don't have those layers, I encourage you to start adding them as time and budget allow. If you want someone watching playing centerfield and your endpoints with this level of vigilance, give us a call.

-Scott Quimby