Weekly Tech Tidbit – It Happened Again. The Dark Side of Third Party Vendor Equipment On Your Network

March 1st, 2019
Weekly Tech Tidbit – It Happened Again.  The Dark Side of Third Party Vendor Equipment On Your Network

We constantly harp on all of you segregating third-party vendor equipment on to vendor VLANs.   The purpose of that is to keep third-party vendor equipment completely away from your internal network.  Ideally, we don't want these vendors to see, touch, or ask your internal network anything.   You contract with them to do whatever they do (i.e. security, HVAC, refrigeration, etc.).  We put them in their lane and hopefully, they do good things for the district.

One of our major concerns is do these vendors understand security, or patching, or routine maintenance?   They may be really smart about HVAC, but they may be completely ignorant about maintaining their own computer equipment.  If their equipment goes south, does the district get compromised with it?

The famous Target breach happened through a third-party vendor that they contracted with.  That vendor had bad things happen to them and then Target allowed their equipment to see the inside of the Target network and the breach happened.

The other incident I like to cite is the college that was attacked and shut down by a denial of service attack because of an infected soda machine with internet access on their network.  In the end, 5,000 IoT devices were taken over on that college's network because they all used default passwords.  The college had to visit each device manually to reset it and regain control!

That is why Bob is so emphatic on this topic.  You being alive is pretty important to us.

We recently had the following situation.  We are doing our routine 6 am Paladin Sentinel Monitoring sweep to help everyone start their day and focus on what needs attention.   A simple alert came in - drive c: low on disk space.    Now, this was the "last call" alert meaning Windows really thought you were scraping the bottom of the barrel and bad things were on the horizon if not addressed.

I sent out my routine morning note to the appropriate people reminding them that this was out there.  It was the holiday week.  No one answered.  A day or two goes by and this is still out there.   I re-send the note widening the recipients.  I get a response.  That the district is short staffed due to the holidays.  I ask if we should intervene to prevent bad things from happening and am told yes.

Christina gets the ticket.  It turns out to be a very important third-party vendor server running very critical functions for the district.   Christina quickly realizes that this server is a real live SQL server.   Furthermore, she finds out that the vendor merely did the default D:\SETUP and installed everything to drive C:!  The vendor has no clue or no special skills in SQL database administration.  They just use SQL to do what they do.

Christina realizes that the "issue" is improper clean up of SQL transaction logs.  She calls the vendor for assistance with their server.  The vendor actually suggested she delete the logs (OMG)!  She asks for the main passwords for SQL so she can properly clean up and configure SQL.  The vendor doesn't know the passwords for the system that they set up!   Christina then gets an "oh by the way", if this runs out of space it will be really bad and really expensive to fix!  All of this was caused by not configuring the server properly upon installation.  Plus this server is sitting on the inside of the network!

The sad reality is that this isn't an isolated incident.  This vendor isn't necessarily dumber or more irresponsible than others.  Once in a while, we see people who are really smart (Infinite Campus comes to mind).   But the sad reality is that you have to look at all of this skeptically and make sure you know what is and isn't going on with these vendors devices on your network.

Here are a few things to think about:

  • Keep these people in their own VLAN lane so they can't hurt the rest of you if they do something stupid beyond taking their service off-line.
  • Make sure all these devices are not using default passwords so you can protect against "soda machine" style IoT attacks.
  • Critically evaluate security changes requested by vendors.  Please ask the impact of these changes.  We actually had a security camera vendor ask the district to gut the DMZ security because they didn't actually know how to configure their own equipment properly!
  • If vendors are putting servers on your network, please understand how these servers are being protected (i.e. anti-virus or advanced endpoint protection (AEP)).  How are they getting updated such as Windows updates?  What is actually being installed (i.e. SQL or advanced software)?   Do you need to do anything special like specialized backup agents?  Who knows server and software management passwords so the system can be maintained?

If you need help sorting this out, please give us a call.  It is vitally important you have this correct.