Weekly Tech Tidbits – Kill the Zombies

During our May Tech Talk meetings one of my summertime recommended tasks was for everyone to find and eliminate “zombie” servers.   Zombie servers are servers which theoretically are retired, but yet that somehow still exist, turned on, in your network.

Since these servers were supposed to be retired, they often have outdated, unsupported operating systems.   Best case they are probably unpatched.   These servers are the ideal place for malware to take root and launch an aggressive attack against your network - from the inside.

This summer they need to be removed from your network so they are no longer a threat.

But they are already retired, so why are they here?

• They were left up after they were retired, “just to make sure” that all the data and programs were working fine. It got busy and people forgot it was even there.
• The BIOS in the physical, retired servers is set to Last State On for Power.
• VMware is set to auto-start the retired VM.
• The servers were never removed from inventory and/or disk in VCenter.

So how should you proceed?

• Unplug the power and NICs and (ideally) remove the physical servers from your racks.
• Examine the network looking for servers that are up that should retired. Remove the servers.
• Go into VCenter and shutdown the servers.
• If the server must hang around, make sure it is not on any auto-startup lists
• Remove them from VCenter inventory.
• If the servers are gone forever, remove them from disk.

However, on Thursday night I was faced with an entirely different type of network Zombie – Retired services that are still running.  I happened to be on a mail server at 10:45pm at night.   It was at 100% CPU.  The obvious answer is “backup is causing high CPU at night”.    I took a peek at task manager.   Sure enough backup was taking 40-50% CPU.  However, an inventory service from a client management server that had been retired was running consuming 40-60% of CPU!   To further complicate the issue the inventory service program was not listed in add/remove programs!   This was truly a rogue service.

My take away from this is:

• In these days of bitcoin mining malware, take high CPU conditions seriously. Paladin Sentinel monitoring will graph CPU usage over time.
• Investigate what services are running on the server.
• Investigate unknown services and unknown software.
• Remember retiring client side management tools is required when retiring the management system.

If you need help rooting out the zombies in your network, give us a call.