Weekly Tech Tidbit – Stupid MDF/IDF Stuff That Will Come Back to Bite You Part II

February 1st, 2019
Weekly Tech Tidbit – Stupid MDF/IDF Stuff That Will Come Back to Bite You Part II

In part I we were talking about UPSes.  In part II we are going to continue our MDF/IDF UPS theme.

I want you to physically audit the power situation in your MDF and IDF closets.   Specifically, I want you to look for the following:

  • UPS load.   If the UPS is overloaded or loaded almost to capacity, you can have a power event and instead of protecting you, the UPS collapses because it cannot support the load.   If you see overloaded UPSes, balance the power load out across UPSes.  Calibration keeps the UPS honest as to what it can and can't do and monitoring keeps you in the loop for when things are heading in the wrong direction,
  • Number and size of UPSes.   If you don't have enough capacity to better balance the load, then it is time to either add, expand or upgrade the UPSes in your closet.   Lisa can help you with figuring out what capacity you require to get things back to a good place.
  • Where do the power plugs go from your equipment?  Ideally, servers, SANs, ESX hosts, and switches with redundant power supplies should have each power supply plugged into its own UPS, That way you can survive a single UPS failure/replacement and not have to bring your equipment completely down.
  • Is there anything plugged into the wall?  If so why?  Does the wall have surge protection?  This is one of the nastier situations in a closet.  You or a vendor ran out of plugs in a UPS in a closet.  We actually had a district which awarded a switch upgrade contract to another vendor.  The vendor came in and found the district had insufficient outlets for the new switches.  The vendor simply went out and bought power strips and plugged the new equipment into the power strips and left - without even telling the customer!  We all found out when closets were mysteriously collapsing and we all found out together the reason why.
  • There are things that might be okay to plug into a wall such as monitors which take a lot of power, but won't cause harm if plugged in the wall.   In some districts where money is tight the redundant power supply may in fact be plugged directly into the wall.  That makes for a lot of monitoring noise for little power abberations that take out the redundant power supply, but if the primary power supply is in a UPS, it is the probably best you can do until you have more capacity.  However, unless you know that the outlets in the wall have built-in surge protection, nothing should be plugged directly into the wall.  It should be plugged into a surge protector power strip and then plugged in the wall.
  • Now, look at the UPS.  Do you have power strips plugged into the back of the outlets because you lack adequate power outlets to plug everything in despite having adequate UPS capacity to protect it all?  If you do, are those power strips surge strips or just random power strips?    The reality is that if you must plug in a strip into your UPS outlets, it *MUST* be a dumb, non-surge protected power strip!  The issue with surge protector strips is that an APC UPS can get confused by true load coming from the surge protected strip.  That can give you false confidence that you have enough capacity.  This is one case where cheaper, dumber is actually better.   APC makes a very nice power strip called a PDU that does some other cool things like allow you to control power by outlet remotely.  Talk to Lisa if you want to explore that.  At the end of the day we want:
    • All redundant power supplies plugged into separate UPSes
    • All items plugged into the wall to be plugged into surge strips
    • All power strips plugged into UPSes be non-surge strips.
    • Adequate capacity to support the load
    • UPS NICs in every UPS to actually have the UPSes tell us what is going on out there.
    • Active UPS monitoring such as our Paladin Sentinel Monitoring so we can easily see and act upon UPS alerts.

In part III we will continue our discussions as to what your plan b and plan c power strategy should be.