Be Careful what you Scan for!

Published: 2014-04-24
Last Updated: 2014-04-24 00:31:08 UTC
by Rob VandenBrink (Version: 1)
10 comment(s)

After some fun and games at one customer site in particular, I found that the SSL services on the earlier versions of the HP Proiliant Servers iLo ports (iL01 and iLO2) are not susceptible to heartbleed.

However, their implementation of SSL is fragile enough that scanning them for the Heartbleed vulnerability will render them inoperable.  This affects Proliants from G1 all the way up to G6, as well as many of the HP Bladesystems. 

A complete power down of the entire system - as in remove both of the AC cables - is required to reset the iLo card and bring it back to life.  While this may seem  like a quick fix for a single server, if that server is running a Hypervisor, or if it's a bladesystem with Hypervisors running on the blades, this can multiply to be a huge issue.  Especially if your client scanned the server subnet, and effectively bricked all their iLO cards before they realized there was a problem (oops)

(And yes, the fact that we worked this out Easter weekend is somewhat ironic.)

Full details are in HP Support Document c04249852

This illustrates that even when scanning for simple things (with NMAP, Nessus or any other scanning tool really), it's best to scan a few test systems first - or if you have a test VLAN that replicates your production systems, even better!   This isn't a problem with the scanners, almost always the problem is the fragility of the service being scanned.  Many services are only written to deal with "the right" inputs, which is not how most scanners (or most attackers) tend to operate.

Safe Scanning Everyone!

========================
Rob VandenBrink
Metafore

Keywords:
10 comment(s)

Comments

Is this for all heartbleed scans or is it unique to how a specific scanning tool does the query?
This is not specific to any tool. Any tool that checks for the heartbleed vulnerability will be using SSL features that these iLO versions are not written to handle well, with unfortunate results.
In my testing, Tenable resolved this issue around 4-15 in the Nessus plugin (73412). The original test simply looked for TLS and then attempted the exploit. The updated query evaluates the STARTTLS services, checks for supported extensions and enabled heartbeats, and then attempts the exploit. This no longer crashes the HP iLO service in our environment, but results may differ.

The HP iLO apparently doesn't respond well to unexpected queries against it's SSL stack - yet another reason why this type of device should be segmented off your network and certainly not internet accessible.
I recall that some BMCs can be reset from the host via a I2C loadable driver module, at least under Linux. Might be worth looking into as this would be far less painful that rebooting production servers. My guess is Windows Server systems are a lost cause.
When you scan any Juniper Netscreen firewall it also takes it down. It impacts all Netscreen devices and I do not believe it is patched yet either.
Rob, I think you should change your name to VandenBRICK ;-)
On the "will every scanner do this" and the "Nessus does not" conversation - both are correct.

Nessus has several decision points along the path when checking for Heartbleed. It checks if the host is up, is the target port open, is SSL running, are heartbeats supported, THEN it checks for Heartbleed.
If the host in question doesn't run SSL or doesn't support the heartbeat extension, then the actual check is never run, because Nessus has determined that it is not vulnerable.
So - correct, Nessus does not crash these identified iLO hosts
but, the actual Heartbleed check is never executed, which is why Nessus does not.

+1 to Tenable for writing one of the more intelligent scanners in the industry. Many of their defaults are set with an eye towards "how can we NOT crash the target". However, once you start scanning "for real", you'll find that you need to change many of these defaults to do the best job. So knowing what you are scanning, scheduling potential service interruption windows and so on is a really good practice, no matter what tool you are scanning with.
I was going to mention this earlier, but how many OOB services have we scanned? I just did all of Supermicro's IPMI/BMC versions through the X10 and found for once that they weren't vulnerable to something.

Anybody try Dell's iDrac?
I guess this kind of thing is not new for HP. In speaking with some of the other security folks here they informed me that they're already working with our server folks based on this article:

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c02629048-2%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&sp4ts.oid=3239482&ac.admitted=1397514098285.876444892.492883150

Seems old iLO just has a lousy networking stack and vulnerability scanners trigger bad things.
It's also worth noting that while iLO doesn't have the Heartbleed vulnerability, certain versions of the HP Insight Management Web Page DO. +1 on Nessus handling this very well.

Diary Archives