When Good CA's go Bad: Other Things to Check in Your Datacenter

When Good CA's go Bad: Other Things to Check in Your Datacenter
The recent problems at DigiNotar (and now GlobalSign) has gotten a lot of folks thinking about what happens when significant events impact our trust of Public Certificate Authorities, and how it affects users of secured services. But aside from the browsers at the desktop, what is affected and what should we look at in our infrastructure? What has been brought up by several of our readers, as well as a lively discussion on several of the SANS email lists, is SSL proxy servers, and any other IDS / IPS device that does "SSL proxy" encryption. If you are not familiar with this concept, in a general way these products work as shown in the diagram: As you can see from the diagram, the person at the client workstation "sees" an HTTPS browser session with a target webserver. However, the client's HTTPS session is actually with the proxy box (using the client's trust in the Private CA to cut a dynamic cert), and the HTTPS session with the target web server is actually between the proxy and the web server, not involving the client at all. In many cases, the private CA for this resides directly on the proxy hardware, allowing certs to be issued very quickly as the clients browse. In any case, the issue that we're seeing is that these units are often not patched as rapidly as servers and desktops, so many of these boxes remain blissfully unaware of all the issues with DigiNotar. If you have an SSL proxy server (or an IDS / IPS unit that handles SSL in this way), it's a good idea to check the trusted CA list on your server, and also check for any recent patches or updates from the vendor. It's probably a good time to do some certificate "housekeeping" - - look at all devices that use public, private or self-signed certificates. Off the top of my head, I'd look at any web or mail servers you might have with certificates, load balancers you have in your web farm that might front-ending any HTTPS web servers, any FTP servers or SSH servers that might use public certificates, or any SSL VPN appliances. What should you look for? Make sure that you're using valid private or public certificates - not self-signed certificates for anything (this is especially common for admin interfaces for datacenter infrastructure). It often makes sense at a time like this to see if it makes sense in your organization to get all your certs with one vendor, in one contract on a common renewal date to simplify the renewals and ensure that nothing gets missed, resulting in an expired cert facing your clients. Or it may make sense to see if it's time to consider an EV (extended validation) cert on some servers, or downgrading an existing EV cert to a standard one. (Look for more on CA nuts and bolts in an upcoming diary). Check renewal dates to ensure that you have them all noted properly. If you've standardized on 3 year certificates, has one of your admins slipped a 1 year cert in by accident (we see this all the time, often a 1 year cert is less than the corporate PO limit, and a 2 or 3 year cert is over). What else should you check? What other devices in the datacenter can you think of that needs to trust a public CA? Mail servers come to mind, but I'm sure that there are others in the mix - please use our comment form to let us know what we've missed. =============== Rob VandenBrink Metafore	Rob VandenBrink 579 Posts ISC Handler Sep 8th 2011
Thread locked Subscribe	Sep 8th 2011 1 decade ago
Thanks... it's great knowledge for me.	Anonymous
Quote	Sep 8th 2011 1 decade ago
Certificates can and often are revoked. Make sure your checking the published CRL's (Certificate Revocation Lists) or have OCSP configured (where supported) for all of the Certificate Authorities that you trust.	Home account 1 Posts
Quote	Sep 8th 2011 1 decade ago
I've written a couple of programs that if you cut them with an SSL Proxy and private CA, it says something along the lines of "what the heck is this certificate" and refuses to work. In a similar vein, I've been considering recommending to Mozilla that the browser needs to know internally the cert for addons.mozilla.org and wherever it gets updates from as an internal cert and reject all others.	Home account 39 Posts
Quote	Sep 8th 2011 1 decade ago
It seems to me that the most important systems to address are those that are 'clients' in the SSL conversation (as an example - users browsers). If you are hosting an SSL web server (or SSL VPN) you don't need to be worried unless one of the maliciously issued certs was for one of your domains. Am I right?	Home account 4 Posts
Quote	Sep 8th 2011 1 decade ago
You should check CRL's and OCSP (as "Home account" writes), however those mechanisms are fundamentally broken. Apart from many other issues, the current mechanism is that the revocation URL's are included in the affected certificate itself (e.g. not in the parent's/issuer certificate). The "Comodo hacker" could probably easily have changed those revocation URL's in the falsified certificates to a server of his liking. Hence this mechanism is no longer guaranteed when CA's get owned, even after they detect the compromise and supply a safe and clean revocation server. With respect to "what else to check": the "Comodo hacker" falsified major CA root certificates and Microsoft certificates. Furthermore he demonstrated his ability to sign files (he signed with a falsified google certificate and published it, see http://pastebin.com/u/ComodoHacker). As the "Comodo hacker" claims to have reverse engineered the Microsoft Windows Update protocols, I would not be surprised if his intentions were to push root CA lists, signed with a falsified Microsoft cert, to Windows PC's (and servers). If Windows does not include hard-coded checks before accepting a new root CA list (which I simply don't know), this may already have happened. He may also be able to push updates of the Windows crypto libraries that have such hard-coded checks removed. Note that we don't know how good the "Comodo hacker" has protected the private keys he generated for the falsified certificates. They may have been stolen by cybercriminals (or he may have simply sold or shared them). I personally believe this matter is seriously underestimated by the security community.	Erik van Straten 129 Posts
Quote	Sep 9th 2011 1 decade ago
Sorry, halfway my comment above: "... he signed with ..." should read "... he signed calc.exe with ..."	Erik van Straten 129 Posts
Quote	Sep 9th 2011 1 decade ago

The recent problems at DigiNotar (and now GlobalSign) has gotten a lot of folks thinking about what happens when significant events impact our trust of Public Certificate Authorities, and how it affects users of secured services. But aside from the browsers at the desktop, what is affected and what should we look at in our infrastructure?

What has been brought up by several of our readers, as well as a lively discussion on several of the SANS email lists, is SSL proxy servers, and any other IDS / IPS device that does "SSL proxy" encryption. If you are not familiar with this concept, in a general way these products work as shown in the diagram:

As you can see from the diagram, the person at the client workstation "sees" an HTTPS browser session with a target webserver. However, the client's HTTPS session is actually with the proxy box (using the client's trust in the Private CA to cut a dynamic cert), and the HTTPS session with the target web server is actually between the proxy and the web server, not involving the client at all.
In many cases, the private CA for this resides directly on the proxy hardware, allowing certs to be issued very quickly as the clients browse.

In any case, the issue that we're seeing is that these units are often not patched as rapidly as servers and desktops, so many of these boxes remain blissfully unaware of all the issues with DigiNotar. If you have an SSL proxy server (or an IDS / IPS unit that handles SSL in this way), it's a good idea to check the trusted CA list on your server, and also check for any recent patches or updates from the vendor.

It's probably a good time to do some certificate "housekeeping" - - look at all devices that use public, private or self-signed certificates. Off the top of my head, I'd look at any web or mail servers you might have with certificates, load balancers you have in your web farm that might front-ending any HTTPS web servers, any FTP servers or SSH servers that might use public certificates, or any SSL VPN appliances. What should you look for? Make sure that you're using valid private or public certificates - not self-signed certificates for anything (this is especially common for admin interfaces for datacenter infrastructure). It often makes sense at a time like this to see if it makes sense in your organization to get all your certs with one vendor, in one contract on a common renewal date to simplify the renewals and ensure that nothing gets missed, resulting in an expired cert facing your clients. Or it may make sense to see if it's time to consider an EV (extended validation) cert on some servers, or downgrading an existing EV cert to a standard one. (Look for more on CA nuts and bolts in an upcoming diary). Check renewal dates to ensure that you have them all noted properly. If you've standardized on 3 year certificates, has one of your admins slipped a 1 year cert in by accident (we see this all the time, often a 1 year cert is less than the corporate PO limit, and a 2 or 3 year cert is over).

What else should you check? What other devices in the datacenter can you think of that needs to trust a public CA? Mail servers come to mind, but I'm sure that there are others in the mix - please use our comment form to let us know what we've missed.

===============
Rob VandenBrink
Metafore

Rob VandenBrink

579 Posts
ISC Handler

Sep 8th 2011

Thanks... it's great knowledge for me.

Anonymous

Certificates can and often are revoked. Make sure your checking the published CRL's (Certificate Revocation Lists) or have OCSP configured (where supported) for all of the Certificate Authorities that you trust.

Home account

1 Posts

I've written a couple of programs that if you cut them with an SSL Proxy and private CA, it says something along the lines of "what the heck is this certificate" and refuses to work.

In a similar vein, I've been considering recommending to Mozilla that the browser needs to know internally the cert for addons.mozilla.org and wherever it gets updates from as an internal cert and reject all others.

Home account
39 Posts

It seems to me that the most important systems to address are those that are 'clients' in the SSL conversation (as an example - users browsers). If you are hosting an SSL web server (or SSL VPN) you don't need to be worried unless one of the maliciously issued certs was for one of your domains. Am I right?

Home account
4 Posts

You should check CRL's and OCSP (as "Home account" writes), however those mechanisms are fundamentally broken.

Apart from many other issues, the current mechanism is that the revocation URL's are included in the affected certificate itself (e.g. not in the parent's/issuer certificate). The "Comodo hacker" could probably easily have changed those revocation URL's in the falsified certificates to a server of his liking. Hence this mechanism is no longer guaranteed when CA's get owned, even after they detect the compromise and supply a safe and clean revocation server.

With respect to "what else to check": the "Comodo hacker" falsified major CA root certificates and Microsoft certificates. Furthermore he demonstrated his ability to sign *files* (he signed with a falsified google certificate and published it, see http://pastebin.com/u/ComodoHacker).

As the "Comodo hacker" claims to have reverse engineered the Microsoft Windows Update protocols, I would not be surprised if his intentions were to push root CA lists, signed with a falsified Microsoft cert, to Windows PC's (and servers). If Windows does not include hard-coded checks before accepting a new root CA list (which I simply don't know), this may already have happened. He may also be able to push updates of the Windows crypto libraries that have such hard-coded checks removed.

Note that we don't know how good the "Comodo hacker" has protected the private keys he generated for the falsified certificates. They may have been stolen by cybercriminals (or he may have simply sold or shared them).

I personally believe this matter is seriously underestimated by the security community.

Erik van Straten

129 Posts

Sorry, halfway my comment above: "... he signed with ..." should read "... he signed calc.exe with ..."

Erik van Straten

129 Posts