SSL certificate chains

I learned something interesting today when an intermediate certificate in our certificate chain expired over the weekend.  Our certificate chain looked like this:

Trusted Certificate Authority (CA) Cert
----> 2nd Level Certificate
--------> 3rd Level Certificate
------------> Machine/Application Certificate

It was the "2nd Level Certificate" that expired for us.  When that happened, our ArcSight console application would no longer allow us to log into the ArcSight ESM.  What was puzzling was that the ArcSight web interface continued to work just fine, as did other web applications who had that certificate in their chains.  We spent about a day working on this and eventually figured out why only the ArcSight Console was failing and that the fix turned out to be relatively simple.

When the browser initiates a secure connection to a server, it requests the application's certificate and walks the certificate chain until it finds a trusted CA certificate.  In this corporate environment, the 2nd level certificate had been revoked and a replacement added as a root certificate.  In addition the "3rd Level Certificate" was also listed as a trusted root certificate.  So as the browser walked up the certificate chain it did the following:

1. Examine "Machine/Application Certificate" -> this certificate is not trusted. Must examine the issuing authority for this certificate, the 3rd level cert. (Side note, adding the application certificate as an accepted root cert is what you do when you use a self-signed certificate.)
2. Examine "3rd Level Certificate" -> this is a trusted root certificate.
3. Certificate is trusted and connection is created.

Note that the expired "2nd Level Certificate" is never examined here.  However, the ArcSight Console application only had the "Trusted CA Cert" at the top level in its cacerts file.  So its flow looked like this:

1. Examine "Machine/Application Certificate" -> this certificate is not trusted.  Must examine the issuing authority for this certificate, the 3rd level cert.
2. Examine "3rd Level Certificate" -> this certificate is not trusted.  Must examine the issuing authority for this certificate, the 2nd level cert.
3. Examine "2nd Level Certificate" -> this certificate is not trusted and is EXPIRED.  Cease walking the certification chain.
4. A trusted connection can't be made.

We added the new "2nd Level Certificate" and the "3rd Level Certificate" to the ArcSight console's cacerts trust store and the problem was solved.  We could have added only the "3rd Level Certificate" and we would have fixed the problem.  However, the ArcSight keytoolgui application will object if you attempt to add a cert to the cacerts store and the certificate for its issuing authority is not present.  (It just throws a warning but will still let you override if you want to, this is how you install a self-signed cert.)  The reason the new "2nd Level Certificate" was installed was to avoid having to override that message when updating the cacerts store.

Are there any security implications to this?  It seems like we're creating a bypass around an expired certificate.

Why does a certificate authority (in this case 2nd level) issue certificates that expire after its own?  That seems like bad practice.