MaSSSoftech

Due to the very nature of secure channel establishment, it is often difficult to even approach troubleshooting and debugging SSL related issues.

This brief article intends to illustrate the challenges, approaches and tools available for debugging these difficult scenarios.

SSL Description

Secure Socket Layer (SSL) is a protocol for providing a secure channel of communication between two computers. It makes provisions for data integrity, confidentiality and authentication. Authentication of the server - by the client - provides an assurance of the fact that the traffic has not been diverted to an attacking server. Mutual authentication requires the client to provide credentials to the server over the secure channel.

SSL Handshake Overview

In order to really be able to troubleshoot and debug SSL related issues, we need an understanding of what the protocol actually does on both the client and server sides. This understanding will enable us to quickly categorize the type of problem being encountered and hopefully a category of approaches for tracking down the root cause.


We will touch on issues and troubleshooting approaches in the following categories:

1. Certificate Validation
2. Trust
3. Configuration

So let's briefly describe the protocol with a bit of focus on these three categories.

The client initiates the SSL connection by requesting a channel through the use of a ClientHello handshake message. This message contains the Cipher Suites that are configured to be supported by the client side and are available for the server to choose in creating the most secure channel configuration possible between the two machines. It also contains a random number to be used by the server in the generation of keys - this random number is a result of the configured or default RNG on a given platform.

The server side, in turn, responds with a ServerHello that includes the Cipher Suite selected by the server as the most appropriately secure suite for the channel. If a suitable cipher suite could not be selected from the list of supported suites provided by the client - the request for an SSL connection is denied by the server. It also includes a random number and the certificate that is to be used for authenticating the server to the client. This certificate must be validated by the client in order for it to be trusted as representing the identity asserted by it.


This validation is based on a number of possible factors (driven by configuration):

1.Whether it is expired
2.Whether it has been revoked
3.Whether it was issued by a trusted Certificate Authority
4.Whether the server name within the certificate matches the host name for the current connection

Where Things Can Go Wrong

There are a number of common scenarios that occur as a result of improperly configured environments, clients, servers and certificates that can be categorized into one or more of the afore mentioned categories.

The following are a few descriptions of these scenarios and what the approach to identifying the root cause might be.

Keystores and Truststores
- Categories: Configuration, Trust, Certificate Validation

The client (for mutual authentication) and server each present the other a certificate that represents the identity of the machine its running on. In order for either to present this certificate - it must be available within the appropriate Keystore.

Tip 1: Determine the default certificate for a machine as appropriate for your server and ensure that it exists within the configured Keystore and is available to the process that needs to present it to the corresponding partner process.

Tip 2: Ensure that the issuer of the presented certificate exists within the appropriate Truststore of the recipient process.

Supported Cipher Suites
- Categories: Configuration

As described earlier, the handshake involves the selection of the most secure Cipher Suite by the server from the list of supported suites presented by the client.

If there isn't a common Cipher Suite between the client and server, then there is no way for the two machines to establish a secure channel - as there is no common language that will be understood buy each party that provides the necessary protection offered by SSL.

Tip 3: Ensure that the appropriate Cipher Suites are enabled on the client and server sides in order to establish this common language for secure message exchange.

Tip 4: Utilize SSL debug information to determine which cipher suites has been selected
...








<17849795 SSL3/TLS MAC>

<17849795 received HANDSHAKE>












































...


Tip 5: Utilize a tool such as SSLDump as necessary to see details of the handshake and application data message exchanges
...

11 1 0.0035 (0.0035) C>S SSLv2 compatible client hello

Version 3.1

cipher suites

TLS_RSA_WITH_RC4_128_MD5

SSL2_CK_RC4

TLS_RSA_WITH_RC4_128_SHA

TLS_DHE_DSS_WITH_RC4_128_SHA

TLS_ECDH_ECDSA_WITH_RC4_128_SHA

Unknown value 0x4e

Unknown value 0x2f

Unknown value 0x35

Unknown value 0x4b

Unknown value 0x4c

TLS_RSA_WITH_3DES_EDE_CBC_SHA

TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA

TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA

TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA

Unknown value 0x50

TLS_RSA_WITH_DES_CBC_SHA

TLS_DHE_DSS_WITH_DES_CBC_SHA

TLS_DHE_RSA_WITH_DES_CBC_SHA

TLS_ECDH_ECDSA_WITH_DES_CBC_SHA

Unknown value 0x4f

TLS_RSA_EXPORT1024_WITH_RC4_56_SHA

TLS_DHE_DSS_WITH_RC2_56_CBC_SHA

TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA

TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA

TLS_RSA_EXPORT_WITH_RC4_40_MD5

SSL2_CK_RC4_EXPORT40

TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_DH_anon_WITH_3DES_EDE_CBC_SHA

TLS_DH_anon_WITH_RC4_128_MD5

TLS_DH_anon_WITH_DES_CBC_SHA

TLS_DH_anon_EXPORT_WITH_RC4_40_MD5

TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_RSA_EXPORT_WITH_DES40_CBC_SHA

TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA

11 2 0.0053 (0.0017) S>C Handshake

ServerHello

Version 3.1

session_id[0]=


cipherSuite TLS_DH_anon_WITH_3DES_EDE_CBC_SHA

compressionMethod NULL

11 3 0.0053 (0.0000) S>C Handshake

ServerKeyExchange

Short read: 0 bytes available (expecting 2)

11 4 0.0065 (0.0012) S>C Handshake

ServerHelloDone

11 5 0.0976 (0.0910) C>S Handshake

ClientKeyExchange

DiffieHellmanClientPublicValue[128]=

8a 23 78 02 8a a5 fc 03 f4 9b 7c 33 05 22 36 91

85 9d 17 e4 bf bf 0a 3e be 45 25 47 07 e0 9c a2

e5 d6 bf 78 95 f1 84 ca cb cc e4 3e f3 d8 d4 9a

3a 01 71 5c 29 0c 0b f9 69 8d 3e a6 f4 08 f0 36

18 fd a7 b9 3e 30 4e a4 a6 19 d9 d3 64 1c 3c 78

d3 c3 fa 83 07 58 f2 be d2 32 80 c0 32 4e 49 4c

bf 73 1a f2 d8 fd f2 16 c7 31 da 48 58 50 bb 99

3f a4 8c 31 6e 5f ed e8 0d d8 91 cf 8f eb fa d8

11 6 0.0976 (0.0000) C>S ChangeCipherSpec

11 7 0.0976 (0.0000) C>S Handshake

11 8 0.0997 (0.0021) S>C ChangeCipherSpec

11 9 0.1000 (0.0002) S>C Handshake

11 10 0.3580 (0.2580) C>S application_data

11 11 0.3580 (0.0000) C>S application_data

11 12 0.3586 (0.0005) S>C application_data

11 13 2.5039 (2.1453) C>S application_data

11 14 2.5039 (0.0000) C>S application_data

11 15 2.5053 (0.0013) S>C application_data

8 20 31.4483 (3.3621) C>S application_data

8 21 31.4483 (0.0000) C>S application_data

8 22 31.4507 (0.0024) S>C application_data

8 23 31.4508 (0.0000) S>C application_data

8 24 32.0824 (0.6316) C>S application_data

8 25 32.0824 (0.0000) C>S application_data

8 26 32.2550 (0.1726) S>C application_data

8 27 32.2550 (0.0000) S>C application_data

8 28 33.1710 (0.9159) C>S application_data

8 29 33.1710 (0.0000) C>S application_data

8 30 33.1745 (0.0035) S>C application_data

8 31 33.1754 (0.0009) C>S application_data
...

Anonymous Cipher Suite
- Categories: Configuration

The failure of a client or server to reject a certificate that is not trusted may present as potential SSL problem. Recall earlier that I describe the process of selecting the most secure Cipher Suite common between both parties.

In a scenario where one of the parties has only the anonymous Cipher Suite enabled and the other party also has it enabled - even if it is one of many - the anonymous cipher suite will be selected and the connection will not be rejected.

Tip 6: see Tip 5 above - in fact, the example ssldump output above is from troubleshooting just such a scenario

Trusted CA's

- Categories: Trust, Configuration

Unless the issuer of a certificate is found in the Truststore of a client or server involved in the establishment of an SSL connection, the certificate validation will fail.

Tip 7: Determine the Truststore/s in use and whether or not the issuer of the presented certificate exists within the configured Truststore

Tip 8: Utilize keytool in order to dump the contents of the Truststores (or keystores for the presented certificates)

Alias name: ttelesecglobalrootclass3ca
Creation date: Feb 10, 2009
Entry type: trustedCertEntry

Owner: CN=T-TeleSec GlobalRoot Class 3, OU=T-Systems Trust Center, O=T-Systems Enterprise Services GmbH, C=DE
Issuer: CN=T-TeleSec GlobalRoot Class 3, OU=T-Systems Trust Center, O=T-Systems Enterprise Services GmbH, C=DE
Serial number: 1
Valid from: Wed Oct 01 03:29:56 PDT 2008 until: Sat Oct 01 16:59:59 PDT 2033
Certificate fingerprints:
MD5: CA:FB:40:A8:4E:39:92:8A:1D:FE:8E:2F:C4:27:EA:EF
SHA1: 55:A6:72:3E:CB:F2:EC:CD:C3:23:74:70:19:9D:2A:BE:11:E3:81:D1
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Crl_Sign
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: B5 03 F7 76 3B 61 82 6A 12 AA 18 53 EB 03 21 94 ...v;a.j...S..!.
0010: BF FE CE CA ....
]
]


Certificate Expiration
- Categories: Certificate Validation, Configuration

Tip 9: see Tip 8 above - In the example keytool output above you can see the dates for which the particular certificate is valid.

Valid from: Wed Oct 01 03:29:56 PDT 2008 until: Sat Oct 01 16:59:59 PDT 2033

Random Number Generation (RNG) Issue
- Categories: Configuration

Performance issues may be encountered due to low or zero entropy on a server. This entropy results in longer than expected blocking in acquiring the random number seeding from /dev/random. There are a couple potential workarounds.

1.use /dev/urandom - NOTE: this may result in degenerated encryption strength and must be investigated by your system/security administrators
2.patches may be available for your particular Linux flavor or Solaris

Tip 10: Observe through SSL debug output whether or not the handshake is timing out as this is an indicator that perhaps we are blocking on the RNG

Tip 11: Ensure that all related patches have been installed on your machine.

Available Tools and Facts

Like any other specialization, troubleshooting and debugging security - and SSL in particular - presents unique challenges and to address these unique challenges we need to be prepared by having appropriate tools and facts at our disposal.

In order to be productive in this area, we need to know certain things about the environment, management consoles, etc.


Debug output
Each middleware platform provides the ability to configure the server to run with SSL debug logging turned on. This configuration enables the viewer of the logs to see pertinent information regarding the configuration and runtime behavior of the handshaking and application data message exchanges in real time.

Tip 12: Determine what the configuration mechanism is for turning on SSL debug information on your platform. On WebLogic Server the following System Properties are used to configure SSL debug information and can be used on the command line or within start up scripts:

-Dssl.debug=false -Dweblogic.StdoutDebugEnabled=true -Dweblogic.security.SSL.verbose=true

Tip 13: When you have access to starting the server with these System properties do so immediately - the information that it creates will be valuable - if not to you then to someone else that is pulled in to help debug - at which time you will be request to do so anyway and go through the whole thing again.

SSLDump

In cases where we don't have access to the server to restart with SSL debug logging or when we would like to supplement that output with additional information SSLDump is hugely valuable.

ssldump is an SSLv3/TLS network protocol analyzer. It identifies TCP connections on the chosen network interface and attempts to interpret them as SSLv3/TLS traffic. When it identifies SSLv3/TLS traffic, it decodes the records and displays them in a textual form to stdout. If provided with the appropriate keying material, it will also decrypt the connections and display the application data traffic.

Tip 14: Download a copy of ssldump from http://www.rtfm.com/ssldump/. You will need to build it on your platform and you may actually need to resolve a couple compilation errors - but it is well worth it.

Examples:

To listen to traffic on port 443:
ssldump -i eth0 port 443

To listen to traffic to the server target on port 443:
ssldump -i eth0 port 443 and host target

To decrypt traffic to the host target server.pem and the password foobar:
ssldump -i eth0 -Ad -k ~/server.pem -p foobar host target

Generic SSL Client

Once you have ssldump built and running on your machine, you can use any SSL client to target the server that you are trying to troubleshoot. Often a browser will suffice - however you may need to build a client more appropriate for your usecase.

Tip 15: Utilize ssldump, SSL debug logging and your SSL client to observe the messages exchanged and the runtime behavior that manifests as a result of your current configuration.

Platform Specific Knowledge

Become intimately familiar with where the appropriate keystores, truststores, configuration files and management consoles are located.

Tip 16: Maintain a checklist of this information and keep it handy so that you don't have to rediscover it every time you encounter SSL issues.

Books

SSL and TLS: Designing and Building Secure Systems, Addison-Wesley, 2001 ISBN 0-201-61598-3

http://www.rtfm.com/sslbook/