TLS/SSL Handshake: RSA, Ephemeral DH, and TLS 1.3

What the handshake does #

The TLS handshake is the process that starts a secure session. Client and server exchange messages to:

1. Agree on a TLS version
2. Agree on a cipher suite
3. Authenticate the server via its certificate and the issuing CA’s signature
4. Establish a shared session key for symmetric encryption

It runs after the TCP handshake completes.

Universal principle: every TLS handshake uses asymmetric crypto to authenticate the server and bootstrap the session, then switches to symmetric encryption (the session key) for the actual data. What differs between methods is whether the private key is used to decrypt a transmitted secret or merely to sign the handshake — and that distinction is what determines forward secrecy.

1. RSA key exchange (legacy, pre-TLS 1.3, now considered insecure) #

1. Client Hello — client sends supported TLS versions, cipher suites, and a random string (“client random”).
2. Server Hello — server replies with its certificate, chosen cipher suite, and a “server random”.
3. Authentication — client verifies the certificate against the issuing CA.
4. Premaster secret — client generates a random “premaster secret,” encrypts it with the server’s public key (from the cert), and sends it. Only the server’s private key can decrypt it.
5. Decrypt — server uses its private key to recover the premaster secret.
6. Session key — both sides derive the same session key from client random + server random + premaster secret.
7. Finished — each side sends a message encrypted with the session key.
8. Secure symmetric encryption begins.

Key trait: the secret is transmitted (encrypted with the public key). Weakness: if the server’s private key is later compromised, an attacker who recorded the encrypted premaster secret can decrypt it and recover every session key ever protected by that key — there is no forward secrecy.

2. Ephemeral Diffie-Hellman (DHE) handshake — the secure pre-1.3 alternative #

1. Client Hello — version, client random, cipher suite list.
2. Server Hello — certificate, chosen cipher suite, server random.
3. Server signs the handshake — the server signs the handshake messages (including its DH parameters) with its long-term private key, proving it owns that key.
4. Signature verified — client checks the signature, confirming server identity.
5. Client DH parameters — client sends its own DH parameters.
6. Both sides independently compute the same premaster secret by combining the exchanged DH parameters — the secret itself is never transmitted.
7. Session key — derived from premaster secret + client random + server random, same as RSA.
8. Finished (both sides) → secure symmetric encryption.

Key difference vs. RSA: the private key is used to sign/authenticate, not to decrypt a transmitted secret. Because the shared secret is derived fresh per session and the ephemeral DH values are discarded afterward, compromising the long-term private key later does not expose past sessions — this is forward secrecy.

3. TLS 1.3 handshake (current) #

TLS 1.3 removes RSA key exchange and static/insecure cipher suites entirely, making ephemeral (Diffie-Hellman/ECDHE) key exchange mandatory, and shortens the handshake to 1 round trip (down from 2 in TLS 1.2):

1. Client Hello — version, client random, a short cipher suite list, and its key-exchange parameters (key share) sent speculatively up front, guessing the server’s preferred method.
2. Server computes the session secret immediately — it already has the client’s random, params, and cipher choice, plus its own server random.
3. Server Hello + “Finished” — server sends its certificate, signature, server random, chosen cipher suite, and (since it already has the secret) its Finished message too.
4. Client final step — client verifies the signature and certificate, computes the session secret, sends its own Finished.
5. Secure symmetric encryption begins.

0-RTT session resumption: if client and server have connected before, they derive a “resumption secret” and the server issues a session ticket. On a later connection, the client can send encrypted application data immediately alongside that ticket — zero round trips to resume.

Caveat: 0-RTT data is replayable. Because the first-flight data isn’t tied to a fresh handshake, an attacker who captures it can resend it, and a naive server may process the same request twice. TLS 1.3 does not protect against this on its own — applications using 0-RTT must ensure requests sent in that window are idempotent, or avoid using 0-RTT for non-idempotent operations.

Quick comparison #

Takeaway: all TLS handshakes use asymmetric crypto to authenticate and bootstrap, then symmetric encryption for the data itself. RSA transmits the secret (no forward secrecy); DH and TLS 1.3 derive it (forward secrecy). TLS 1.3 removes RSA, trims cipher choices, shortens the handshake to 1-RTT, and adds 0-RTT resumption — at the cost of a replay risk that applications must handle themselves.

Sources:

• RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3
• RFC 7919 — Negotiated Finite Field Diffie-Hellman Ephemeral Parameters for TLS
• TLS 1.3 0-RTT replay risk discussion