HAProxy SSL Termination and Pass Through
technial security network haproxy
Published: 2021-05-20

Correct way of configuring HAProxy to terminate SSL and passthrough, at the same time.

Summary

Use HAProxy to:

  • Provide TLS offloading (decrypt)
  • Route TLS traffic to server (without decrypt)

Motivation

I have two services requiring TLS for security.

  • Gitea, a private Git server, configured to be HTTP only.
  • Nextcloud, due to its complex Nginx conf, HTTPS is provided by Nginx.

Therefore, HAProxy:

  • Can provide fast and unified interface for enabling HTTPS
    • Much less management overhead
    • Obtain/Renew multiple Let’s Encrypt cert at once
    • Unified TLS Cipher config
  • Allow reuse of precious 443/TCP
    • Route TLS traffic based on SNI (See below)

Basic Concepts

SSL Termination

Simply speaking,

  • SSL Termination = Decrypt TLS Traffic at HAProxy, optionally re-encrypt to backend. Requires a valid cert.
  • TLS passthrough = Pass the TLS traffic as-is, no decryption. Traffic routing is based on Server Name Indication (SNI).

For more info, read HAProxy doc.

Reference

For TLS passthrough, some config is from https://scriptthe.net/2015/02/08/pass-through-ssl-with-haproxy/

TLS offloading

ssl passthrough sequence diagram

  1. A Frontend listens on 0.0.0.0:443
  2. For TLS traffic matching SNI, e.g. c.2isk.in
  3. Pass the traffic to corresponding endpoint, e.g. 127.0.0.1:443

Config

frontend 443
  bind 162.250.188.59:443 tfo
  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend nginx if { req.ssl_sni -i c.2isk.in }
  default_backend ssl_termination

backend nginx
  mode tcp
  server nginx_ 127.0.0.1:443

SSL Termination

This is the problematic part.

ssl termination sequence diagram

  1. A Frontend listens on 0.0.0.0:443
  2. For TLS traffic matching SNI git.2isk.in
  3. Pass the traffic to backend ssl_termination
  4. Backend ssl_termination passes traffic to server haproxy_bounce_back, server address = 127.0.0.1:4431
  5. A Frontend listens on 127.0.0.1:4431, which terminates TLS to HTTP
  6. Frontend pass HTTP traffic to corresponding backend and then server

Config

Append to the config in the section above.

frontend ssl_termination
  bind 127.0.0.1:4431 ssl crt /etc/haproxy/tracking.2isk.in.pem crt /etc/haproxy/git.2isk.in.pem strict-sni alpn h2
  mode http
  #reqadd X-Forwarded-Proto:\ https
  option http-server-close
  option forwardfor
  use_backend tracking if { ssl_fc_sni -i tracking.2isk.in }
  use_backend gitea if { ssl_fc_sni -i git.2isk.in }

backend ssl_termination
  mode tcp
  server haproxy_bounce_back 127.0.0.1:4431

backend nginx
  mode tcp
  server nginx_ 127.0.0.1:443

Problem

tl;dr Change mode http under section defaults to mode tcp, if SSL passthrough is needed.

This HAProxy is fresh and latest stable installation from its official repo.

  • Failed to connect ssl passthrough services.

The following cmd is used to test, -servername sets SNI for the TLS request.

$ openssl s_client -connect c.2isk.in:443 -servername c.2isk.in

CONNECTED(00000003)
140014773765504:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 311 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Troubleshooting

  • First, to confirm the traffic is routed from “frontend 443” to “nginx”,

    • Changed SNI for the request
    • Such that the traffic should now be “frontend 443” to default backend “ssl_termination”.
$ openssl s_client -connect c.2isk.in:443 -servername c2.2isk.in

CONNECTED(00000003)
139822070588800:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1543:SSL alert number 70
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 312 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

The error is now different, which can be confirmed in the HAProxy log.

  • After crossing out access control rules, check the HAProxy

    May 20 09:55:12 s haproxy[9563]: xxx.xxx.xxx.xxx:41728 [20/May/2021:09:55:12.438] 443 nginx/<NOSRV> -1/-1/0 196 PR 1/1/0/0/3 0/0
    • For nginx/<NOSRV>, nginx shows the backend is correct (nginx)
    • <NOSRV> caused many confusions
      • There is only one server for the backend, with no ACL and health check
      • Even if health check is enabled later on, and shown “up” in stats page
    • PR is the error flag made no sense

From HAProxy config manual, the meaning of error flag:

TCP and HTTP logs provide a session termination indicator in the
"termination_state" field, just before the number of active connections. It is
2-characters long in TCP mode, and is extended to 4 characters in HTTP mode,
each of which has a special meaning

What does PR means:

PR   The proxy blocked the client's HTTP request, either because of an
     invalid HTTP syntax, in which case it returned an HTTP 400 error to
     the client, or because a deny filter matched, in which case it
     returned an HTTP 403 error.

  • To this point, I still cannot found the cause. I reviewed config file again and again.

    • To create even more confusions, the “relevant” frontend, backend and server is same as the working conf of HAProxy on old host.

  • The The proxy blocked the client's HTTP request really caught my attention

    • It should be pass through mode, all TCP packets should be passed as-if
    • How there is any “HTTP request”, especially it is encrypted

  • After confirming frontend/backend/server is quite correct, I inspect the default section, which is unchanged after installation

    • I change mode http to mode tcp
    • But I thought the mode is already defined in all frontend and backend section
    • -> mode in default is useless

Result

Changing default mode works.

To confirm,

$ openssl s_client -connect c.2isk.in:443 -servername c.2isk.in

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = c.2isk.in
verify return:1

Nice.

Lesson learned: Never overlook default setting.

Bonus

  • TLSv1.3 only (it is 2021, all my devices supported, no one else will use anyway)
  • Remove AES-256 cipher, 128 bit is enough for me
  • Certbot integration, certbot certonly --standalone --preferred-challenges http-01 --http-01-port 8745 -d c.2isk.in
  • Gzip offload for HTTP backend
  • Add HSTS header
  • HAProxy stats page on 10.192.12.1:5654
  • External port supports TCP Fast Open
  • Redirect 80/TCP HTTP to 443/TCP HTTPS

Full Config

defaults
     #log    global
     no log
     mode    tcp

global
  log /dev/log    local0
  log /dev/log    local1 notice
  chroot /var/lib/haproxy
  maxconn 65536
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

  ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

defaults
  #log    global
  no log
  mode    tcp
  option  httplog
  option  dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend stats
  bind 10.192.12.1:5654 tfo
  mode http
  stats enable
  stats uri /stats
  stats refresh 10s
  stats admin if TRUE
  stats auth user:pass

frontend 443
  bind 162.250.188.59:443 tfo
  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend nginx if { req.ssl_sni -i c.2isk.in }
  default_backend ssl_termination

frontend ssl_termination
  bind 127.0.0.1:4431 ssl crt /etc/haproxy/tracking.2isk.in.pem crt /etc/haproxy/git.2isk.in.pem strict-sni alpn h2
  mode http
  #reqadd X-Forwarded-Proto:\ https
  option http-server-close
  option forwardfor
  use_backend tracking if { ssl_fc_sni -i tracking.2isk.in }
  use_backend gitea if { ssl_fc_sni -i git.2isk.in }

frontend 80
  bind 162.250.188.59:80 tfo
  option http-server-close
  option forwardfor

  acl is_well_known_cert path_beg -i /.well-known/acme-challenge

  redirect scheme https code 301 if !is_well_known_cert !{ ssl_fc }

  use_backend certbot if is_well_known_cert

backend ssl_termination
  mode tcp
  server haproxy_bounce_back 127.0.0.1:4431

backend tracking
  mode http
  option httpchk
  server tracking 127.0.0.1:8000 check

backend gitea
  mode http
  #option httpchk GET /
  #server gitea 127.0.0.1:3000 check
  http-response add-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
  compression algo gzip
  server gitea 127.0.0.1:3000

backend certbot
  mode tcp
  server certbot 127.0.0.1:8745

backend nginx
  mode tcp
  server nginx_ 127.0.0.1:443

Contact

Anything wrong?

Email me: adam \a\ 2isk.in, replace \a\ with @