Adventures in HAProxy

Table of Contents

For those of you that do not know, HAProxy is an amazing piece of kit that can proxy HTTP and arbitrary TCP connections. It’s also so customizable that I’m practically using it as my main entry point to my network, and do indeed refer to it internally as the “border gateway.” With two exceptions (SSH and SMTP, more on that later), everything that comes into the TD-StorageBay network, yes, that includes this site too, passes through that one process. However, HA cannot do everything… and I feel that I’m pushing its limits. Not in the “sheer workload” sense, no, I am way far off from that. I mean in the old Mythbusters style “using things in ways for which they were never intended” manner.

First problem: SSH

For those unaware, SSH (Secure SHell) is a method of gaining a shell (login and console) on a remote server. And yes, it’s secure. Who would have guessed? Two TDSB servers have external-facing SSH access: the primary domain (GitLab) and Gen (literally everything else except mail and IRC). Not a gripe with HAProxy exactly, but it’s relevant: SSH has no SNI or Host mechanism like TLS or HTTP does. You connect to a port and begin an immediate exchange, there is no “I’m looking for gen.tdstoragebay.com, not tdstoragebay.com” in that. This can easily be solved by mapping one to a different port instead of the default (port 22), that won’t cause too much user hassle. That’s actually not the issue.

The issue here is that, along with any other remote login service, it’s a rather high value target for hackers to find valid SSH credentials and gain access to a system. As such, SSH services are very prone to attack. This would be okay, except… well I can’t log it. If SSH passes through HAProxy, then suddenly HAProxy’s IP is all over the logs. And with the advent of systems like fail2ban, I’ll just ban my proxy, which isn’t really useful.

Now some of you are probably about to point out the PROXY protocol. Yes, I know it exists. And nope, it doesn’t work. sshd, the SSH server, does not support PROXY. Here’s the start of an example connection:

C: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
S: SSH-2.0-OpenSSH_7.4
<key exchange follows>
...

And here is what happens when I try to send a PROXY string using telnet:

$ telnet 192.168.5.20 22
Trying 192.168.5.20...
Connected to 192.168.5.20.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.4
PROXY TCP4 45.89.234.81 192.168.5.20 45310 22
Protocol mismatch.
Connection closed by foreign host.

No dice, cue the sad trombone. sshd will not accept the PROXY header, and… as far as I can tell, it’s really not a priority feature.

Workaround

Simple: I already have NAT in front of HA. I’ll just DNAT the two SSH ports to bypass it entirely, problem solved.

This also clears up a few things, since HA wasn’t really meant for long-running TCP connections, since it only logs at the end of a session… unless you specified option logasap, but that makes my log parsing a pain, and takes off a fair amount of useful information (like byte counts).

Additionally: Mail

SMTP uses a ton of seucurity features to combat spam, almost all of which need the client IP. And just like SSH, Postfix (the server I use) doesn’t support the PROXY protocol. And as far as I know at this point, there is no workaround for that one. I’m sending SMTP straight over with NAT, and there’s nothing else that can be done there, meaning I have almost no logging about raw SMTP connection statistics other than what Postfix itselt emits during the transaction.

Second Problem: Conditional Response Headers

HA is set up to add a LOT of response headers on any connection. HSTS if secure, and a few other famous ones: Content-Security-Policy, X-XSS-Protection, X-Frame-Options, Expect-CT, NEL, there’s quite a few. And well… sometimes I don’t want them there. Prime example: mail server.

The mail server

mail.tdstoragebay.com is special… RoundCube (the webmailer) displays the contents of your emails in an iframe, which works until you block iframes. It uses inline scripts and styles, which work, until the CSP rejects them. And because how HAProxy handles requests, I do not have access to request information in the response. This means that I cannot use an ACL with req.hdr(Host) -m beg mail for my http-response set-header because the request is not available at this processing step. I can’t (easily, to the best of my knowledge) just filter on backend servers, so I made a workaround… add a new header.

In my Apache config, there’s a line that looks something like this:

Header set always Lax-Mail-Security true

This means that any response from the mail server will include Lax-Mail-Security: true in it. If HAProxy sees this header, then it will change it’s various security headers, and just delete that one.

A very weird way of doing things if you ask me. Why do I need to edit each server’s configuration to add an identification header when… well, host already exists, unless I’m supposed to just echo that on the backend?

Third Problem: Dealing with Multiple Domains

Note that when I say multiple domains, I do not mean subdomains. That’s rather easy to accomplish. No, I’m talking about two completely different domains…. like tdstoragebay.com and teknikaldomain.me going through the same proxy.

Header Troubles

So first off… tdstoragebay.com has a lot of security headers… I mean just look at this:

HTTP/2 302
date: Sat, 14 Sep 2019 22:28:53 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
location: https://tdstoragebay.com/users/sign_in
x-request-id: Ltyfta2FZN9
x-runtime: 0.022560
x-ua-compatible: IE=edge
vary: Accept-Encoding
x-varnish: 729557
age: 0
via: 1.1 varnish (Varnish/6.2)
content-length: 104
content-security-policy: default-src https: 'self' 'unsafe-inline'; img-src * data:; script-src 'self' https: 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri https://tdstoragebay.report-uri.com/r/d/csp/enforce
server: TD-StorageBay Border Ingress
report-to: {group:default,max_age:31536000,endpoints:[{url:https://tdstoragebay.report-uri.com/a/d/g}],include_subdomains:true}
nel: {report_to:default,max_age:31536000,include_subdomains:true}
expect-ct: enforce, max-age=86400, report-uri=https://tdstoragebay.report-uri.com/r/d/ct/enforce
strict-transport-security: max-age=31536000;includeSubdomains;preload
referrer-policy: strict-origin-when-cross-origin
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: sameorigin
x-permitted-cross-domain-policies: none
x-xss-protection: 1; mode=block; report=https://tdstoragebay.report-uri.com/r/d/xss/enforce
x-robots-tag: noindex, nofollow

Everything from content-security-policy on down is for security, maybe I’ll make a post at a later date detailing what each of those does. For now though, it’s sufficient to know that they’re here, and they are specific to tdstoragebay.com and not any other domain name!

Just as an example, at the time of writing, when vising teknikaldomain.me (right here!), the content-security-policy header prevents you from loading all the required fonts and icons, strict-transport-security is advertising preload, server shows TDSB, though that’s not actually a security header (nor is it important that it technically has the wrong name, it’s pretty much ignored), the nel and report-to headers, for reporting errors and CSP violations, are for reporting tdstoragebay.com infractions which muddy my reports, and half the time are just plain useless.

Not to mention, since other domains actually use Cloudflare for more than just DNS, CF’s headers and systems start to clash with mine, and it’s a mess.

So as seen above, with conditional response headers, the problem becomes how to actually modify headers when the primary way of identifying what the headers go to cannot be seen. Well, there’s a few solutions.

Solution 1: Internal Headers

And this was my solution for the above section: send a header that’s checked by the proxy, used as a conditional, and then deleted before it reaches the client. This works well, but this also means I’d need to edit every TDSB server to send a header, or I’d need to edit every non-TDSB server to send a header, and use that as the conditional. While yes, this would definitely work, for one, I cannot (easily) add custom headers to GitLab output. The counterpart, make every non-TDSB server send a header is easy: I have one server, and since this particular one is actually running NGINX, not Apache, that’s easy to add. The problem is that it’s not easily able to be scaled. Every new server is going to need to implement this header. Because of this… not my solution.

Solution 2: Further Upstream

HAProxy isn’t the only step in request routing, there’s also a local Varnish cache that everything passes through. Yes, it is configured to bypass it if it goes down (though if Varnish is down, that usually means we have other issues.) I could simply use a lot of set resp.http.<header> = <value>; in my top level VCL (More on modular VCL in another post, maybe), and take HAProxy out of this entirely. The issue is that any change requires reloading and swapping VCL modules, which, while not hard, isn’t as simple as changing one single line and systemctl reload haproxy. So while entirely possible, I didn’t go with this only because it’s just more complexity. Plus, some headers are best set by HAProxy which actually knows details about the incoming client request, so then I’d have two things setting headers, and…. yeah no, I’ll pass.

Solution 3: Varnish to HAProxy Internal Header.

This is probably the most hacky of them all. You see, Varnish is only used for tdstoragebay.com, other domains use Cloudflare for most of the work, let’s me focus most of my configuration and effort on my main website. This also means that caching on my end is technically not required, and so Varnish is only used for tdstoragebay.com. What this means is that in my top level VCL I can add one line… say set resp.http.tdsb = "1"; or something, some header that will never occur naturally (value is unimportant), and test for its existence with HAProxy and then set headers there. This is the most flexible solution, since it works by default for any other domain, and to get the headers to work, I just need to make sure requests are routed to the varnish backend in HA, and Varnish itself is configured to accept them. There is, however, potentially one better solution.

Solution 4: Set Headers in Backend

Let’s look at two chunks of my HAProxy configuration:

listen web
    description Singular HTTP entrypoint for all domains
    mode http
    timeout tunnel 60m
    acl mail-backend res.hdr(Lax-Mail-Security) -m found
    acl cache-down nbsrv(varnish) lt 1
    acl has-host req.hdr(Host) -m sub tdstoragebay.com
    acl acme path_beg /.well-known/acme-challenge
    acl vcs req.hdr(Host) -m beg cvs svn darcs hg bk mtn bzr
    option http-use-htx
    option forwardfor
    bind :80
    bind :443 alpn h2,http/1.1 ssl crt /etc/haproxy/wildcard.pem crt /etc/haproxy/tdstoragebay.pem crt /etc/haproxy/mattermost.pem crt /etc/haproxy/gen.pem crt /etc/haproxy/mail.pem crt /etc/haproxy/teknikaldomain.pem ssl-min-ver TLSv1.1 ssl-max-ver TLSv1.3
    bind :9000 alpn http/1.1 ssl crt /etc/haproxy/wildcard.pem ssl-min-ver TLSv1.1 ssl-max-ver TLSv1.3
    bind :5050 alpn http/1.1 ssl crt /etc/haproxy/tdstoragebay.pem ssl-min-ver TLSv1.1 ssl-max-ver TLSv1.3
    http-request redirect scheme https code 301 if !{ ssl_fc } !acme
    http-request redirect location https://plantuml.tdstoragebay.com/plantuml/ code 302 if { req.hdr(Host) -m beg plantuml } !{ path -m beg /plantuml/ }
    http-request redirect location https://cvs.tdstoragebay.com/viewvc code 302 if { req.hdr(Host) -m beg cvs } { path / }
    http-request redirect location https://svn.tdstoragebay.com/viewvc code 302 if { req.hdr(Host) -m beg svn } { path / }
    http-request redirect location https://darcs.tdstoragebay.com/darcs code 302 if { req.hdr(Host) -m beg darcs } { path / }
    http-request redirect location https://hg.tdstoragebay.com/hg code 302 if { req.hdr(Host) -m beg hg } { path / }
    http-request redirect location https://monitoring.tdstoreagebay.com:9000/ code 302 if !{ dst_port 9000 } { req.hdr(Host) -m sub monitoring.tdstoragebay.com }
    http-request tarpit if { req.hdr(User-Agent) -m sub SemrushBot AhrefsBot Googlebot bingbot Baiduspider YandexBot AwarioRssBot }
    http-request deny if { path -m beg /server-status }
    http-request deny deny_status 405 if { method TRACE CONNECT }
    default_backend varnish
    use_backend graylog   if { req.hdr(Host) -m sub monitoring.tdstoragebay.com }
    use_backend myblog    if { req.hdr(Host) -m sub teknikaldomain.me }
    use_backend lab       if { req.hdr(Host) -m sub mattermost.tdstoragebay.com } or { dst_port 5050 }
    use_backend local     if acme
    use_backend vcs-web   if vcs { req.hdr(User-Agent) -m sub mercurial darcs bzr SVN BitKeeper }
    use_backend lab       if cache-down !has-host
    use_backend lab       if cache-down { req.hdr(Host) -m beg tdstoragebay.com }
    use_backend mail-web  if cache-down has-host { req.hdr(Host) -m sub mail.tdstoragebay.com }
    use_backend vcs-web   if cache-down has-host vcs
    capture request  header Host len 32
    capture request  header User-Agent len 64
    capture response header X-Varnish len 32
    capture response header Age len 16

    http-response add-header Access-Control-Allow-Origin * if { res.hdr(Web-Key-Directory) -m found }

    http-response set-header Content-Security-Policy default-src\ https:\ \'self\'\ \'unsafe-inline\';\ img-src\ *\ data:;\ script-src\ \'self\'\ https:\ \'unsafe-inline\'\ \'unsafe-eval\';\ upgrade-insecure-requests;\ report-uri\ https://tdstoragebay.report-uri.com/r/d/csp/enforce
    http-response set-header Server TD-StorageBay\ Border\ Ingress
    http-response set-header Report-To {"group":"default","max_age":31536000,"endpoints":[{"url":"https://tdstoragebay.report-uri.com/a/d/g"}],"include_subdomains":true}
    http-response set-header NEL {"report_to":"default","max_age":31536000,"include_subdomains":true}
    http-response set-header Expect-CT enforce,\ max-age=86400,\ report-uri="https://tdstoragebay.report-uri.com/r/d/ct/enforce"
    http-response set-header Strict-Transport-Security max-age=31536000;includeSubdomains;preload if { ssl_fc }
    http-response set-header Referrer-Policy strict-origin-when-cross-origin
    http-response set-header X-Content-Type-Options nosniff
    http-response set-header X-Download-Options noopen
    http-response set-header X-Frame-Options sameorigin
    http-response set-header X-Permitted-Cross-Domain-Policies none
    http-response set-header X-XSS-Protection 1;\ mode=block;\ report=https://tdstoragebay.report-uri.com/r/d/xss/enforce
    http-response set-header X-Robots-Tag noindex,\ nofollow

    http-response del-header Pragma
    http-response del-header X-Powered-By
    http-response del-header Lax-Mail-Security if mail-backend

backend varnish
    description Web cache servers
    mode http
    option httpchk
    server      local 127.0.0.1:6081 check send-proxy-v2 proto h2

You may not be able to understand most of it, but I’ll explain one thing:

Towards the bottom of the first block there’s a few http-response blocks… one for set-header, one for add-header, and one for del-header. Using set and add will add headers (difference isn’t important here), and del removes them. As you can see, most headers here are unconditional: there is no if or unless on those lines. And because this is the frontend declaration, this means that all requests, period, get these tacked on.

Now here’s a thing about HAProxy: I could just as easily add these header modifications in the backend section, meaning only requests that pass through that backend will have the headers added in their response. By copying all the lines to the varnish block, I’ve essentially replicated the same results as solution 3, but with 0 configuration changes to Varnish. Hence why I’ll be using this solution going forward.