Secure custom domains with Caddy

If you want to run a SaaS app where your customers can each have one of their own domains pointing to your app, then you will encounter two problems:

1. Getting those custom domain names to resolve to your app
2. Serving the content for those domains over SSL

For example, let's say you wanted to provide a status page service hosted at yourapp.com that allows customers to set up their own domain (like status.acmecorp.com) to be served by your app. You will need to have your customer update their domain name records (DNS settings) to have status.acmecorp.com resolve to yourapp.com (problem #1), and then you'll need to have an SSL certificate you can use to convince the browser that you are allowed to serve https://status.acmecorp.com (problem #2). Let's dive into the solutions.

Configuring customer domain names

The first step is getting your customer's domain name to point to your domain. Your customer will need to create a CNAME record for their domain (status.acmecorp.com) in their DNS settings. The value of that record should be the domain where you will be hosting Caddy to serve the customer's content. In our case, we decided to pick a separate domain from our main domain: status.hbuptime.com. We went with a separate domain to keep the Caddy configuration simple. All it needs to do is serve status pages — it doesn't need to have any other configuration for our app that lives at app.honeybadger.io. Using a separate domain also allows us to scale status page traffic separately from our main app.

As we'll cover in a minute, to serve content over SSL for status.acmecorp.com, you will need to generate an SSL certificate for that domain name. We'll be doing that as an automated process, which means it will also be subject to abuse and potential denial of service attacks. What if someone decided to try to overwhelm your server by making it generate certificates for a.example.com, b.example.com, c.example.com, and so on? How would you know which certificates you should generate and which ones you shouldn't? We solved this problem by requiring the customer to verify their CNAME before we attempt to serve traffic for it.

In our application, we ask the customer to tell us what domain they will be using for their status page. We store that domain name in our database, and we try to resolve the domain name to check whether it points back to our domain when the customer triggers the verify action:

class StatusPage < ApplicationRecord
  CNAME_DOMAIN = "status.hbuptime.com"

  def valid_cname?
    return false unless domain.present?
    Resolv::DNS.new.getresources(domain, Resolv::DNS::Resource::IN::CNAME).first&.name&.to_s == StatusPage::CNAME_DOMAIN
  end
end

class StatusPagesController < ApplicationController
  def verify
    if @status_page.valid_cname?
      flash[:notice] = "Your custom domain has been verified."
      @status_page.update(domain_verified_at: Time.now)
      Thread.new { @status_page.fetch } # Go ahead and pre-fetch the domain to get Caddy to generate the SSL certificate
    else
      flash[:alert] = "Your custom domain could not be verified. Please check your DNS settings and try again."
    end

    redirect_back fallback_location: @status_page
  end
end

Configuring Caddy to generate SSL certificates on demand

The Caddy web server has an awesome feature that uses Let's Encrypt to generate certificates automatically for you. It does this by requesting a certificate for a given domain, responding to Let's Encrypt's request back to the domain to verify that it is serving the content for that domain (as we verified in the DNS setup discussed above), and then storing the SSL certificate that Let's Encrypt generates. Here's what that looks like in a Caddyfile:

https:// {
  # Generate SSL certificates when the first request for customerdomain.com arrives
  tls letsencrypt@example.com {
    on_demand
  }
}

This configuration tells Caddy to listen at port 443 for SSL requests for any domain and generate an SSL certificate as needed, using an email address you have previously registered with Let's Encrypt. While this works, it doesn't prevent the abuse potential mentioned previously. Let's address that with some additional configuration for how to determine when to generate a certificate:

{
  on_demand_tls {
    ask http://web.internal:5000/confirm_domain
    interval 1m
    burst 10
  }
}

Adding this configuration to your Caddy file will cause Caddy to first make a request to http://web.internal:5000/confirm_domain before generating a certificate. If that URL returns a successful (200) response, the certificate will be requested. If not, it won't. The other settings control how often that backend request will be made (to help avoid rapid, repeated requests to generate certificates).

In our case, we have a Rails app that responds to these requests from Caddy, and it looks something like this:

class ConfirmController < ApplicationController
  def confirm
    # If the record isn't found, a 404 is returned, which causes Caddy to refuse to generate the certificate
    StatusPage.where.not(domain_verified_at: nil).find_by!(domain: params[:domain])
    head :ok
  end
end

Once the customer domain is verified, Caddy will get the SSL certificate. How do we serve the right content for requests to status.acmecorp.com? Let's go back to our Caddyfile and add a proxy configuration:

https:// {
  tls letsencrypt@example.com {
    on_demand
  }

  reverse_proxy web.internal:5000 {
    header_up X-Via-Caddy {host}
  }
}

The reverse_proxy block will pass all requests to the backend while adding a header with the requested customer domain. We can use that in our Rails app to set up our routes and to serve the right content in our controller:

Honeybadger::Application.routes.draw do
  constraints ->(req) { req.headers["X-Via-Caddy"].present? } do
    root to: "public_status_pages#show", as: :public_status_page_show
  end
end

class PublicStatusPagesController < ApplicationController
  def show
    @status_page = StatusPage.where.not(domain_verified_at: nil).find_by!(domain: request.headers["X-Via-Caddy"])
  end
end

Bonus: Making it scale

Caddy will store the certificate it gets back from Let's Encrypt to avoid repeatedly requesting that the same certificate be generated. The default location for storing the certificates is the file system, which doesn't work well if multiple Caddy instances serve this traffic. We use the Redis storage module for the certificate content so that all Caddy instances can access the same certificate cache, rather than each Caddy instance having to request a certificate. Here's how you configure that in the Caddyfile:

{
  # Avoid DoS attacks by confirming with a backend app that a requested domain should have an on-demand certificate generated
  on_demand_tls {
    ask http://web.internal:5000/confirm_domain
    interval 1m
    burst 10
  }

  # Store generated SSL certificates in redis instead of local file system so all app instances have access to them
  storage redis {
    db "1"
    host "redis.internal"
  }
}

I used "1" for the db value because we are already using database 0 (the default Redis database) for other stuff, and I didn't want the keys to get mixed together.

The only snag when using Redis for certificate storage is that the standard Caddy binary doesn't include that module. You'll need to compile your own Caddy yourself. Fortunately, doing so is pretty easy. Here's the Dockerfile we use for that:

FROM ubuntu:latest

RUN apt-get -y update
RUN apt-get -y install wget

RUN wget -q -O /usr/bin/caddy "https://caddyserver.com/api/download?os=linux&arch=amd64&p=github.com%2Fpberkel%2Fcaddy-storage-redis&idempotency=$(date '+%s')" && chmod a+x /usr/bin/caddy
RUN mkdir -p /etc/caddy
COPY Caddyfile /etc/caddy/Caddyfile

CMD ["/usr/bin/caddy", "run", "--config", "/etc/caddy/Caddyfile"]

And here is the full Caddyfile that is referenced in that Dockerfile:

{
  # Caddy is running behind an application load balancer hosted at AWS, so this configures Caddy to trust the headers set by it
  servers {
    trusted_proxies static private_ranges
  }

  # Avoid DoS attacks by confirming with a backend app that a requested domain should have an on-demand certificate generated
  on_demand_tls {
    ask http://web.internal:5000/confirm_domain
    interval 1m
    burst 10
  }

  # Store generated SSL certificates in redis instead of local file system so all app instances have access to them
  storage redis {
    db "1"
    host "redis.internal"
  }
}

https:// {
  # Generate SSL certificates when the first request for customerdomain.com arrives
  tls letsencrypt@example.com {
    on_demand
  }

  # A request to customerdomain.com will get forwarded to the backend with the hostname passed as a header, so the backend knows what content to serve
  reverse_proxy web.internal:5000 {
    header_up X-Via-Caddy {host}
  }
}

# Load balancer health check
:4431 {
  # Use the rails app's health check as our health check, and rewrite any requests we get to the health check route
  handle_path /* {
    rewrite * /pages/health
    reverse_proxy web.internal:5000
  }
}

Check out Honeybadger for a live example

There you have it — a complete configuration for Caddy to serve customer domains securely from your application. This approach has served us well in serving our customers' status pages from their domains. If you want to see it in action, you can create a Honeybadger account and set up your own status page with a custom domain.