Adding config to AWS ECS tasks

When deploying Docker containers to AWS ECS, you can encounter a situation where you want to run an image that requires some configuration. For example, let's say you wanted to run Vector¹ as a sidecar to your main application so you can ship your application's metrics to a service like Honeybadger Insights. To run Vector, you only need to provide one configuration file (/etc/vector/vector.yaml) to the image available on Docker Hub. However, creating your own image that just adds one file would be a hassle. It would be easier if you could pull the public image, add your config, and deploy that. But ECS doesn't allow you to mount a file when running the container like you can when running Docker on your laptop or a VM. There is a way to do it on ECS, though — let's check it out.

Services and Tasks

But first, a little terminology. Running a Docker container on ECS requires you to create a task definition that specifies what image(s) you want to run, what the command should be, what the environment variables are, etc. Continuing our example, a task definition that runs Vector looks like this:

{
  "containerDefinitions":[
    {
      "name": "vector",
      "image": "timberio/vector:0.38.0-alpine",
      "essential": true,
      "environment": []
    }
  ]
}

Of course, this configuration won't do us much good as-is — it will run Vector, but there won't be any Vector configuration, so Vector won't be doing anything at all. We'll fix that in a bit. :)

An ECS service runs your tasks (made up of one or more images) on your own EC2 instances or instances managed by AWS (known as Fargate). We'll assume you're using Fargate for this tutorial. Each service definition specifies how many copies of the task definition you want to run (e.g., two or more for redundancy), what security group to use, the ports to forward to the containers, and so on. In other words, your task definition specifies the Docker-specific stuff like the image to use, and the service specifies how to run it in the AWS environment.

With that out of the way, we can return to the task at hand (pun intended).

Configuring a container

You might have a container that's configured entirely by environment variables. If that's the case, then you can use the environment section of the task definition to handle that:

  "environment": [
    {
      "name": "ENVIRONMENT",
      "value": "production"
    },
    {
      "name": "LOG_LEVEL",
      "value": "info"
    }
  ]

But you have to do a bit more work to get a configuration file to show up. I'll drop a task definition on you, then walk through the key points.

{
  "containerDefinitions":[
    {
      "name": "vector",
      "image": "timberio/vector:0.38.0-alpine",
      "mountPoints": [
        {
          "sourceVolume": "vector-config",
          "containerPath": "/etc/vector"
        }
      ],
      "dependsOn": [
        {
          "containerName": "vector-config",
          "condition": "COMPLETE"
        }
      ],
    },
    {
      "name": "vector-config",
      "image": "bash",
      "essential": false,
      "command": [
        "sh",
        "-c",
        "echo $VECTOR_CONFIG | base64 -d - | tee /etc/vector/vector.yaml"
      ],
      "environment": [
        {
          "name": "VECTOR_CONFIG",
          "value": "Contents of a config file go here"
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "vector-config",
          "containerPath": "/etc/vector"
        }
      ]
    }
  ]
}

There are a few things to notice here:

• There are two containers instead of just one. This is how you run a sidecar (running an app container and a logging container side by side) or, in this case, bootstrapping one container with another one.
• Both containers share a mountpoint (vector-config) at the same location (/etc/vector). The containerPath doesn't have to be the same, but the sourceVolume does. This allows one container to write to a file and the other container to be able to read that same file.
• The vector container depends on the vector-config container and waits to boot until the vector-config container has run its command.
• The command for the vector-config container populates a configuration file with the contents of an environment variable called VECTOR_CONFIG.

That's the bones of getting a file mounted for the Docker container. An initializer container creates the file on a shared volume; then, another container can read the file. But how do we get the contents of our config file into that environment variable, and what's with the base64 -d - thing?

Terraform it

Terraform is a handy tool for automating the deployment of cloud infrastructure. It works with all kinds of clouds and is great for documenting and tracking your infrastructure changes. For this tutorial, we'll focus on just one Terraform resource — the one that can create our task definition and populate the configuration:

resource "aws_ecs_task_definition" "vector" {
  family                   = "vector"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"

  volume {
    name = "vector-config"
  }

  container_definitions = jsonencode([
    {
      name      = "vector"
      image     = "timberio/vector:0.38.0-alpine"
      essential = true
      mountPoints = [
        {
          sourceVolume  = "vector-config"
          containerPath = "/etc/vector"
        }
      ],
      dependsOn = [
        {
          containerName = "vector-config"
          condition     = "COMPLETE"
        }
      ]
    },
    {
      name      = "vector-config"
      image     = "bash"
      essential = false
      command = [
        "sh",
        "-c",
        "echo $VECTOR_CONFIG | base64 -d - | tee /etc/vector/vector.yaml"
      ],
      environment = [
        {
          name  = "VECTOR_CONFIG"
          value = base64encode(file("vector.yaml"))
        }
      ],
      mountPoints = [
        {
          sourceVolume  = "vector-config"
          containerPath = "/etc/vector"
        }
      ],
    }
  ])
}

That looks pretty familiar, right? Terraform does a good job of sticking closely to the formats used by the various cloud providers. In this case, the aws_ecs_task_definition resource looks like the JSON used in task definitions. Note how the VECTOR_CONFIG environment variable is populated. Terraform provides file and base64encode helpers to read a file's contents and encode it, respectively².

Our actual Vector config (that ends up at /etc/vector/vector.yaml) is stored in a file next to our Terraform config. It could look something like this:

sources:
  app_metrics:
    type: prometheus_scrape
    endpoints:
      - http://localhost:9090/metrics

sinks:
  honeybadger_insights:
    type: "http"
    inputs: ["app_metrics"]
    uri: "https://api.honeybadger.io/v1/events"
    request:
      headers:
        X-API-Key: "hbp_123"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"

Diving into how Vector works could be a whole 'nother blog post, but here's a quick run-down on what we're configuring our Vector sidecar to do. We first define a source, or in other words, something that emits some data for Vector to process. Vector supports many sources, like S3 buckets, Kafka topics, etc. We're telling Vector to scrape Prometheus metrics served by our application on port 9090³. The sink configuration sends data from Vector to someplace else — in this case, to Honeybadger Insights.

That's a wrap

So, that's how you can deploy a Docker image to AWS ECS with a custom configuration without having to build and host a custom image. All it takes is a little bit of Terraform!