Nomad Workshop 3 - Storing Configuration in Consul

Re-deploying hello-world every time we want to change the name we're saying hello to seems a little heavy handed when we really just need to update our greet config file template and restart our greet task. So, how can we accomplish this without another deployment? Consul to the rescue!

It's best if you follow the documentation here to update your job specification at 1_HELLO_WORLD/job.go and your vars file at 1_HELLO_WORLD/vars.go, but if you get lost you can see the final product under 3_HELLO_CONSUL/job.go and 3_HELLO_CONSUL/vars.go.

Nomad ships with a tool called consul-template that we've actually already been making good use of. For example, the template stanza of our greet uses consul-template to template our config.yml file.

template {
  data        = var.config-yml-template
  destination = "${NOMAD_ALLOC_DIR}/config.yml"
  change_mode = "restart"
}

We can instruct consul-template to retrieve the name of the person we're saying hello to from the Consul K/V store while deploying our greeter allocations. After an initial deploy, Nomad will then watch the Consul K/V path for changes. If a change is detected, Nomad will re-run consul-template with the updated value and then take the action specified by the change_mode attribute of our template stanza. In our case, it will restart the greet task.

Modify our greet config file template to source from Consul

Our template var in 1_HELLO_WORLD/vars.go is currently:

config-yml-template = <<-EOF
  ---
  name: "YOUR NAME"
  port: {{ env "NOMAD_ALLOC_PORT_http" }}
  
EOF

We should edit it like so:

config-yml-template = <<-EOF
  {{ with $v := key "hello-world/config" | parseYAML }}
  ---
  name: "{{ $v.name }}"
  port: {{ env "NOMAD_ALLOC_PORT_http" }}
  {{ end }}
  
EOF

Here we're setting a variable v with the parsed contents of the YAML stored at the Consul K/V path of hello-world/config. We're then templating the value of the name key.

Note: make sure you leave an empty newline at the end of your vars file otherwise the Nomad CLI won't be able to parse it properly.

Push our YAML formatted config to Consul

We could do this with the Consul web UI but using the consul CLI is much
faster.

$ consul kv put 'hello-world/config' 'name: "Samantha"'
Success! Data written to: hello-world/config

$ consul kv get 'hello-world/config'
name: "Samantha"

Here we've pushed a name key with a value of "Samantha" to the
hello-world/config Consul K/V. We have also fetched it just to be sure.

Check the plan output for our updated hello-world job

$ nomad job plan -verbose -var-file=./1_HELLO_WORLD/vars.go ./1_HELLO_WORLD/job.go
+/- Job: "hello-world"
+/- Task Group: "greeter" (1 create/destroy update, 1 ignore)
  +/- Task: "greet" (forces create/destroy update)
    +/- Template {
          ChangeMode:   "restart"
          ChangeSignal: ""
          DestPath:     "${NOMAD_ALLOC_DIR}/config.yml"
      +/- EmbeddedTmpl: "---\nname: \"Samantha\"\nport: {{ env \"NOMAD_ALLOC_PORT_http\" }}\n\n" => "{{ with $v := key \"hello-world/config\" | parseYAML }}\n---\nname: \"{{ $v.name }}\"\nport: {{ env \"NOMAD_ALLOC_PORT_http\" }}\n{{ end }}\n  \n"
          Envvars:      "false"
          LeftDelim:    "{{"
          Perms:        "0644"
          RightDelim:   "}}"
          SourcePath:   ""
          Splay:        "5000000000"
          VaultGrace:   "0"
        }

Scheduler dry-run:
- All tasks successfully allocated.

Alright this looks like it should work.

Run our updated hello-world job

$ nomad job run -verbose -var-file=./1_HELLO_WORLD/vars.go ./1_HELLO_WORLD/job.go

Let's fetch the ports of our 2 new greeter allocations

$ dig @127.0.0.1 -p 8600 hello-world-greeter.service.dev-general.consul. SRV | grep hello-world-greeter.service
; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 8600 hello-world-greeter.service.dev-general.consul. SRV
;hello-world-greeter.service.dev-general.consul.	IN SRV
hello-world-greeter.service.dev-general.consul.	0 IN SRV 1 1 30226 7f000001.addr.dev-general.consul.
hello-world-greeter.service.dev-general.consul.	0 IN SRV 1 1 28843 7f000001.addr.dev-general.consul.

You should be able to browse to http://localhost:30226 or http://localhost:28843
and be greeted.

Update the value of name in Consul

$ consul kv put 'hello-world/config' 'name: "SAMANTHA"'
Success! Data written to: hello-world/config

$ consul kv get 'hello-world/config'
name: "SAMANTHA"

Browse to one of our greeter allocation URLs again

If you reload http://localhost:30226 or http://localhost:28843 you should be greeted by your updated name. This did not require a deployment; Nomad was notified that the value at the Consul K/V path of hello-world/config had been updated. Nomad re-templated our greet config file and then restarted our greet task just like we asked (in the template stanza).

Note: you may have observed a pause of about 5 seconds between when the greet task being stopped and when it was started again. This is the default wait time between stop and start operations and it's entirely configurable on a per group or task basis.

Ready to configure a Load Balancer using Consul?

Continue on to Nomad Workshop 4 - Load Balancing with Consul and Traefik.