uncategorized

Rethinking nginx.conf

In earlier posts I discussed generating the nginx.conf file dynamically and then the modifications necessary when all the services do not need an upstream block.

The procedure works but has three undesirable attributes:

  1. The list of services that need an upstream is tied to the links in the nginx service
  2. It requires an external program yaml2json
  3. Retrieval of the server names requires a node plugin

With the release of containerbuddy 1.0 we can eliminate those dependencies by utilizing tags for a service which are now exposed when the consul template is rendered.

Links have a purpose and the fact that, in most cases, all the links require an upstream is helpful but there is no guarantee that there is a one to one mapping. We need another mechanism to define the required upstreams.

By making use of comments in our docker-compose file we can define the services that need an upstream (those services which are using nginx as a proxy)

If we add comment like

1
## 'forupstream+"consul" "crypto" "hexo" "nginx" "ninja" "socketserver" "notify"+'"

and then parse it during the template generation

1
declare -a rArray=$(echo '('; cat  docker-compose.yml |grep 'forupstream' |sed 's/\([^+]*\)\+\(.*\)\+\(.*\)$/\2/'|sed 's/\\//g'; echo ')')

we can create an array of services that need an upstream without coupling it to links or depending on yaml2json
The remainder of the template generation code is unchanged.

Generating server blocks

If we want to route nginx to a given upstream based on host names, we need to create a server block for those hosts. In the earlier implementation I used a node plugin to look up the host names in a configuration file, requiring that you have node on the nginx container. Since our template now exposes tags we can use them to replace the node lookup.

1
{{range services }}
{{if (or (or (or (or (or (or (or (eq .Name "consul")) (eq .Name "crypto")) (eq .Name "hexo")) (eq .Name "nginx")) (eq .Name "ninja")) (eq .Name "socketserver")) (eq .Name "notify")) }}
{{if .Tags}}
 server {
 server_name {{range .Tags }} {{.}} {{end}} ;
 listen 80 ;
 location / {
 proxy_redirect off;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 proxy_set_header Host $host;
 proxy_set_header X-Forwarded-Host $host;
 proxy_set_header X-Forwarded-Server $host;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_pass http://{{.Name}}/;
 }
 location = "" {
 return 302 /;
 }
 }
 {{end}}

{{end}}
{{end}}

The second line

1
{{if (or (or (or (or (or (or (or (eq .Name "consul")) (eq .Name "crypto")) (eq .Name "hexo")) (eq .Name "nginx")) (eq .Name "ninja")) (eq .Name "socketserver")) (eq .Name "notify")) }}

is generated from our comment line in the docker-compose file.
For any service that has a non empty tags array we generate the server block and echo the tags to produce the server_name

1
server_name {{range .Tags }} {{.}} {{end}} ;

The associated service definition file is of the form:

1

{
	"consul"  : "consul:8500",
	"services": [
		{
			"name"  : "hexo",
			"port"  : 4000,
			"tags"  : [
				"joyent.blog.vawter.com",
				"aws.blog.vawter.com",
				"blog.vawter.com",
				"etsyhosts.com",
				"www.etsyhosts.com",
				"www.vawter.com",
				"vawter.com",
				"www.vawterconsultingservices.com",
				"vawterconsultingservices.com"
			],
			"health": "/usr/bin/curl --fail -s -o /dev/null http://localhost:4000/",
			"poll"  : 10,
			"ttl"   : 25
		}
	],
	"backends": [
		{
			"name"    : "nginx",
			"poll"    : 7,
			"onChange": "/opt/containerbuddy/reload-hexo.sh"
		}
	]
}

Summary

This approach allows us to generate the nginx conf file without using external programs and gives us complete control of the upstream and server blocks presence by modifying only the service definition files and adding a comment line in the docker-compose.

One caveat on this approach. Since we use “.”‘s in the tag names we will be unable to use consul to query by tag. If you use that capability or make other use of tags you would need to a)replace the . with and replace the with . when generating the server name and b)use some convention to indicate which tags refer to server names and only iterate over those tags.

The complete code for the template generation can be found here: Template

Share