uncategorized

Mending a broken Nginx configuration

Everything was going swimmingly. All my containers were registering with Consul, my scripts were using consul templates to update the nginx conf file for the currently active services, and when nginx reloaded it automatically updated the DNS records at CloudFlare. Easy peasy now I can stop wearing my devops hat and get back to writing application code. Not so fast. I created a container which had no need to use nginx. It didn’t expose any ports and didn’t need to be externally visible. BOOM! Even though the container was not using nginx the generated conf file broke.

Why? The conf file was generated by looping over the healthy services as reported by consul. Of course, the suspect container came along for the ride, it was healthy. Clearly we should only be considering containers using nginx but consul does not know anything about that. If the container does not expose any ports you could skip the service when .Port was 0 but what about cases where you have a redis server that exposes a port but you stil don’t want nginx to proxy it because you are not sending http requests.

We need to know which services nginx controls when we generate the template. Fortunately that information is in the docker-compose.yml file. The services it controls are shown as links. I suppose I could use awk to read the file and set a flag when we reached the nginx section and pull them that way but I decided there must be any easier way.

Fortunately there is an npm package which will convert yaml to json and if you install it globally it is available in bash scripts. I also used the jq package to parse the json.

1
npm install -g yamljs

and for jq:
Get it here

Once you have those tools getting the services is a one liner

1
declare -a rArray=$(  yaml2json  docker-compose.yml |jq ".nginx.links" |awk -F: 'BEGIN{out="\"nginx\""}/:/{out=out$1"\" "}END{print("("out")")}'|sed 's/,)/\)/')

We have to add “nginx” at the start because it won’t be in the links and there is a little sed cleanup to strip the trailing comma.

We can now use that information to create our consul-template

What we would like (psuedo code) is

1
{{range services }}
{{if .Name in ("nginx","consul","socketserver"...)
   ...do stuff
{{end}}
{{end}}

Unfortunately I can no construct in the consul-template language that does that.
I have resorted to using nested “or” and “eq” functions. It is ugly but you can easily script it.
The results yield a template in the form of:

1
{{range services }}
{{if (or (or (or (or (or (or (or (eq .Name "nginx")) (eq .Name "consul")) (eq .Name "hexo")) (eq .Name "socketserver")) (eq .Name "crypto")) (eq .Name "ninja")) (eq .Name "notify")) }}
upstream {{.Name}} {
 {{range service .Name}}
 server {{.Address}}:{{.Port}};
 {{end}}
 }

{{end}}
{{end}}

I warned you it wasn’t pretty. If anyone has an elegant solution please let me know but at least this is functional.

With the above changes nginx is happy again.

Conclusion

Bad assumptions lead to broken code.
Assuming that we can generate an nginx conf file by just looping over all the consul services is incorrect. If you have internal services you need to keep those out of the nginx file. Fortunately it is not difficult to do.

If you are interested in seeing the code you can just download the npm module.

1
npm install  deploy-joyent

That will, of course, put everying in a node_modules directory which is probably not where you want it since it has nothing to do with node. You can just move it wherever you like.

Share