uncategorized

Updating Nginx on Joyent Triton Using ContainerBuddy, Consul, Nginx, and Manta

Sometimes it takes a village. I have been moving my applications to Docker Containers and am trying out Joyent’s Triton which runs the containers on bare metal rather than within a VM. This comes with some challenges, mostly in one’s thought process, because rather than having multiple containers in a VM each container runs separately and you must set up some sort of discovery mechanism so the containers can communicate. In a VM scenario this is very easy. Add Nginx to the VM, using the great container developed by jwilder, expose a couple of envars and you are good to go. In this post I will discuss configuring nginx. In a later post I will discuss updating dns automatically when nginx has changed.

Joyent has some nice tools to assist you.

  1. Joyent ContainerBuddy which communcates with Consul to register and monitor your app container.
  2. An Nginx container set up to have a dynamic configuration file (see above)

ContainerBuddy is a go application which handles all of the coordination for the containers.

Creating the nginx conf file

We need to update our nginx conf file as services are added to consul. When a service is added or it’s ip has changed it causes a consul template to be rendered which then is copied to the nginx container and nginx is restarted. The consul template has access to all the registered services and their network information and can easily generate then conf file. Unfortunately, it doesn’t have any knowledge of fully qualified domain names so if you want to direct www.dailycrytogram.com to the “crypto” service rather than have users go to www.dailycryptogram.com/crypto we need to have a more powerful script. And if you want nginx to control multiple domains you definitely need to modify the script. What we need to do is add a server block with the server_name being the fully qualified domain name and pass requests to the appropriate service.

The problem is, how do you get this domain information into the template. Fortunately the consul templating language allows the use of plugins to inject code. We will create a node plugin to provide the information. You could use bash or provide a web service to provide the data. Node is my bread and butter and I used that instead of brushing up on my bash skills. Let’s look at the template:

1
{{range services}}
upstream {{.Name}} {
    # write the health service address:port pairs for this backend
    {{range service .Name}}
    server {{.Address}}:{{.Port}};
    {{end}}
}
{{end}}
server {
    listen       80;
    server_name  _;

    # if you have http_stub_status_module compiled-in, then
    # this would be a good place to use /nginx_status
    location /health.txt {
        add_header content-type "text/html";
        alias /usr/share/nginx/html/index.html;
    }
    {{range services}}


    location /{{.Name}}/ {
        proxy_pass http://{{.Name}}/;
        proxy_redirect off;
        proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Server $host;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Real-IP $remote_addr;
    }
      location = /{{.Name}} {
            return 302 /{{.Name}}/;
        }
    {{end}}
}

{{range services}}
	{{if .Name}}
			server {
					server_name {{plugin "node" "/application/index.js" "--task" "serverAlias" "--service" .Name}};
					listen 80 ;
					location / {
                       proxy_redirect off;
                              proxy_http_version 1.1;
                      		proxy_set_header Upgrade $http_upgrade;
                      		proxy_set_header Connection "upgrade";
                      		proxy_set_header Host $host;
                      		proxy_set_header X-Forwarded-Host $host;
                      		proxy_set_header X-Forwarded-Server $host;
                      		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                      		proxy_set_header X-Real-IP $remote_addr;
                      proxy_pass http://{{.Name}}/;
					}

			}
	{{end}}
{{end}}

The top section sets the upstream for each service and then creates a default server.
The bottom section (starting with the { {range services} } loop is where we create our server blocks.
Notice that “node” is called and it will insert the appropriate domain names.
Now lets look at the rendered template.

1
upstream ninja {
    # write the health service address:port pairs for this backend
    
    server 192.168.130.149:3500;
    
}

server {
    listen       80;
    server_name  _;

    # if you have http_stub_status_module compiled-in, then
    # this would be a good place to use /nginx_status
    location /health.txt {
        add_header content-type "text/html";
        alias /usr/share/nginx/html/index.html;
    }
    
    location /ninja/ {
        proxy_pass http://ninja/;
        proxy_redirect off;
        proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Server $host;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Real-IP $remote_addr;
    }
      location = /ninja {
            return 302 /ninja/;
        }
    
}
	
server {
		server_name joyent.lendingclubninja.com joyent2.lendingclubninja.com www.lendingclubninja.com lendingclubninja.com;
		listen 80 ;
		location / {
		   proxy_redirect off;
				  proxy_http_version 1.1;
				proxy_set_header Upgrade $http_upgrade;
				proxy_set_header Connection "upgrade";
				proxy_set_header Host $host;
				proxy_set_header X-Forwarded-Host $host;
				proxy_set_header X-Forwarded-Server $host;
				proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
				proxy_set_header X-Real-IP $remote_addr;
		  proxy_pass http://ninja/;
		}

}

I have included only the snippet for a single service. Other services would have similar code with a different upstream, location blocks and server block. In this case the base domain is the same for all aliases but that is not necessary.

If you tail the access log for nginx you will something similar to the following

1
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
199.27.133.92 - - [02/Jan/2016:18:30:35 +0000] "GET /templates/templ_listednotesgrid.html HTTP/1.1" 304 0 "http://www.lendingclubninja.com/members.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
199.27.133.92 - - [02/Jan/2016:18:30:35 +0000] "GET /templates/templ_filterRow.html HTTP/1.1" 304 0 "http://www.lendingclubninja.com/members.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
199.27.133.92 - - [02/Jan/2016:18:30:35 +0000] "POST /LoadFilters/?rnd=0.08654024917632341 HTTP/1.1" 200 2763 "http://www.lendingclubninja.com/members.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
199.27.133.92 - - [02/Jan/2016:18:30:35 +0000] "GET /GetListedNotesColumns/?rnd=0.3991539857815951 HTTP/1.1" 200 2714 "http://www.lendingclubninja.com/members.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
199.27.133.92 - - [02/Jan/2016:18:30:36 +0000] "GET /GetListedNotes/?rnd=0.7869115453213453 HTTP/1.1" 200 245370 "http://www.lendingclubninja.com/members.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "73.153.27.94"
127.0.0.1 - - [02/Jan/2016:18:30:37 +0000] "GET /health.txt HTTP/1.1" 200 867 "-" "curl/7.38.0" "-"
127.0.0.1 - - [02/Jan/2016:18:30:47 +0000] "GET /health.txt HTTP/1.1" 200 867 "-" "curl/7.38.0" "-"
127.0.0.1 - - [02/Jan/2016:18:30:57 +0000] "GET /health.txt HTTP/1.1" 200 867 "-" "curl/7.38.0" "-"
127.0.0.1 - - [02/Jan/2016:18:31:07 +0000] "GET /health.txt HTTP/1.1" 200 867 "-" "curl/7.38.0" "-"

Notice that requests to www.lendingclubninja.com are routed appropriately. Also, you see requests from consul as it performs health tests. This confirms that both internal and external requests are handled correctly.

Storing Configuration

In order to provide the domain names I store the configurations in a file using Joyent’s Manta Object Store. [Manta] ( https://apidocs.joyent.com/manta/nodesdk.html “Manta api”).

1
{
	"notify"      : {
		"upstream"   : "joyent.notify.dailycryptogram.com",
		"serverAlias": [
			"joyent.notify.dailycryptogram.com",
			"notify.dailycryptogram.com"
		]
	},
	"hexo"        : {
		"upstream"   : "joyent.blog.vawter.com",
		"serverAlias": [
			"joyent.blog.vawter.com"
		]
	},
	"ninja": {
		"upstream"   : "joyent.lendingclubninja.com",
		"serverAlias": [
			"joyent.lendingclubninja.com",
			"joyent2.lendingclubninja.com",
			"www.lendingclubninja.com",
			"lendingclubninja.com"
		]
	}	
}

Here is the snippet of code that retrieves the file from Manta.

1
let retrieveFromManta = ()=>new Promise(function (resolve, reject) {
	let client = manta.createClient({
		sign: manta.privateKeySigner({
			key  : fs.readFileSync(__dirname + '/id_rsa', 'utf8'),
			keyId: process.env.MANTA_KEY_ID,
			user : process.env.MANTA_USER
		}),
		user: process.env.MANTA_USER,
		url : process.env.MANTA_URL
	});
	client.get('~~/stor/dns/dnsConfig.json', (err, stream)=> {
		if (err) {
			//console.log(err);
			reject(err);
		} else {
			stream.pipe(fs.createWriteStream("/tmp/dnsConfig.json"));
			stream.setEncoding('utf8');
			stream.on('end', resolve);
			stream.on('error', reject);
		}
	});
});

Manta returns the file as a stream so we just pipe the stream to a file. I am using promises so we resolve when the stream is finished processing.

Extracting the Server Alias for consul

Consul plugins are required to send results on stdout.

1
let processConfig = (configObject)=> {
	let obj = JSON.parse(configObject);
	switch (argv.task) {
		...
		case 'serverAlias':
			if (obj[argv.service]) {
				process.stdout.write(obj[argv.service].serverAlias.join(" "))
			} else {
				process.stdout.write("");
			}
			process.exit(0);
			break;
		default:
			process.stdout.write("");
			process.exit(0);
	}
}

All we are doing is finding the array of serverAliases for the service, joining them and writing the string to stdout.

This post has covered configuring nginx dynamically in conjunction with Consul and ContainerBuddy. In the next article I will discuss updating the dns records when nginx is changed. The implementation will use the CloudFlare DNS server but any dns server which provides an api could be used.

Share