Distributed Testing Using AWS SQS and NodeJS

We have many different Node applications and wanted a way to run tests on all of them and consolidate the results,
view the results, create exception reports etc.

##Requirements

  1. Test multiple apps seamlessly
  2. Allow multiple monitors to run simultaneously
  3. View results in browser
  4. Create exception reports
  5. Not add any code to the application being tested (except tests themselves, of course)
  6. Test runner and applications can be on separate networks etc.
  7. Can run Jasmine tests
  8. Can run custom non Jasmine Tests written in Node.
  9. Able to be incorporated in automated process ( e.g. Jenkins)
  10. Be flexible enough to add non Node based tests.

##Solution

The key is how to communicate requests for tests to the various applications and how to get the results back. Since we cannot be assured the requestor has access to the application we ruled out communicating via socket.io which is our usual choice for inter applications communication.

Then the light bulb lit up. There is an app for that - AWS SQS. While not as performant as direct communication it still offers a viable mechanism for communication.

Our solution was to have a worker process on each machine(VM, Docker container) for which an application exists that has tests. That process must have access to the tests and be able to execute them. The worker responds to incoming SQS messages. Each worker has a unique queue, named by convention. If the queue does not exist the worker creates it on launch. The worker queries SQS for incoming messages at a configurable interval.

On launch each monitoring app creates a unique queue for this session and always sends that name with any message so responders know where to respond. At the close of the session the queue is deleted (somewhat problematic on browser based monitors). Since those queues are named by convention it is easy to have a maintenance process that deletes stale queues.

The monitoring app queries SQS for a list of queues (named by convention to filter). It then sends a message to the chosen queue requesting avaiable tests. The worker then responds with a list of tests and the monitor can send a request to run a test. In the current implementation it runs all the tests but there is no need to limit it to that.

The worker app consists or a node module installed globally on deployment, and a configuration file which defines the location of test files.

##Implementation

All communication with AWS leverages aws-sdk which is available both to node and to clients in a browser. The authentication is handled with environment variables, although if everything was based in EC2 containers we could handle authentication by inheritance.

Lets look at a couple of snippets from the browser viewer (a simple static html page with some javascript which doesn’t even need to be hosted. You can just open the file in a browser.) to illustrate.

1
 $(document).ready(function () {
      AWS.config.update({
        accessKeyId    : ${ACCESS_KEY},
        secretAccessKey: ${SECRET_ACCESS_KEY}
      });
//       Configure your region
      AWS.config.region = ${REGION};
      var sqs = new AWS.SQS();
      var inUrl = void(0);
      var testerInstances = [];
      //Query to get available queues
      sqs.listQueues({}, function (err, data) {
        data.QueueUrls.forEach(function (queue) {
          var aMachine = queue.split("/");
          var queueName = aMachine[aMachine.length - 1];
          //By convention monitor queues start with "tester"
          //so exclude them from dropdown of available machines
          if (!(queueName.indexOf("tester") === 0)) {
            $("#machines").append("<option value='" + queue + "'>" + queueName + "</option>");
          }
          //if a monitor queue tester_XXX add it's number
          if (queueName.indexOf("tester") === 0) {
            var numb = Number(queueName.replace(/[^0-9]/g, ""));
            if (numb === Number(numb)) {
              testerInstances.push(numb);
            }
          }
        });
          var qNumber = testerInstances.length ? Math.max.apply(null, testerInstances) : 0;
          qNumber++;
          //create a new queue with a unique number
          sqs.createQueue({QueueName: "tester_" + qNumber}, function (err, data) {
            if (err) {
              console.log(err);
            } else {
              inUrl = data.QueueUrl;
              $(window).on("unload",function(){
                sqs.deleteQueue({QueueUrl:inUrl},function(err,data){
                  console.log(err,data);
                })
              });

				...

The above code sets up an instance of aws-sdk, queries sqs for available queues, builds a list of machines which have tests, and creates a new queue for this session

When the viewer selects a machine from the dropdown we ask that machine for a list of available tests.

1
$("body").on("change", "#machines", function () {
               var outUrl = $("#machines").val();
               if ((outUrl.indexOf("__") === 0)||(outUrl.indexOf("test") === 0)) {
                 return;
               }
               $("#availabletests").html("Available Tests:<br/");
               $("#testresults").html("Test Results<br/");
               var payload = {request: "getTests",testerUrl:inUrl};
              sqs.sendMessage({
                 QueueUrl   : outUrl,
                 MessageBody: JSON.stringify(payload)
               }, function (err, data) {
                  console.log(err);
                  //the data is of no use to us
               });

We now have to poll AWS listening for a response

1
setInterval(function () {
               sqs.receiveMessage({QueueUrl: inUrl}, function (err, data) {
                 if (data && data.Messages) {
                   data.Messages.forEach(function (message) {
                     var params = {
                       QueueUrl     :inUrl,
                       ReceiptHandle: message.ReceiptHandle
                     };
                     sqs.deleteMessage(params, function (err, data) {
                     });

When a message is received we capture it (message) and delete it from queue so AWS doesn’t resend it.
We can then process the message body, display the available tests, and send a message to the queue requesting it to run the tests. When we receive those results we display them on page. There is nothing interesting in the code so it is omitted here.

Here is typical output:
Viewer

##The worker application
The worker listens to the SQS queue for incoming messages, processes the message, and returns the results in another SQS message. If the incoming message requests to run tests, it does so, captures the results and sends them.

1
class Worker {
 ....
	startListener() {
		let self = this;
		this.listenQueue = new Queue(this.receiveOptions);
		this.listenQueue.initializeQueue()
			.then((q)=> {
				let ql = Promise.coroutine(function* () {
					while (true) {
						yield Promise.delay(self.config.pollInterval || 10000);
						yield self.listenQueue.readMessage()
							.then((results)=> {
								// read the messages
								results.forEach((result)=> {
									var message = result;
									var body = JSON.parse(message.Body);
									switch (body.request) {
										case "getTests":
										{
											self.getTests(body.testerUrl);
											self.listenQueue.removeMessage(message)
												.then((result)=>console.log(result))
												.catch((err)=>console.log(err, err.stack));
											break;
										}
										case "runJasmineTests":
										{
											require("glob-promise")(body.files)
												.then((files)=> {
													require("serial-jasmine").runTasks(files, null, true)
														.then((results)=> {
															let res = writeResults(results);
															let payload = {request: body.request, results: res};
															self.getSendQueue(body.testerUrl).sendMessage(JSON.stringify(payload))
																.then((results)=>console.log(results))
																.catch((err)=>console.log(err));

														})
														.catch((err)=> {
															self.getSendQueue(body.testerUrl).sendMessage(JSON.stringify(err))
																.then((results)=>console.log(results))
																.catch((err)=>console.log(err));
														})
														.finally(()=> {
															self.listenQueue.removeMessage(message)
																.then((result)=>console.log(result))
																.catch((err)=>console.log(err, err.stack));
														});
												})
...

The Worker class has a method startListener which creates an infinite generator. We use the bluebird promise library and their coroutine which allows you to yield a promise in a generator. Notice that we also yield to Promise.delay to avoid continuous polling.
A lot of other cases are left out of case statement here but the runJasmineTests illustrates the process:

  1. Gets the test files to run using glob-promise
  2. Calls runTasks on serial-jasmine
  3. When the tests are complete, serial-jasmine resolves
  4. We then process those results, create a message with the results as a payload
  5. Send the message to the requesting queue
  6. Delete the incomming message
  7. resolve the promise to return to the generator (not shown here)

The worker module is available on npm as aws-test-worker.
That has a description of how to use the worker from the command line and prepare the conifuration file but to summarize it’s usage:

1
npm install aws-test-worker -g

and to start it on a machine:

1
aws-test-worker --configfile <path to json file containing configuration> &

##Conclusion

We have developed an architecture which allows us to monitor the results of running tests on multiple applications from a remote location using AWS SQS to communicate. There is no modification to the applications being tested. Installing aws-test-worker via npm is the only addition to the host machine. The workers can easily be started from an automation script since they have a command line interface.

Currently we have only implemented a simple viewer but extending this to use the test results from a node application would be very easy. You would just set up a listener to the queues of interest and process the messages containing test results appropriately.

It would also be easy to extend aws-test-worker to run tests from another test framework. As long as that framework is promise based the implementation should be trivial.

Share