uncategorized

Cron jobs in Docker Containers

The final piece in moving everything to Joyent is to handle scheduled tasks, which I usually do using cron.

The tasks I have:

  1. Fetch and parse RSS feeds for quotations (4 times a day)
  2. Send cryptograms by email to weekly subscribers (once a week)
  3. Send cryptograms by email to daily subscribers (once a day)
  4. Post a random quotation to the daily cryptogram site via http post (every two minutes)

It is not necessary to run these task at Joyent or in a Docker container. I can run them from my local machine and have for a long time. Doing so, however, requires that my local machine be running 24/7. Practically that is not an issue but it still creates a single point of failure. For that reason I decided to move the tasks to Joyent where a container can automatically be restarted.

The next question is whether to put the tasks in an existing container or in a new one. It would be cheaper to just add the tasks to another container which already runs 24/7. Since customers really don’t like getting multiple copies of email you certainly would not want the task on a container that has multiple instances. Using an existing container violates my philosopy of having single purpose containers. Since the cost of an additional container is only about $7.00/mo I am not willing to compromise on that philosophy.

Requirements for the cron container

  1. Has node installed with code for the tasks.
  2. Can run in detached mode without exiting.
  3. Makes the environment variables visible to cron.
  4. Provides a health check to consul.

All my containers require node (except consul). In order to keep the version of node consistent across containers I have a base node image from which all other images inherit. That image inherits from the official Docker container node:argon. If a minor node patch is realeased I just rebuild nodebase and the next time any container image is rebuilt it will have the new version of node. I also install basic packages which most containers use (nano, net-tools, jq, json). If you don’t need the additional packages you can just inherit from node:argon.

In order for a detached container to run it must keep a foreground process running. I accomplish this by installing monit. I think you could probably accomplish the same thing by running cron not in daemon mode but I haven’t tried this.

Cron normally does not see environment variables that are in .bashrc or .bash_profile. You could always source those files in your task but unfortunately when you specify environment variables in your DockerFile (or DockerCompose yaml file) they do not get copied to those files anyway. I get around this by adding this line to my DockerFile after I have defined the environment variables:

1
RUN env |sed 's/^\(.*\)$/export \1/g' >/root/.profile

I then source .proile in my cron tasks. Just be aware that cron doesn’t understand “source”. It does understand the equivalent “.” though.

The downside of this approach is that you cannot change the environment variables when you launch the container. If someone has a better solution please let me know.
If I have the need to change the values I just shell in and edit .proile. Not elegant and not automatic but I rarely need to change them.

For a consul health check I just have the container ping google because we all know that when google is down we are in a world of hurt anyway.

Here is the final docker file:

1
FROM donniev/nodebase:latest

COPY dist  /application/
WORKDIR /application
RUN chmod 775 start.sh
ADD .npmrc .

ADD opt/containerbuddy/ /opt/containerbuddy/
RUN  npm install --production && npm install --production -g json && npm install --production -g bunyan
RUN apt-get update
RUN apt-get -y --fix-missing install monit cron
RUN echo '0 0,6,12,18 * * *  root . /root/.profile;/usr/local/bin/node /application/rssfetch/index.js' >>/etc/crontab
RUN echo '45 7 * * 1          root . /root/.profile;/usr/local/bin/node /application/emailer/runWeekly.js' >>/etc/crontab
RUN echo '5 7 * * *         root . /root/.profile;/usr/local/bin/node /application/emailer/runDaily.js' >>/etc/crontab
RUN echo '*/2 * * * *       root curl -s http://www.dailycryptogram.com/getrandomquote?forMessage=yes >/dev/null 2>&1'  >>/etc/crontab

#ENTRYPOINT ["/application/start.sh"]
ENV AWS_SECRET_ACCESS_KEY=<obfuscated>
ENV AWS_REGION=us-east-1
ENV AWS_ACCESS_KEY_ID=<obfuscated>
ENV BUCKET=<obfuscated>
ENV CREDSFILE=<obfuscated>
ENV APPCONFIGFILE=<obfuscated>
ENV COMMONCONFIGFILE=<obfuscated>
ENV SOCKETSERVERURL=http://socketserver.icryptogram.com
ENV SOCKETSERVERPORT=80
ENV logRoot=/mylogs
ENV useSES=true
ENV forproduction=true
ENV TERM=xterm-256color
RUN env |sed 's/^\(.*\)$/export \1/g' >/root/.profile

Notice that we just append our cron tasks to /etc/crontab

Here is the corresponding entry in docker-compose used by container buddy:

1
cron:
    image: donniev/projects:cron.latest
    mem_limit: 128m
    links:
    - consul:consul
    restart: always
    command: >
      /opt/containerbuddy/containerbuddy
      -config file:///opt/containerbuddy/cron.json
      /bin/bash /application/start.sh

And start.sh which starts monit.

1
#!/usr/bin/env bash

/usr/sbin/service cron start
/usr/bin/monit -I

The -I flag runs monit in the foreground

Finally cron.json for container buddy

1
{
  "consul": "consul:8500",
  "onStart": "/opt/containerbuddy/reload-cron.sh",
  "services": [
    {
      "name": "cron",
      "health": "/bin/ping -c1 google.com",
      "poll": 300,
      "ttl": 600
    }
  ],
  "backends": [
  ]
}

This approach allows me to eliminate any dependencies on my local machine. That is good because when I finally take a vacation this spring (to the Lake District in Italy) I won’t have to worry about my local machine.

Share