Michael Trojanek (relativkreativ) — Bootstrapper and creator of things

This article was published on August 5th 2014 and takes about 6 minutes to read.

Use it with caution — it is probably still valid, but it has not been updated for over a year.

Use cron to add newsletter subscribers in the background

For simple background jobs (like interacting with your email marketing service provider's API to subscribe people to your mailing list), a full fledged background stack is not always necessary.

Until recently, adding new subscribers to my newsletter was kind of a hack I was not very proud of. Subscribing to my newsletter worked this way:

It was easy to implement and I needed a working solution quickly, but there is so much wrong with this approach:

This had to be reworked.

Switching to an ActiveRecord model

The first thing I did was refactoring the code, making "subscription" an ActiveRecord model (prior to that it was just a Ruby class with ActiveRecord's validations mixed in). Moving the interaction with MailChimp's API to an after_save-hook made the controller code clean again and I got a local copy of my subscribers list for free.

However, it was still slow and tightly coupled. Introducing a service object would have removed the tight coupling and made testing easier (while still being slow) but I decided to completely remove the interaction with the third party API from my application.

When you think about it, there is no need for the visitor to wait for a response from MailChimp's API: I have his email address and his first name, which is all I need. Storing this information in my application's database happens instantly. What happens afterwards really is my job and should not affect the visitor's experience with my website.

So: Background jobs!

I took a look at Sidekiq which is a really neat solution for implementing background jobs in your Rails application. Just tell a worker what to do, throw it into a queue and forget about it.

The downside is that you have to have a Redis server running which has to be monitored (in case it dies and needs to be restarted). This is one more moving part in the already complex Rails stack and I decided that it was overkill for my needs.

cron to the rescue

All I needed was a way to mark new subscribers as "unhandled" (when their email address needs to be sent to MailChimp's API) and later "opted_in" (when a subscriber actually joins the newsletter list by clicking the link in the confirmation email) in my application's database.

A small Ruby script would then periodically execute two tasks:

This is the script I came up with in its first iteration:

#! /usr/bin/env ruby

require 'active_record'
require 'yaml'
require 'mailchimp'

# ==============================================================================
# Basic subscription class
# ==============================================================================

class Subscription < ActiveRecord::Base
  scope :unhandled, -> { where(handled: false) }
  scope :not_yet_opted_in, -> { where(handled: true, opted_in: false) }

# ==============================================================================
# Configuration
# ==============================================================================

environment = ENV['RAILS_ENV'] || 'development'
db_config = YAML::load(File.open(File.join(Dir.pwd, 'config', 'database.yml')))[environment]
mailchimp = Mailchimp::API.new('00000000000000000000000000000000-us8')
list_id = '0000000000'


# ==============================================================================
# Handle new subscriptions
# ==============================================================================

if Subscription.unhandled.any?
  unhandled_subscriptions = Subscription.unhandled.map do |subscription|
      :email      => { 'email' => subscription.email_address },
      :merge_vars => { 'fname' => subscription.first_name },
      :email_type => 'html'

  # Send the list to MailChimp and get an array of added email addresses back
  new_email_addresses = mailchimp.lists.batch_subscribe(list_id, unhandled_subscriptions, true, false, true)['adds'].collect { |subscription| subscription['email'] }

  # Mark all subscriptions as handled whose email addresses were added
  Subscription.where(email_address: new_email_addresses).update_all(handled: true)

# ==============================================================================
# Handle new Opt-Ins
# ==============================================================================

# Get an array of all email addresses that MailChimp has for this list
opted_in = mailchimp.lists.members(list_id)['data'].select { |member| member['status'] == 'subscribed' }.map { |member| member['email'] }

# Mark all subscriptions whose email addresses are in MailChimps list (and thus
# have opted in) as opted in
Subscription.not_yet_opted_in.where(email_address: opted_in).update_all(opted_in: true)


All that's left to do is configure cron to run this script every 5 minutes. My applications all run under the app user, so I opened its crontab (crontab -e) and added the following line:

*/5 * * * * /bin/bash -lc /home/app/relativkreativ.at/codebase/subscription.rb

Note that a cron job runs with basically no environment. Since I use rbenv in production, I have to make sure that the rbenv environment is loaded (by starting a non-interactive login-shell with bash -lc) and all required variables are set (my RAILS_ENV is set system-wide via export RAILS_ENV=production in the file /etc/profile.d/rails_env.sh) before calling the script.

Now, at the latest 5 minutes after a visitor subscribes to my newsletter, the new email address is added to my newsletter-list in MailChimp which in turn sends the opt-in confirmation email. The first time the script runs after the new subscriber has clicked the confirmation link, my database is updated and I know I have a new adequate subscriber.

What's missing in this solution is taking care of people who unsubscribe. Of course these need to be removed from the local database.

The conclusion

Even though there are loads of excellent gems out there, you do not need to resort to one for every simple task. Think about your requirements and use the right tool for the job™.

Expand your DevOps skills!

Join hundreds of Rails developers and operators on my email list and get my ebook Build Your Own Rails Server as a free welcome gift.

No spam — guaranteed. You can leave at any time.