How to Backup Google Mail for Your Domain (Free + Open Source)

I’m currently migrating a company from using Google Mail to an Office 365 hosted exchange server. Before getting started, I wanted to make a full backup of all mails of all users. Now there’s several commercial offers and also several shady-looking free tools to accomplish this task, but I didn’t feel comfortable trusting either one. So I built a very simple free and open source solution.

I already had a backup script which I forked from thefloweringash on Github early 2011 and since modified to suit my own needs. Using my backup script, I could backup all emails from an IMAP server. However, my script (until now) needed full credentials to access the email account. For backing up the full Google Apps domain, I would need another authentication method which does not require the user’s passwords. Luckily, Google supports the use of so-called “service accounts” for precisely this purpose.

Introducing gmail-imap-backup

You can download my backup script from github:

https://github.com/fxtentacle/gmail-imap-backup

The script will perform backups for every config-*.yml configuration file which it finds. It uses lockfiles to prevent two processes from accessing the same IMAP account at the same time.

For authentication, you can use:

  • plain text username & password
  • OAuth tokens (used for personal GMail accounts)
  • Google service accounts (which use OAuth2 to access a full domain)

Setting up the service account with Google

First, we’ll need to create a service account that you (the admin) control.

  1. Login to the Google developer console. I used my private Google account for this.
  2. Create a new project.
  3. Open the project. Navigate to “APIs & auth” –> “Credentials”
  4. Click the red “Create new client ID” button under OAuth. Choose type “Service account”
  5. Download the private key file and write down the private key’s password.
  6. Write down the “Client ID” and the “Email address” of the service account you just created. These will be needed later (along with the key file and its password).

Granting the service account access to your Google Apps domain

Now we need to give your new service account the permission to access the Google Apps domain that you wish to backup.

  1. Go to the Google Apps admin console. Login using your admin account for the domain that you want to backup.
  2. Open the security settings. These might be hidden behind “More controls” at the bottom of the dashboard.
  3. Navigate to “Advanced Settings” –> “Manage OAuth client access”
  4. In the “Client Name” field, enter the client ID of your service account. In the “One or More API Scopes” field, enter “https://mail.google.com/”. Click authorize.
  5. Your service account should now have access to all email accounts in your domain through OAuth2.

Configuring gmail-imap-backup

Now you’ll need to create a configuration file for every user that you wish to backup. Basically, the configuration file states which folders to backup and where to save the backup.

Here’s an example config.yml to get you started:

1
2
3
4
5
6
7

---
email: [email protected]
mailbox: '[Gmail]/All messages'
destination_root: '/mnt/email-backups/com.example-user'
service_account_clientid: '[email protected]'
service_account_key_file: '/mnt/email-backups/service-account-key.p12'
service_account_key_pass: 'notasecret'

I think email and destination_root are pretty self-explanatory.

If the mailbox has subfolders, they will be backupped recursively. To exclude subfolders, you may use mailbox_skip which is an array of strings. To enumerate all folders, create a config with mailbox set to '' and then run:

1

bundle exec ruby list_mailboxes.rb

service_account_key_file should be the full path to the private key file you downloaded when creating the service account. service_account_key_pass should be the password that Google gave you to unlock this keyfile. service_account_clientid is a slight misnomer. It’s the client ID that my backup tool sends to GMail for authentication, but GMail actually expects you to specify the “Email address” that you got when creating your service account. Mine looks like (number)@developer.gserviceaccount.com.

Running the backup

Now it’s time to execute:

1

bundle exec ruby run_backup.rb

My backups script will then iterate over all config files it can find and then backup all emails in all mailboxes that you specified (or their subfolders). For authentication, it’ll use your service account with the keyfile and passphrase which means you don’t need to know your user’s login data to backup their emails :)