Encrypted Backup Shootout (acha.ninja)

https://acha.ninja/blog/encrypted_backup_shootout/

aDfbrtVt 9 months ago [–]

I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. I've found encrypted Borg very difficult to get wrong and setup is very simple. I've also successfully recovered two systems with the tool without issue.

Not saying that Borg is necessarily the best solution, just that we should evaluate the important metrics. ///

Good discussion re: backup software, including comments from tarsnap author cperciva

Recently I have been spending time on improving the performance of bupstash (my encrypted backup tool), and wanted to compare it to some existing tools to try and find its relative performance in the backup tool landscape.

This post compares bupstash, restic, borg backup and plain old tar + gzip + GPG across a series of simple benchmarks.

What do all these tools have in common?

• They encrypt data at rest.
• They compress data.
• They have some form of incremental and/or deduplicated snapshotting.
• They are all pretty great backup systems.

To create a printable key, either paste the contents of your keyfile or a key export in the text field below, or select a key export file.

To create a key export use

borg key export /path/to/repository exportfile.txt

If you are using keyfile mode, keyfiles are usually stored in $HOME/.config/borg/keys/

You can edit the parts with light blue border in the print preview below by click into them.

Key security: This print template will never send anything to remote servers. But keep in mind, that printing might involve computers that can store the printed image, for example with cloud printing services, or networked printers.

• Restic is multi-threaded, borg is not. This translates to restic being extremely fast in comparison to borg, but borg having less impact on average on CPU usage while running. This limitation in borg is actually a direct consequence of the next point.
• Borg does actual deduplication, while restic only does classic incremental backups. With restic, you store a copy of every file, but the files are reference counted so that each version of a file only gets stored once. Borg, however, operates on blocks, not files, and deduplicates within individual backups. So if you have a dozen copies of the same data in your backup, restic stores each copy, but borg only stores the first and makes all the others references to that. The main benefit of this is that borg produces much smaller backups when you have lots of duplicate data and actually does more space efficient incremental backups (because it only stores what actually changed, not the whole changed file).
• Borg supports compression, while Restic seemingly does not (and doesn't handle sparse files very well either). This too has a huge impact on space efficiency, and may explain why restic is lightning fast on my systems when compared to borg. //

Austin I have to correct you:

Restic does indeed do deduplication on blocklevel. It uses a rolling hash algorithm called rabin as a chunker.

In short, a rolling hash algorithm reacts to patterns within the file and cuts it. If two files have the same patterns there's a high chance to have the cuts at the same positions, giving it the ability to deduplicate files which's data is not aligned to any specific block size.

backup strategies -- cold storage on encrypted drives managed with git-annex, remote repositories managed with borg and rclone, optical storage of archival photos, encrypted financials, etc.

For years the core of my backup strategy has been rsnapshot via cryptshot to various external drives for local backups, and Tarsnap for remote backups.

Tarsnap, however, can be slow. It tends to take somewhere between 15 to 20 minutes to create my dozen or so archives, even if little has changed since the last run. My impression is that this is simply due to the number of archives I have stored and the number of files I ask it to archive. Once it has decided what to do, the time spent transferring data is negligible. I run Tarsnap hourly. Twenty minutes out of every hour seems like a lot of time spent Tarsnapping.

Initially I played with borgmatic to perform and maintain the backups. Unfortunately it seems to have issues with signal handling, which caused me to end up with annoying lock files left over from interrupted backups. Borg itself has good documentation and is easy to use, and I think it is useful to build familiarity with the program itself instead of only interacting with it through something else. So I did away with borgmatic and wrote a small bash script to handle my use case.

Creating the backups is simple enough. Borg disables compression by default, but after a little experimentation I found that LZ4 seemed to be a decent compromise between compression and performance.

Pruning backups is equally easy. I knew I wanted to match roughly what I had with Tarsnap: hourly backups for a day or so, daily backups for a week or so, then a month or two of weekly backups, and finally a year or so of monthly backups.

Vorta is a backup client for macOS and Linux desktops. It integrates the mighty Borg Backup with your favorite desktop environment to protect your data from disk failure, ransomware and theft.

• Encrypted, deduplicated and compressed backups using Borg as backend.
• No vendor lock-in – back up to local drives, your own server or BorgBase, a hosting service for Borg backups.
• Open source – free to use, modify, improve and audit.
• Flexible profiles to group source folders, backup destinations and schedules.
• One place to view all point-in-time archives and restore individual files.

Installation

Vorta should work on all platforms that support Qt and Borg. This includes macOS, Ubuntu, Debian, Fedora, Arch Linux and many others. Windows is currently not supported by Borg, but this may change in the future.

rsync.net accounts have full support for borg backup

borg creates and maintains encrypted, remote backups.

• • Your data is encrypted with keys that only you hold
• • rsync.net cannot see your data.
• • Backups are fast, bandwidth efficient and compressed/deduplicated.
• • borg is fully open source and is in active, current development

Specific borg Features

• You may access the account with any tool that runs over SSH - not just borg.

• You may create and maintain an unlimited number of borg repositories.

• You have full control over your authorized_keys file to restrict IP and command access - or to enforce append-only mode.

• You may configure custom alerts to generate email, SMS, or Pushover warnings - or call a webhook. Or all of the above.

• You may set your account to be immutable (read-only) and accessible only by SSH key (disabled passwords).

We support legacy borg versions for backward compatibility - currently 0.29 and 1.x branches.

Special Pricing for borg Accounts

Special "borg accounts" are available at a very deep discount for technically proficient users.

• • Free ZFS filesystem snapshots are not included since you'll be doing versioning and retention with borg.
• • We will not configure subaccounts, or additional logins, for these borg accounts.
• • There is NO borg specific technical support or integration engineering. You're here because you're an expert.

Choose any location

• • NO Charges for ingress/egress
• • Unlimited borg Repositories
• • Start with 100 GB for $18/year
• $0.015/GB/Month

Simple and Secure Offsite Backups
Hosting for Borg Repositories. From $2/month, 5 GB free Trial.

$25/year
Billed annually, 30 days refund
100GB, then $0.01/GB/month
10 repositories

$80/year
1TB, then $0.007/GB/month

Configuration Assistant
Quick setup with pre-filled Borg commands and templates for Borgmatic.

BorgBackup (short: Borg) is a deduplicating backup program. Compression and authenticated encryption are also supported as options.

Borg's main goal is to provide an efficient and secure backup solution. Thanks to deduplication, the backup process with Borg is very fast and makes Borg very interesting for daily backups. You may notice that Borg is significantly quicker than some other methods, depending on the amount of data and the number of changes you need to back up. With Borg, all data is already encrypted on the client side, which makes Borg a good choice for hosted systems.

borgmatic is simple, configuration-driven backup software for servers and workstations. Backup all of your machines from the command-line or scheduled jobs. No GUI required. Built atop Borg Backup, borgmatic initiates a backup, prunes any old backups according to a retention policy, and validates backups for consistency. borgmatic supports specifying your settings in a declarative configuration file, rather than having to put them all on the command-line, and handles common errors.

Borg Backup is a Linux command-line utility to create backups of your computers. It's de-duplication and speed can't be beat. However, when you are dealing with a large number of machines to backup up, it quickly becomes obvious that you don't have a good way to manage all your client machines from a single place. It's also time-consuming and tedious to setup large number of machines. So, we've created Borg Backup Server.

Borg Backup Server makes it easy to install and maintain Borg on each client machine from a single server GUI (Graphical User Interface). Some of the powerful features include:

Ultimately, I landed on a combination of BorgBackup, Rclone, and Wasabi cloud storage, and I couldn't be happier with my decision. Borg fits all my criteria and has a pretty healthy community of users and contributors. It offers deduplication and compression, and works great on PC, Mac, and Linux. I use Rclone to synchronize the backup repositories from the Borg host to S3-compatible storage on Wasabi. Any S3-compatible storage will work, but I chose Wasabi because its price can't be beat and it outperforms Amazon's S3. With this setup, I can restore files from the local Borg host or from Wasabi. //

Each machine has a backup.sh script (see below) that is kicked off by cron at regular intervals; it will make only one backup set per day, but it doesn't hurt to try a few times in the same day. The laptops are set to try every two hours, because there's no guarantee they will be on at a certain time, but it's very likely they'll be on during one of those times. //

I could skip the cron job and provide a relatively easy way for each user to trigger a backup using BorgWeb, but I really don't want anyone to have to remember to back things up. I tend to forget to click that backup button until I'm in dire need of a restoration (at which point it's way too late!).

The backup script I'm using came from the Borg quick start docs, plus I added a little check at the top to see if Borg is already running, which will exit the script if the previous backup run is still in progress. //

Restoring files is not as easy as it was with CrashPlan, but it is relatively straightforward. The fastest approach is to restore from the backup stored on the Borg backup server. Here are some example commands used to restore:

Recently, I’ve migrated my personal backup from Backblaze Backup to B2 (an online S3-style file storage, also by Backblaze).I’ve heard of Arq Backup for a few years now but had not tried it yet. Being a native macOS app, it has a very nice UI and meets all the requirements mentioned above.

For the record, I am writing down other softwares I have considered and why I did not use them. Note they are all server-oriented and need to be scheduled using cron or similar software. All of them also support encryption.
borg

A fork of attic, which was known as the holy grail of backups. It supports compression, block-based incremental and is open-source. However, the only remote backup it supports is SSH. Rsync.net provides an attic-specific package for $0.03/GB/month, which is considerably higher than B2’s $0.005/GB/month.