Daily Shaarli
October 4, 2021
The purpose of this page is to distribute releases of Alpine, as well as to provide users with documentation on Alpine. All documentation needed to configure Alpine can be found in the binary, and this page may provide extra documentation. In any case, you are not required, nor encouraged to access this site to run Alpine. You can run Alpine in your computer without the need to ever visit this site. All documentation needed to configure Alpine can be found in the binary, and can be read by either pressing the "?" command or by pressing "^G" in Alpine.
Encrypted, Deduplicated, and Compressed Data Backups Using Your Own Cloud Storage
KopiaUI
Kopia comes with a user-friendly desktop app for Windows, macOS and Linux which allows you to create snapshots, define policies and restore files quickly.
rsync.net is not built on a cloud platform - we own (and have built) all of our own platform. Our storage arrays, as we call them, are 2U “head units” with x16 2.5" drive slots and one or more SAS HBAs.
ZFS makes good use of fast cache storage so, after using up two drive slots for our boot mirror (which is always a mix of two totally different SSDs) we have room for up to 14 more SSDs for read (L2ARC) and write (SLOG) cache. ZFS is also RAM hungry so we choose a motherboard that can support up to 2TB of RAM.
Attached to these head units, externally, are one or more JBOD chassis filled with hard drives. JBOD stands for “just a bunch of disks” and they are typically 4U chassis containing 45 or 60 drives that attach back to the head unit with SAS cables.
rsync.net has no firewalls and no routers. In each location we connect to our IP provider with a dumb, unmanaged switch.
This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security.
- Restic is multi-threaded, borg is not. This translates to restic being extremely fast in comparison to borg, but borg having less impact on average on CPU usage while running. This limitation in borg is actually a direct consequence of the next point.
- Borg does actual deduplication, while restic only does classic incremental backups. With restic, you store a copy of every file, but the files are reference counted so that each version of a file only gets stored once. Borg, however, operates on blocks, not files, and deduplicates within individual backups. So if you have a dozen copies of the same data in your backup, restic stores each copy, but borg only stores the first and makes all the others references to that. The main benefit of this is that borg produces much smaller backups when you have lots of duplicate data and actually does more space efficient incremental backups (because it only stores what actually changed, not the whole changed file).
- Borg supports compression, while Restic seemingly does not (and doesn't handle sparse files very well either). This too has a huge impact on space efficiency, and may explain why restic is lightning fast on my systems when compared to borg. //
Austin I have to correct you:
Restic does indeed do deduplication on blocklevel. It uses a rolling hash algorithm called rabin as a chunker.
In short, a rolling hash algorithm reacts to patterns within the file and cuts it. If two files have the same patterns there's a high chance to have the cuts at the same positions, giving it the ability to deduplicate files which's data is not aligned to any specific block size.