Introducing rsync-time-machine.py - a Python port of the rsync-time-backup script, offering Time Machine-style backups using rsync. It creates incremental backups of files and directories to the destination of your choice. The backups are structured in a way that makes it easy to recover any file at any point in time. 🚀
It works on Linux, macOS, and Windows (via WSL or Cygwin). The main advantage over Time Machine is flexibility, as it can backup from/to any filesystem and works on any platform. You can also backup to a Truecrypt drive without any problem. 😃
rsync-time-machine.py is fully tested, has no external dependencies (only Python ≥3.7 🐍), is fully compatible with rsync-time-backup, offers pretty terminal output, and is fully typed! 🎉
[ToC] 📚
- 📁 Each backup is in its own folder named after the current timestamp.
- 🔒 Backup to/from remote destinations over SSH.
- 🔗 Files that haven't changed from one backup to the next are hard-linked to the previous backup, saving space.
⚠️ Safety check - the backup will only happen if the destination has explicitly been marked as a backup destination.- 🔄 Resume feature - if a backup has failed or was interrupted, the tool will resume from there on the next backup.
- 🚫 Exclude file - support for pattern-based exclusion via the
--exclude-fromrsync parameter. - 🧹 Automatically purge old backups based on a configurable expiration strategy.
- 🔗 "latest" symlink that points to the latest successful backup.
To use rsync-time-machine.py, you'll need to provide source and destination paths, along with any desired options:
rsync-time-machine --helpShows the help message:
usage: rsync-time-machine [-h] [-p PORT] [-i ID_RSA] [--rsync-get-flags]
[--rsync-set-flags RSYNC_SET_FLAGS]
[--rsync-append-flags RSYNC_APPEND_FLAGS]
[--log-dir LOG_DIR] [--strategy STRATEGY]
[--no-auto-expire] [--allow-host-only]
[--exclude-from EXCLUDE_FROM] [-v]
src_folder dest_folder [exclusion_file]
A script for creating and managing time-stamped backups using rsync.
positional arguments:
src_folder Source folder for backup. Format: [USER@HOST:]SOURCE
dest_folder Destination folder for backup. Format:
[USER@HOST:]DESTINATION
exclusion_file Path to the file containing exclude patterns. Cannot
be used together with `--exclude-from`.
options:
-h, --help show this help message and exit
-p PORT, --port PORT SSH port.
-i ID_RSA, --id_rsa ID_RSA
Specify the private ssh key to use.
--rsync-get-flags Display the default rsync flags that are used for
backup. If using remote drive over SSH, --compress
will be added.
--rsync-set-flags RSYNC_SET_FLAGS
Set the rsync flags that are going to be used for
backup.
--rsync-append-flags RSYNC_APPEND_FLAGS
Append the rsync flags that are going to be used for
backup.
--log-dir LOG_DIR Set the log file directory. If this flag is set,
generated files will not be managed by the script - in
particular they will not be automatically deleted.
Default: $HOME/.rsync-time-backup
--strategy STRATEGY Set the expiration strategy. Default: "1:1 30:7
365:30" means after one day, keep one backup per day.
After 30 days, keep one backup every 7 days. After 365
days keep one backup every 30 days.
--no-auto-expire Disable automatically deleting backups when out of
space. Instead, an error is logged, and the backup is
aborted.
--allow-host-only By default, the script expects a 'USER@HOST' pattern
for specifying SSH connections. When this flag is
used, it allows for the 'HOST' pattern without a
specified user. This is useful if you want to use
configurations from the `.ssh/config` file or rely on
the current username. Note: this option will not
enforce SSH usage, it only broadens the accepted input
formats.
--exclude-from EXCLUDE_FROM
Path to the file containing exclude patterns.
Alternative to the positional `exclusion_file`. Not to
be used with `exclusion_file`.
-v, --verbose Enable verbose output. This will slow down the backup
process (in simple tests by 2x).Please refer to the original rsync-time-backup README for a list of options, as they have been preserved in the Python port.
To install rsync-time-machine.py, simply clone the repository:
pip install rsync-time-machineand use it like rsync-time-machine --help
Or just copy the script to your computer:
wget https://raw.githubusercontent.com/basnijholt/rsync-time-machine.py/main/rsync_time_machine.pyand use it like ./rsync_time_machine.py --help
- Backup the home folder to backup_drive:
./rsync_time_machine.py /home /mnt/backup_drive
- Backup with exclusion list:
./rsync_time_machine.py /home /mnt/backup_drive excluded_patterns.txt
For more examples and detailed usage instructions, please refer to the original rsync-time-backup README.
Backup sets are automatically deleted following a simple expiration strategy defined with the --strategy flag. The default strategy is 1:1 30:7 365:30. Please see the original README for a detailed explanation.
An optional exclude file can be provided as a third parameter, compatible with the --exclude-from parameter of rsync.
The --exclude-from option in rsync-time-machine.py allows you to exclude specific files or directories from the backup process. You can provide an exclusion file containing patterns for files or directories that should be excluded.
📖🔽 Click here to expand the docs on --exclude-from 🔽📖
Here's how to use the --exclude-from feature in rsync-time-machine.py:
- Create a text file named
exclusion_file.txt(or any other name you prefer) in your preferred location. - Add the exclusion patterns to the file, one pattern per line. Patterns can be literal strings, wildcards, or character ranges.
- Save the file.
To use this exclusion file while performing a backup with rsync-time-machine.py, include it as the third positional argument in your command (or with --exclude-from exclusion_file.txt). For example:
rsync-time-machine.py /home /mnt/backup_drive exclusion_file.txtIn this example, /home is the source folder, /mnt/backup_drive is the destination folder, and exclusion_file.txt contains the exclude patterns.
Here's a sample exclusion_file.txt:
+ /home/.fileA
- /home/.*
- /home/junk/
In this example:
+ /home/.fileA: Include the file.fileAfrom thehomedirectory.- /home/.*: Exclude all hidden files (files starting with a dot) from thehomedirectory.- /home/junk/: Exclude the entirejunkdirectory from thehomedirectory.
Remember that the order of patterns matters, as rsync reads the file top-down and acts on the first matching pattern it encounters.
See this tutorial for more information.
The script is designed so that only one backup operation can be active for a given directory, avoiding conflicts.
To display, add, or remove rsync options, use the --rsync-get-flags, --rsync-append-flags, or --rsync-set-flags options.
Use the --no-auto-expire flag to disable the default behavior of purging old backups when out of space.
Restoring files from the backup is simple, as the script creates a backup in a regular directory. You can easily copy the files back to the original directory using a command like:
rsync -aP /path/to/last/backup/ /path/to/restore/to/
Consider using the --dry-run option to check what exactly is going to be copied. If you want to delete files that exist in the destination but not in the backup, use the --delete option. Be extra cautious when using this option to avoid data loss.
You can also restore files using any file explorer, including Finder on macOS or the command line.
- the Real Python podcast: Episode 158: Building Python CI With Docker & Applying for a Hacker Initiative Grant @ 00:26:28
- Y Combinator Hacker News: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups (self-posted)
- Reddit /rpython: Ported a popular (untested) 600+ Line Bash Script 📜 to Python 🐍: Introducing rsync-time-machine.py for Time Machine-Style Backups Using Rsync 🔄⏰ (self-posted)
We appreciate your feedback and contributions! If you encounter any issues or have suggestions for improvements, please file an issue on the GitHub repository. We also welcome pull requests for bug fixes or new features.
Happy backing up! 💾🕰️🎉