Compare commits

...
85 Commits
Author SHA1 Message Date
Dan Helfman 7b77fd2510 Fix compatibility issue between pykwalify and ruamel.yaml 0.15.52 (#85). 2018-08-11 13:59:27 -07:00
Dan Helfman ece5608677 Rev for release. 2018-07-28 22:27:39 -07:00
Dan Helfman 4644f613b2 Fix typo in README. 2018-07-28 22:24:24 -07:00
Dan Helfman 3afa5ac76d Document hooks (#81). 2018-07-28 22:22:25 -07:00
Dan Helfman 27f8a1df04 Switch to non-raw link to sample cron job. 2018-07-28 20:29:55 -07:00
Dan Helfman 8e5b0bbf17 Remove errant ctrl-F character from docs. 2018-07-28 20:27:18 -07:00
Dan Helfman 282e9565c9 Mentioning new --info --json option in NEWS. 2018-07-28 20:24:19 -07:00
Thomas LÉVEILandGitea b714ffd48b add support for --info --json (#83) 2018-07-29 03:17:45 +00:00
Dan Helfman 9968a15ef8 Clarifying code style for multiline constructs. 2018-07-28 15:21:19 -07:00
Dan Helfman d93da55ce9 Add code style guidelines to the documention, and reformat some code accordingly. 2018-07-28 15:02:17 -07:00
Thomas LÉVEILandGitea 789bcd402a add support for --list --json (#74) 2018-07-28 21:21:38 +00:00
Dan HelfmanandGitea cf6ab60d2e Use XDG_CONFIG_HOME for user configuration directory, if set. (Thanks to floli.) (#71)
Thanks! This will go out in the next release.
2018-07-25 01:34:05 +00:00
Dan Helfman 64364b20ff Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files, editor swap files, etc. (#77) 2018-07-22 12:08:49 -07:00
Dan Helfman d29c7956bc Upgrade ruamel.yaml compatibility version range and fix support for Python 3.7 (#38, #76). 2018-07-22 11:25:06 -07:00
Dan Helfman e5ef485d6b Merge branch 'master' of ssh://projects.torsion.org:3022/witten/borgmatic 2018-07-01 14:54:15 -07:00
Dan Helfman fc8046edc4 Adding NEWS item about skipping before/after backup hooks. 2018-07-01 14:51:57 -07:00
Dan HelfmanandGitea 4538017206 Merge branch 'fix-72-hooks-are-executed-when-list-or-info' of thomasleveil/borgmatic into master
Thanks for fixing this!

I agree about more specific hooks if and when `--check` or `--prune`-specific hook use cases are needed. I think what you've done here is fine until then.
2018-07-01 21:47:39 +00:00
Thomas LEVEIL d664b6d253 only run hooks when creating an archive
fix #72
2018-07-01 21:09:45 +02:00
Dan Helfman f42aa0a6f2 Revving version for development. 2018-06-17 15:26:53 -07:00
Dan Helfman 9d4ba66f6e Revving version for release. 2018-06-17 15:14:45 -07:00
Dan Helfman cf846ab8ac Support for Borg prune --umask option (#69). 2018-06-17 15:12:43 -07:00
Dan Helfman 219e287c6c Document how to develop on and contribute to borgmatic. 2018-06-17 14:55:57 -07:00
Dan Helfman dede8f9d4b News for: ~/.config/borgmatic/config.yaml. 2018-06-17 14:30:47 -07:00
Dan HelfmanandGitea 7a1e3f5639 Merge branch 'add_user_config' of floli/borgmatic into master 2018-06-17 21:26:36 +00:00
Florian Lindner 9bd77292ff Add default path for user configuration 2018-06-10 15:03:23 +02:00
Dan Helfman f1a143de5b Adding list and info Borg sub-commands to find-unsupported-borg-options script. 2018-05-26 20:53:03 -07:00
Dan Helfman 765e343c71 Support for Borg --info via borgmatic command-line (#61). 2018-05-26 16:19:05 -07:00
Dan Helfman af4b91a048 Support for Borg --list option via borgmatic command-line to list all archives (#61). 2018-05-26 16:09:08 -07:00
Dan Helfman cc9044487b Support for Borg --nobsdflags option to skip recording bsdflags (e.g. NODUMP, IMMUTABLE) in archive (#63). 2018-05-26 15:09:23 -07:00
Dan Helfman 11c30001c3 Add "Persistent" flag to systemd timer example. (#60) 2018-05-20 22:20:21 -07:00
Dan HelfmanandGitea ac9161035a Merge branch 'master' of floli/borgmatic into master
Thanks for taking the time to add this! Makes sense.
2018-05-21 05:18:13 +00:00
Dan Helfman 007ec0644c Ignore "check_last" and consistency "prefix" when "archives" not in consistency checks. (#59) 2018-05-20 22:11:40 -07:00
Dan Helfman 1db808fb3d Link to OpenBSD port of borgmatic. 2018-05-19 16:16:54 -07:00
Dan Helfman 76656275c3 Update README to mention other ways of installing borgmatic. (#62) 2018-05-19 16:06:54 -07:00
Florian Lindner 64bdbc4bf0 Add Persistent, so that the timer is triggered if missed last time. 2018-05-17 21:47:58 +02:00
Dan Helfman 61033bb4e5 Update tox.ini to only assume Python 3.x instead of Python 3.4 specifically. 2018-04-09 20:34:59 -07:00
Dan Helfman e608b7924a Adding note about executable location. 2018-04-08 12:06:15 -07:00
Dan Helfman f7f852a28b Fix tests broken by addition of check --prefix default. 2018-03-03 22:36:51 -08:00
Dan Helfman 9b9c4c4abb Clarifying note in schema about adding prefix to consistency section. 2018-03-03 22:33:34 -08:00
Dan Helfman 1b59f5b190 Changing version in warning to correspond with next release version. 2018-03-03 22:30:30 -08:00
Dan Helfman 65ab230961 Noting new Borg check --prefix feature in release notes. 2018-03-03 22:21:48 -08:00
Nick WhyteandDan Helfman c64d0100d5 Only check archives with matching prefix. 2018-03-03 22:17:39 -08:00
Dan Helfman 0112407250 Add introductory screencast link to documentation. 2018-02-19 17:44:20 -08:00
Dan Helfman 2d3f5fa05d Support for Borg --lock-wait option for the maximum wait for a repository/cache lock (#56). 2018-02-19 15:51:04 -08:00
Dan Helfman a87036ee46 Support for using tilde in exclude_patterns to reference home directory (#58). 2018-02-18 15:34:19 -08:00
Dan Helfman a72f5ff69a Tests for --dry-run + --verbosity fix. 2018-02-18 14:26:51 -08:00
newtonneandDan Helfman bb99009191 Fix issue when using both --dry-run and -v options. 2018-02-18 14:18:25 -08:00
Dan Helfman 4c45d60529 Mentioning ssh_command for additional SSH configuration.. 2018-02-18 13:39:05 -08:00
Dan Helfman 2211f959ae AUTHORS addition for recent encryption pass command changes. 2018-01-18 21:13:20 -08:00
Dan Helfman cc1d6f53a0 55: Fix for missing tags/releases from Gitea and GitHub project hosting. 2018-01-17 20:27:09 -08:00
Dan Helfman 389778c716 Adding BORG_PASSCOMMAND update to NEWS. 2018-01-16 21:05:53 -08:00
newtonneandDan Helfman e55e9e8139 Add encryption_passcommand configuration option 2018-01-16 21:03:25 -08:00
Dan Helfman ef76e87477 Bumping version for release. 2018-01-15 20:55:49 -08:00
Dan Helfman 62526038d6 47: Support for Borg --dry-run option via borgmatic command-line. 2018-01-15 20:55:27 -08:00
Dan Helfman bf2f39623e 49: Rename incorrect --pattern-from option to correct --patterns-from. 2018-01-15 20:22:53 -08:00
Dan Helfman 28c890a52d Bumping version for release. 2018-01-14 16:37:02 -08:00
Dan Helfman cd189c4fe4 48: Add "local_path" to configuration for specifying an alternative Borg executable path. 2018-01-14 16:35:24 -08:00
Dan Helfman b8f6bab12d 49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed includes/excludes. 2018-01-14 15:52:19 -08:00
Dan Helfman 50b3240c4f 54: Fix for incorrect consistency check flags passed to Borg when all three checks in borgmatic config. 2018-01-14 14:09:20 -08:00
Dan Helfman 18fbc75e16 Revising history to account for off-by-one error when importing issue numbers into Gitea. 2018-01-04 21:49:43 -08:00
Dan Helfman 0881da4a82 New issue tracker. 2018-01-04 21:37:43 -08:00
Dan fa210766a2 Update for release. 2018-01-02 20:36:52 -08:00
Dan d4f52e3137 Update AUTHORS with recent changes + sort. 2018-01-02 20:33:27 -08:00
Dan 8b2ebdc5f7 Simplifying example. 2018-01-02 20:31:06 -08:00
Dan HelfmanandGitea a00407256d Merge branch 'keep_minutely' of thomasleveil/borgmatic into master 2018-01-03 04:30:13 +00:00
Thomas LEVEIL 24b5eccefc add support for Borg --keep-minutely prune option 2018-01-03 00:13:44 +01:00
Dan 815fb39a05 Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version" rule errors. 2017-11-26 10:30:31 -08:00
Dan 24c196d2a4 Script to find unsupported Borg options in borgmatic, to assist with #13. 2017-11-11 16:07:54 -08:00
Dan 3e26e70d0c Fix for incorrect /etc/borgmatic.d/ configuration path probing on macOS. 2017-11-10 21:33:29 -08:00
Dan 5ce25e2790 Re-fixing logo image. 2017-11-04 11:55:48 -07:00
Dan 8243552c8c Fixing PNG path. 2017-11-04 11:54:29 -07:00
Dan 425e27dee5 Add "ssh_command" to configuration for specifying a custom SSH command or options. 2017-11-03 22:01:04 -07:00
Dan 9ec9269a18 Link to repository encryption section of Borg Quick Start. 2017-11-03 20:28:31 -07:00
Dan bf5cbd1deb Mentioning use of BORG_PASSPHRASE environment variable. 2017-11-03 20:27:21 -07:00
Dan 4c09cbf1a4 Releasing. 2017-11-02 22:38:16 -07:00
Dan fc077af4ce Mentioning borgmatic --config option in docs. 2017-11-02 22:28:53 -07:00
Dan ca4312bb85 Support for Borg --remote-ratelimit for limiting upload rate. And log Borg commands. 2017-11-02 22:22:40 -07:00
Dan fc3b1fccba Support for Borg --files-cache option for setting the files cache operation mode. 2017-11-02 22:03:11 -07:00
Dan f83346b9b3 Support for using tilde in repository paths to reference home directory. 2017-11-02 21:34:04 -07:00
Dan 63c7241aec Typo in comment. 2017-10-31 22:21:49 -07:00
Dan fd77dc579e Pass through several more Unix signals that Borg happens to consume. 2017-10-31 22:10:00 -07:00
Dan f017ed648f Disabling code coverage on this one-line functions. 2017-10-31 22:01:18 -07:00
Dan 27a6745743 Passing the Unix SIGTERM signal through to child processes like Borg. 2017-10-31 21:58:35 -07:00
Dan 95be0c8e46 Removing broken download URL. 2017-10-29 21:42:01 -07:00
Dan 17a774ba7e Removing .hg* files. 2017-10-29 20:32:21 -07:00
38 changed files with 1748 additions and 302 deletions
+1
View File
@@ -3,6 +3,7 @@
*.swp
.cache
.coverage
.pytest_cache
.tox
build
dist
-9
View File
@@ -1,9 +0,0 @@
syntax: glob
*.egg-info
*.pyc
*.swp
.cache
.coverage
.tox
build
dist
-40
View File
@@ -1,40 +0,0 @@
467d3a3ce9185e84ee51ca9156499162efd94f9a 0.0.2
7730ae34665c0dedf46deab90b32780abf6dbaff 0.0.3
4bb2e81fc77038be4499b7ea6797ab7d109460e0 0.0.4
b31d51b633701554e84f996cc0c73bad2990780b 0.0.5
b31d51b633701554e84f996cc0c73bad2990780b 0.0.5
aa8a807f4ba28f0652764ed14713ffea2fd6922d 0.0.5
aa8a807f4ba28f0652764ed14713ffea2fd6922d 0.0.5
569aef47a9b25c55b13753f94706f5d330219995 0.0.5
569aef47a9b25c55b13753f94706f5d330219995 0.0.5
a03495a8e8b471da63b5e2ae79d3ff9065839c2a 0.0.5
7ea93ca83f426ec0a608a68580c72c0775b81f86 0.0.6
cf4c7065f0711deda1cba878398bc05390e2c3f9 0.0.7
38d72677343f0a5d6845f4ac50d6778397083d45 0.1.0
ac5dfa01e9d14d09845f5e94c2c679e21c5eb2f9 0.1.1
ac5dfa01e9d14d09845f5e94c2c679e21c5eb2f9 0.1.1
7b6c87dca7ea312b2257ac1b46857b3f8c56b39c 0.1.1
83067f995dd391e38544a7722dc3b254b59c5521 0.1.2
acc7fb61566fe8028c179f43ecc735c851220b06 0.1.3
6dda59c12de88f060eb7244e6d330173985a9639 0.1.4
6dda59c12de88f060eb7244e6d330173985a9639 0.1.4
e58246fc92bb22c2b2fd8b86a1227de69d2d0315 0.1.4
0afff209b902698c2266986129d6dc9f5f913101 0.1.5
4c63f3d90ec2bf6af1714a3acec84654a7c9edf3 0.1.6
5a458ebef804be14e30d7375e3e9fbc26aedb80d 0.1.7
977f19c2f6a515be6c5ef69cf17b0e0989532209 github/yaml_config_files
0000000000000000000000000000000000000000 github/yaml_config_files
28434dd0440cc8da44c2f3e9bd7e9402a59c3b40 github/master
dbc96d3f83bd5570b6826537616d4160b3374836 0.1.8
0e1fbee9358de4f062fa9539e1355db83db70caa 1.0.0
de2d7721cdec93a52d20222a9ddd579ed93c1017 1.0.1
9603d13910b32d57a887765cab694ac5d0acc1f4 1.0.2
32c6341dda9fad77a3982641bce8a3a45821842e 1.0.3
5a003056a8ff4709c5bd4d6d33354199423f8a1d 1.1.0
7d3d11eff6c0773883c48f221431f157bc7995eb 1.1.1
f052a77a8ad5a0fea7fa86a902e0e401252f7d80 1.1.2
3f838f661546e04529b453aa443529b432afc243 1.1.3
3d605962d891731a0f372b903b556ac7a8c8359f 1.1.4
64ca13bfe050f656b44ed2eb1c3db045bfddd133 1.1.5
4daa944c122c572b9b56bfcac3f4e2869181c630 1.1.6
ec7949a14a2051616f7cdcb8e05555f02e024ae8 1.1.7
+4 -1
View File
@@ -2,7 +2,10 @@ Dan Helfman <witten@torsion.org>: Main developer
Alexander Görtz: Python 3 compatibility
Henning Schroeder: Copy editing
Johannes Feichtner: Support for user hooks
Michele Lazzeri: Custom archive names
newtonne: Read encryption password from external file
Robin `ypid` Schneider: Support additional options of Borg
Scott Squires: Custom archive names
Johannes Feichtner: Support for user hooks
Thomas LÉVEIL: Support for a keep_minutely prune option. Support for the --json option
Nick Whyte: Support prefix filtering for archive consistency checks
+92 -20
View File
@@ -1,7 +1,79 @@
1.2.2
* #85: Fix compatibility issue between pykwalify and ruamel.yaml 0.15.52, which manifested in
borgmatic as a pykwalify RuleError.
1.2.1
* Skip before/after backup hooks when only doing --prune, --check, --list, and/or --info.
* #71: Support for XDG_CONFIG_HOME environment variable for specifying alternate user ~/.config/
path.
* #74, #83: Support for Borg --json option via borgmatic command-line to --list archives or show
archive --info in JSON format, ideal for programmatic consumption.
* #38, #76: Upgrade ruamel.yaml compatibility version range and fix support for Python 3.7.
* #77: Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files,
editor swap files, etc.
* #81: Document user-defined hooks run before/after backup, or on error.
* Add code style guidelines to the documention.
1.2.0
* #61: Support for Borg --list option via borgmatic command-line to list all archives.
* #61: Support for Borg --info option via borgmatic command-line to display summary information.
* #62: Update README to mention other ways of installing borgmatic.
* Support for Borg --prefix option for consistency checks via "prefix" option in borgmatic's
consistency configuration.
* Add introductory screencast link to documentation.
* #59: Ignore "check_last" and consistency "prefix" when "archives" not in consistency checks.
* #60: Add "Persistent" flag to systemd timer example.
* #63: Support for Borg --nobsdflags option to skip recording bsdflags (e.g. NODUMP, IMMUTABLE) in
archive.
* #69: Support for Borg prune --umask option using value of existing "umask" option in borgmatic's
storage configuration.
* Update tox.ini to only assume Python 3.x instead of Python 3.4 specifically.
* Add ~/.config/borgmatic/config.yaml to default configuration path probing.
* Document how to develop on and contribute to borgmatic.
1.1.15
* Support for Borg BORG_PASSCOMMAND environment variable to read a password from an external file.
* Fix for Borg create error when using borgmatic's --dry-run and --verbosity options together.
Work-around for behavior introduced in Borg 1.1.3: https://github.com/borgbackup/borg/issues/3298
* #55: Fix for missing tags/releases on Gitea and GitHub project hosting.
* #56: Support for Borg --lock-wait option for the maximum wait for a repository/cache lock.
* #58: Support for using tilde in exclude_patterns to reference home directory.
1.1.14
* #49: Fix for typo in --patterns-from option.
* #47: Support for Borg --dry-run option via borgmatic command-line.
1.1.13
* #54: Fix for incorrect consistency check flags passed to Borg when all three checks ("repository",
"archives", and "extract") are specified in borgmatic configuration.
* #48: Add "local_path" to configuration for specifying an alternative Borg executable path.
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
includes/excludes.
* Moved issue tracker from Taiga to integrated Gitea tracker at
https://projects.torsion.org/witten/borgmatic/issues
1.1.12
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"
rule errors.
* Support for Borg --keep-minutely prune option.
1.1.11
* #26: Add "ssh_command" to configuration for specifying a custom SSH command or options.
* Fix for incorrect /etc/borgmatic.d/ configuration path probing on macOS. This problem manifested
as an error on startup: "[Errno 2] No such file or directory: '/etc/borgmatic.d'".
1.1.10
* Pass several Unix signals through to child processes like Borg. This means that Borg now properly
shuts down if borgmatic is terminated (e.g. due to a system suspend).
* #30: Support for using tilde in repository paths to reference home directory.
* #43: Support for Borg --files-cache option for setting the files cache operation mode.
* #45: Support for Borg --remote-ratelimit option for limiting upload rate.
* Log invoked Borg commands when at highest verbosity level.
1.1.9
* #16, #38: Support for user-defined hooks before/after backup, or on error.
* #33: Improve clarity of logging spew at high verbosity levels.
* #29: Support for using tilde in source directory path to reference home directory.
* #17, #39: Support for user-defined hooks before/after backup, or on error.
* #34: Improve clarity of logging spew at high verbosity levels.
* #30: Support for using tilde in source directory path to reference home directory.
* Require "prefix" in retention section when "archive_name_format" is set. This is to avoid
accidental pruning of archives with a different archive name format. For similar reasons, default
"prefix" to "{hostname}-" if not specified.
@@ -9,43 +81,43 @@
* Update dead links to Borg documentation.
1.1.8
* #39: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default
* #40: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default
config paths.
1.1.7
* #28: Add "archive_name_format" to configuration for customizing archive names.
* #29: Add "archive_name_format" to configuration for customizing archive names.
* Fix for traceback when "exclude_from" value is empty in configuration file.
* When pruning, make highest verbosity level list archives kept and pruned.
* Clarification of Python 3 pip usage in documentation.
1.1.6
* #12, #35: Support for Borg --exclude-from, --exclude-caches, and --exclude-if-present options.
* #13, #36: Support for Borg --exclude-from, --exclude-caches, and --exclude-if-present options.
1.1.5
* #34: New "extract" consistency check that performs a dry-run extraction of the most recent
* #35: New "extract" consistency check that performs a dry-run extraction of the most recent
archive.
1.1.4
* #17: Added command-line flags for performing a borgmatic run with only pruning, creating, or
* #18: Added command-line flags for performing a borgmatic run with only pruning, creating, or
checking enabled. This supports use cases like running consistency checks from a different cron
job with a different frequency, or running pruning with a different verbosity level.
1.1.3
* #14: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run.
* #15: Support for running multiple config files in /etc/borgmatic.d/ from a single borgmatic run.
* Fix for generate-borgmatic-config writing config with invalid one_file_system value.
1.1.2
* #32: Fix for passing check_last as integer to subprocess when calling Borg.
* #33: Fix for passing check_last as integer to subprocess when calling Borg.
1.1.1
* Part of #32: Fix for upgrade-borgmatic-config converting check_last option as a string instead of
* Part of #33: Fix for upgrade-borgmatic-config converting check_last option as a string instead of
an integer.
* Fix for upgrade-borgmatic-config erroring when consistency checks option is not present.
@@ -54,8 +126,8 @@
* Switched config file format to YAML. Run upgrade-borgmatic-config to upgrade.
* Added generate-borgmatic-config command for initial config creation.
* Dropped Python 2 support. Now Python 3 only.
* #18: Fix for README mention of sample files not included in package.
* #22: Sample files for triggering borgmatic from a systemd timer.
* #19: Fix for README mention of sample files not included in package.
* #23: Sample files for triggering borgmatic from a systemd timer.
* Support for backing up to multiple repositories.
* To free up space, now pruning backups prior to creating a new backup.
* Enabled test coverage output during tox runs.
@@ -63,15 +135,15 @@
1.0.3
* #21: Fix for verbosity flag not actually causing verbose output.
* #22: Fix for verbosity flag not actually causing verbose output.
1.0.2
* #20: Fix for traceback when remote_path option is missing.
* #21: Fix for traceback when remote_path option is missing.
1.0.1
* #19: Support for Borg's --remote-path option to use an alternate Borg
* #20: Support for Borg's --remote-path option to use an alternate Borg
executable. See sample/config.
1.0.0
@@ -93,13 +165,13 @@
0.1.7
* #11: Fixed parsing of punctuation in configuration file.
* #12: Fixed parsing of punctuation in configuration file.
* Better error message when configuration file is missing.
0.1.6
* #9: New configuration option for the encryption passphrase.
* #10: Support for Borg's new archive compression feature.
* #10: New configuration option for the encryption passphrase.
* #11: Support for Borg's new archive compression feature.
0.1.5
@@ -111,7 +183,7 @@
0.1.3
* #1: Add support for "borg check --last N" to Borg backend.
* #2: Add support for "borg check --last N" to Borg backend.
0.1.2
+137 -26
View File
@@ -1,4 +1,4 @@
<img src="https://projects.torsion.org/witten/borgmatic/raw/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
<img src="https://projects.torsion.org/witten/borgmatic/raw/branch/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
## Overview
@@ -44,16 +44,26 @@ borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
available](https://projects.torsion.org/witten/borgmatic). It's also mirrored
on [GitHub](https://github.com/witten/borgmatic) for convenience.
<a href="https://asciinema.org/a/164143" target="_blank"><img src="https://asciinema.org/a/164143.png" width="100%" /></a>
## Installation
To get up and running, follow the [Borg Quick
Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create
a repository on a local or remote host. Note that if you plan to run
borgmatic on a schedule with cron, and you encrypt your Borg repository with
a passphrase instead of a key file, you'll need to set the borgmatic
`encryption_passphrase` configuration variable. See the repository encryption
section of the Quick Start for more info.
a repository on a local or remote host. Note that if you plan to run borgmatic
on a schedule with cron, and you encrypt your Borg repository with a
passphrase instead of a key file, you'll either need to set the borgmatic
`encryption_passphrase` configuration variable or set the `BORG_PASSPHRASE`
environment variable. See the [repository encryption
section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
of the Quick Start for more info.
Alternatively, the passphrase can be specified programatically by setting
either the borgmatic `encryption_passcommand` configuration variable or the
`BORG_PASSCOMMAND` environment variable. See the [Borg Security
FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
for more info.
If the repository is on a remote host, make sure that your local root user has
key-based ssh access to the desired user account on the remote host.
@@ -67,11 +77,18 @@ sudo pip3 install --upgrade borgmatic
Note that your pip binary may have a different name than "pip3". Make sure
you're using Python 3, as borgmatic does not support Python 2.
### Docker
### Other ways to install
If you would like to run borgmatic within Docker, please take a look at
[b3vis/borgmatic](https://hub.docker.com/r/b3vis/borgmatic/) for more
information.
* [A borgmatic Docker image](https://hub.docker.com/r/b3vis/borgmatic/) based
on Alpine Linux.
* [Another borgmatic Docker image](https://hub.docker.com/r/coaxial/borgmatic/) based
on Alpine Linux, updated automatically whenever the Alpine image updates.
* [A borgmatic package for
Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic).
* [A borgmatic package for Arch
Linux](https://aur.archlinux.org/packages/borgmatic/).
* [A borgmatic package for OpenBSD](http://ports.su/sysutils/borgmatic).
<br><br>
## Configuration
@@ -81,6 +98,9 @@ After you install borgmatic, generate a sample configuration file:
sudo generate-borgmatic-config
```
If that command is not found, then it may be installed in a location that's
not in your system `PATH`. Try looking in `/usr/local/bin/`.
This generates a sample configuration file at /etc/borgmatic/config.yaml (by
default). You should edit the file to suit your needs, as the values are just
representative. All fields are optional except where indicated, so feel free
@@ -113,6 +133,35 @@ When you set up multiple configuration files like this, borgmatic will run
each one in turn from a single borgmatic invocation. This includes, by
default, the traditional /etc/borgmatic/config.yaml as well.
And if you need even more customizability, you can specify alternate
configuration paths on the command-line with borgmatic's `--config` option.
See `borgmatic --help` for more information.
### Hooks
If you find yourself performing prepraration tasks before your backup runs, or
cleanup work afterwards, borgmatic hooks may be of interest. They're simply
shell commands that borgmatic executes for you at various points, and they're
configured in the `hooks` section of your configuration file.
For instance, you can specify `before_backup` hooks to dump a database to file
before backing it up, and specify `after_backup` hooks to delete the temporary
file afterwards.
borgmatic hooks run once per configuration file. `before_backup` hooks run
prior to backups of all repositories. `after_backup` hooks run afterwards, but
not if an error occurs in a previous hook or in the backups themselves. And
borgmatic runs `on_error` hooks if an error occurs.
An important security note about hooks: borgmatic executes all hook commands
with the user permissions of borgmatic itself. So to prevent potential shell
injection or privilege escalation, do not forget to set secure permissions
(chmod 0700) on borgmatic configuration files and scripts invoked by hooks.
See the sample generated configuration file mentioned above for specifics
about hook configuration syntax.
## Upgrading
@@ -251,7 +300,7 @@ configure a job runner to invoke it periodically.
### cron
If you're using cron, download the [sample cron
file](https://projects.torsion.org/witten/borgmatic/raw/master/sample/cron/borgmatic).
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
Then, from the directory where you downloaded it:
```bash
@@ -280,17 +329,89 @@ Feel free to modify the timer file based on how frequently you'd like
borgmatic to run.
## Running tests
## Support and contributing
First install tox, which is used for setting up testing environments:
### Issues
```bash
pip3 install tox
You've got issues? Or an idea for a feature enhancement? We've got an [issue
tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to
create a new issue or comment on an issue, you'll need to [login
first](https://projects.torsion.org/user/login). Note that you can login with
an existing GitHub account if you prefer.
Other questions or comments? Contact <mailto:witten@torsion.org>.
### Contributing
If you'd like to contribute to borgmatic development, please feel free to
submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
your thing. In general, contributions are very welcome. We don't bite!
### Code style
Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
the following deviations from it:
* For strings, prefer single quotes over double quotes.
* Limit all lines to a maximum of 100 characters.
* Use trailing commas within multiline values or argument lists.
* For multiline constructs, put opening and closing delimeters on lines
separate from their contents.
* Within multiline constructs, use standard four-space indentation. Don't align
indentation with an opening delimeter.
### Development
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
```
git clone https://projects.torsion.org/witten/borgmatic.git
```
Then, to actually run tests, run:
Or:
```bash
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
```
Then, install borgmatic
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
so that you can easily run borgmatic commands while you're hacking on them to
make sure your changes work.
```bash
cd borgmatic/
pip3 install --editable --user .
```
Note that this will typically install the borgmatic commands into
`~/.local/bin`, which may or may not be on your PATH. There are other ways to
install borgmatic editable as well, for instance into the system Python
install (so without `--user`, as root), or even into a
[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
borgmatic is up to you, but generally an editable install makes development
and testing easier.
### Running tests
Assuming you've cloned the borgmatic source code as described above, and
you're in the `borgmatic/` working copy, install tox, which is used for
setting up testing environments:
```bash
sudo pip3 install tox
```
Finally, to actually run tests, run:
```bash
cd borgmatic
tox
```
@@ -329,13 +450,3 @@ YAML library. If so, not to worry. borgmatic should install and function
correctly even without the C YAML library. And borgmatic won't be any faster
with the C library present, so you don't need to go out of your way to install
it.
## Issues and feedback
Got an issue or an idea for a feature enhancement? Check out the [borgmatic
issue tracker](https://tree.taiga.io/project/witten-borgmatic/issues?page=1&status=399951,399952,399955). In
order to create a new issue or comment on an issue, you'll need to [login
first](https://tree.taiga.io/login).
Other questions or comments? Contact <mailto:witten@torsion.org>.
+39 -12
View File
@@ -1,3 +1,4 @@
import logging
import os
import subprocess
@@ -6,6 +7,10 @@ from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
DEFAULT_CHECKS = ('repository', 'archives')
DEFAULT_PREFIX = '{hostname}-'
logger = logging.getLogger(__name__)
def _parse_checks(consistency_config):
@@ -30,7 +35,7 @@ def _parse_checks(consistency_config):
return tuple(check for check in checks if check.lower() not in ('disabled', '')) or DEFAULT_CHECKS
def _make_check_flags(checks, check_last=None):
def _make_check_flags(checks, check_last=None, prefix=None):
'''
Given a parsed sequence of checks, transform it into tuple of command-line flags.
@@ -42,44 +47,66 @@ def _make_check_flags(checks, check_last=None):
('--repository-only',)
Additionally, if a check_last value is given, a "--last" flag will be added.
However, if both "repository" and "archives" are in checks, then omit them from the returned
flags because Borg does both checks by default.
Additionally, if a check_last value is given and "archives" is in checks, then include a
"--last" flag. And if a prefix value is given and "archives" is in checks, then include a
"--prefix" flag.
'''
last_flag = ('--last', str(check_last)) if check_last else ()
if checks == DEFAULT_CHECKS:
return last_flag
if 'archives' in checks:
last_flags = ('--last', str(check_last)) if check_last else ()
prefix_flags = ('--prefix', prefix) if prefix else ('--prefix', DEFAULT_PREFIX)
else:
last_flags = ()
prefix_flags = ()
if check_last:
logger.warn('Ignoring check_last option, as "archives" is not in consistency checks.')
if prefix:
logger.warn('Ignoring consistency prefix option, as "archives" is not in consistency checks.')
if set(DEFAULT_CHECKS).issubset(set(checks)):
return last_flags + prefix_flags
return tuple(
'--{}-only'.format(check) for check in checks
if check in DEFAULT_CHECKS
) + last_flag
) + last_flags + prefix_flags
def check_archives(verbosity, repository, consistency_config, remote_path=None):
def check_archives(verbosity, repository, storage_config, consistency_config, local_path='borg',
remote_path=None):
'''
Given a verbosity flag, a local or remote repository path, a consistency config dict, and a
command to run, check the contained Borg archives for consistency.
Given a verbosity flag, a local or remote repository path, a storage config dict, a consistency
config dict, and a local/remote commands to run, check the contained Borg archives for
consistency.
If there are no consistency checks to run, skip running them.
'''
checks = _parse_checks(consistency_config)
check_last = consistency_config.get('check_last', None)
lock_wait = None
if set(checks).intersection(set(DEFAULT_CHECKS)):
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
lock_wait = storage_config.get('lock_wait', None)
lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
verbosity_flags = {
VERBOSITY_SOME: ('--info',),
VERBOSITY_LOTS: ('--debug',),
}.get(verbosity, ())
prefix = consistency_config.get('prefix')
full_command = (
'borg', 'check',
local_path, 'check',
repository,
) + _make_check_flags(checks, check_last) + remote_path_flags + verbosity_flags
) + _make_check_flags(checks, check_last, prefix) + remote_path_flags + lock_wait_flags + verbosity_flags
# The check command spews to stdout/stderr even without the verbose flag. Suppress it.
stdout = None if verbosity_flags else open(os.devnull, 'w')
logger.debug(' '.join(full_command))
subprocess.check_call(full_command, stdout=stdout, stderr=subprocess.STDOUT)
if 'extract' in checks:
extract.extract_last_archive_dry_run(verbosity, repository, remote_path)
extract.extract_last_archive_dry_run(verbosity, repository, lock_wait, local_path, remote_path)
+96 -42
View File
@@ -1,5 +1,6 @@
import glob
import itertools
import logging
import os
import subprocess
import tempfile
@@ -7,12 +8,22 @@ import tempfile
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def initialize(storage_config):
passphrase = storage_config.get('encryption_passphrase')
logger = logging.getLogger(__name__)
def initialize_environment(storage_config):
passcommand = storage_config.get('encryption_passcommand')
if passcommand:
os.environ['BORG_PASSCOMMAND'] = passcommand
passphrase = storage_config.get('encryption_passphrase')
if passphrase:
os.environ['BORG_PASSPHRASE'] = passphrase
ssh_command = storage_config.get('ssh_command')
if ssh_command:
os.environ['BORG_RSH'] = ssh_command
def _expand_directory(directory):
'''
@@ -24,28 +35,61 @@ def _expand_directory(directory):
return glob.glob(expanded_directory) or [expanded_directory]
def _write_exclude_file(exclude_patterns=None):
def _expand_directories(directories):
'''
Given a sequence of exclude patterns, write them to a named temporary file and return it. Return
None if no patterns are provided.
Given a sequence of directory paths, expand tildes and globs in each one. Return all the
resulting directories as a single flattened tuple.
'''
if not exclude_patterns:
if directories is None:
return ()
return tuple(
itertools.chain.from_iterable(
_expand_directory(directory)
for directory in directories
)
)
def _write_pattern_file(patterns=None):
'''
Given a sequence of patterns, write them to a named temporary file and return it. Return None
if no patterns are provided.
'''
if not patterns:
return None
exclude_file = tempfile.NamedTemporaryFile('w')
exclude_file.write('\n'.join(exclude_patterns))
exclude_file.flush()
pattern_file = tempfile.NamedTemporaryFile('w')
pattern_file.write('\n'.join(patterns))
pattern_file.flush()
return exclude_file
return pattern_file
def _make_exclude_flags(location_config, exclude_patterns_filename=None):
def _make_pattern_flags(location_config, pattern_filename=None):
'''
Given a location config dict with a potential pattern_from option, and a filename containing any
additional patterns, return the corresponding Borg flags for those files as a tuple.
'''
pattern_filenames = tuple(location_config.get('patterns_from') or ()) + (
(pattern_filename,) if pattern_filename else ()
)
return tuple(
itertools.chain.from_iterable(
('--patterns-from', pattern_filename)
for pattern_filename in pattern_filenames
)
)
def _make_exclude_flags(location_config, exclude_filename=None):
'''
Given a location config dict with various exclude options, and a filename containing any exclude
patterns, return the corresponding Borg flags as a tuple.
'''
exclude_filenames = tuple(location_config.get('exclude_from') or ()) + (
(exclude_patterns_filename,) if exclude_patterns_filename else ()
(exclude_filename,) if exclude_filename else ()
)
exclude_from_flags = tuple(
itertools.chain.from_iterable(
@@ -61,45 +105,55 @@ def _make_exclude_flags(location_config, exclude_patterns_filename=None):
def create_archive(
verbosity, repository, location_config, storage_config,
verbosity, dry_run, repository, location_config, storage_config, local_path='borg', remote_path=None,
):
'''
Given a vebosity flag, a local or remote repository path, a location config dict, and a storage
config dict, create a Borg archive.
Given vebosity/dry-run flags, a local or remote repository path, a location config dict, and a
storage config dict, create a Borg archive.
'''
sources = tuple(
itertools.chain.from_iterable(
_expand_directory(directory)
for directory in location_config['source_directories']
)
)
sources = _expand_directories(location_config['source_directories'])
exclude_patterns_file = _write_exclude_file(location_config.get('exclude_patterns'))
exclude_flags = _make_exclude_flags(
location_config,
exclude_patterns_file.name if exclude_patterns_file else None,
)
pattern_file = _write_pattern_file(location_config.get('patterns'))
exclude_file = _write_pattern_file(_expand_directories(location_config.get('exclude_patterns')))
compression = storage_config.get('compression', None)
compression_flags = ('--compression', compression) if compression else ()
remote_rate_limit = storage_config.get('remote_rate_limit', None)
umask = storage_config.get('umask', None)
umask_flags = ('--umask', str(umask)) if umask else ()
one_file_system_flags = ('--one-file-system',) if location_config.get('one_file_system') else ()
remote_path = location_config.get('remote_path')
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
verbosity_flags = {
VERBOSITY_SOME: ('--info', '--stats',),
VERBOSITY_LOTS: ('--debug', '--list', '--stats'),
}.get(verbosity, ())
lock_wait = storage_config.get('lock_wait', None)
files_cache = location_config.get('files_cache')
default_archive_name_format = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
archive_name_format = storage_config.get('archive_name_format', default_archive_name_format)
full_command = (
'borg', 'create',
'{repository}::{archive_name_format}'.format(
repository=repository,
archive_name_format=archive_name_format,
),
) + sources + exclude_flags + compression_flags + one_file_system_flags + \
remote_path_flags + umask_flags + verbosity_flags
(
local_path, 'create',
'{repository}::{archive_name_format}'.format(
repository=repository,
archive_name_format=archive_name_format,
),
)
+ sources
+ _make_pattern_flags(
location_config,
pattern_file.name if pattern_file else None,
)
+ _make_exclude_flags(
location_config,
exclude_file.name if exclude_file else None,
)
+ (('--compression', compression) if compression else ())
+ (('--remote-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ())
+ (('--one-file-system',) if location_config.get('one_file_system') else ())
+ (('--nobsdflags',) if location_config.get('bsd_flags') is False else ())
+ (('--files-cache', files_cache) if files_cache else ())
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ {
VERBOSITY_SOME: ('--info',) if dry_run else ('--info', '--stats',),
VERBOSITY_LOTS: ('--debug', '--list',) if dry_run else ('--debug', '--list', '--stats',),
}.get(verbosity, ())
+ (('--dry-run',) if dry_run else ())
)
logger.debug(' '.join(full_command))
subprocess.check_call(full_command)
+11 -5
View File
@@ -1,25 +1,30 @@
import logging
import sys
import subprocess
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def extract_last_archive_dry_run(verbosity, repository, remote_path=None):
logger = logging.getLogger(__name__)
def extract_last_archive_dry_run(verbosity, repository, lock_wait=None, local_path='borg', remote_path=None):
'''
Perform an extraction dry-run of just the most recent archive. If there are no archives, skip
the dry-run.
'''
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
verbosity_flags = {
VERBOSITY_SOME: ('--info',),
VERBOSITY_LOTS: ('--debug',),
}.get(verbosity, ())
full_list_command = (
'borg', 'list',
local_path, 'list',
'--short',
repository,
) + remote_path_flags + verbosity_flags
) + remote_path_flags + lock_wait_flags + verbosity_flags
list_output = subprocess.check_output(full_list_command).decode(sys.stdout.encoding)
@@ -29,12 +34,13 @@ def extract_last_archive_dry_run(verbosity, repository, remote_path=None):
list_flag = ('--list',) if verbosity == VERBOSITY_LOTS else ()
full_extract_command = (
'borg', 'extract',
local_path, 'extract',
'--dry-run',
'{repository}::{last_archive_name}'.format(
repository=repository,
last_archive_name=last_archive_name,
),
) + remote_path_flags + verbosity_flags + list_flag
) + remote_path_flags + lock_wait_flags + verbosity_flags + list_flag
logger.debug(' '.join(full_extract_command))
subprocess.check_call(full_extract_command)
+33
View File
@@ -0,0 +1,33 @@
import logging
import subprocess
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
logger = logging.getLogger(__name__)
def display_archives_info(
verbosity, repository, storage_config, local_path='borg', remote_path=None, json=False
):
'''
Given a verbosity flag, a local or remote repository path, and a storage config dict,
display summary information for Borg archives in the repository.
'''
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'info', repository)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--json',) if json else ())
+ {
VERBOSITY_SOME: ('--info',),
VERBOSITY_LOTS: ('--debug',),
}.get(verbosity, ())
)
logger.debug(' '.join(full_command))
output = subprocess.check_output(full_command)
return output.decode() if output is not None else None
+33
View File
@@ -0,0 +1,33 @@
import logging
import subprocess
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
logger = logging.getLogger(__name__)
def list_archives(
verbosity, repository, storage_config, local_path='borg', remote_path=None, json=False
):
'''
Given a verbosity flag, a local or remote repository path, and a storage config dict,
list Borg archives in the repository.
'''
lock_wait = storage_config.get('lock_wait', None)
full_command = (
(local_path, 'list', repository)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ (('--json',) if json else ())
+ {
VERBOSITY_SOME: ('--info',),
VERBOSITY_LOTS: ('--debug',),
}.get(verbosity, ())
)
logger.debug(' '.join(full_command))
output = subprocess.check_output(full_command)
return output.decode() if output is not None else None
+29 -15
View File
@@ -1,8 +1,12 @@
import logging
import subprocess
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
logger = logging.getLogger(__name__)
def _make_prune_flags(retention_config):
'''
Given a retention config dict mapping from option name to value, tranform it into an iterable of
@@ -28,24 +32,34 @@ def _make_prune_flags(retention_config):
)
def prune_archives(verbosity, repository, retention_config, remote_path=None):
def prune_archives(verbosity, dry_run, repository, storage_config, retention_config,
local_path='borg', remote_path=None):
'''
Given a verbosity flag, a local or remote repository path, a retention config dict, prune Borg
archives according the the retention policy specified in that configuration.
Given verbosity/dry-run flags, a local or remote repository path, a storage config dict, and a
retention config dict, prune Borg archives according to the retention policy specified in that
configuration.
'''
remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
verbosity_flags = {
VERBOSITY_SOME: ('--info', '--stats',),
VERBOSITY_LOTS: ('--debug', '--stats', '--list'),
}.get(verbosity, ())
umask = storage_config.get('umask', None)
lock_wait = storage_config.get('lock_wait', None)
full_command = (
'borg', 'prune',
repository,
) + tuple(
element
for pair in _make_prune_flags(retention_config)
for element in pair
) + remote_path_flags + verbosity_flags
(
local_path, 'prune',
repository,
) + tuple(
element
for pair in _make_prune_flags(retention_config)
for element in pair
)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--umask', str(umask)) if umask else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ {
VERBOSITY_SOME: ('--info', '--stats',),
VERBOSITY_LOTS: ('--debug', '--stats', '--list'),
}.get(verbosity, ())
+ (('--dry-run',) if dry_run else ())
)
logger.debug(' '.join(full_command))
subprocess.check_call(full_command)
+136 -29
View File
@@ -1,13 +1,15 @@
from argparse import ArgumentParser
import json
import logging
import os
from subprocess import CalledProcessError
import sys
from borgmatic.borg import check, create, prune
from borgmatic.borg import check as borg_check, create as borg_create, prune as borg_prune, \
list as borg_list, info as borg_info
from borgmatic.commands import hook
from borgmatic.config import collect, convert, validate
from borgmatic.signals import configure_signals
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS, verbosity_to_log_level
@@ -22,6 +24,8 @@ def parse_arguments(*arguments):
Given command-line arguments with which this script was invoked, parse the arguments and return
them as an ArgumentParser instance.
'''
config_paths = collect.get_default_config_paths()
parser = ArgumentParser(
description=
'''
@@ -34,8 +38,8 @@ def parse_arguments(*arguments):
'-c', '--config',
nargs='+',
dest='config_paths',
default=collect.DEFAULT_CONFIG_PATHS,
help='Configuration filenames or directories, defaults to: {}'.format(' '.join(collect.DEFAULT_CONFIG_PATHS)),
default=config_paths,
help='Configuration filenames or directories, defaults to: {}'.format(' '.join(config_paths)),
)
parser.add_argument(
'--excludes',
@@ -60,6 +64,31 @@ def parse_arguments(*arguments):
action='store_true',
help='Check archives for consistency',
)
parser.add_argument(
'-l', '--list',
dest='list',
action='store_true',
help='List archives',
)
parser.add_argument(
'-i', '--info',
dest='info',
action='store_true',
help='Display summary information on archives',
)
parser.add_argument(
'--json',
dest='json',
default=False,
action='store_true',
help='Output results from the --list option as json',
)
parser.add_argument(
'-n', '--dry-run',
dest='dry_run',
action='store_true',
help='Go through the motions, but do not actually write to any repositories',
)
parser.add_argument(
'-v', '--verbosity',
type=int,
@@ -68,14 +97,22 @@ def parse_arguments(*arguments):
args = parser.parse_args(arguments)
# If any of the three action flags in the given parse arguments have been explicitly requested,
# leave them as-is. Otherwise, assume defaults: Mutate the given arguments to enable all the
# actions.
if not args.prune and not args.create and not args.check:
args.prune = True
args.create = True
args.check = True
if args.json and not (args.list or args.info):
raise ValueError('The --json option can only be used with the --list or --info options')
if args.json and args.list and args.info:
raise ValueError(
'With the --json option, options --list and --info cannot be used together'
)
# If any of the action flags are explicitly requested, leave them as-is. Otherwise, assume
# defaults: Mutate the given arguments to enable the default actions.
if args.prune or args.create or args.check or args.list or args.info:
return args
args.prune = True
args.create = True
args.check = True
return args
@@ -92,34 +129,104 @@ def run_configuration(config_filename, args): # pragma: no cover
)
try:
local_path = location.get('local_path', 'borg')
remote_path = location.get('remote_path')
create.initialize(storage)
hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
borg_create.initialize_environment(storage)
for repository in location['repositories']:
if args.prune:
logger.info('{}: Pruning archives'.format(repository))
prune.prune_archives(args.verbosity, repository, retention, remote_path=remote_path)
if args.create:
logger.info('{}: Creating archive'.format(repository))
create.create_archive(
args.verbosity,
repository,
location,
storage,
)
if args.check:
logger.info('{}: Running consistency checks'.format(repository))
check.check_archives(args.verbosity, repository, consistency, remote_path=remote_path)
if args.create:
hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
hook.execute_hook(hooks.get('after_backup'), config_filename, 'post-backup')
_run_commands(args, consistency, local_path, location, remote_path, retention, storage)
if args.create:
hook.execute_hook(hooks.get('after_backup'), config_filename, 'post-backup')
except (OSError, CalledProcessError):
hook.execute_hook(hooks.get('on_error'), config_filename, 'on-error')
raise
def _run_commands(args, consistency, local_path, location, remote_path, retention, storage):
json_results = []
for unexpanded_repository in location['repositories']:
_run_commands_on_repository(
args, consistency, json_results, local_path, location, remote_path, retention, storage,
unexpanded_repository,
)
if args.json:
sys.stdout.write(json.dumps(json_results))
def _run_commands_on_repository(
args, consistency, json_results, local_path, location, remote_path,
retention, storage, unexpanded_repository,
): # pragma: no cover
repository = os.path.expanduser(unexpanded_repository)
dry_run_label = ' (dry run; not making any changes)' if args.dry_run else ''
if args.prune:
logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
borg_prune.prune_archives(
args.verbosity,
args.dry_run,
repository,
storage,
retention,
local_path=local_path,
remote_path=remote_path,
)
if args.create:
logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
borg_create.create_archive(
args.verbosity,
args.dry_run,
repository,
location,
storage,
local_path=local_path,
remote_path=remote_path,
)
if args.check:
logger.info('{}: Running consistency checks'.format(repository))
borg_check.check_archives(
args.verbosity,
repository,
storage,
consistency,
local_path=local_path,
remote_path=remote_path,
)
if args.list:
logger.info('{}: Listing archives'.format(repository))
output = borg_list.list_archives(
args.verbosity,
repository,
storage,
local_path=local_path,
remote_path=remote_path,
json=args.json,
)
if args.json:
json_results.append(json.loads(output))
else:
sys.stdout.write(output)
if args.info:
logger.info('{}: Displaying summary info for archives'.format(repository))
output = borg_info.display_archives_info(
args.verbosity,
repository,
storage,
local_path=local_path,
remote_path=remote_path,
json=args.json,
)
if args.json:
json_results.append(json.loads(output))
else:
sys.stdout.write(output)
def main(): # pragma: no cover
try:
configure_signals()
args = parse_arguments(*sys.argv[1:])
logging.basicConfig(level=verbosity_to_log_level(args.verbosity), format='%(message)s')
+23 -6
View File
@@ -1,23 +1,40 @@
import os
DEFAULT_CONFIG_PATHS = ['/etc/borgmatic/config.yaml', '/etc/borgmatic.d']
def get_default_config_paths():
'''
Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
default configuration paths. This includes both system-wide configuration and configuration in
the current user's home directory.
'''
user_config_directory = (
os.getenv('XDG_CONFIG_HOME') or os.path.expandvars(os.path.join('$HOME', '.config'))
)
return [
'/etc/borgmatic/config.yaml',
'/etc/borgmatic.d',
'%s/borgmatic/config.yaml' % user_config_directory,
]
def collect_config_filenames(config_paths):
'''
Given a sequence of config paths, both filenames and directories, resolve that to just an
iterable of files. Accomplish this by listing any given directories looking for contained config
files. This is non-recursive, so any directories within the given directories are ignored.
files (ending with the ".yaml" extension). This is non-recursive, so any directories within the
given directories are ignored.
Return paths even if they don't exist on disk, so the user can find out about missing
configuration paths. However, skip /etc/borgmatic.d if it's missing, so the user doesn't have to
create it unless they need it.
configuration paths. However, skip a default config path if it's missing, so the user doesn't
have to create a default config path unless they need it.
'''
real_default_config_paths = set(map(os.path.realpath, get_default_config_paths()))
for path in config_paths:
exists = os.path.exists(path)
if os.path.realpath(path) in DEFAULT_CONFIG_PATHS and not exists:
if os.path.realpath(path) in real_default_config_paths and not exists:
continue
if not os.path.isdir(path) or not exists:
@@ -26,5 +43,5 @@ def collect_config_filenames(config_paths):
for filename in os.listdir(path):
full_filename = os.path.join(path, filename)
if not os.path.isdir(full_filename):
if full_filename.endswith('.yaml') and not os.path.isdir(full_filename):
yield full_filename
+78 -7
View File
@@ -22,6 +22,21 @@ map:
type: bool
desc: Stay in same file system (do not cross mount points).
example: true
bsd_flags:
type: bool
desc: Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive. Defaults to true.
example: true
files_cache:
type: scalar
desc: |
Mode in which to operate the files cache. See
https://borgbackup.readthedocs.io/en/stable/usage/create.html#description for
details.
example: ctime,size,inode
local_path:
type: scalar
desc: Alternate Borg local executable. Defaults to "borg".
example: borg1
remote_path:
type: scalar
desc: Alternate Borg remote executable. Defaults to "borg".
@@ -31,26 +46,49 @@ map:
seq:
- type: scalar
desc: |
Paths to local or remote repositories (required). Multiple repositories are
backed up to in sequence.
Paths to local or remote repositories (required). Tildes are expanded. Multiple
repositories are backed up to in sequence. See ssh_command for SSH options like
identity file or port.
example:
- user@backupserver:sourcehostname.borg
patterns:
seq:
- type: scalar
desc: |
Any paths matching these patterns are included/excluded from backups. Globs are
expanded. (Tildes are not.) Note that Borg considers this option experimental.
See the output of "borg help patterns" for more details. Quote any value if it
contains leading punctuation, so it parses correctly.
example:
- 'R /'
- '- /home/*/.cache'
- '+ /home/susan'
- '- /home/*'
patterns_from:
seq:
- type: scalar
desc: |
Read include/exclude patterns from one or more separate named files, one pattern
per line. Note that Borg considers this option experimental. See the output of
"borg help patterns" for more details.
example:
- /etc/borgmatic/patterns
exclude_patterns:
seq:
- type: scalar
desc: |
Any paths matching these patterns are excluded from backups. Globs are expanded.
See the output of "borg help patterns" for more details.
Any paths matching these patterns are excluded from backups. Globs and tildes
are expanded. See the output of "borg help patterns" for more details.
example:
- '*.pyc'
- /home/*/.cache
- ~/*/.cache
- /etc/ssl
exclude_from:
seq:
- type: scalar
desc: |
Read exclude patterns from one or more separate named files, one pattern per
line.
line. See the output of "borg help patterns" for more details.
example:
- /etc/borgmatic/excludes
exclude_caches:
@@ -70,6 +108,14 @@ map:
https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
details.
map:
encryption_passcommand:
type: scalar
desc: |
The standard output of this command is used to unlock the encryption key. Only
use on repositories that were initialized with passcommand/repokey encryption.
Note that if both encryption_passcommand and encryption_passphrase are set,
then encryption_passphrase takes precedence.
example: "secret-tool lookup borg-repository repo-name"
encryption_passphrase:
type: scalar
desc: |
@@ -85,10 +131,22 @@ map:
https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.
Defaults to no compression.
example: lz4
remote_rate_limit:
type: int
desc: Remote network upload rate limit in kiBytes/second.
example: 100
ssh_command:
type: scalar
desc: Command to use instead of just "ssh". This can be used to specify ssh options.
example: ssh -i /path/to/private/key
umask:
type: scalar
desc: Umask to be used for borg create.
example: 0077
lock_wait:
type: int
desc: Maximum seconds to wait for acquiring a repository/cache lock.
example: 5
archive_name_format:
type: scalar
desc: |
@@ -96,17 +154,23 @@ map:
"borg help placeholders" for details. Default is
"{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
also specify a prefix in the retention section to avoid accidental pruning of
archives with a different archive name format.
archives with a different archive name format. And you should also specify a
prefix in the consistency section as well.
example: "{hostname}-documents-{now}"
retention:
desc: |
Retention policy for how many backups to keep in each category. See
https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
At least one of the "keep" options is required for pruning to work.
map:
keep_within:
type: scalar
desc: Keep all archives within this time interval.
example: 3H
keep_minutely:
type: int
desc: Number of minutely archives to keep.
example: 60
keep_hourly:
type: int
desc: Number of hourly archives to keep.
@@ -159,6 +223,13 @@ map:
desc: Restrict the number of checked archives to the last n. Applies only to the
"archives" check.
example: 3
prefix:
type: scalar
desc: |
When performing the "archives" check, only consider archive names starting with
this prefix. Borg placeholders can be used. See the output of
"borg help placeholders" for details. Default is "{hostname}-".
example: sourcehostname
hooks:
desc: |
Shell commands or scripts to execute before and after a backup or if an error has occurred.
+10 -2
View File
@@ -8,6 +8,9 @@ import pykwalify.errors
from ruamel import yaml
logger = logging.getLogger(__name__)
def schema_filename():
'''
Path to the installed YAML configuration schema file, used to validate and parse the
@@ -50,6 +53,11 @@ def apply_logical_validation(config_filename, parsed_configuration):
)
)
consistency_prefix = parsed_configuration.get('consistency', {}).get('prefix')
if archive_name_format and not consistency_prefix:
logger.warning('Since version 1.1.16, if you provide `archive_name_format`, you should also'
' specify `consistency.prefix`.')
def parse_configuration(config_filename, schema_filename):
'''
@@ -66,8 +74,8 @@ def parse_configuration(config_filename, schema_filename):
logging.getLogger('pykwalify').setLevel(logging.ERROR)
try:
config = yaml.round_trip_load(open(config_filename))
schema = yaml.round_trip_load(open(schema_filename))
config = yaml.safe_load(open(config_filename))
schema = yaml.safe_load(open(schema_filename))
except yaml.error.YAMLError as error:
raise Validation_error(config_filename, (str(error),))
+18
View File
@@ -0,0 +1,18 @@
import os
import signal
def _handle_signal(signal_number, frame): # pragma: no cover
'''
Send the signal to all processes in borgmatic's process group, which includes child process.
'''
os.killpg(os.getpgrp(), signal_number)
def configure_signals(): # pragma: no cover
'''
Configure borgmatic's signal handlers to pass relevant signals through to any child processes
like Borg. Note that SIGINT gets passed through even without these changes.
'''
for signal_number in (signal.SIGHUP, signal.SIGTERM, signal.SIGUSR1, signal.SIGUSR2):
signal.signal(signal_number, _handle_signal)
@@ -7,14 +7,20 @@ from borgmatic.commands import borgmatic as module
def test_parse_arguments_with_no_arguments_uses_defaults():
config_paths = ['default']
flexmock(module.collect).should_receive('get_default_config_paths').and_return(config_paths)
parser = module.parse_arguments()
assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
assert parser.config_paths == config_paths
assert parser.excludes_filename == None
assert parser.verbosity is None
assert parser.json is False
def test_parse_arguments_with_path_arguments_overrides_defaults():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
assert parser.config_paths == ['myconfig']
@@ -23,6 +29,8 @@ def test_parse_arguments_with_path_arguments_overrides_defaults():
def test_parse_arguments_with_multiple_config_paths_parses_as_list():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
parser = module.parse_arguments('--config', 'myconfig', 'otherconfig')
assert parser.config_paths == ['myconfig', 'otherconfig']
@@ -30,14 +38,24 @@ def test_parse_arguments_with_multiple_config_paths_parses_as_list():
def test_parse_arguments_with_verbosity_flag_overrides_default():
config_paths = ['default']
flexmock(module.collect).should_receive('get_default_config_paths').and_return(config_paths)
parser = module.parse_arguments('--verbosity', '1')
assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
assert parser.config_paths == config_paths
assert parser.excludes_filename == None
assert parser.verbosity == 1
def test_parse_arguments_with_json_flag_overrides_default():
parser = module.parse_arguments('--list', '--json')
assert parser.json is True
def test_parse_arguments_with_no_actions_defaults_to_all_actions_enabled():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
parser = module.parse_arguments()
assert parser.prune is True
@@ -46,6 +64,8 @@ def test_parse_arguments_with_no_actions_defaults_to_all_actions_enabled():
def test_parse_arguments_with_prune_action_leaves_other_actions_disabled():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
parser = module.parse_arguments('--prune')
assert parser.prune is True
@@ -54,6 +74,8 @@ def test_parse_arguments_with_prune_action_leaves_other_actions_disabled():
def test_parse_arguments_with_multiple_actions_leaves_other_action_disabled():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
parser = module.parse_arguments('--create', '--check')
assert parser.prune is False
@@ -62,5 +84,22 @@ def test_parse_arguments_with_multiple_actions_leaves_other_action_disabled():
def test_parse_arguments_with_invalid_arguments_exits():
flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
with pytest.raises(SystemExit):
module.parse_arguments('--posix-me-harder')
def test_parse_arguments_with_json_flag_with_list_or_info_flag_does_not_raise_any_error():
module.parse_arguments('--list', '--json')
module.parse_arguments('--info', '--json')
def test_parse_arguments_with_json_flag_but_no_list_or_info_flag_raises_value_error():
with pytest.raises(ValueError):
module.parse_arguments('--json')
def test_parse_arguments_with_json_flag_and_both_list_and_info_flag_raises_value_error():
with pytest.raises(ValueError):
module.parse_arguments('--list', '--info', '--json')
@@ -44,6 +44,8 @@ def test_parse_configuration_transforms_file_into_mapping():
- hostname.borg
retention:
keep_minutely: 60
keep_hourly: 24
keep_daily: 7
consistency:
@@ -57,7 +59,7 @@ def test_parse_configuration_transforms_file_into_mapping():
assert result == {
'location': {'source_directories': ['/home', '/etc'], 'repositories': ['hostname.borg']},
'retention': {'keep_daily': 7},
'retention': {'keep_daily': 7, 'keep_hourly': 24, 'keep_minutely': 60},
'consistency': {'checks': ['repository', 'archives']},
}
+127 -22
View File
@@ -42,34 +42,64 @@ def test_parse_checks_with_disabled_returns_no_checks():
assert checks == ()
def test_make_check_flags_with_checks_returns_flags():
def test_make_check_flags_with_repository_check_returns_flag():
flags = module._make_check_flags(('repository',))
assert flags == ('--repository-only',)
def test_make_check_flags_with_extract_check_does_not_make_extract_flag():
def test_make_check_flags_with_extract_omits_extract_flag():
flags = module._make_check_flags(('extract',))
assert flags == ()
def test_make_check_flags_with_default_checks_returns_no_flags():
def test_make_check_flags_with_default_checks_returns_default_flags():
flags = module._make_check_flags(module.DEFAULT_CHECKS)
assert flags == ()
assert flags == ('--prefix', module.DEFAULT_PREFIX)
def test_make_check_flags_with_checks_and_last_returns_flags_including_last():
def test_make_check_flags_with_all_checks_returns_default_flags():
flags = module._make_check_flags(module.DEFAULT_CHECKS + ('extract',))
assert flags == ('--prefix', module.DEFAULT_PREFIX)
def test_make_check_flags_with_archives_check_and_last_includes_last_flag():
flags = module._make_check_flags(('archives',), check_last=3)
assert flags == ('--archives-only', '--last', '3', '--prefix', module.DEFAULT_PREFIX)
def test_make_check_flags_with_repository_check_and_last_omits_last_flag():
flags = module._make_check_flags(('repository',), check_last=3)
assert flags == ('--repository-only', '--last', '3')
assert flags == ('--repository-only',)
def test_make_check_flags_with_default_checks_and_last_returns_last_flag():
def test_make_check_flags_with_default_checks_and_last_includes_last_flag():
flags = module._make_check_flags(module.DEFAULT_CHECKS, check_last=3)
assert flags == ('--last', '3')
assert flags == ('--last', '3', '--prefix', module.DEFAULT_PREFIX)
def test_make_check_flags_with_archives_check_and_prefix_includes_prefix_flag():
flags = module._make_check_flags(('archives',), prefix='foo-')
assert flags == ('--archives-only', '--prefix', 'foo-')
def test_make_check_flags_with_repository_check_and_prefix_omits_prefix_flag():
flags = module._make_check_flags(('repository',), prefix='foo-')
assert flags == ('--repository-only',)
def test_make_check_flags_with_default_checks_and_prefix_includes_prefix_flag():
flags = module._make_check_flags(module.DEFAULT_CHECKS, prefix='foo-')
assert flags == ('--prefix', 'foo-')
@pytest.mark.parametrize(
@@ -81,11 +111,11 @@ def test_make_check_flags_with_default_checks_and_last_returns_last_flag():
('repository', 'archives', 'other'),
),
)
def test_check_archives_should_call_borg_with_parameters(checks):
def test_check_archives_calls_borg_with_parameters(checks):
check_last = flexmock()
consistency_config = flexmock().should_receive('get').and_return(check_last).mock
consistency_config = {'check_last': check_last}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last).and_return(())
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last, None).and_return(())
stdout = flexmock()
insert_subprocess_mock(
('borg', 'check', 'repo'),
@@ -97,14 +127,15 @@ def test_check_archives_should_call_borg_with_parameters(checks):
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
def test_check_archives_with_extract_check_should_call_extract_only():
def test_check_archives_with_extract_check_calls_extract_only():
checks = ('extract',)
check_last = flexmock()
consistency_config = flexmock().should_receive('get').and_return(check_last).mock
consistency_config = {'check_last': check_last}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').never()
flexmock(module.extract).should_receive('extract_last_archive_dry_run').once()
@@ -113,13 +144,14 @@ def test_check_archives_with_extract_check_should_call_extract_only():
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
def test_check_archives_with_verbosity_some_should_call_borg_with_info_parameter():
def test_check_archives_with_verbosity_some_calls_borg_with_info_parameter():
checks = ('repository',)
consistency_config = flexmock().should_receive('get').and_return(None).mock
consistency_config = {'check_last': None}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').and_return(())
insert_subprocess_mock(
@@ -130,13 +162,14 @@ def test_check_archives_with_verbosity_some_should_call_borg_with_info_parameter
module.check_archives(
verbosity=VERBOSITY_SOME,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
def test_check_archives_with_verbosity_lots_should_call_borg_with_debug_parameter():
def test_check_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
checks = ('repository',)
consistency_config = flexmock().should_receive('get').and_return(None).mock
consistency_config = {'check_last': None}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').and_return(())
insert_subprocess_mock(
@@ -147,28 +180,53 @@ def test_check_archives_with_verbosity_lots_should_call_borg_with_debug_paramete
module.check_archives(
verbosity=VERBOSITY_LOTS,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
def test_check_archives_without_any_checks_should_bail():
consistency_config = flexmock().should_receive('get').and_return(None).mock
def test_check_archives_without_any_checks_bails():
consistency_config = {'check_last': None}
flexmock(module).should_receive('_parse_checks').and_return(())
insert_subprocess_never()
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
def test_check_archives_with_remote_path_should_call_borg_with_remote_path_parameters():
def test_check_archives_with_local_path_calls_borg_via_local_path():
checks = ('repository',)
check_last = flexmock()
consistency_config = flexmock().should_receive('get').and_return(check_last).mock
consistency_config = {'check_last': check_last}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last).and_return(())
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last, None).and_return(())
stdout = flexmock()
insert_subprocess_mock(
('borg1', 'check', 'repo'),
stdout=stdout, stderr=STDOUT,
)
flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
flexmock(module.os).should_receive('devnull')
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
local_path='borg1',
)
def test_check_archives_with_remote_path_calls_borg_with_remote_path_parameters():
checks = ('repository',)
check_last = flexmock()
consistency_config = {'check_last': check_last}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last, None).and_return(())
stdout = flexmock()
insert_subprocess_mock(
('borg', 'check', 'repo', '--remote-path', 'borg1'),
@@ -180,6 +238,53 @@ def test_check_archives_with_remote_path_should_call_borg_with_remote_path_param
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
remote_path='borg1',
)
def test_check_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
checks = ('repository',)
check_last = flexmock()
consistency_config = {'check_last': check_last}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last, None).and_return(())
stdout = flexmock()
insert_subprocess_mock(
('borg', 'check', 'repo', '--lock-wait', '5'),
stdout=stdout, stderr=STDOUT,
)
flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
flexmock(module.os).should_receive('devnull')
module.check_archives(
verbosity=None,
repository='repo',
storage_config={'lock_wait': 5},
consistency_config=consistency_config,
)
def test_check_archives_with_retention_prefix():
checks = ('repository',)
check_last = flexmock()
prefix = 'foo-'
consistency_config = {'check_last': check_last, 'prefix': prefix}
flexmock(module).should_receive('_parse_checks').and_return(checks)
flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last, prefix).and_return(())
stdout = flexmock()
insert_subprocess_mock(
('borg', 'check', 'repo'),
stdout=stdout, stderr=STDOUT,
)
flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
flexmock(module.os).should_receive('devnull')
module.check_archives(
verbosity=None,
repository='repo',
storage_config={},
consistency_config=consistency_config,
)
+347 -44
View File
@@ -6,24 +6,48 @@ from borgmatic.borg import create as module
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def test_initialize_with_passphrase_should_set_environment():
def test_initialize_environment_with_passcommand_should_set_environment():
orig_environ = os.environ
try:
os.environ = {}
module.initialize({'encryption_passphrase': 'pass'})
module.initialize_environment({'encryption_passcommand': 'command'})
assert os.environ.get('BORG_PASSCOMMAND') == 'command'
finally:
os.environ = orig_environ
def test_initialize_environment_with_passphrase_should_set_environment():
orig_environ = os.environ
try:
os.environ = {}
module.initialize_environment({'encryption_passphrase': 'pass'})
assert os.environ.get('BORG_PASSPHRASE') == 'pass'
finally:
os.environ = orig_environ
def test_initialize_without_passphrase_should_not_set_environment():
def test_initialize_environment_with_ssh_command_should_set_environment():
orig_environ = os.environ
try:
os.environ = {}
module.initialize({})
module.initialize_environment({'ssh_command': 'ssh -C'})
assert os.environ.get('BORG_RSH') == 'ssh -C'
finally:
os.environ = orig_environ
def test_initialize_environment_without_configuration_should_not_set_environment():
orig_environ = os.environ
try:
os.environ = {}
module.initialize_environment({})
assert os.environ.get('BORG_PASSCOMMAND') == None
assert os.environ.get('BORG_PASSPHRASE') == None
assert os.environ.get('BORG_RSH') == None
finally:
os.environ = orig_environ
@@ -46,7 +70,22 @@ def test_expand_directory_with_glob_expands():
assert paths == ['foo', 'food']
def test_write_exclude_file_does_not_raise():
def test_expand_directories_flattens_expanded_directories():
flexmock(module).should_receive('_expand_directory').with_args('~/foo').and_return(['/root/foo'])
flexmock(module).should_receive('_expand_directory').with_args('bar*').and_return(['bar', 'barf'])
paths = module._expand_directories(('~/foo', 'bar*'))
assert paths == ('/root/foo', 'bar', 'barf')
def test_expand_directories_considers_none_as_no_directories():
paths = module._expand_directories(None)
assert paths == ()
def test_write_pattern_file_does_not_raise():
temporary_file = flexmock(
name='filename',
write=lambda mode: None,
@@ -54,11 +93,11 @@ def test_write_exclude_file_does_not_raise():
)
flexmock(module.tempfile).should_receive('NamedTemporaryFile').and_return(temporary_file)
module._write_exclude_file(['exclude'])
module._write_pattern_file(['exclude'])
def test_write_exclude_file_with_empty_exclude_patterns_does_not_raise():
module._write_exclude_file([])
def test_write_pattern_file_with_empty_exclude_patterns_does_not_raise():
module._write_pattern_file([])
def insert_subprocess_mock(check_call_command, **kwargs):
@@ -66,17 +105,50 @@ def insert_subprocess_mock(check_call_command, **kwargs):
subprocess.should_receive('check_call').with_args(check_call_command, **kwargs).once()
def test_make_pattern_flags_includes_pattern_filename_when_given():
pattern_flags = module._make_pattern_flags(
location_config={'patterns': ['R /', '- /var']},
pattern_filename='/tmp/patterns',
)
assert pattern_flags == ('--patterns-from', '/tmp/patterns')
def test_make_pattern_flags_includes_patterns_from_filenames_when_in_config():
pattern_flags = module._make_pattern_flags(
location_config={'patterns_from': ['patterns', 'other']},
)
assert pattern_flags == ('--patterns-from', 'patterns', '--patterns-from', 'other')
def test_make_pattern_flags_includes_both_filenames_when_patterns_given_and_patterns_from_in_config():
pattern_flags = module._make_pattern_flags(
location_config={'patterns_from': ['patterns']},
pattern_filename='/tmp/patterns',
)
assert pattern_flags == ('--patterns-from', 'patterns', '--patterns-from', '/tmp/patterns')
def test_make_pattern_flags_considers_none_patterns_from_filenames_as_empty():
pattern_flags = module._make_pattern_flags(
location_config={'patterns_from': None},
)
assert pattern_flags == ()
def test_make_exclude_flags_includes_exclude_patterns_filename_when_given():
exclude_flags = module._make_exclude_flags(
location_config={'exclude_patterns': ['*.pyc', '/var']},
exclude_patterns_filename='/tmp/excludes',
exclude_filename='/tmp/excludes',
)
assert exclude_flags == ('--exclude-from', '/tmp/excludes')
def test_make_exclude_flags_includes_exclude_from_filenames_when_in_config():
flexmock(module).should_receive('_write_exclude_file').and_return(None)
exclude_flags = module._make_exclude_flags(
location_config={'exclude_from': ['excludes', 'other']},
@@ -86,19 +158,15 @@ def test_make_exclude_flags_includes_exclude_from_filenames_when_in_config():
def test_make_exclude_flags_includes_both_filenames_when_patterns_given_and_exclude_from_in_config():
flexmock(module).should_receive('_write_exclude_file').and_return(None)
exclude_flags = module._make_exclude_flags(
location_config={'exclude_from': ['excludes']},
exclude_patterns_filename='/tmp/excludes',
exclude_filename='/tmp/excludes',
)
assert exclude_flags == ('--exclude-from', 'excludes', '--exclude-from', '/tmp/excludes')
def test_make_exclude_flags_considers_none_exclude_from_filenames_as_empty():
flexmock(module).should_receive('_write_exclude_file').and_return(None)
exclude_flags = module._make_exclude_flags(
location_config={'exclude_from': None},
)
@@ -141,13 +209,15 @@ CREATE_COMMAND = ('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'fo
def test_create_archive_calls_borg_with_parameters():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND)
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -158,15 +228,38 @@ def test_create_archive_calls_borg_with_parameters():
)
def test_create_archive_with_patterns_calls_borg_with_patterns():
pattern_flags = ('--patterns-from', 'patterns')
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(flexmock(name='/tmp/patterns')).and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(pattern_flags)
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + pattern_flags)
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'patterns': ['pattern'],
},
storage_config={},
)
def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
exclude_flags = ('--exclude-from', 'excludes')
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(flexmock(name='/tmp/excludes'))
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(('exclude',))
flexmock(module).should_receive('_write_pattern_file').and_return(None).and_return(flexmock(name='/tmp/excludes'))
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(exclude_flags)
insert_subprocess_mock(CREATE_COMMAND + exclude_flags)
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -178,13 +271,16 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
def test_create_archive_with_verbosity_some_calls_borg_with_info_parameter():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--info', '--stats',))
module.create_archive(
verbosity=VERBOSITY_SOME,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -196,13 +292,78 @@ def test_create_archive_with_verbosity_some_calls_borg_with_info_parameter():
def test_create_archive_with_verbosity_lots_calls_borg_with_debug_parameter():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--stats'))
module.create_archive(
verbosity=VERBOSITY_LOTS,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--dry-run',))
module.create_archive(
verbosity=None,
dry_run=True,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_dry_run_and_verbosity_some_calls_borg_without_stats_parameter():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--info', '--dry-run'))
module.create_archive(
verbosity=VERBOSITY_SOME,
dry_run=True,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_dry_run_and_verbosity_lots_calls_borg_without_stats_parameter():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--dry-run'))
module.create_archive(
verbosity=VERBOSITY_LOTS,
dry_run=True,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -214,13 +375,15 @@ def test_create_archive_with_verbosity_lots_calls_borg_with_debug_parameter():
def test_create_archive_with_compression_calls_borg_with_compression_parameters():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--compression', 'rle'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -231,14 +394,36 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
)
def test_create_archive_with_remote_rate_limit_calls_borg_with_remote_ratelimit_parameters():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--remote-ratelimit', '100'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={'remote_rate_limit': 100},
)
def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_parameters():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--one-file-system',))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -250,33 +435,121 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
)
def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
def test_create_archive_with_bsd_flags_true_calls_borg_without_nobsdflags_parameter():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
insert_subprocess_mock(CREATE_COMMAND)
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'remote_path': 'borg1',
'bsd_flags': True,
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_bsd_flags_false_calls_borg_with_nobsdflags_parameter():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--nobsdflags',))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'bsd_flags': False,
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--files-cache', 'ctime,size'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'files_cache': 'ctime,size',
'exclude_patterns': None,
},
storage_config={},
)
def test_create_archive_with_local_path_calls_borg_via_local_path():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg1',) + CREATE_COMMAND[1:])
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={},
local_path='borg1',
)
def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={},
remote_path='borg1',
)
def test_create_archive_with_umask_calls_borg_with_umask_parameters():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--umask', '740'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -287,15 +560,37 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
)
def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(CREATE_COMMAND + ('--lock-wait', '5'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
},
storage_config={'lock_wait': 5},
)
def test_create_archive_with_source_directories_glob_expands():
flexmock(module).should_receive('_expand_directory').and_return(['foo', 'food'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'food')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'food'))
flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo*'],
@@ -307,14 +602,16 @@ def test_create_archive_with_source_directories_glob_expands():
def test_create_archive_with_non_matching_source_directories_glob_passes_through():
flexmock(module).should_receive('_expand_directory').and_return(['foo*'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo*',)).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo*'))
flexmock(module.glob).should_receive('glob').with_args('foo*').and_return([])
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo*'],
@@ -326,13 +623,15 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
def test_create_archive_with_glob_calls_borg_with_expanded_directories():
flexmock(module).should_receive('_expand_directory').and_return(['foo', 'food'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'food')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'food'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo*'],
@@ -344,13 +643,15 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg', 'create', 'repo::ARCHIVE_NAME', 'foo', 'bar'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
@@ -364,13 +665,15 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
flexmock(module).should_receive('_write_exclude_file').and_return(None)
flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return([])
flexmock(module).should_receive('_write_pattern_file').and_return(None)
flexmock(module).should_receive('_make_pattern_flags').and_return(())
flexmock(module).should_receive('_make_exclude_flags').and_return(())
insert_subprocess_mock(('borg', 'create', 'repo::Documents_{hostname}-{now}', 'foo', 'bar'))
module.create_archive(
verbosity=None,
dry_run=False,
repository='repo',
location_config={
'source_directories': ['foo', 'bar'],
+40
View File
@@ -34,6 +34,7 @@ def test_extract_last_archive_dry_run_should_call_borg_with_last_archive():
module.extract_last_archive_dry_run(
verbosity=None,
repository='repo',
lock_wait=None,
)
@@ -48,6 +49,7 @@ def test_extract_last_archive_dry_run_without_any_archives_should_bail():
module.extract_last_archive_dry_run(
verbosity=None,
repository='repo',
lock_wait=None,
)
@@ -64,6 +66,7 @@ def test_extract_last_archive_dry_run_with_verbosity_some_should_call_borg_with_
module.extract_last_archive_dry_run(
verbosity=VERBOSITY_SOME,
repository='repo',
lock_wait=None,
)
@@ -80,6 +83,25 @@ def test_extract_last_archive_dry_run_with_verbosity_lots_should_call_borg_with_
module.extract_last_archive_dry_run(
verbosity=VERBOSITY_LOTS,
repository='repo',
lock_wait=None,
)
def test_extract_last_archive_dry_run_should_call_borg_via_local_path():
flexmock(sys.stdout).encoding = 'utf-8'
insert_subprocess_check_output_mock(
('borg1', 'list', '--short', 'repo'),
result='archive1\narchive2\n'.encode('utf-8'),
)
insert_subprocess_mock(
('borg1', 'extract', '--dry-run', 'repo::archive2'),
)
module.extract_last_archive_dry_run(
verbosity=None,
repository='repo',
lock_wait=None,
local_path='borg1',
)
@@ -96,5 +118,23 @@ def test_extract_last_archive_dry_run_should_call_borg_with_remote_path_paramete
module.extract_last_archive_dry_run(
verbosity=None,
repository='repo',
lock_wait=None,
remote_path='borg1',
)
def test_extract_last_archive_dry_run_should_call_borg_with_lock_wait_parameters():
flexmock(sys.stdout).encoding = 'utf-8'
insert_subprocess_check_output_mock(
('borg', 'list', '--short', 'repo', '--lock-wait', '5'),
result='archive1\narchive2\n'.encode('utf-8'),
)
insert_subprocess_mock(
('borg', 'extract', '--dry-run', 'repo::archive2', '--lock-wait', '5'),
)
module.extract_last_archive_dry_run(
verbosity=None,
repository='repo',
lock_wait=5,
)
+88
View File
@@ -0,0 +1,88 @@
from collections import OrderedDict
from flexmock import flexmock
from borgmatic.borg import info as module
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def insert_subprocess_mock(check_call_command, **kwargs):
subprocess = flexmock(module.subprocess)
subprocess.should_receive('check_output').with_args(check_call_command, **kwargs).once()
INFO_COMMAND = ('borg', 'info', 'repo')
def test_display_archives_info_calls_borg_with_parameters():
insert_subprocess_mock(INFO_COMMAND)
module.display_archives_info(
verbosity=None,
repository='repo',
storage_config={},
)
def test_display_archives_info_with_verbosity_some_calls_borg_with_info_parameter():
insert_subprocess_mock(INFO_COMMAND + ('--info',))
module.display_archives_info(
repository='repo',
storage_config={},
verbosity=VERBOSITY_SOME,
)
def test_display_archives_info_with_verbosity_lots_calls_borg_with_debug_parameter():
insert_subprocess_mock(INFO_COMMAND + ('--debug',))
module.display_archives_info(
repository='repo',
storage_config={},
verbosity=VERBOSITY_LOTS,
)
def test_display_archives_info_with_json_calls_borg_with_json_parameter():
insert_subprocess_mock(INFO_COMMAND + ('--json',))
module.display_archives_info(
verbosity=None,
repository='repo',
storage_config={},
json=True,
)
def test_display_archives_info_with_local_path_calls_borg_via_local_path():
insert_subprocess_mock(('borg1',) + INFO_COMMAND[1:])
module.display_archives_info(
verbosity=None,
repository='repo',
storage_config={},
local_path='borg1',
)
def test_display_archives_info_with_remote_path_calls_borg_with_remote_path_parameters():
insert_subprocess_mock(INFO_COMMAND + ('--remote-path', 'borg1'))
module.display_archives_info(
verbosity=None,
repository='repo',
storage_config={},
remote_path='borg1',
)
def test_display_archives_info_with_lock_wait_calls_borg_with_lock_wait_parameters():
storage_config = {'lock_wait': 5}
insert_subprocess_mock(INFO_COMMAND + ('--lock-wait', '5'))
module.display_archives_info(
verbosity=None,
repository='repo',
storage_config=storage_config,
)
+88
View File
@@ -0,0 +1,88 @@
from collections import OrderedDict
from flexmock import flexmock
from borgmatic.borg import list as module
from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def insert_subprocess_mock(check_call_command, **kwargs):
subprocess = flexmock(module.subprocess)
subprocess.should_receive('check_output').with_args(check_call_command, **kwargs).once()
LIST_COMMAND = ('borg', 'list', 'repo')
def test_list_archives_calls_borg_with_parameters():
insert_subprocess_mock(LIST_COMMAND)
module.list_archives(
verbosity=None,
repository='repo',
storage_config={},
)
def test_list_archives_with_verbosity_some_calls_borg_with_info_parameter():
insert_subprocess_mock(LIST_COMMAND + ('--info',))
module.list_archives(
repository='repo',
storage_config={},
verbosity=VERBOSITY_SOME,
)
def test_list_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
insert_subprocess_mock(LIST_COMMAND + ('--debug',))
module.list_archives(
repository='repo',
storage_config={},
verbosity=VERBOSITY_LOTS,
)
def test_list_archives_with_json_calls_borg_with_json_parameter():
insert_subprocess_mock(LIST_COMMAND + ('--json',))
module.list_archives(
verbosity=None,
repository='repo',
storage_config={},
json=True,
)
def test_list_archives_with_local_path_calls_borg_via_local_path():
insert_subprocess_mock(('borg1',) + LIST_COMMAND[1:])
module.list_archives(
verbosity=None,
repository='repo',
storage_config={},
local_path='borg1',
)
def test_list_archives_with_remote_path_calls_borg_with_remote_path_parameters():
insert_subprocess_mock(LIST_COMMAND + ('--remote-path', 'borg1'))
module.list_archives(
verbosity=None,
repository='repo',
storage_config={},
remote_path='borg1',
)
def test_list_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
storage_config = {'lock_wait': 5}
insert_subprocess_mock(LIST_COMMAND + ('--lock-wait', '5'))
module.list_archives(
verbosity=None,
repository='repo',
storage_config=storage_config,
)
+80 -5
View File
@@ -18,7 +18,7 @@ BASE_PRUNE_FLAGS = (
)
def test_make_prune_flags_should_return_flags_from_config_plus_default_prefix():
def test_make_prune_flags_returns_flags_from_config_plus_default_prefix():
retention_config = OrderedDict(
(
('keep_daily', 1),
@@ -55,7 +55,7 @@ PRUNE_COMMAND = (
)
def test_prune_archives_should_call_borg_with_parameters():
def test_prune_archives_calls_borg_with_parameters():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
@@ -64,12 +64,14 @@ def test_prune_archives_should_call_borg_with_parameters():
module.prune_archives(
verbosity=None,
dry_run=False,
repository='repo',
storage_config={},
retention_config=retention_config,
)
def test_prune_archives_with_verbosity_some_should_call_borg_with_info_parameter():
def test_prune_archives_with_verbosity_some_calls_borg_with_info_parameter():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
@@ -78,12 +80,14 @@ def test_prune_archives_with_verbosity_some_should_call_borg_with_info_parameter
module.prune_archives(
repository='repo',
storage_config={},
verbosity=VERBOSITY_SOME,
dry_run=False,
retention_config=retention_config,
)
def test_prune_archives_with_verbosity_lots_should_call_borg_with_debug_parameter():
def test_prune_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
@@ -92,12 +96,47 @@ def test_prune_archives_with_verbosity_lots_should_call_borg_with_debug_paramete
module.prune_archives(
repository='repo',
storage_config={},
verbosity=VERBOSITY_LOTS,
dry_run=False,
retention_config=retention_config,
)
def test_prune_archives_with_remote_path_should_call_borg_with_remote_path_parameters():
def test_prune_archives_with_dry_run_calls_borg_with_dry_run_parameter():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(PRUNE_COMMAND + ('--dry-run',))
module.prune_archives(
repository='repo',
storage_config={},
verbosity=None,
dry_run=True,
retention_config=retention_config,
)
def test_prune_archives_with_local_path_calls_borg_via_local_path():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(('borg1',) + PRUNE_COMMAND[1:])
module.prune_archives(
verbosity=None,
dry_run=False,
repository='repo',
storage_config={},
retention_config=retention_config,
local_path='borg1',
)
def test_prune_archives_with_remote_path_calls_borg_with_remote_path_parameters():
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
@@ -106,7 +145,43 @@ def test_prune_archives_with_remote_path_should_call_borg_with_remote_path_param
module.prune_archives(
verbosity=None,
dry_run=False,
repository='repo',
storage_config={},
retention_config=retention_config,
remote_path='borg1',
)
def test_prune_archives_with_umask_calls_borg_with_umask_parameters():
storage_config = {'umask': '077'}
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(PRUNE_COMMAND + ('--umask', '077'))
module.prune_archives(
verbosity=None,
dry_run=False,
repository='repo',
storage_config=storage_config,
retention_config=retention_config,
)
def test_prune_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
storage_config = {'lock_wait': 5}
retention_config = flexmock()
flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
BASE_PRUNE_FLAGS,
)
insert_subprocess_mock(PRUNE_COMMAND + ('--lock-wait', '5'))
module.prune_archives(
verbosity=None,
dry_run=False,
repository='repo',
storage_config=storage_config,
retention_config=retention_config,
)
@@ -0,0 +1,52 @@
import json
import sys
from flexmock import flexmock
import pytest
from borgmatic.commands import borgmatic
def test__run_commands_handles_multiple_json_outputs_in_array():
(
flexmock(borgmatic)
.should_receive('_run_commands_on_repository')
.times(3)
.replace_with(
lambda args, consistency, json_results, local_path, location, remote_path, retention,
storage,
unexpanded_repository: json_results.append({"whatever": unexpanded_repository})
)
)
(
flexmock(sys.stdout)
.should_call("write")
.with_args(
json.dumps(
json.loads(
'''
[
{"whatever": "fake_repo1"},
{"whatever": "fake_repo2"},
{"whatever": "fake_repo3"}
]
''',
)
)
)
)
borgmatic._run_commands(
args=flexmock(json=True),
consistency=None,
local_path=None,
location={'repositories': [
'fake_repo1',
'fake_repo2',
'fake_repo3'
]},
remote_path=None,
retention=None,
storage=None,
)
@@ -3,6 +3,22 @@ from flexmock import flexmock
from borgmatic.config import collect as module
def test_get_default_config_paths_includes_absolute_user_config_path():
flexmock(module.os, environ={'XDG_CONFIG_HOME': None, 'HOME': '/home/user'})
config_paths = module.get_default_config_paths()
assert '/home/user/.config/borgmatic/config.yaml' in config_paths
def test_get_default_config_paths_prefers_xdg_config_home_for_user_config_path():
flexmock(module.os, environ={'XDG_CONFIG_HOME': '/home/user/.etc', 'HOME': '/home/user'})
config_paths = module.get_default_config_paths()
assert '/home/user/.etc/borgmatic/config.yaml' in config_paths
def test_collect_config_filenames_collects_given_files():
config_paths = ('config.yaml', 'other.yaml')
flexmock(module.os.path).should_receive('isdir').and_return(False)
@@ -32,6 +48,21 @@ def test_collect_config_filenames_collects_files_from_given_directories_and_igno
)
def test_collect_config_filenames_collects_files_from_given_directories_and_ignores_non_yaml_filenames():
config_paths = ('/etc/borgmatic.d',)
mock_path = flexmock(module.os.path)
mock_path.should_receive('exists').and_return(True)
mock_path.should_receive('isdir').with_args('/etc/borgmatic.d').and_return(True)
mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/foo.yaml').and_return(False)
mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/bar.yaml~').and_return(False)
mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/baz.txt').and_return(False)
flexmock(module.os).should_receive('listdir').and_return(['foo.yaml', 'bar.yaml~', 'baz.txt'])
config_filenames = tuple(module.collect_config_filenames(config_paths))
assert config_filenames == ('/etc/borgmatic.d/foo.yaml',)
def test_collect_config_filenames_skips_etc_borgmatic_config_dot_yaml_if_it_does_not_exist():
config_paths = ('config.yaml', '/etc/borgmatic/config.yaml')
mock_path = flexmock(module.os.path)
@@ -58,6 +89,19 @@ def test_collect_config_filenames_skips_etc_borgmatic_dot_d_if_it_does_not_exist
assert config_filenames == ('config.yaml',)
def test_collect_config_filenames_skips_non_canonical_etc_borgmatic_dot_d_if_it_does_not_exist():
config_paths = ('config.yaml', '/etc/../etc/borgmatic.d')
mock_path = flexmock(module.os.path)
mock_path.should_receive('exists').with_args('config.yaml').and_return(True)
mock_path.should_receive('exists').with_args('/etc/../etc/borgmatic.d').and_return(False)
mock_path.should_receive('isdir').with_args('config.yaml').and_return(False)
mock_path.should_receive('isdir').with_args('/etc/../etc/borgmatic.d').and_return(True)
config_filenames = tuple(module.collect_config_filenames(config_paths))
assert config_filenames == ('config.yaml',)
def test_collect_config_filenames_includes_other_directory_if_it_does_not_exist():
config_paths = ('config.yaml', '/my/directory')
mock_path = flexmock(module.os.path)
+31 -1
View File
@@ -1,4 +1,5 @@
import pytest
from flexmock import flexmock
from borgmatic.config import validate as module
@@ -23,13 +24,42 @@ def test_apply_logical_validation_raises_if_archive_name_format_present_without_
},
)
def test_apply_logical_validation_raises_if_archive_name_format_present_without_retention_prefix():
with pytest.raises(module.Validation_error):
module.apply_logical_validation(
'config.yaml',
{
'storage': {'archive_name_format': '{hostname}-{now}'},
'retention': {'keep_daily': 7},
'consistency': {'prefix': '{hostname}-'}
},
)
def test_apply_logical_validation_warns_if_archive_name_format_present_without_consistency_prefix():
logger = flexmock(module.logger)
logger.should_receive('warning').once()
def test_apply_logical_validation_does_not_raise_if_archive_name_format_and_prefix_present():
module.apply_logical_validation(
'config.yaml',
{
'storage': {'archive_name_format': '{hostname}-{now}'},
'retention': {'prefix': '{hostname}-'},
'consistency': {},
},
)
def test_apply_logical_validation_does_not_raise_or_warn_if_archive_name_format_and_prefix_present():
logger = flexmock(module.logger)
logger.should_receive('warning').never()
module.apply_logical_validation(
'config.yaml',
{
'storage': {'archive_name_format': '{hostname}-{now}'},
'retention': {'prefix': '{hostname}-'},
'consistency': {'prefix': '{hostname}-'}
},
)
+2 -2
View File
@@ -7,5 +7,5 @@ def test_verbosity_to_log_level_maps_known_verbosity_to_log_level():
assert module.verbosity_to_log_level(module.VERBOSITY_SOME) == logging.INFO
def test_verbosity_to_log_level_maps_unknown_verbosity_to_error_level():
assert module.verbosity_to_log_level('my pants') == logging.ERROR
def test_verbosity_to_log_level_maps_unknown_verbosity_to_warning_level():
assert module.verbosity_to_log_level('my pants') == logging.WARNING
+1 -1
View File
@@ -12,4 +12,4 @@ def verbosity_to_log_level(verbosity):
return {
VERBOSITY_SOME: logging.INFO,
VERBOSITY_LOTS: logging.DEBUG,
}.get(verbosity, logging.ERROR)
}.get(verbosity, logging.WARNING)
+1
View File
@@ -3,6 +3,7 @@ Description=Run borgmatic backup
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
+49
View File
@@ -0,0 +1,49 @@
#!/bin/bash
set -o nounset
# For each Borg sub-command that borgmatic uses, print out the Borg flags that borgmatic does not
# appear to support yet. This script isn't terribly robust. It's intended as a basic tool to ferret
# out unsupported Borg options so that they can be considered for addition to borgmatic.
# Generate a sample borgmatic configuration with all options set.
generate-borgmatic-config --destination temp.yaml
# For each sub-command (prune, create, and check), collect the Borg command-line flags that result
# from running borgmatic with the generated configuration. Then, collect the full set of available
# Borg flags as reported by "borg --help" for that sub-command. Finally, compare the two lists of
# flags to determine which Borg flags borgmatic doesn't yet support.
for sub_command in prune create check list info; do
echo "********** borg $sub_command **********"
for line in $(borgmatic --config temp.yaml --$sub_command -v 2 2>&1 | grep "borg\w* $sub_command") ; do
echo "$line" | grep '^-' >> borgmatic_borg_flags
done
sort borgmatic_borg_flags > borgmatic_borg_flags.sorted
mv borgmatic_borg_flags.sorted borgmatic_borg_flags
for line in $(borg $sub_command --help | awk -v RS= '/^usage:/') ; do
# Exclude a bunch of flags that borgmatic actually supports, but don't get exercised by the
# generated sample config, and also flags that don't make sense to support.
echo "$line" | grep -- -- | sed -r 's/(\[|\])//g' \
| grep -v '^-h$' \
| grep -v '^--archives-only$' \
| grep -v '^--repository-only$' \
| grep -v '^--stats$' \
| grep -v '^--list$' \
| grep -v '^--critical$' \
| grep -v '^--error$' \
| grep -v '^--warning$' \
| grep -v '^--info$' \
| grep -v '^--debug$' \
>> all_borg_flags
done
sort all_borg_flags > all_borg_flags.sorted
mv all_borg_flags.sorted all_borg_flags
comm -13 borgmatic_borg_flags all_borg_flags
rm ./*_borg_flags
done
rm temp.yaml
+2 -2
View File
@@ -2,5 +2,5 @@
set -e
git push -u origin
git push -u github
git push -u origin master
git push -u github master
+5
View File
@@ -2,6 +2,11 @@
set -e
version=$(head --lines=1 NEWS)
git tag $version
git push origin $version
git push github $version
rm -fr dist
python3 setup.py bdist_wheel
python3 setup.py sdist
+3 -4
View File
@@ -1,7 +1,7 @@
from setuptools import setup, find_packages
VERSION = '1.1.9'
VERSION = '1.2.2'
setup(
@@ -11,7 +11,6 @@ setup(
author='Dan Helfman',
author_email='witten@torsion.org',
url='https://torsion.org/borgmatic',
download_url='https://projects.torsion.org/witten/borgmatic/archive/%s.tar.gz' % VERSION,
classifiers=(
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
@@ -33,8 +32,8 @@ setup(
'atticmatic',
],
install_requires=(
'pykwalify',
'ruamel.yaml<=0.15',
'pykwalify>=1.6.0',
'ruamel.yaml>0.15.0,<0.16.0',
'setuptools',
),
tests_require=(
+3 -3
View File
@@ -1,5 +1,5 @@
flexmock==0.10.2
pykwalify==1.6.0
pytest==2.9.1
pykwalify==1.6.1
pytest==3.6.3
pytest-cov==2.5.1
ruamel.yaml==0.15.18
ruamel.yaml>0.15.0,<0.16.0
+1 -1
View File
@@ -1,5 +1,5 @@
[tox]
envlist=py34
envlist=py3
skipsdist=True
[testenv]