Bump version.

Closes #94.

.gitignore

@@ -3,6 +3,7 @@
 *.swp
 .cache
 .coverage
+.pytest_cache
 .tox
 build
 dist

AUTHORS

@@ -1,11 +1,12 @@
 Dan Helfman <witten@torsion.org>: Main developer
 
 Alexander Görtz: Python 3 compatibility
+Florian Lindner: Logging rewrite
 Henning Schroeder: Copy editing
 Johannes Feichtner: Support for user hooks
 Michele Lazzeri: Custom archive names
+Nick Whyte: Support prefix filtering for archive consistency checks
 newtonne: Read encryption password from external file
 Robin `ypid` Schneider: Support additional options of Borg
 Scott Squires: Custom archive names
-Thomas LÉVEIL: Support for a keep_minutely prune option
-Nick Whyte: Support prefix filtering for archive consistency checks
+Thomas LÉVEIL: Support for a keep_minutely prune option. Support for the --json option

NEWS

@@ -1,3 +1,34 @@
+1.2.4
+ * Fix for archive checking traceback due to parameter mismatch.
+
+1.2.3
+ * #64, #90, #92: Rewrite of logging system. Now verbosity flags passed to Borg are derived from
+   borgmatic's log level. Note that the output of borgmatic might slightly change.
+ * Part of #80: Support for Borg create --read-special via "read_special" option in borgmatic's
+   location configuration.
+ * #87: Support for Borg create --checkpoint-interval via "checkpoint_interval" option in
+   borgmatic's storage configuration.
+ * #88: Fix declared pykwalify compatibility version range in setup.py to prevent use of ancient
+   versions of pykwalify with large version numbers.
+ * #89: Pass --show-rc option to Borg when at highest verbosity level.
+ * #94: Support for Borg --json option via borgmatic command-line to --create archives.
+
+1.2.2
+ * #85: Fix compatibility issue between pykwalify and ruamel.yaml 0.15.52, which manifested in
+   borgmatic as a pykwalify RuleError.
+
+1.2.1
+ * Skip before/after backup hooks when only doing --prune, --check, --list, and/or --info.
+ * #71: Support for XDG_CONFIG_HOME environment variable for specifying alternate user ~/.config/
+   path.
+ * #74, #83: Support for Borg --json option via borgmatic command-line to --list archives or show
+   archive --info in JSON format, ideal for programmatic consumption.
+ * #38, #76: Upgrade ruamel.yaml compatibility version range and fix support for Python 3.7.
+ * #77: Skip non-"*.yaml" config filenames in /etc/borgmatic.d/ so as not to parse backup files,
+   editor swap files, etc.
+ * #81: Document user-defined hooks run before/after backup, or on error.
+ * Add code style guidelines to the documention.
+
 1.2.0
  * #61: Support for Borg --list option via borgmatic command-line to list all archives.
  * #61: Support for Borg --info option via borgmatic command-line to display summary information.

README.md

@@ -1,13 +1,17 @@
-<img src="https://projects.torsion.org/witten/borgmatic/raw/branch/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
-
+---
+title: borgmatic
+permalink: borgmatic/index.html
+---
 ## Overview
 
-borgmatic (formerly atticmatic) is a simple Python wrapper script for the
-[Borg](https://borgbackup.readthedocs.org/en/stable/) backup software that
-initiates a backup, prunes any old backups according to a retention policy,
-and validates backups for consistency. The script supports specifying your
-settings in a declarative configuration file rather than having to put them
-all on the command-line, and handles common errors.
+<img src="https://projects.torsion.org/witten/borgmatic/raw/branch/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
+
+borgmatic is a simple Python wrapper script for the
+[Borg](https://www.borgbackup.org/) backup software that initiates a backup,
+prunes any old backups according to a retention policy, and validates backups
+for consistency. The script supports specifying your settings in a declarative
+configuration file rather than having to put them all on the command-line, and
+handles common errors.
 
 Here's an example config file:
 
@@ -138,6 +142,31 @@ configuration paths on the command-line with borgmatic's `--config` option.
 See `borgmatic --help` for more information.
 
 
+### Hooks
+
+If you find yourself performing prepraration tasks before your backup runs, or
+cleanup work afterwards, borgmatic hooks may be of interest. They're simply
+shell commands that borgmatic executes for you at various points, and they're
+configured in the `hooks` section of your configuration file.
+
+For instance, you can specify `before_backup` hooks to dump a database to file
+before backing it up, and specify `after_backup` hooks to delete the temporary
+file afterwards.
+
+borgmatic hooks run once per configuration file. `before_backup` hooks run
+prior to backups of all repositories. `after_backup` hooks run afterwards, but
+not if an error occurs in a previous hook or in the backups themselves. And
+borgmatic runs `on_error` hooks if an error occurs.
+
+An important security note about hooks: borgmatic executes all hook commands
+with the user permissions of borgmatic itself. So to prevent potential shell
+injection or privilege escalation, do not forget to set secure permissions
+(`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
+
+See the sample generated configuration file mentioned above for specifics
+about hook configuration syntax.
+
+
 ## Upgrading
 
 In general, all you should need to do to upgrade borgmatic is run the
@@ -266,6 +295,20 @@ any number of them. This supports use cases like running consistency checks
 from a different cron job with a different frequency, or running pruning with
 a different verbosity level.
 
+Additionally, borgmatic provides convenient flags for Borg's
+[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
+[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
+functionality:
+
+
+```bash
+borgmatic --list
+borgmatic --info
+```
+
+You can include an optional `--json` flag with `--create`, `--list`, or
+`--info` to get the output formatted as JSON.
+
 
 ## Autopilot
 
@@ -275,7 +318,7 @@ configure a job runner to invoke it periodically.
 ### cron
 
 If you're using cron, download the [sample cron
-file](https://projects.torsion.org/witten/borgmatic/raw/master/sample/cron/borgmatic).
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
 Then, from the directory where you downloaded it:
 
 ```bash
@@ -326,11 +369,25 @@ to discuss your idea. We also accept Pull Requests on GitHub, if that's more
 your thing. In general, contributions are very welcome. We don't bite! 
 
 
+### Code style
+
+Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
+the following deviations from it:
+
+ * For strings, prefer single quotes over double quotes.
+ * Limit all lines to a maximum of 100 characters.
+ * Use trailing commas within multiline values or argument lists.
+ * For multiline constructs, put opening and closing delimeters on lines
+   separate from their contents.
+ * Within multiline constructs, use standard four-space indentation. Don't align
+   indentation with an opening delimeter.
+
+
 ### Development
 
 To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
 
-```
+```bash
 git clone https://projects.torsion.org/witten/borgmatic.git
 ```
 
@@ -341,7 +398,7 @@ git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
 ```
 
 Then, install borgmatic
-"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
+"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
 so that you can easily run borgmatic commands while you're hacking on them to
 make sure your changes work.
 

borgmatic/borg/check.py

@@ -3,7 +3,6 @@ import os
 import subprocess
 
 from borgmatic.borg import extract
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
 
 DEFAULT_CHECKS = ('repository', 'archives')
@@ -74,12 +73,10 @@ def _make_check_flags(checks, check_last=None, prefix=None):
     ) + last_flags + prefix_flags
 
 
-def check_archives(verbosity, repository, storage_config, consistency_config, local_path='borg',
-                   remote_path=None):
+def check_archives(repository, storage_config, consistency_config, local_path='borg', remote_path=None):
     '''
-    Given a verbosity flag, a local or remote repository path, a storage config dict, a consistency
-    config dict, and a local/remote commands to run, check the contained Borg archives for
-    consistency.
+    Given a local or remote repository path, a storage config dict, a consistency config dict, 
+    and a local/remote commands to run, check the contained Borg archives for consistency.
 
     If there are no consistency checks to run, skip running them.
     '''
@@ -91,10 +88,13 @@ def check_archives(verbosity, repository, storage_config, consistency_config, lo
         remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
         lock_wait = storage_config.get('lock_wait', None)
         lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
-        verbosity_flags = {
-            VERBOSITY_SOME: ('--info',),
-            VERBOSITY_LOTS: ('--debug',),
-        }.get(verbosity, ())
+
+        verbosity_flags = ()
+        if logger.isEnabledFor(logging.INFO):
+            verbosity_flags = ('--info',)
+        if logger.isEnabledFor(logging.DEBUG):
+            verbosity_flags = ('--debug', '--show-rc')
+
         prefix = consistency_config.get('prefix')
 
         full_command = (
@@ -109,4 +109,4 @@ def check_archives(verbosity, repository, storage_config, consistency_config, lo
         subprocess.check_call(full_command, stdout=stdout, stderr=subprocess.STDOUT)
 
     if 'extract' in checks:
-        extract.extract_last_archive_dry_run(verbosity, repository, lock_wait, local_path, remote_path)
+        extract.extract_last_archive_dry_run(repository, lock_wait, local_path, remote_path)

borgmatic/borg/create.py

@@ -5,8 +5,6 @@ import os
 import subprocess
 import tempfile
 
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
-
 
 logger = logging.getLogger(__name__)
 
@@ -105,8 +103,7 @@ def _make_exclude_flags(location_config, exclude_filename=None):
 
 
 def create_archive(
-    verbosity, dry_run, repository, location_config, storage_config, local_path='borg', remote_path=None,
-):
+        dry_run, repository, location_config, storage_config, local_path='borg', remote_path=None, json=False):
     '''
     Given vebosity/dry-run flags, a local or remote repository path, a location config dict, and a
     storage config dict, create a Borg archive.
@@ -115,6 +112,7 @@ def create_archive(
 
     pattern_file = _write_pattern_file(location_config.get('patterns'))
     exclude_file = _write_pattern_file(_expand_directories(location_config.get('exclude_patterns')))
+    checkpoint_interval = storage_config.get('checkpoint_interval', None)
     compression = storage_config.get('compression', None)
     remote_rate_limit = storage_config.get('remote_rate_limit', None)
     umask = storage_config.get('umask', None)
@@ -140,19 +138,22 @@ def create_archive(
             location_config,
             exclude_file.name if exclude_file else None,
         )
+        + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--compression', compression) if compression else ())
         + (('--remote-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ())
         + (('--one-file-system',) if location_config.get('one_file_system') else ())
+        + (('--read-special',) if location_config.get('read_special') else ())
         + (('--nobsdflags',) if location_config.get('bsd_flags') is False else ())
         + (('--files-cache', files_cache) if files_cache else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + {
-            VERBOSITY_SOME: ('--info',) if dry_run else ('--info', '--stats',),
-            VERBOSITY_LOTS: ('--debug', '--list',) if dry_run else ('--debug', '--list', '--stats',),
-        }.get(verbosity, ())
+        + (('--list', '--filter', 'AME',) if logger.isEnabledFor(logging.INFO) else ())
+        + (( '--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--stats',) if not dry_run and logger.isEnabledFor(logging.INFO) else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
+        + (('--json',) if json else ())
     )
 
     logger.debug(' '.join(full_command))

borgmatic/borg/extract.py

@@ -2,23 +2,23 @@ import logging
 import sys
 import subprocess
 
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
-
 
 logger = logging.getLogger(__name__)
 
 
-def extract_last_archive_dry_run(verbosity, repository, lock_wait=None, local_path='borg', remote_path=None):
+def extract_last_archive_dry_run(repository, lock_wait=None, local_path='borg', remote_path=None):
     '''
     Perform an extraction dry-run of just the most recent archive. If there are no archives, skip
     the dry-run.
     '''
     remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
     lock_wait_flags = ('--lock-wait', str(lock_wait)) if lock_wait else ()
-    verbosity_flags = {
-        VERBOSITY_SOME: ('--info',),
-        VERBOSITY_LOTS: ('--debug',),
-    }.get(verbosity, ())
+    verbosity_flags = ()
+    if logger.isEnabledFor(logging.DEBUG):
+        verbosity_flags = ('--debug', '--show-rc')
+    elif logger.isEnabledFor(logging.INFO):
+        verbosity_flags = ('--info',)
+    
 
     full_list_command = (
         local_path, 'list',
@@ -32,7 +32,7 @@ def extract_last_archive_dry_run(verbosity, repository, lock_wait=None, local_pa
     if not last_archive_name:
         return
 
-    list_flag = ('--list',) if verbosity == VERBOSITY_LOTS else ()
+    list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
     full_extract_command = (
         local_path, 'extract',
         '--dry-run',

borgmatic/borg/info.py

@@ -1,16 +1,13 @@
 import logging
 import subprocess
 
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
-
 
 logger = logging.getLogger(__name__)
 
 
-def display_archives_info(verbosity, repository, storage_config, local_path='borg',
-                          remote_path=None):
+def display_archives_info(repository, storage_config, local_path='borg', remote_path=None, json=False):
     '''
-    Given a verbosity flag, a local or remote repository path, and a storage config dict,
+    Given a local or remote repository path, and a storage config dict,
     display summary information for Borg archives in the repository.
     '''
     lock_wait = storage_config.get('lock_wait', None)
@@ -19,11 +16,12 @@ def display_archives_info(verbosity, repository, storage_config, local_path='bor
         (local_path, 'info', repository)
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + {
-            VERBOSITY_SOME: ('--info',),
-            VERBOSITY_LOTS: ('--debug',),
-        }.get(verbosity, ())
+        + (('--info',)  if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (('--json',) if json else ())
     )
 
     logger.debug(' '.join(full_command))
-    subprocess.check_call(full_command)
+
+    output = subprocess.check_output(full_command)
+    return output.decode() if output is not None else None

borgmatic/borg/list.py

@@ -1,15 +1,13 @@
 import logging
 import subprocess
 
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
-
 
 logger = logging.getLogger(__name__)
 
 
-def list_archives(verbosity, repository, storage_config, local_path='borg', remote_path=None):
+def list_archives(repository, storage_config, local_path='borg', remote_path=None, json=False):
     '''
-    Given a verbosity flag, a local or remote repository path, and a storage config dict,
+    Given a local or remote repository path, and a storage config dict,
     list Borg archives in the repository.
     '''
     lock_wait = storage_config.get('lock_wait', None)
@@ -18,11 +16,11 @@ def list_archives(verbosity, repository, storage_config, local_path='borg', remo
         (local_path, 'list', repository)
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + {
-            VERBOSITY_SOME: ('--info',),
-            VERBOSITY_LOTS: ('--debug',),
-        }.get(verbosity, ())
+        + (('--info',)  if logger.getEffectiveLevel() == logging.INFO  else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (('--json',) if json else ())
     )
-
     logger.debug(' '.join(full_command))
-    subprocess.check_call(full_command)
+
+    output = subprocess.check_output(full_command)
+    return output.decode() if output is not None else None

borgmatic/borg/prune.py

@@ -1,7 +1,6 @@
 import logging
 import subprocess
 
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
 
 logger = logging.getLogger(__name__)
@@ -32,10 +31,9 @@ def _make_prune_flags(retention_config):
     )
 
 
-def prune_archives(verbosity, dry_run, repository, storage_config, retention_config,
-                   local_path='borg', remote_path=None):
+def prune_archives(dry_run, repository, storage_config, retention_config, local_path='borg', remote_path=None):
     '''
-    Given verbosity/dry-run flags, a local or remote repository path, a storage config dict, and a
+    Given dry-run flag, a local or remote repository path, a storage config dict, and a
     retention config dict, prune Borg archives according to the retention policy specified in that
     configuration.
     '''
@@ -54,10 +52,9 @@ def prune_archives(verbosity, dry_run, repository, storage_config, retention_con
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + {
-            VERBOSITY_SOME: ('--info', '--stats',),
-            VERBOSITY_LOTS: ('--debug', '--stats', '--list'),
-        }.get(verbosity, ())
+        + (('--stats',) if not dry_run and logger.isEnabledFor(logging.INFO) else ())
+        + (( '--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
     )
 

borgmatic/commands/borgmatic.py

@@ -1,5 +1,5 @@
-
 from argparse import ArgumentParser
+import json
 import logging
 import os
 from subprocess import CalledProcessError
@@ -10,7 +10,7 @@ from borgmatic.borg import check as borg_check, create as borg_create, prune as
 from borgmatic.commands import hook
 from borgmatic.config import collect, convert, validate
 from borgmatic.signals import configure_signals
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS, verbosity_to_log_level
+from borgmatic.verbosity import verbosity_to_log_level
 
 
 logger = logging.getLogger(__name__)
@@ -24,6 +24,8 @@ def parse_arguments(*arguments):
     Given command-line arguments with which this script was invoked, parse the arguments and return
     them as an ArgumentParser instance.
     '''
+    config_paths = collect.get_default_config_paths()
+
     parser = ArgumentParser(
         description=
             '''
@@ -36,8 +38,8 @@ def parse_arguments(*arguments):
         '-c', '--config',
         nargs='+',
         dest='config_paths',
-        default=collect.DEFAULT_CONFIG_PATHS,
-        help='Configuration filenames or directories, defaults to: {}'.format(' '.join(collect.DEFAULT_CONFIG_PATHS)),
+        default=config_paths,
+        help='Configuration filenames or directories, defaults to: {}'.format(' '.join(config_paths)),
     )
     parser.add_argument(
         '--excludes',
@@ -74,6 +76,13 @@ def parse_arguments(*arguments):
         action='store_true',
         help='Display summary information on archives',
     )
+    parser.add_argument(
+        '--json',
+        dest='json',
+        default=False,
+        action='store_true',
+        help='Output results from the --create, --list, or --info options as json',
+    )
     parser.add_argument(
         '-n', '--dry-run',
         dest='dry_run',
@@ -83,11 +92,21 @@ def parse_arguments(*arguments):
     parser.add_argument(
         '-v', '--verbosity',
         type=int,
+        choices=range(0,3),
+        default=0,
         help='Display verbose progress (1 for some, 2 for lots)',
     )
 
     args = parser.parse_args(arguments)
 
+    if args.json and not (args.create or args.list or args.info):
+        raise ValueError('The --json option can only be used with the --create, --list, or --info options')
+
+    if args.json and args.list and args.info:
+        raise ValueError(
+            'With the --json option, options --list and --info cannot be used together'
+        )
+
     # If any of the action flags are explicitly requested, leave them as-is. Otherwise, assume
     # defaults: Mutate the given arguments to enable the default actions.
     if args.prune or args.create or args.check or args.list or args.info:
@@ -115,68 +134,93 @@ def run_configuration(config_filename, args):  # pragma: no cover
         local_path = location.get('local_path', 'borg')
         remote_path = location.get('remote_path')
         borg_create.initialize_environment(storage)
-        hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
 
-        for unexpanded_repository in location['repositories']:
-            repository = os.path.expanduser(unexpanded_repository)
-            dry_run_label = ' (dry run; not making any changes)' if args.dry_run else ''
-            if args.prune:
-                logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
-                borg_prune.prune_archives(
-                    args.verbosity,
-                    args.dry_run,
-                    repository,
-                    storage,
-                    retention,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
-            if args.create:
-                logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
-                borg_create.create_archive(
-                    args.verbosity,
-                    args.dry_run,
-                    repository,
-                    location,
-                    storage,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
-            if args.check:
-                logger.info('{}: Running consistency checks'.format(repository))
-                borg_check.check_archives(
-                   args.verbosity,
-                    repository,
-                    storage,
-                    consistency,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
-            if args.list:
-                logger.info('{}: Listing archives'.format(repository))
-                borg_list.list_archives(
-                    args.verbosity,
-                    repository,
-                    storage,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
-            if args.info:
-                logger.info('{}: Displaying summary info for archives'.format(repository))
-                borg_info.display_archives_info(
-                    args.verbosity,
-                    repository,
-                    storage,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
+        if args.create:
+            hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
 
-        hook.execute_hook(hooks.get('after_backup'), config_filename, 'post-backup')
+        _run_commands(args, consistency, local_path, location, remote_path, retention, storage)
+
+        if args.create:
+            hook.execute_hook(hooks.get('after_backup'), config_filename, 'post-backup')
     except (OSError, CalledProcessError):
         hook.execute_hook(hooks.get('on_error'), config_filename, 'on-error')
         raise
 
 
+def _run_commands(args, consistency, local_path, location, remote_path, retention, storage):
+    json_results = []
+    for unexpanded_repository in location['repositories']:
+        _run_commands_on_repository(
+            args, consistency, json_results, local_path, location, remote_path, retention, storage,
+            unexpanded_repository,
+        )
+    if args.json:
+        sys.stdout.write(json.dumps(json_results))
+
+
+def _run_commands_on_repository(
+    args, consistency, json_results, local_path, location, remote_path,
+    retention, storage, unexpanded_repository,
+):  # pragma: no cover
+    repository = os.path.expanduser(unexpanded_repository)
+    dry_run_label = ' (dry run; not making any changes)' if args.dry_run else ''
+    if args.prune:
+        logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
+        borg_prune.prune_archives(
+            args.dry_run,
+            repository,
+            storage,
+            retention,
+            local_path=local_path,
+            remote_path=remote_path,
+        )
+    if args.create:
+        logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
+        borg_create.create_archive(
+            args.dry_run,
+            repository,
+            location,
+            storage,
+            local_path=local_path,
+            remote_path=remote_path,
+        )
+    if args.check:
+        logger.info('{}: Running consistency checks'.format(repository))
+        borg_check.check_archives(
+            repository,
+            storage,
+            consistency,
+            local_path=local_path,
+            remote_path=remote_path
+        )
+    if args.list:
+        logger.info('{}: Listing archives'.format(repository))
+        output = borg_list.list_archives(
+            repository,
+            storage,
+            local_path=local_path,
+            remote_path=remote_path,
+            json=args.json,
+        )
+        if args.json:
+            json_results.append(json.loads(output))
+        else:
+            sys.stdout.write(output)
+    if args.info:
+        logger.info('{}: Displaying summary info for archives'.format(repository))
+        output = borg_info.display_archives_info(
+            repository,
+            storage,
+            local_path=local_path,
+            remote_path=remote_path,
+            json=args.json,
+        )
+        if args.json:
+            json_results.append(json.loads(output))
+        else:
+            sys.stdout.write(output)
+
+
 def main():  # pragma: no cover
     try:
         configure_signals()

borgmatic/config/collect.py

@@ -1,24 +1,35 @@
 import os
 
 
-DEFAULT_CONFIG_PATHS = [
-    '/etc/borgmatic/config.yaml',
-    '/etc/borgmatic.d',
-    os.path.expanduser('~/.config/borgmatic/config.yaml'),
-]
+def get_default_config_paths():
+    '''
+    Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
+    default configuration paths. This includes both system-wide configuration and configuration in
+    the current user's home directory.
+    '''
+    user_config_directory = (
+        os.getenv('XDG_CONFIG_HOME') or os.path.expandvars(os.path.join('$HOME', '.config'))
+    )
+
+    return [
+        '/etc/borgmatic/config.yaml',
+        '/etc/borgmatic.d',
+        '%s/borgmatic/config.yaml' % user_config_directory,
+    ]
 
 
 def collect_config_filenames(config_paths):
     '''
     Given a sequence of config paths, both filenames and directories, resolve that to just an
     iterable of files. Accomplish this by listing any given directories looking for contained config
-    files. This is non-recursive, so any directories within the given directories are ignored.
+    files (ending with the ".yaml" extension). This is non-recursive, so any directories within the
+    given directories are ignored.
 
     Return paths even if they don't exist on disk, so the user can find out about missing
     configuration paths. However, skip a default config path if it's missing, so the user doesn't
     have to create a default config path unless they need it.
     '''
-    real_default_config_paths = set(map(os.path.realpath, DEFAULT_CONFIG_PATHS))
+    real_default_config_paths = set(map(os.path.realpath, get_default_config_paths()))
 
     for path in config_paths:
         exists = os.path.exists(path)
@@ -32,5 +43,5 @@ def collect_config_filenames(config_paths):
 
         for filename in os.listdir(path):
             full_filename = os.path.join(path, filename)
-            if not os.path.isdir(full_filename):
+            if full_filename.endswith('.yaml') and not os.path.isdir(full_filename):
                 yield full_filename

borgmatic/config/schema.yaml

@@ -22,6 +22,13 @@ map:
                 type: bool
                 desc: Stay in same file system (do not cross mount points).
                 example: true
+            read_special:
+                type: bool
+                desc: |
+                    Use Borg's --read-special flag to allow backup of block and other special
+                    devices. Use with caution, as it will lead to problems if used when
+                    backing up special devices such as /dev/zero.
+                example: false
             bsd_flags:
                 type: bool
                 desc: Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive. Defaults to true.
@@ -124,6 +131,13 @@ map:
                     punctuation, so it parses correctly. And backslash any quote or backslash
                     literals as well.
                 example: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
+            checkpoint_interval:
+                type: int
+                desc: |
+                    Number of seconds between each checkpoint during a long-running backup. See
+                    https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there
+                    for details. Defaults to checkpoints every 1800 seconds (30 minutes).
+                example: 1800
             compression:
                 type: scalar
                 desc: |

borgmatic/config/validate.py

@@ -10,6 +10,7 @@ from ruamel import yaml
 
 logger = logging.getLogger(__name__)
 
+
 def schema_filename():
     '''
     Path to the installed YAML configuration schema file, used to validate and parse the
@@ -73,8 +74,8 @@ def parse_configuration(config_filename, schema_filename):
     logging.getLogger('pykwalify').setLevel(logging.ERROR)
 
     try:
-        config = yaml.round_trip_load(open(config_filename))
-        schema = yaml.round_trip_load(open(schema_filename))
+        config = yaml.safe_load(open(config_filename))
+        schema = yaml.safe_load(open(schema_filename))
     except yaml.error.YAMLError as error:
         raise Validation_error(config_filename, (str(error),))
 

borgmatic/tests/integration/commands/test_borgmatic.py

@@ -7,37 +7,55 @@ from borgmatic.commands import borgmatic as module
 
 
 def test_parse_arguments_with_no_arguments_uses_defaults():
+    config_paths = ['default']
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(config_paths)
+
     parser = module.parse_arguments()
 
-    assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
+    assert parser.config_paths == config_paths
     assert parser.excludes_filename == None
-    assert parser.verbosity is None
+    assert parser.verbosity is 0
+    assert parser.json is False
 
 
 def test_parse_arguments_with_path_arguments_overrides_defaults():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
 
     assert parser.config_paths == ['myconfig']
     assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity is None
+    assert parser.verbosity is 0
 
 
 def test_parse_arguments_with_multiple_config_paths_parses_as_list():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     parser = module.parse_arguments('--config', 'myconfig', 'otherconfig')
 
     assert parser.config_paths == ['myconfig', 'otherconfig']
-    assert parser.verbosity is None
+    assert parser.verbosity is 0
 
 
 def test_parse_arguments_with_verbosity_flag_overrides_default():
+    config_paths = ['default']
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(config_paths)
+
     parser = module.parse_arguments('--verbosity', '1')
 
-    assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
+    assert parser.config_paths == config_paths
     assert parser.excludes_filename == None
     assert parser.verbosity == 1
 
 
+def test_parse_arguments_with_json_flag_overrides_default():
+    parser = module.parse_arguments('--list', '--json')
+    assert parser.json is True
+
+
 def test_parse_arguments_with_no_actions_defaults_to_all_actions_enabled():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     parser = module.parse_arguments()
 
     assert parser.prune is True
@@ -46,6 +64,8 @@ def test_parse_arguments_with_no_actions_defaults_to_all_actions_enabled():
 
 
 def test_parse_arguments_with_prune_action_leaves_other_actions_disabled():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     parser = module.parse_arguments('--prune')
 
     assert parser.prune is True
@@ -54,6 +74,8 @@ def test_parse_arguments_with_prune_action_leaves_other_actions_disabled():
 
 
 def test_parse_arguments_with_multiple_actions_leaves_other_action_disabled():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     parser = module.parse_arguments('--create', '--check')
 
     assert parser.prune is False
@@ -62,5 +84,22 @@ def test_parse_arguments_with_multiple_actions_leaves_other_action_disabled():
 
 
 def test_parse_arguments_with_invalid_arguments_exits():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
     with pytest.raises(SystemExit):
         module.parse_arguments('--posix-me-harder')
+
+
+def test_parse_arguments_with_json_flag_with_list_or_info_flag_does_not_raise_any_error():
+    module.parse_arguments('--list', '--json')
+    module.parse_arguments('--info', '--json')
+
+
+def test_parse_arguments_with_json_flag_but_no_list_or_info_flag_raises_value_error():
+    with pytest.raises(ValueError):
+        module.parse_arguments('--json')
+
+
+def test_parse_arguments_with_json_flag_and_both_list_and_info_flag_raises_value_error():
+    with pytest.raises(ValueError):
+        module.parse_arguments('--list', '--info', '--json')

borgmatic/tests/unit/borg/test_check.py

@@ -1,12 +1,11 @@
 from subprocess import STDOUT
-import sys
+import logging, sys
 
 from flexmock import flexmock
 import pytest
 
 from borgmatic.borg import check as module
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
-
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess = flexmock(module.subprocess)
@@ -125,7 +124,6 @@ def test_check_archives_calls_borg_with_parameters(checks):
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
@@ -142,43 +140,42 @@ def test_check_archives_with_extract_check_calls_extract_only():
     insert_subprocess_never()
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
     )
 
 
-def test_check_archives_with_verbosity_some_calls_borg_with_info_parameter():
+def test_check_archives_with_log_info_calls_borg_with_info_parameter():
     checks = ('repository',)
     consistency_config = {'check_last': None}
     flexmock(module).should_receive('_parse_checks').and_return(checks)
     flexmock(module).should_receive('_make_check_flags').and_return(())
+    insert_logging_mock(logging.INFO)
     insert_subprocess_mock(
         ('borg', 'check', 'repo', '--info'),
         stdout=None, stderr=STDOUT,
     )
 
     module.check_archives(
-        verbosity=VERBOSITY_SOME,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
     )
 
 
-def test_check_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
+def test_check_archives_with_log_debug_calls_borg_with_debug_parameter():
     checks = ('repository',)
     consistency_config = {'check_last': None}
     flexmock(module).should_receive('_parse_checks').and_return(checks)
     flexmock(module).should_receive('_make_check_flags').and_return(())
+    insert_logging_mock(logging.DEBUG)
     insert_subprocess_mock(
-        ('borg', 'check', 'repo', '--debug'),
+        ('borg', 'check', 'repo', '--debug', '--show-rc'),
         stdout=None, stderr=STDOUT,
     )
 
     module.check_archives(
-        verbosity=VERBOSITY_LOTS,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
@@ -191,7 +188,6 @@ def test_check_archives_without_any_checks_bails():
     insert_subprocess_never()
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
@@ -213,7 +209,6 @@ def test_check_archives_with_local_path_calls_borg_via_local_path():
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
@@ -236,7 +231,6 @@ def test_check_archives_with_remote_path_calls_borg_with_remote_path_parameters(
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,
@@ -259,7 +253,6 @@ def test_check_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={'lock_wait': 5},
         consistency_config=consistency_config,
@@ -283,7 +276,6 @@ def test_check_archives_with_retention_prefix():
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         consistency_config=consistency_config,

borgmatic/tests/unit/borg/test_create.py

@@ -1,9 +1,9 @@
-import os
+import logging, os
 
 from flexmock import flexmock
 
 from borgmatic.borg import create as module
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 
 def test_initialize_environment_with_passcommand_should_set_environment():
@@ -216,7 +216,6 @@ def test_create_archive_calls_borg_with_parameters():
     insert_subprocess_mock(CREATE_COMMAND)
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -237,7 +236,6 @@ def test_create_archive_with_patterns_calls_borg_with_patterns():
     insert_subprocess_mock(CREATE_COMMAND + pattern_flags)
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -258,7 +256,6 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
     insert_subprocess_mock(CREATE_COMMAND + exclude_flags)
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -270,16 +267,16 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
     )
 
 
-def test_create_archive_with_verbosity_some_calls_borg_with_info_parameter():
+def test_create_archive_with_log_info_calls_borg_with_info_parameter():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--info', '--stats',))
-
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME', '--info', '--stats',))
+    insert_logging_mock(logging.INFO)
+    
     module.create_archive(
-        verbosity=VERBOSITY_SOME,
         dry_run=False,
         repository='repo',
         location_config={
@@ -291,15 +288,15 @@ def test_create_archive_with_verbosity_some_calls_borg_with_info_parameter():
     )
 
 
-def test_create_archive_with_verbosity_lots_calls_borg_with_debug_parameter():
+def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--stats'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME','--stats', '--debug', '--show-rc'))
+    insert_logging_mock(logging.DEBUG)
 
     module.create_archive(
-        verbosity=VERBOSITY_LOTS,
         dry_run=False,
         repository='repo',
         location_config={
@@ -320,7 +317,6 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
     insert_subprocess_mock(CREATE_COMMAND + ('--dry-run',))
 
     module.create_archive(
-        verbosity=None,
         dry_run=True,
         repository='repo',
         location_config={
@@ -332,16 +328,18 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
     )
 
 
-def test_create_archive_with_dry_run_and_verbosity_some_calls_borg_without_stats_parameter():
+def test_create_archive_with_dry_run_and_log_info_calls_borg_without_stats_parameter():
+    """ --dry-run and --stats are mutually exclusive, see:
+    https://borgbackup.readthedocs.io/en/stable/usage/create.html#description """
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--info', '--dry-run'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME', '--info', '--dry-run'))
+    insert_logging_mock(logging.INFO)
 
     module.create_archive(
-        verbosity=VERBOSITY_SOME,
         dry_run=True,
         repository='repo',
         location_config={
@@ -353,16 +351,18 @@ def test_create_archive_with_dry_run_and_verbosity_some_calls_borg_without_stats
     )
 
 
-def test_create_archive_with_dry_run_and_verbosity_lots_calls_borg_without_stats_parameter():
+def test_create_archive_with_dry_run_and_log_debug_calls_borg_without_stats_parameter():
+    """ --dry-run and --stats are mutually exclusive, see:
+    https://borgbackup.readthedocs.io/en/stable/usage/create.html#description """
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--dry-run'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME', '--debug', '--show-rc', '--dry-run'))
+    insert_logging_mock(logging.DEBUG)
 
     module.create_archive(
-        verbosity=VERBOSITY_LOTS,
         dry_run=True,
         repository='repo',
         location_config={
@@ -374,6 +374,25 @@ def test_create_archive_with_dry_run_and_verbosity_lots_calls_borg_without_stats
     )
 
 
+def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_interval_parameters():
+    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
+    flexmock(module).should_receive('_write_pattern_file').and_return(None)
+    flexmock(module).should_receive('_make_pattern_flags').and_return(())
+    flexmock(module).should_receive('_make_exclude_flags').and_return(())
+    insert_subprocess_mock(CREATE_COMMAND + ('--checkpoint-interval', '600'))
+
+    module.create_archive(
+        dry_run=False,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'checkpoint_interval': 600},
+    )
+
+
 def test_create_archive_with_compression_calls_borg_with_compression_parameters():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
@@ -382,7 +401,6 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
     insert_subprocess_mock(CREATE_COMMAND + ('--compression', 'rle'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -402,7 +420,6 @@ def test_create_archive_with_remote_rate_limit_calls_borg_with_remote_ratelimit_
     insert_subprocess_mock(CREATE_COMMAND + ('--remote-ratelimit', '100'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -422,7 +439,6 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
     insert_subprocess_mock(CREATE_COMMAND + ('--one-file-system',))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -435,6 +451,26 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
     )
 
 
+def test_create_archive_with_read_special_calls_borg_with_read_special_parameter():
+    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
+    flexmock(module).should_receive('_write_pattern_file').and_return(None)
+    flexmock(module).should_receive('_make_pattern_flags').and_return(())
+    flexmock(module).should_receive('_make_exclude_flags').and_return(())
+    insert_subprocess_mock(CREATE_COMMAND + ('--read-special',))
+
+    module.create_archive(
+        dry_run=False,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'read_special': True,
+            'exclude_patterns': None,
+        },
+        storage_config={},
+    )
+
+
 def test_create_archive_with_bsd_flags_true_calls_borg_without_nobsdflags_parameter():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
@@ -443,7 +479,6 @@ def test_create_archive_with_bsd_flags_true_calls_borg_without_nobsdflags_parame
     insert_subprocess_mock(CREATE_COMMAND)
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -464,7 +499,6 @@ def test_create_archive_with_bsd_flags_false_calls_borg_with_nobsdflags_paramete
     insert_subprocess_mock(CREATE_COMMAND + ('--nobsdflags',))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -485,7 +519,6 @@ def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters(
     insert_subprocess_mock(CREATE_COMMAND + ('--files-cache', 'ctime,size'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -506,7 +539,6 @@ def test_create_archive_with_local_path_calls_borg_via_local_path():
     insert_subprocess_mock(('borg1',) + CREATE_COMMAND[1:])
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -527,7 +559,6 @@ def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters(
     insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -548,7 +579,6 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
     insert_subprocess_mock(CREATE_COMMAND + ('--umask', '740'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -568,7 +598,6 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
     insert_subprocess_mock(CREATE_COMMAND + ('--lock-wait', '5'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -580,6 +609,26 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
     )
 
 
+def test_create_archive_with_json_calls_borg_with_json_parameter():
+    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
+    flexmock(module).should_receive('_write_pattern_file').and_return(None)
+    flexmock(module).should_receive('_make_pattern_flags').and_return(())
+    flexmock(module).should_receive('_make_exclude_flags').and_return(())
+    insert_subprocess_mock(CREATE_COMMAND + ('--json',))
+
+    module.create_archive(
+        dry_run=False,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        json=True
+    )
+
+
 def test_create_archive_with_source_directories_glob_expands():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'food')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)
@@ -589,7 +638,6 @@ def test_create_archive_with_source_directories_glob_expands():
     flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -610,7 +658,6 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
     flexmock(module.glob).should_receive('glob').with_args('foo*').and_return([])
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -630,7 +677,6 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
     insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'food'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -650,7 +696,6 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
     insert_subprocess_mock(('borg', 'create', 'repo::ARCHIVE_NAME', 'foo', 'bar'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={
@@ -672,7 +717,6 @@ def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
     insert_subprocess_mock(('borg', 'create', 'repo::Documents_{hostname}-{now}', 'foo', 'bar'))
 
     module.create_archive(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         location_config={

borgmatic/tests/unit/borg/test_extract.py

@@ -1,9 +1,10 @@
-import sys
+import logging, sys
 
 from flexmock import flexmock
 
 from borgmatic.borg import extract as module
 from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 
 def insert_subprocess_mock(check_call_command, **kwargs):
@@ -32,7 +33,6 @@ def test_extract_last_archive_dry_run_should_call_borg_with_last_archive():
     )
 
     module.extract_last_archive_dry_run(
-        verbosity=None,
         repository='repo',
         lock_wait=None,
     )
@@ -47,13 +47,12 @@ def test_extract_last_archive_dry_run_without_any_archives_should_bail():
     insert_subprocess_never()
 
     module.extract_last_archive_dry_run(
-        verbosity=None,
         repository='repo',
         lock_wait=None,
     )
 
 
-def test_extract_last_archive_dry_run_with_verbosity_some_should_call_borg_with_info_parameter():
+def test_extract_last_archive_dry_run_with_log_info_should_call_borg_with_info_parameter():
     flexmock(sys.stdout).encoding = 'utf-8'
     insert_subprocess_check_output_mock(
         ('borg', 'list', '--short', 'repo', '--info'),
@@ -62,26 +61,26 @@ def test_extract_last_archive_dry_run_with_verbosity_some_should_call_borg_with_
     insert_subprocess_mock(
         ('borg', 'extract', '--dry-run', 'repo::archive2', '--info'),
     )
+    insert_logging_mock(logging.INFO)
 
     module.extract_last_archive_dry_run(
-        verbosity=VERBOSITY_SOME,
         repository='repo',
         lock_wait=None,
     )
 
 
-def test_extract_last_archive_dry_run_with_verbosity_lots_should_call_borg_with_debug_parameter():
+def test_extract_last_archive_dry_run_with_log_debug_should_call_borg_with_debug_parameter():
     flexmock(sys.stdout).encoding = 'utf-8'
     insert_subprocess_check_output_mock(
-        ('borg', 'list', '--short', 'repo', '--debug'),
+        ('borg', 'list', '--short', 'repo', '--debug', '--show-rc'),
         result='archive1\narchive2\n'.encode('utf-8'),
     )
     insert_subprocess_mock(
-        ('borg', 'extract', '--dry-run', 'repo::archive2', '--debug', '--list'),
+        ('borg', 'extract', '--dry-run', 'repo::archive2', '--debug', '--show-rc', '--list'),
     )
+    insert_logging_mock(logging.DEBUG)
 
     module.extract_last_archive_dry_run(
-        verbosity=VERBOSITY_LOTS,
         repository='repo',
         lock_wait=None,
     )
@@ -98,7 +97,6 @@ def test_extract_last_archive_dry_run_should_call_borg_via_local_path():
     )
 
     module.extract_last_archive_dry_run(
-        verbosity=None,
         repository='repo',
         lock_wait=None,
         local_path='borg1',
@@ -116,7 +114,6 @@ def test_extract_last_archive_dry_run_should_call_borg_with_remote_path_paramete
     )
 
     module.extract_last_archive_dry_run(
-        verbosity=None,
         repository='repo',
         lock_wait=None,
         remote_path='borg1',
@@ -134,7 +131,6 @@ def test_extract_last_archive_dry_run_should_call_borg_with_lock_wait_parameters
     )
 
     module.extract_last_archive_dry_run(
-        verbosity=None,
         repository='repo',
         lock_wait=5,
     )

borgmatic/tests/unit/borg/test_info.py

@@ -1,14 +1,15 @@
+import logging
 from collections import OrderedDict
 
 from flexmock import flexmock
 
 from borgmatic.borg import info as module
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 
 def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess = flexmock(module.subprocess)
-    subprocess.should_receive('check_call').with_args(check_call_command, **kwargs).once()
+    subprocess.should_receive('check_output').with_args(check_call_command, **kwargs).once()
 
 
 INFO_COMMAND = ('borg', 'info', 'repo')
@@ -18,29 +19,37 @@ def test_display_archives_info_calls_borg_with_parameters():
     insert_subprocess_mock(INFO_COMMAND)
 
     module.display_archives_info(
-        verbosity=None,
         repository='repo',
         storage_config={},
     )
 
 
-def test_display_archives_info_with_verbosity_some_calls_borg_with_info_parameter():
+def test_display_archives_info_with_log_info_calls_borg_with_info_parameter():
     insert_subprocess_mock(INFO_COMMAND + ('--info',))
-
+    insert_logging_mock(logging.INFO)
     module.display_archives_info(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_SOME,
     )
 
 
-def test_display_archives_info_with_verbosity_lots_calls_borg_with_debug_parameter():
-    insert_subprocess_mock(INFO_COMMAND + ('--debug',))
+def test_display_archives_info_with_log_debug_calls_borg_with_debug_parameter():
+    insert_subprocess_mock(INFO_COMMAND + ('--debug', '--show-rc'))
+    insert_logging_mock(logging.DEBUG)
 
     module.display_archives_info(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_LOTS,
+    )
+
+
+def test_display_archives_info_with_json_calls_borg_with_json_parameter():
+    insert_subprocess_mock(INFO_COMMAND + ('--json',))
+
+    module.display_archives_info(
+        repository='repo',
+        storage_config={},
+        json=True,
     )
 
 
@@ -48,7 +57,6 @@ def test_display_archives_info_with_local_path_calls_borg_via_local_path():
     insert_subprocess_mock(('borg1',) + INFO_COMMAND[1:])
 
     module.display_archives_info(
-        verbosity=None,
         repository='repo',
         storage_config={},
         local_path='borg1',
@@ -59,7 +67,6 @@ def test_display_archives_info_with_remote_path_calls_borg_with_remote_path_para
     insert_subprocess_mock(INFO_COMMAND + ('--remote-path', 'borg1'))
 
     module.display_archives_info(
-        verbosity=None,
         repository='repo',
         storage_config={},
         remote_path='borg1',
@@ -71,7 +78,6 @@ def test_display_archives_info_with_lock_wait_calls_borg_with_lock_wait_paramete
     insert_subprocess_mock(INFO_COMMAND + ('--lock-wait', '5'))
 
     module.display_archives_info(
-        verbosity=None,
         repository='repo',
         storage_config=storage_config,
     )

borgmatic/tests/unit/borg/test_list.py

@@ -1,14 +1,15 @@
+import logging
 from collections import OrderedDict
 
 from flexmock import flexmock
 
 from borgmatic.borg import list as module
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 
 def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess = flexmock(module.subprocess)
-    subprocess.should_receive('check_call').with_args(check_call_command, **kwargs).once()
+    subprocess.should_receive('check_output').with_args(check_call_command, **kwargs).once()
 
 
 LIST_COMMAND = ('borg', 'list', 'repo')
@@ -18,29 +19,38 @@ def test_list_archives_calls_borg_with_parameters():
     insert_subprocess_mock(LIST_COMMAND)
 
     module.list_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
     )
 
 
-def test_list_archives_with_verbosity_some_calls_borg_with_info_parameter():
+def test_list_archives_with_log_info_calls_borg_with_info_parameter():
     insert_subprocess_mock(LIST_COMMAND + ('--info',))
+    insert_logging_mock(logging.INFO)
 
     module.list_archives(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_SOME,
     )
 
 
-def test_list_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
-    insert_subprocess_mock(LIST_COMMAND + ('--debug',))
+def test_list_archives_with_log_debug_calls_borg_with_debug_parameter():
+    insert_subprocess_mock(LIST_COMMAND + ('--debug', '--show-rc'))
+    insert_logging_mock(logging.DEBUG)
 
     module.list_archives(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_LOTS,
+    )
+
+
+def test_list_archives_with_json_calls_borg_with_json_parameter():
+    insert_subprocess_mock(LIST_COMMAND + ('--json',))
+
+    module.list_archives(
+        repository='repo',
+        storage_config={},
+        json=True,
     )
 
 
@@ -48,7 +58,6 @@ def test_list_archives_with_local_path_calls_borg_via_local_path():
     insert_subprocess_mock(('borg1',) + LIST_COMMAND[1:])
 
     module.list_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         local_path='borg1',
@@ -59,7 +68,6 @@ def test_list_archives_with_remote_path_calls_borg_with_remote_path_parameters()
     insert_subprocess_mock(LIST_COMMAND + ('--remote-path', 'borg1'))
 
     module.list_archives(
-        verbosity=None,
         repository='repo',
         storage_config={},
         remote_path='borg1',
@@ -71,7 +79,6 @@ def test_list_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
     insert_subprocess_mock(LIST_COMMAND + ('--lock-wait', '5'))
 
     module.list_archives(
-        verbosity=None,
         repository='repo',
         storage_config=storage_config,
     )

borgmatic/tests/unit/borg/test_prune.py

@@ -1,9 +1,10 @@
+import logging
 from collections import OrderedDict
 
 from flexmock import flexmock
 
 from borgmatic.borg import prune as module
-from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
+from borgmatic.tests.unit.test_verbosity import insert_logging_mock
 
 
 def insert_subprocess_mock(check_call_command, **kwargs):
@@ -11,6 +12,7 @@ def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess.should_receive('check_call').with_args(check_call_command, **kwargs).once()
 
 
+
 BASE_PRUNE_FLAGS = (
     ('--keep-daily', '1'),
     ('--keep-weekly', '2'),
@@ -63,7 +65,6 @@ def test_prune_archives_calls_borg_with_parameters():
     insert_subprocess_mock(PRUNE_COMMAND)
 
     module.prune_archives(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         storage_config={},
@@ -71,33 +72,33 @@ def test_prune_archives_calls_borg_with_parameters():
     )
 
 
-def test_prune_archives_with_verbosity_some_calls_borg_with_info_parameter():
+def test_prune_archives_with_log_info_calls_borg_with_info_parameter():
     retention_config = flexmock()
     flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
         BASE_PRUNE_FLAGS,
     )
-    insert_subprocess_mock(PRUNE_COMMAND + ('--info', '--stats',))
+    insert_subprocess_mock(PRUNE_COMMAND + ('--stats', '--info',))
+    insert_logging_mock(logging.INFO)
 
     module.prune_archives(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_SOME,
         dry_run=False,
         retention_config=retention_config,
     )
 
 
-def test_prune_archives_with_verbosity_lots_calls_borg_with_debug_parameter():
+def test_prune_archives_with_log_debug_calls_borg_with_debug_parameter():
     retention_config = flexmock()
     flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
         BASE_PRUNE_FLAGS,
     )
-    insert_subprocess_mock(PRUNE_COMMAND + ('--debug', '--stats', '--list'))
+    insert_subprocess_mock(PRUNE_COMMAND + ('--stats', '--debug', '--list', '--show-rc'))
+    insert_logging_mock(logging.DEBUG)
 
     module.prune_archives(
         repository='repo',
         storage_config={},
-        verbosity=VERBOSITY_LOTS,
         dry_run=False,
         retention_config=retention_config,
     )
@@ -113,7 +114,6 @@ def test_prune_archives_with_dry_run_calls_borg_with_dry_run_parameter():
     module.prune_archives(
         repository='repo',
         storage_config={},
-        verbosity=None,
         dry_run=True,
         retention_config=retention_config,
     )
@@ -127,7 +127,6 @@ def test_prune_archives_with_local_path_calls_borg_via_local_path():
     insert_subprocess_mock(('borg1',) + PRUNE_COMMAND[1:])
 
     module.prune_archives(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         storage_config={},
@@ -144,7 +143,6 @@ def test_prune_archives_with_remote_path_calls_borg_with_remote_path_parameters(
     insert_subprocess_mock(PRUNE_COMMAND + ('--remote-path', 'borg1'))
 
     module.prune_archives(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         storage_config={},
@@ -162,7 +160,6 @@ def test_prune_archives_with_umask_calls_borg_with_umask_parameters():
     insert_subprocess_mock(PRUNE_COMMAND + ('--umask', '077'))
 
     module.prune_archives(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         storage_config=storage_config,
@@ -179,7 +176,6 @@ def test_prune_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
     insert_subprocess_mock(PRUNE_COMMAND + ('--lock-wait', '5'))
 
     module.prune_archives(
-        verbosity=None,
         dry_run=False,
         repository='repo',
         storage_config=storage_config,

borgmatic/tests/unit/commands/test_borgmatic.py

@@ -0,0 +1,52 @@
+import json
+import sys
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.commands import borgmatic
+
+
+def test__run_commands_handles_multiple_json_outputs_in_array():
+    (
+        flexmock(borgmatic)
+        .should_receive('_run_commands_on_repository')
+        .times(3)
+        .replace_with(
+            lambda args, consistency, json_results, local_path, location, remote_path, retention,
+            storage,
+            unexpanded_repository: json_results.append({"whatever": unexpanded_repository})
+        )
+    )
+
+    (
+        flexmock(sys.stdout)
+        .should_call("write")
+        .with_args(
+            json.dumps(
+                json.loads(
+                    '''
+                        [
+                            {"whatever": "fake_repo1"},
+                            {"whatever": "fake_repo2"},
+                            {"whatever": "fake_repo3"}
+                        ]
+                    ''',
+                )
+            )
+        )
+    )
+
+    borgmatic._run_commands(
+        args=flexmock(json=True),
+        consistency=None,
+        local_path=None,
+        location={'repositories': [
+            'fake_repo1',
+            'fake_repo2',
+            'fake_repo3'
+        ]},
+        remote_path=None,
+        retention=None,
+        storage=None,
+    )

borgmatic/tests/unit/config/test_collect.py

@@ -3,6 +3,22 @@ from flexmock import flexmock
 from borgmatic.config import collect as module
 
 
+def test_get_default_config_paths_includes_absolute_user_config_path():
+    flexmock(module.os, environ={'XDG_CONFIG_HOME': None, 'HOME': '/home/user'})
+
+    config_paths = module.get_default_config_paths()
+
+    assert '/home/user/.config/borgmatic/config.yaml' in config_paths
+
+
+def test_get_default_config_paths_prefers_xdg_config_home_for_user_config_path():
+    flexmock(module.os, environ={'XDG_CONFIG_HOME': '/home/user/.etc', 'HOME': '/home/user'})
+
+    config_paths = module.get_default_config_paths()
+
+    assert '/home/user/.etc/borgmatic/config.yaml' in config_paths
+
+
 def test_collect_config_filenames_collects_given_files():
     config_paths = ('config.yaml', 'other.yaml')
     flexmock(module.os.path).should_receive('isdir').and_return(False)
@@ -32,6 +48,21 @@ def test_collect_config_filenames_collects_files_from_given_directories_and_igno
     )
 
 
+def test_collect_config_filenames_collects_files_from_given_directories_and_ignores_non_yaml_filenames():
+    config_paths = ('/etc/borgmatic.d',)
+    mock_path = flexmock(module.os.path)
+    mock_path.should_receive('exists').and_return(True)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d').and_return(True)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/foo.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/bar.yaml~').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic.d/baz.txt').and_return(False)
+    flexmock(module.os).should_receive('listdir').and_return(['foo.yaml', 'bar.yaml~', 'baz.txt'])
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == ('/etc/borgmatic.d/foo.yaml',)
+
+
 def test_collect_config_filenames_skips_etc_borgmatic_config_dot_yaml_if_it_does_not_exist():
     config_paths = ('config.yaml', '/etc/borgmatic/config.yaml')
     mock_path = flexmock(module.os.path)

borgmatic/tests/unit/test_verbosity.py

@@ -1,10 +1,19 @@
 import logging
 
+from flexmock import flexmock
+
 from borgmatic import verbosity as module
 
+def insert_logging_mock(log_level):
+    """ Mocks the isEnabledFor from python logging. """
+    logging = flexmock(module.logging.Logger)
+    logging.should_receive('isEnabledFor').replace_with(lambda lvl : lvl >= log_level)
+    logging.should_receive('getEffectiveLevel').replace_with(lambda: log_level)
+
 
 def test_verbosity_to_log_level_maps_known_verbosity_to_log_level():
     assert module.verbosity_to_log_level(module.VERBOSITY_SOME) == logging.INFO
+    assert module.verbosity_to_log_level(module.VERBOSITY_LOTS) == logging.DEBUG
 
 
 def test_verbosity_to_log_level_maps_unknown_verbosity_to_warning_level():

borgmatic/verbosity.py

@@ -1,6 +1,7 @@
 import logging
 
 
+VERBOSITY_WARNING = 0
 VERBOSITY_SOME = 1
 VERBOSITY_LOTS = 2
 
@@ -10,6 +11,7 @@ def verbosity_to_log_level(verbosity):
     Given a borgmatic verbosity value, return the corresponding Python log level.
     '''
     return {
+        VERBOSITY_WARNING: logging.WARNING,
         VERBOSITY_SOME: logging.INFO,
         VERBOSITY_LOTS: logging.DEBUG,
     }.get(verbosity, logging.WARNING)

sample/cron-alpine/borgmatic

@@ -0,0 +1,3 @@
+# You can drop this file into /etc/cron.d/ to run borgmatic nightly.
+
+0 3 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic

scripts/find-unsupported-borg-options

@@ -22,22 +22,30 @@ for sub_command in prune create check list info; do
     sort borgmatic_borg_flags > borgmatic_borg_flags.sorted
     mv borgmatic_borg_flags.sorted borgmatic_borg_flags
 
-    for line in $(borg $sub_command --help | awk -v RS= '/^usage:/') ; do
+    for word in $(borg $sub_command --help | grep '^  -') ; do
         # Exclude a bunch of flags that borgmatic actually supports, but don't get exercised by the
         # generated sample config, and also flags that don't make sense to support.
-        echo "$line" | grep -- -- | sed -r 's/(\[|\])//g' \
-            | grep -v '^-h$' \
+        echo "$word" | grep ^-- | sed -e 's/,$//' \
             | grep -v '^--archives-only$' \
-            | grep -v '^--repository-only$' \
-            | grep -v '^--stats$' \
-            | grep -v '^--list$' \
             | grep -v '^--critical$' \
-            | grep -v '^--error$' \
-            | grep -v '^--warning$' \
-            | grep -v '^--info$' \
             | grep -v '^--debug$' \
+            | grep -v '^--dry-run$' \
+            | grep -v '^--error$' \
+            | grep -v '^--help$' \
+            | grep -v '^--info$' \
+            | grep -v '^--list$' \
+            | grep -v '^--nobsdflags$' \
+            | grep -v '^--pattern$' \
+            | grep -v '^--read-special$' \
+            | grep -v '^--repository-only$' \
+            | grep -v '^--show-rc$' \
+            | grep -v '^--stats$' \
+            | grep -v '^--verbose$' \
+            | grep -v '^--warning$' \
+            | grep -v '^-h$' \
             >> all_borg_flags
     done
+
     sort all_borg_flags > all_borg_flags.sorted
     mv all_borg_flags.sorted all_borg_flags
 

setup.py

@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 
 
-VERSION = '1.2.0'
+VERSION = '1.2.4'
 
 
 setup(
@@ -32,8 +32,8 @@ setup(
         'atticmatic',
     ],
     install_requires=(
-        'pykwalify>=1.6.0',
-        'ruamel.yaml<=0.15',
+        'pykwalify>=1.6.0,<14.06',
+        'ruamel.yaml>0.15.0,<0.16.0',
         'setuptools',
     ),
     tests_require=(

test_requirements.txt

@@ -1,5 +1,5 @@
 flexmock==0.10.2
-pykwalify==1.6.0
-pytest==2.9.1
+pykwalify==1.6.1
+pytest==3.7.4
 pytest-cov==2.5.1
-ruamel.yaml==0.15.18
+ruamel.yaml>0.15.0,<0.16.0