Bumping version for release.

* Added storage.archive_name_format to config

.gitignore

@@ -1,6 +1,8 @@
 *.egg-info
 *.pyc
 *.swp
+.cache
+.coverage
 .tox
 build
 dist

.hgtags

@@ -37,3 +37,4 @@ f052a77a8ad5a0fea7fa86a902e0e401252f7d80 1.1.2
 3d605962d891731a0f372b903b556ac7a8c8359f 1.1.4
 64ca13bfe050f656b44ed2eb1c3db045bfddd133 1.1.5
 4daa944c122c572b9b56bfcac3f4e2869181c630 1.1.6
+ec7949a14a2051616f7cdcb8e05555f02e024ae8 1.1.7

AUTHORS

@@ -5,3 +5,4 @@ Henning Schroeder: Copy editing
 Michele Lazzeri: Custom archive names
 Robin `ypid` Schneider: Support additional options of Borg
 Scott Squires: Custom archive names
+Johannes Feichtner: Support for user hooks

NEWS

@@ -1,3 +1,17 @@
+1.1.9
+ * #16, #38: Support for user-defined hooks before/after backup, or on error.
+ * #33: Improve clarity of logging spew at high verbosity levels.
+ * #29: Support for using tilde in source directory path to reference home directory.
+ * Require "prefix" in retention section when "archive_name_format" is set. This is to avoid
+   accidental pruning of archives with a different archive name format. For similar reasons, default
+   "prefix" to "{hostname}-" if not specified.
+ * Convert main source repository from Mercurial to Git.
+ * Update dead links to Borg documentation.
+
+1.1.8
+ * #39: Fix to make /etc/borgmatic/config.yaml optional rather than required when using the default
+   config paths.
+
 1.1.7
 
  * #28: Add "archive_name_format" to configuration for customizing archive names.

README.md

@@ -1,4 +1,4 @@
-<img src="https://cdn.rawgit.com/witten/borgmatic/master/static/borgmatic.svg" width="150px" style="float: right; padding-left: 1em;">
+<img src="https://projects.torsion.org/witten/borgmatic/raw/master/static/borgmatic.png" width="150px" style="float: right; padding-left: 1em;">
 
 ## Overview
 
@@ -41,9 +41,8 @@ consistency:
 ```
 
 borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
-available](https://torsion.org/hg/borgmatic). It's also mirrored on
-[GitHub](https://github.com/witten/borgmatic) and
-[BitBucket](https://bitbucket.org/dhelfman/borgmatic) for convenience.
+available](https://projects.torsion.org/witten/borgmatic). It's also mirrored
+on [GitHub](https://github.com/witten/borgmatic) for convenience.
 
 
 ## Installation
@@ -61,16 +60,26 @@ key-based ssh access to the desired user account on the remote host.
 
 To install borgmatic, run the following command to download and install it:
 
-    sudo pip3 install --upgrade borgmatic
+```bash
+sudo pip3 install --upgrade borgmatic
+```
 
 Note that your pip binary may have a different name than "pip3". Make sure
 you're using Python 3, as borgmatic does not support Python 2.
 
+### Docker
+
+If you would like to run borgmatic within Docker, please take a look at
+[b3vis/borgmatic](https://hub.docker.com/r/b3vis/borgmatic/) for more
+information.
+
 ## Configuration
 
 After you install borgmatic, generate a sample configuration file:
 
-    sudo generate-borgmatic-config
+```bash
+sudo generate-borgmatic-config
+```
 
 This generates a sample configuration file at /etc/borgmatic/config.yaml (by
 default). You should edit the file to suit your needs, as the values are just
@@ -78,7 +87,7 @@ representative. All fields are optional except where indicated, so feel free
 to remove anything you don't need.
 
 You can also have a look at the [full configuration
-schema](https://torsion.org/hg/borgmatic/file/tip/borgmatic/config/schema.yaml)
+schema](https://projects.torsion.org/witten/borgmatic/src/master/borgmatic/config/schema.yaml)
 for the authoritative set of all configuration options. This is handy if
 borgmatic has added new options since you originally created your
 configuration file.
@@ -89,9 +98,11 @@ configuration file.
 A more advanced usage is to create multiple separate configuration files and
 place each one in an /etc/borgmatic.d directory. For instance:
 
-    sudo mkdir /etc/borgmatic.d
-    sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
-    sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
+```bash
+sudo mkdir /etc/borgmatic.d
+sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
+sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
+```
 
 With this approach, you can have entirely different backup policies for
 different applications on your system. For instance, you may want one backup
@@ -108,7 +119,9 @@ default, the traditional /etc/borgmatic/config.yaml as well.
 In general, all you should need to do to upgrade borgmatic is run the
 following:
 
-    sudo pip3 install --upgrade borgmatic
+```bash
+sudo pip3 install --upgrade borgmatic
+```
 
 However, see below about special cases.
 
@@ -124,12 +137,16 @@ As of version 1.1.0, borgmatic no longer supports Python 2. If you were
 already running borgmatic with Python 3, then you can simply upgrade borgmatic
 in-place:
 
-    sudo pip3 install --upgrade borgmatic
+```bash
+sudo pip3 install --upgrade borgmatic
+```
 
 But if you were running borgmatic with Python 2, uninstall and reinstall instead:
 
-    sudo pip uninstall borgmatic
-    sudo pip3 install borgmatic
+```bash
+sudo pip uninstall borgmatic
+sudo pip3 install borgmatic
+```
 
 The pip binary names for different versions of Python can differ, so the above
 commands may need some tweaking to work on your machine.
@@ -137,7 +154,9 @@ commands may need some tweaking to work on your machine.
 
 Once borgmatic is upgraded, run:
 
-    sudo upgrade-borgmatic-config
+```bash
+sudo upgrade-borgmatic-config
+```
 
 That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
 (by default) using the values from both your existing configuration and
@@ -159,8 +178,10 @@ your borgmatic configuration files.
 If you were already using Borg with atticmatic, then you can easily upgrade
 from atticmatic to borgmatic. Simply run the following commands:
 
-    sudo pip3 uninstall atticmatic
-    sudo pip3 install borgmatic
+```bash
+sudo pip3 uninstall atticmatic
+sudo pip3 install borgmatic
+```
 
 That's it! borgmatic will continue using your /etc/borgmatic configuration
 files.
@@ -171,14 +192,18 @@ files.
 You can run borgmatic and start a backup simply by invoking it without
 arguments:
 
-    borgmatic
+```bash
+borgmatic
+```
 
 This will also prune any old backups as per the configured retention policy,
 and check backups for consistency problems due to things like file damage.
 
 If you'd like to see the available command-line arguments, view the help:
 
-    borgmatic --help
+```bash
+borgmatic --help
+```
 
 Note that borgmatic prunes archives *before* creating an archive, so as to
 free up space for archiving. This means that when a borgmatic run finishes,
@@ -191,20 +216,26 @@ By default, the backup will proceed silently except in the case of errors. But
 if you'd like to to get additional information about the progress of the
 backup as it proceeds, use the verbosity option:
 
-    borgmatic --verbosity 1
+```bash
+borgmatic --verbosity 1
+```
 
 Or, for even more progress spew:
 
-    borgmatic --verbosity 2
+```bash
+borgmatic --verbosity 2
+```
 
 ### À la carte
 
 If you want to run borgmatic with only pruning, creating, or checking enabled,
 the following optional flags are available:
 
-    borgmatic --prune
-    borgmatic --create
-    borgmatic --check
+```bash
+borgmatic --prune
+borgmatic --create
+borgmatic --check
+```
 
 You can run with only one of these flags provided, or you can mix and match
 any number of them. This supports use cases like running consistency checks
@@ -220,11 +251,13 @@ configure a job runner to invoke it periodically.
 ### cron
 
 If you're using cron, download the [sample cron
-file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/cron/borgmatic).
+file](https://projects.torsion.org/witten/borgmatic/raw/master/sample/cron/borgmatic).
 Then, from the directory where you downloaded it:
 
-    sudo mv borgmatic /etc/cron.d/borgmatic
-    sudo chmod +x /etc/cron.d/borgmatic
+```bash
+sudo mv borgmatic /etc/cron.d/borgmatic
+sudo chmod +x /etc/cron.d/borgmatic
+```
 
 You can modify the cron file if you'd like to run borgmatic more or less frequently.
 
@@ -232,14 +265,16 @@ You can modify the cron file if you'd like to run borgmatic more or less frequen
 
 If you're using systemd instead of cron to run jobs, download the [sample
 systemd service
-file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.service)
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
 and the [sample systemd timer
-file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.timer).
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
 Then, from the directory where you downloaded them:
 
-    sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
-    sudo systemctl enable borgmatic.timer
-    sudo systemctl start borgmatic.timer
+```bash
+sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
+sudo systemctl enable borgmatic.timer
+sudo systemctl start borgmatic.timer
+```
 
 Feel free to modify the timer file based on how frequently you'd like
 borgmatic to run.
@@ -249,11 +284,15 @@ borgmatic to run.
 
 First install tox, which is used for setting up testing environments:
 
-    pip3 install tox
+```bash
+pip3 install tox
+```
 
 Then, to actually run tests, run:
 
-    tox
+```bash
+tox
+```
 
 
 ## Troubleshooting
@@ -264,14 +303,18 @@ When running borgmatic on a large remote repository, you may receive errors
 like the following, particularly while "borg check" is validating backups for
 consistency:
 
+```text
     Write failed: Broken pipe
     borg: Error: Connection closed by remote host
+```
 
 This error can be caused by an ssh timeout, which you can rectify by adding
 the following to the ~/.ssh/config file on the client:
 
+```text
     Host *
         ServerAliveInterval 120
+```
 
 This should make the client keep the connection alive while validating
 backups.

borgmatic/borg/create.py

@@ -14,6 +14,16 @@ def initialize(storage_config):
         os.environ['BORG_PASSPHRASE'] = passphrase
 
 
+def _expand_directory(directory):
+    '''
+    Given a directory path, expand any tilde (representing a user's home directory) and any globs
+    therein. Return a list of one or more resulting paths.
+    '''
+    expanded_directory = os.path.expanduser(directory)
+
+    return glob.glob(expanded_directory) or [expanded_directory]
+
+
 def _write_exclude_file(exclude_patterns=None):
     '''
     Given a sequence of exclude patterns, write them to a named temporary file and return it. Return
@@ -59,7 +69,7 @@ def create_archive(
     '''
     sources = tuple(
         itertools.chain.from_iterable(
-            glob.glob(directory) or [directory]
+            _expand_directory(directory)
             for directory in location_config['source_directories']
         )
     )
@@ -92,4 +102,4 @@ def create_archive(
     ) + sources + exclude_flags + compression_flags + one_file_system_flags + \
         remote_path_flags + umask_flags + verbosity_flags
 
-    subprocess.check_call(full_command)
+    subprocess.check_call(full_command)

borgmatic/borg/prune.py

@@ -19,6 +19,9 @@ def _make_prune_flags(retention_config):
             ('--keep-monthly', '6'),
         )
     '''
+    if not retention_config.get('prefix'):
+        retention_config['prefix'] = '{hostname}-'
+
     return (
         ('--' + option_name.replace('_', '-'), str(retention_config[option_name]))
         for option_name, value in retention_config.items()

borgmatic/commands/borgmatic.py

@@ -1,16 +1,20 @@
-from __future__ import print_function
+
 from argparse import ArgumentParser
+import logging
 import os
 from subprocess import CalledProcessError
 import sys
 
 from borgmatic.borg import check, create, prune
+from borgmatic.commands import hook
 from borgmatic.config import collect, convert, validate
+from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS, verbosity_to_log_level
+
+
+logger = logging.getLogger(__name__)
 
 
 LEGACY_CONFIG_PATH = '/etc/borgmatic/config'
-DEFAULT_CONFIG_PATHS = ['/etc/borgmatic/config.yaml', '/etc/borgmatic.d']
-DEFAULT_EXCLUDES_PATH = '/etc/borgmatic/excludes'
 
 
 def parse_arguments(*arguments):
@@ -30,8 +34,8 @@ def parse_arguments(*arguments):
         '-c', '--config',
         nargs='+',
         dest='config_paths',
-        default=DEFAULT_CONFIG_PATHS,
-        help='Configuration filenames or directories, defaults to: {}'.format(' '.join(DEFAULT_CONFIG_PATHS)),
+        default=collect.DEFAULT_CONFIG_PATHS,
+        help='Configuration filenames or directories, defaults to: {}'.format(' '.join(collect.DEFAULT_CONFIG_PATHS)),
     )
     parser.add_argument(
         '--excludes',
@@ -75,37 +79,59 @@ def parse_arguments(*arguments):
     return args
 
 
+def run_configuration(config_filename, args):  # pragma: no cover
+    '''
+    Parse a single configuration file, and execute its defined pruning, backups, and/or consistency
+    checks.
+    '''
+    logger.info('{}: Parsing configuration file'.format(config_filename))
+    config = validate.parse_configuration(config_filename, validate.schema_filename())
+    (location, storage, retention, consistency, hooks) = (
+        config.get(section_name, {})
+        for section_name in ('location', 'storage', 'retention', 'consistency', 'hooks')
+    )
+
+    try:
+        remote_path = location.get('remote_path')
+        create.initialize(storage)
+        hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
+
+        for repository in location['repositories']:
+            if args.prune:
+                logger.info('{}: Pruning archives'.format(repository))
+                prune.prune_archives(args.verbosity, repository, retention, remote_path=remote_path)
+            if args.create:
+                logger.info('{}: Creating archive'.format(repository))
+                create.create_archive(
+                    args.verbosity,
+                    repository,
+                    location,
+                    storage,
+                )
+            if args.check:
+                logger.info('{}: Running consistency checks'.format(repository))
+                check.check_archives(args.verbosity, repository, consistency, remote_path=remote_path)
+
+        hook.execute_hook(hooks.get('after_backup'), config_filename, 'post-backup')
+    except (OSError, CalledProcessError):
+        hook.execute_hook(hooks.get('on_error'), config_filename, 'on-error')
+        raise
+
+
 def main():  # pragma: no cover
     try:
         args = parse_arguments(*sys.argv[1:])
+        logging.basicConfig(level=verbosity_to_log_level(args.verbosity), format='%(message)s')
+
         config_filenames = tuple(collect.collect_config_filenames(args.config_paths))
+        logger.debug('Ensuring legacy configuration is upgraded')
         convert.guard_configuration_upgraded(LEGACY_CONFIG_PATH, config_filenames)
 
         if len(config_filenames) == 0:
             raise ValueError('Error: No configuration files found in: {}'.format(' '.join(args.config_paths)))
 
         for config_filename in config_filenames:
-            config = validate.parse_configuration(config_filename, validate.schema_filename())
-            (location, storage, retention, consistency) = (
-                config.get(section_name, {})
-                for section_name in ('location', 'storage', 'retention', 'consistency')
-            )
-            remote_path = location.get('remote_path')
-
-            create.initialize(storage)
-
-            for repository in location['repositories']:
-                if args.prune:
-                    prune.prune_archives(args.verbosity, repository, retention, remote_path=remote_path)
-                if args.create:
-                    create.create_archive(
-                        args.verbosity,
-                        repository,
-                        location,
-                        storage,
-                    )
-                if args.check:
-                    check.check_archives(args.verbosity, repository, consistency, remote_path=remote_path)
+            run_configuration(config_filename, args)
     except (ValueError, OSError, CalledProcessError) as error:
         print(error, file=sys.stderr)
         sys.exit(1)

borgmatic/commands/convert_config.py

@@ -1,4 +1,3 @@
-from __future__ import print_function
 from argparse import ArgumentParser
 import os
 from subprocess import CalledProcessError

borgmatic/commands/generate_config.py

@@ -1,4 +1,3 @@
-from __future__ import print_function
 from argparse import ArgumentParser
 import os
 from subprocess import CalledProcessError

borgmatic/commands/hook.py

@@ -0,0 +1,20 @@
+import logging
+import subprocess
+
+
+logger = logging.getLogger(__name__)
+
+
+def execute_hook(commands, config_filename, description):
+    if not commands:
+        logger.debug('{}: No commands to run for {} hook'.format(config_filename, description))
+        return
+
+    if len(commands) == 1:
+        logger.info('{}: Running command for {} hook'.format(config_filename, description))
+    else:
+        logger.info('{}: Running {} commands for {} hook'.format(config_filename, len(commands), description))
+
+    for command in commands:
+        logger.debug('{}: Hook command: {}'.format(config_filename, command))
+        subprocess.check_call(command, shell=True)

borgmatic/config/collect.py

@@ -1,6 +1,9 @@
 import os
 
 
+DEFAULT_CONFIG_PATHS = ['/etc/borgmatic/config.yaml', '/etc/borgmatic.d']
+
+
 def collect_config_filenames(config_paths):
     '''
     Given a sequence of config paths, both filenames and directories, resolve that to just an
@@ -14,7 +17,7 @@ def collect_config_filenames(config_paths):
     for path in config_paths:
         exists = os.path.exists(path)
 
-        if os.path.realpath(path) == '/etc/borgmatic.d' and not exists:
+        if os.path.realpath(path) in DEFAULT_CONFIG_PATHS and not exists:
             continue
 
         if not os.path.isdir(path) or not exists:

borgmatic/config/generate.py

@@ -24,7 +24,7 @@ def _schema_to_sample_configuration(schema, level=0):
     for each section based on the schema "desc" description.
     '''
     example = schema.get('example')
-    if example:
+    if example is not None:
         return example
 
     config = yaml.comments.CommentedMap([

borgmatic/config/schema.yaml

@@ -12,7 +12,8 @@ map:
                 required: true
                 seq:
                     - type: scalar
-                desc: List of source directories to backup (required). Globs are expanded.
+                desc: |
+                    List of source directories to backup (required). Globs and tildes are expanded.
                 example:
                     - /home
                     - /etc
@@ -39,8 +40,7 @@ map:
                     - type: scalar
                 desc: |
                     Any paths matching these patterns are excluded from backups. Globs are expanded.
-                    See https://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-patterns
-                    for details.
+                    See the output of "borg help patterns" for more details.
                 example:
                     - '*.pyc'
                     - /home/*/.cache
@@ -67,7 +67,8 @@ map:
         desc: |
             Repository storage options. See
             https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
-            https://borgbackup.readthedocs.io/en/stable/usage.html#environment-variables for details.
+            https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
+            details.
         map:
             encryption_passphrase:
                 type: scalar
@@ -91,9 +92,11 @@ map:
             archive_name_format:
                 type: scalar
                 desc: |
-                    Name of the archive. Borg placeholders can be used. See
-                    https://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-placeholders
-                    Default is "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
+                    Name of the archive. Borg placeholders can be used. See the output of
+                    "borg help placeholders" for details. Default is
+                    "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
+                    also specify a prefix in the retention section to avoid accidental pruning of
+                    archives with a different archive name format.
                 example: "{hostname}-documents-{now}"
     retention:
         desc: |
@@ -128,8 +131,8 @@ map:
                 type: scalar
                 desc: |
                     When pruning, only consider archive names starting with this prefix.
-                    Borg placeholders can be used. See
-                    https://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-placeholders
+                    Borg placeholders can be used. See the output of "borg help placeholders" for
+                    details. Default is "{hostname}-".
                 example: sourcehostname
     consistency:
         desc: |
@@ -156,3 +159,28 @@ map:
                 desc: Restrict the number of checked archives to the last n. Applies only to the
                       "archives" check.
                 example: 3
+    hooks:
+        desc: |
+            Shell commands or scripts to execute before and after a backup or if an error has occurred.
+            IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic.
+            Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to
+            prevent potential shell injection or privilege escalation.
+        map:
+            before_backup:
+                seq:
+                    - type: scalar
+                desc: List of one or more shell commands or scripts to execute before creating a backup.
+                example:
+                    - echo "`date` - Starting a backup job."
+            after_backup:
+                seq:
+                    - type: scalar
+                desc: List of one or more shell commands or scripts to execute after creating a backup.
+                example:
+                    - echo "`date` - Backup created."
+            on_error:
+                seq:
+                    - type: scalar
+                desc: List of one or more shell commands or scripts to execute in case an exception has occurred.
+                example:
+                    - echo "`date` - Error while creating a backup."

borgmatic/config/validate.py

@@ -25,6 +25,31 @@ class Validation_error(ValueError):
         self.config_filename = config_filename
         self.error_messages = error_messages
 
+    def __str__(self):
+        '''
+        Render a validation error as a user-facing string.
+        '''
+        return 'An error occurred while parsing a configuration file at {}:\n'.format(
+            self.config_filename
+        ) + '\n'.join(self.error_messages)
+
+
+def apply_logical_validation(config_filename, parsed_configuration):
+    '''
+    Given a parsed and schematically valid configuration as a data structure of nested dicts (see
+    below), run through any additional logical validation checks. If there are any such validation
+    problems, raise a Validation_error.
+    '''
+    archive_name_format = parsed_configuration.get('storage', {}).get('archive_name_format')
+    prefix = parsed_configuration.get('retention', {}).get('prefix')
+
+    if archive_name_format and not prefix:
+        raise Validation_error(
+            config_filename, (
+                'If you provide an archive_name_format, you must also specify a retention prefix.',
+            )
+        )
+
 
 def parse_configuration(config_filename, schema_filename):
     '''
@@ -38,6 +63,8 @@ def parse_configuration(config_filename, schema_filename):
     Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
     have permissions to read the file, or Validation_error if the config does not match the schema.
     '''
+    logging.getLogger('pykwalify').setLevel(logging.ERROR)
+
     try:
         config = yaml.round_trip_load(open(config_filename))
         schema = yaml.round_trip_load(open(schema_filename))
@@ -48,7 +75,7 @@ def parse_configuration(config_filename, schema_filename):
     # simply remove all examples before passing the schema to pykwalify.
     for section_name, section_schema in schema['map'].items():
         for field_name, field_schema in section_schema['map'].items():
-            field_schema.pop('example')
+            field_schema.pop('example', None)
 
     validator = pykwalify.core.Core(source_data=config, schema_data=schema)
     parsed_result = validator.validate(raise_exception=False)
@@ -56,19 +83,6 @@ def parse_configuration(config_filename, schema_filename):
     if validator.validation_errors:
         raise Validation_error(config_filename, validator.validation_errors)
 
+    apply_logical_validation(config_filename, parsed_result)
+
     return parsed_result
-
-
-def display_validation_error(validation_error):
-    '''
-    Given a Validation_error, display its error messages to stderr.
-    '''
-    print(
-        'An error occurred while parsing a configuration file at {}:'.format(
-            validation_error.config_filename
-        ),
-        file=sys.stderr,
-    )
-
-    for error in validation_error.error_messages:
-        print(error, file=sys.stderr)

borgmatic/tests/integration/commands/test_borgmatic.py

@@ -9,7 +9,7 @@ from borgmatic.commands import borgmatic as module
 def test_parse_arguments_with_no_arguments_uses_defaults():
     parser = module.parse_arguments()
 
-    assert parser.config_paths == module.DEFAULT_CONFIG_PATHS
+    assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
     assert parser.excludes_filename == None
     assert parser.verbosity is None
 
@@ -32,7 +32,7 @@ def test_parse_arguments_with_multiple_config_paths_parses_as_list():
 def test_parse_arguments_with_verbosity_flag_overrides_default():
     parser = module.parse_arguments('--verbosity', '1')
 
-    assert parser.config_paths == module.DEFAULT_CONFIG_PATHS
+    assert parser.config_paths == module.collect.DEFAULT_CONFIG_PATHS
     assert parser.excludes_filename == None
     assert parser.verbosity == 1
 

borgmatic/tests/integration/config/test_validate.py

@@ -15,13 +15,18 @@ def test_schema_filename_returns_plausable_path():
     assert schema_path.endswith('/schema.yaml')
 
 
-def mock_config_and_schema(config_yaml):
+def mock_config_and_schema(config_yaml, schema_yaml=None):
     '''
-    Set up mocks for the config config YAML string and the default schema so that the code under
-    test consumes them when parsing the configuration.
+    Set up mocks for the given config config YAML string and the schema YAML string, or the default
+    schema if no schema is provided. The idea is that that the code under test consumes these mocks
+    when parsing the configuration.
     '''
     config_stream = io.StringIO(config_yaml)
-    schema_stream = open(module.schema_filename())
+    if schema_yaml is None:
+        schema_stream = open(module.schema_filename())
+    else:
+        schema_stream = io.StringIO(schema_yaml)
+
     builtins = flexmock(sys.modules['builtins'])
     builtins.should_receive('open').with_args('config.yaml').and_return(config_stream)
     builtins.should_receive('open').with_args('schema.yaml').and_return(schema_stream)
@@ -81,6 +86,35 @@ def test_parse_configuration_passes_through_quoted_punctuation():
     }
 
 
+def test_parse_configuration_with_schema_lacking_examples_does_not_raise():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+
+            repositories:
+                - hostname.borg
+        ''',
+        '''
+        map:
+            location:
+                required: true
+                map:
+                    source_directories:
+                        required: true
+                        seq:
+                            - type: scalar
+                    repositories:
+                        required: true
+                        seq:
+                            - type: scalar
+        '''
+    )
+
+    module.parse_configuration('config.yaml', 'schema.yaml')
+
+
 def test_parse_configuration_raises_for_missing_config_file():
     with pytest.raises(FileNotFoundError):
         module.parse_configuration('config.yaml', 'schema.yaml')
@@ -114,10 +148,3 @@ def test_parse_configuration_raises_for_validation_error():
 
     with pytest.raises(module.Validation_error):
         module.parse_configuration('config.yaml', 'schema.yaml')
-
-
-def test_display_validation_error_does_not_raise():
-    flexmock(sys.modules['builtins']).should_receive('print')
-    error = module.Validation_error('config.yaml', ('oops', 'uh oh'))
-
-    module.display_validation_error(error)

borgmatic/tests/unit/borg/test_create.py

@@ -28,6 +28,24 @@ def test_initialize_without_passphrase_should_not_set_environment():
         os.environ = orig_environ
 
 
+def test_expand_directory_with_basic_path_passes_it_through():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo')
+    flexmock(module.glob).should_receive('glob').and_return([])
+
+    paths = module._expand_directory('foo')
+
+    assert paths == ['foo']
+
+
+def test_expand_directory_with_glob_expands():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo*')
+    flexmock(module.glob).should_receive('glob').and_return(['foo', 'food'])
+
+    paths = module._expand_directory('foo*')
+
+    assert paths == ['foo', 'food']
+
+
 def test_write_exclude_file_does_not_raise():
     temporary_file = flexmock(
         name='filename',
@@ -122,7 +140,8 @@ DEFAULT_ARCHIVE_NAME = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
 CREATE_COMMAND = ('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'bar')
 
 
-def test_create_archive_should_call_borg_with_parameters():
+def test_create_archive_calls_borg_with_parameters():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND)
@@ -139,8 +158,9 @@ def test_create_archive_should_call_borg_with_parameters():
     )
 
 
-def test_create_archive_with_exclude_patterns_should_call_borg_with_excludes():
+def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
     exclude_flags = ('--exclude-from', 'excludes')
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(flexmock(name='/tmp/excludes'))
     flexmock(module).should_receive('_make_exclude_flags').and_return(exclude_flags)
     insert_subprocess_mock(CREATE_COMMAND + exclude_flags)
@@ -157,7 +177,8 @@ def test_create_archive_with_exclude_patterns_should_call_borg_with_excludes():
     )
 
 
-def test_create_archive_with_verbosity_some_should_call_borg_with_info_parameter():
+def test_create_archive_with_verbosity_some_calls_borg_with_info_parameter():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--info', '--stats',))
@@ -174,7 +195,8 @@ def test_create_archive_with_verbosity_some_should_call_borg_with_info_parameter
     )
 
 
-def test_create_archive_with_verbosity_lots_should_call_borg_with_debug_parameter():
+def test_create_archive_with_verbosity_lots_calls_borg_with_debug_parameter():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--stats'))
@@ -191,7 +213,8 @@ def test_create_archive_with_verbosity_lots_should_call_borg_with_debug_paramete
     )
 
 
-def test_create_archive_with_compression_should_call_borg_with_compression_parameters():
+def test_create_archive_with_compression_calls_borg_with_compression_parameters():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--compression', 'rle'))
@@ -208,7 +231,8 @@ def test_create_archive_with_compression_should_call_borg_with_compression_param
     )
 
 
-def test_create_archive_with_one_file_system_should_call_borg_with_one_file_system_parameters():
+def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_parameters():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--one-file-system',))
@@ -226,7 +250,8 @@ def test_create_archive_with_one_file_system_should_call_borg_with_one_file_syst
     )
 
 
-def test_create_archive_with_remote_path_should_call_borg_with_remote_path_parameters():
+def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
@@ -244,7 +269,8 @@ def test_create_archive_with_remote_path_should_call_borg_with_remote_path_param
     )
 
 
-def test_create_archive_with_umask_should_call_borg_with_umask_parameters():
+def test_create_archive_with_umask_calls_borg_with_umask_parameters():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(CREATE_COMMAND + ('--umask', '740'))
@@ -262,6 +288,7 @@ def test_create_archive_with_umask_should_call_borg_with_umask_parameters():
 
 
 def test_create_archive_with_source_directories_glob_expands():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo', 'food'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'food'))
@@ -280,6 +307,7 @@ def test_create_archive_with_source_directories_glob_expands():
 
 
 def test_create_archive_with_non_matching_source_directories_glob_passes_through():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo*'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo*'))
@@ -297,11 +325,11 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
     )
 
 
-def test_create_archive_with_glob_should_call_borg_with_expanded_directories():
+def test_create_archive_with_glob_calls_borg_with_expanded_directories():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo', 'food'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(('borg', 'create', 'repo::{}'.format(DEFAULT_ARCHIVE_NAME), 'foo', 'food'))
-    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
         verbosity=None,
@@ -315,7 +343,8 @@ def test_create_archive_with_glob_should_call_borg_with_expanded_directories():
     )
 
 
-def test_create_archive_with_archive_name_format_without_placeholders():
+def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(('borg', 'create', 'repo::ARCHIVE_NAME', 'foo', 'bar'))
@@ -335,6 +364,7 @@ def test_create_archive_with_archive_name_format_without_placeholders():
 
 
 def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
+    flexmock(module).should_receive('_expand_directory').and_return(['foo']).and_return(['bar'])
     flexmock(module).should_receive('_write_exclude_file').and_return(None)
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(('borg', 'create', 'repo::Documents_{hostname}-{now}', 'foo', 'bar'))

borgmatic/tests/unit/borg/test_hook.py

@@ -0,0 +1,22 @@
+from flexmock import flexmock
+
+from borgmatic.commands import hook as module
+
+
+def test_execute_hook_invokes_each_command():
+    subprocess = flexmock(module.subprocess)
+    subprocess.should_receive('check_call').with_args(':', shell=True).once()
+
+    module.execute_hook([':'], 'config.yaml', 'pre-backup')
+
+
+def test_execute_hook_with_multiple_commands_invokes_each_command():
+    subprocess = flexmock(module.subprocess)
+    subprocess.should_receive('check_call').with_args(':', shell=True).once()
+    subprocess.should_receive('check_call').with_args('true', shell=True).once()
+
+    module.execute_hook([':', 'true'], 'config.yaml', 'pre-backup')
+
+
+def test_execute_hook_with_empty_commands_does_not_raise():
+    module.execute_hook([], 'config.yaml', 'post-backup')

borgmatic/tests/unit/borg/test_prune.py

@@ -18,7 +18,7 @@ BASE_PRUNE_FLAGS = (
 )
 
 
-def test_make_prune_flags_should_return_flags_from_config():
+def test_make_prune_flags_should_return_flags_from_config_plus_default_prefix():
     retention_config = OrderedDict(
         (
             ('keep_daily', 1),
@@ -29,7 +29,7 @@ def test_make_prune_flags_should_return_flags_from_config():
 
     result = module._make_prune_flags(retention_config)
 
-    assert tuple(result) == BASE_PRUNE_FLAGS
+    assert tuple(result) == BASE_PRUNE_FLAGS + (('--prefix', '{hostname}-'),)
 
 
 def test_make_prune_flags_accepts_prefix_with_placeholders():

borgmatic/tests/unit/config/test_collect.py

@@ -32,6 +32,19 @@ def test_collect_config_filenames_collects_files_from_given_directories_and_igno
     )
 
 
+def test_collect_config_filenames_skips_etc_borgmatic_config_dot_yaml_if_it_does_not_exist():
+    config_paths = ('config.yaml', '/etc/borgmatic/config.yaml')
+    mock_path = flexmock(module.os.path)
+    mock_path.should_receive('exists').with_args('config.yaml').and_return(True)
+    mock_path.should_receive('exists').with_args('/etc/borgmatic/config.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('config.yaml').and_return(False)
+    mock_path.should_receive('isdir').with_args('/etc/borgmatic/config.yaml').and_return(True)
+
+    config_filenames = tuple(module.collect_config_filenames(config_paths))
+
+    assert config_filenames == ('config.yaml',)
+
+
 def test_collect_config_filenames_skips_etc_borgmatic_dot_d_if_it_does_not_exist():
     config_paths = ('config.yaml', '/etc/borgmatic.d')
     mock_path = flexmock(module.os.path)
@@ -45,7 +58,7 @@ def test_collect_config_filenames_skips_etc_borgmatic_dot_d_if_it_does_not_exist
     assert config_filenames == ('config.yaml',)
 
 
-def test_collect_config_filenames_includes_directory_if_it_does_not_exist():
+def test_collect_config_filenames_includes_other_directory_if_it_does_not_exist():
     config_paths = ('config.yaml', '/my/directory')
     mock_path = flexmock(module.os.path)
     mock_path.should_receive('exists').with_args('config.yaml').and_return(True)

borgmatic/tests/unit/config/test_validate.py

@@ -0,0 +1,43 @@
+import pytest
+
+from borgmatic.config import validate as module
+
+
+def test_validation_error_str_contains_error_messages_and_config_filename():
+    error = module.Validation_error('config.yaml', ('oops', 'uh oh'))
+
+    result = str(error)
+
+    assert 'config.yaml' in result
+    assert 'oops' in result
+    assert 'uh oh' in result
+
+
+def test_apply_logical_validation_raises_if_archive_name_format_present_without_prefix():
+    with pytest.raises(module.Validation_error):
+        module.apply_logical_validation(
+            'config.yaml',
+            {
+                'storage': {'archive_name_format': '{hostname}-{now}'},
+                'retention': {'keep_daily': 7},
+            },
+        )
+
+
+def test_apply_logical_validation_does_not_raise_if_archive_name_format_and_prefix_present():
+    module.apply_logical_validation(
+        'config.yaml',
+        {
+            'storage': {'archive_name_format': '{hostname}-{now}'},
+            'retention': {'prefix': '{hostname}-'},
+        },
+    )
+
+
+def test_apply_logical_validation_does_not_raise_otherwise():
+    module.apply_logical_validation(
+        'config.yaml',
+        {
+            'retention': {'keep_secondly': 1000},
+        },
+    )

borgmatic/tests/unit/test_verbosity.py

@@ -0,0 +1,11 @@
+import logging
+
+from borgmatic import verbosity as module
+
+
+def test_verbosity_to_log_level_maps_known_verbosity_to_log_level():
+    assert module.verbosity_to_log_level(module.VERBOSITY_SOME) == logging.INFO
+
+
+def test_verbosity_to_log_level_maps_unknown_verbosity_to_error_level():
+    assert module.verbosity_to_log_level('my pants') == logging.ERROR

borgmatic/verbosity.py

@@ -1,2 +1,15 @@
+import logging
+
+
 VERBOSITY_SOME = 1
 VERBOSITY_LOTS = 2
+
+
+def verbosity_to_log_level(verbosity):
+    '''
+    Given a borgmatic verbosity value, return the corresponding Python log level.
+    '''
+    return {
+        VERBOSITY_SOME: logging.INFO,
+        VERBOSITY_LOTS: logging.DEBUG,
+    }.get(verbosity, logging.ERROR)

scripts/push

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -e
+
+git push -u origin
+git push -u github

scripts/release

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+rm -fr dist
+python3 setup.py bdist_wheel
+python3 setup.py sdist
+twine upload -r pypi dist/borgmatic-*.tar.gz
+twine upload -r pypi dist/borgmatic-*-py3-none-any.whl

setup.py

@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 
 
-VERSION = '1.1.7'
+VERSION = '1.1.9'
 
 
 setup(
@@ -11,7 +11,7 @@ setup(
     author='Dan Helfman',
     author_email='witten@torsion.org',
     url='https://torsion.org/borgmatic',
-    download_url='https://torsion.org/hg/borgmatic/archive/%s.tar.gz' % VERSION,
+    download_url='https://projects.torsion.org/witten/borgmatic/archive/%s.tar.gz' % VERSION,
     classifiers=(
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',