Bump version for release.

SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests.
SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to check.

.bandit

@@ -0,0 +1,3 @@
+[bandit]
+exclude=tests
+skips=S105,S404

.gitea/workflows/build.yaml

@@ -17,6 +17,7 @@ jobs:
   docs:
     needs: [test]
     runs-on: host
+    if: gitea.event_name == 'push'
     env:
       IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
 

.github/pull_request_template.md

@@ -0,0 +1,12 @@
+## Hold up, GitHub users
+
+Thanks for your contribution! 
+
+If you're submitting this PR on **projects.torsion.org**: Please ignore the following. You can safely delete this text before submitting your PR.
+
+If you're submitting this PR on **GitHub**: Unfortunately, we don't use GitHub pull requests to manage code contributions to this repository (and GitHub doesn't have any way to disable pull requests entirely). Instead, please see:
+
+https://torsion.org/borgmatic/#contributing
+
+... which provides full instructions on how to submit pull requests. You can even use your GitHub account to login.
+

NEWS

@@ -1,3 +1,85 @@
+2.0.7
+ * #1032: Fix a bug in which a Borg archive gets created even when a database hook fails.
+ * #1043: Support Btrfs subvolume paths in "source_directories" even when the subvolume is mounted
+   elsewhere.
+ * #1048: Ignore Btrfs subvolumes whose read-only status can't be determined.
+ * #1083: Add "debug_passphrase"/"display_passphrase" options and a "{unixtime}" placeholder in
+   support of Borg 2 features.
+ * #1099: Clarify documentation on command hooks order of execution.
+ * #1100: Fix a bug in which "borg --version" failing isn't considered a "fail" state in a command
+   hook.
+ * #1108: Fix a bug in which quoted "extra_borg_options" values containing spaces are passed to Borg
+   incorrectly.
+ * #1108: Add a "--comment" flag to the "create" action for creating an archive with a comment.
+ * Use the Bandit security analysis tool when running tests.
+ * SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests, e.g. due to
+   a compromised monitoring server holding requests open.
+ * SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to
+   check.
+
+2.0.6
+ * #1068: Fix a warning from LVM about leaked file descriptors.
+ * #1086: Fix for the "spot" check breaking when the "--progress" flag is used.
+ * #1089: Fix for the "spot" check erroring when a checksum command errors.
+ * #1091: Fix for the "config generate" action generating invalid configuration when upgrading
+   deprecated command hooks.
+ * #1093: Fix for the LVM hook erroring when the "--dry-run" flag is used.
+ * #1094: Fix incorrect documentation about customizing Borg exit codes:
+   https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
+ * #1095: Fix for the "spot" check's "xxh64sum_command" option erroring on commands containing
+   spaces.
+ * Add support for Borg 2's "s3:" and "b2:" repository URLs, so you can backup to S3 or B2 cloud
+   storage services even without using Rclone.
+ * During the "spot" check, truncate log messages containing many file paths.
+
+2.0.5
+ * #1033: Add a "password_transport" option to the MariaDB and MySQL database hooks for customizing
+   how borgmatic transmits passwords to the database client.
+ * #1078: Add "keep_3monthly" and "keep_13weekly" options for customizing "prune" action archive
+   retention.
+ * #1078: Add a "use_chunks_archive" option for controlling whether Borg uses its chunks cache
+   directory.
+ * #1078: For the "compact" action, pass "--dry-run" through to Borg.
+ * #1085: Fix a regression in which the default monitoring verbosity is 0 (warnings only) instead of
+   1 (info about steps borgmatic is taking). This prevented logs from showing up in monitoring
+   services like Healthchecks unless you had an explicit monitoring verbosity set.
+ * Move Mastodon social hosting from Fosstodon to FLOSS.social: https://floss.social/@borgmatic
+ * The borgmatic project no longer accepts pull requests on GitHub. But see
+   https://torsion.org/borgmatic/#contributing for how you can still submit pull requests. You can
+   even use your GitHub account to login.
+
+2.0.4
+ * #1072: Fix path rewriting for non-root patterns in the ZFS, Btrfs, and LVM hooks.
+ * #1073: Clarify the documentation about when an "after: error" command hook runs and how it
+   differs from other hooks:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #1075: Fix an incorrect warning about Borg placeholders being unsupported in a command hook.
+ * #1080: If the exact same "everything" command hook is present in multiple configuration files,
+   only run it once.
+
+2.0.3
+ * #1065: Fix a regression in monitoring hooks in which an error pinged the finish state instead of
+   the fail state.
+ * #1066: Add a "states" option to command hooks, so you can optionally skip an "after" hook if
+   borgmatic encounters an error.
+ * #1071: Fix an error in the LVM hook when removing a snapshot directory.
+
+2.0.2
+ * #1035: Document potential performance issues and workarounds with the ZFS, Btrfs, and LVM hooks:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #1053: Display a nicer error message when the "recreate" action encounters an archive that
+   already exists.
+ * #1059: Fix a regression in which soft failure exit codes in command hooks were not respected.
+ * #1060: Fix action command hooks getting run too many times when multiple borgmatic actions are
+   executed (implicitly or explicitly).
+ * #1060: Don't run action command hooks for actions listed in the "skip_actions" option.
+ * #1062: Fix a regression that broke environment variable interpolation.
+ * #1063: List the configured "when" action names in the log entries for command hooks.
+
+2.0.1
+ * #1057: Fix argument parsing to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
 2.0.0
  * TL;DR: More flexible, completely revamped command hooks. All configuration options settable on
    the command-line. New configuration options for many command-line flags (including verbosity!).
@@ -30,7 +112,7 @@
  * #837: Add custom command options for the MongoDB hook.
  * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
  * #1020: Document a database use case involving a temporary database client container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #1037: Fix an error with the "extract" action when both a remote repository and a
    "working_directory" are used.
  * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
@@ -280,7 +362,7 @@
    paths when a "working_directory" is set.
  * #906: Add documentation details for how to run custom database dump commands using binaries from
    running containers:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * Fix a regression in which the "color" option had no effect.
  * Add a recent contributors section to the documentation, because credit where credit's due! See:
    https://torsion.org/borgmatic/#recent-contributors
@@ -336,7 +418,7 @@
  * Fix handling of the NO_COLOR environment variable to ignore an empty value.
  * Add documentation about backing up containerized databases by configuring borgmatic to exec into
    a container to run a dump command:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 
 1.8.9
  * #311: Add custom dump/restore command options for MySQL and MariaDB.
@@ -606,7 +688,7 @@
    at the command-line. See the configuration reference for more information:
    https://torsion.org/borgmatic/docs/reference/configuration/
  * #649: Add documentation on backing up a database running in a container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #655: Fix error when databases are configured and a source directory doesn't exist.
  * Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
    To join in the pedantry, refresh your test environment with "tox --recreate".

README.md

@@ -40,8 +40,10 @@ checks:
       frequency: 2 weeks
 
 # Custom preparation scripts to run.
-before_backup:
-    - prepare-for-backup.sh
+commands:
+    - before: action
+      when: [create]
+      run: [prepare-for-backup.sh]
 
 # Databases to dump and include in backups.
 postgresql_databases:
@@ -136,7 +138,8 @@ issues.
 
 ### Social
 
-Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
+Follow <a rel="me" href="https://floss.social/@borgmatic">borgmatic on
+Mastodon</a>.
 
 
 ### Chat
@@ -157,21 +160,21 @@ Other questions or comments? Contact
 ### Contributing
 
 borgmatic [source code is
-available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
-on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
+available](https://projects.torsion.org/borgmatic-collective/borgmatic) and also
+has a read-only mirror on
+[GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
 
 borgmatic is licensed under the GNU General Public License version 3 or any
 later version.
 
-If you'd like to contribute to borgmatic development, please feel free to
-submit a [Pull
-Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
+If you'd like to contribute to borgmatic development, please feel free to submit
+a [pull
+request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
 open an
 [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) to
 discuss your idea. Note that you'll need to
 [register](https://projects.torsion.org/user/sign_up?invite_code=borgmatic)
-first. We also accept Pull Requests on GitHub, if that's more your thing. In
-general, contributions are very welcome. We don't bite!
+first. In general, contributions are very welcome. We don't bite!
 
 Also, please check out the [borgmatic development
 how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for

borgmatic/actions/check.py

@@ -6,9 +6,11 @@ import logging
 import os
 import pathlib
 import random
+import shlex
 import shutil
+import textwrap
 
-import borgmatic.actions.create
+import borgmatic.actions.pattern
 import borgmatic.borg.check
 import borgmatic.borg.create
 import borgmatic.borg.environment
@@ -372,9 +374,13 @@ def collect_spot_check_source_paths(
         borgmatic.borg.create.make_base_create_command(
             dry_run=True,
             repository_path=repository['path'],
-            config=dict(config, list_details=True),
-            patterns=borgmatic.actions.create.process_patterns(
-                borgmatic.actions.create.collect_patterns(config),
+            # Omit "progress" because it interferes with "list_details".
+            config=dict(
+                {option: value for option, value in config.items() if option != 'progress'},
+                list_details=True,
+            ),
+            patterns=borgmatic.actions.pattern.process_patterns(
+                borgmatic.actions.pattern.collect_patterns(config),
                 working_directory,
             ),
             local_borg_version=local_borg_version,
@@ -454,7 +460,7 @@ def collect_spot_check_archive_paths(
     )
 
 
-SAMPLE_PATHS_SUBSET_COUNT = 10000
+SAMPLE_PATHS_SUBSET_COUNT = 5000
 
 
 def compare_spot_check_hashes(
@@ -480,7 +486,7 @@ def compare_spot_check_hashes(
     sample_count = max(
         int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)), 1
     )
-    source_sample_paths = tuple(random.sample(source_paths, sample_count))
+    source_sample_paths = tuple(random.SystemRandom().sample(source_paths, sample_count))
     working_directory = borgmatic.config.paths.get_working_directory(config)
     hashable_source_sample_path = {
         source_path
@@ -508,7 +514,10 @@ def compare_spot_check_hashes(
             break
 
         hash_output = borgmatic.execute.execute_command_and_capture_output(
-            (spot_check_config.get('xxh64sum_command', 'xxh64sum'),)
+            tuple(
+                shlex.quote(part)
+                for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
+            )
             + tuple(
                 path for path in source_sample_paths_subset if path in hashable_source_sample_path
             ),
@@ -562,6 +571,9 @@ def compare_spot_check_hashes(
     return tuple(failing_paths)
 
 
+MAX_SPOT_CHECK_PATHS_LENGTH = 1000
+
+
 def spot_check(
     repository,
     config,
@@ -629,9 +641,12 @@ def spot_check(
     logger.debug(f'{len(archive_paths)} total archive paths for spot check')
 
     if len(source_paths) == 0:
-        logger.debug(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+        truncated_archive_paths = textwrap.shorten(
+            ', '.join(set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
         )
+        logger.debug(f'Paths in latest archive but not source paths: {truncated_archive_paths}')
         raise ValueError(
             'Spot check failed: There are no source paths to compare against the archive'
         )
@@ -642,11 +657,21 @@ def spot_check(
 
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
         rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
-        logger.debug(
-            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
+        truncated_exclusive_source_paths = textwrap.shorten(
+            ', '.join(rootless_source_paths - set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
         )
         logger.debug(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
+            f'Paths in source paths but not latest archive: {truncated_exclusive_source_paths}'
+        )
+        truncated_exclusive_archive_paths = textwrap.shorten(
+            ', '.join(set(archive_paths) - rootless_source_paths) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.debug(
+            f'Paths in latest archive but not source paths: {truncated_exclusive_archive_paths}'
         )
         raise ValueError(
             f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
@@ -669,8 +694,13 @@ def spot_check(
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
+        truncated_failing_paths = textwrap.shorten(
+            ', '.join(failing_paths),
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
         logger.debug(
-            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {truncated_failing_paths}'
         )
         raise ValueError(
             f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'

borgmatic/actions/create.py

@@ -1,272 +1,17 @@
-import glob
-import itertools
 import logging
-import os
-import pathlib
 
 import borgmatic.actions.json
 import borgmatic.borg.create
-import borgmatic.borg.pattern
+import borgmatic.borg.rename
+import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
-import borgmatic.hooks.command
 import borgmatic.hooks.dispatch
+from borgmatic.actions import pattern
 
 logger = logging.getLogger(__name__)
 
 
-def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_style.NONE):
-    '''
-    Given a Borg pattern as a string, parse it into a borgmatic.borg.pattern.Pattern instance and
-    return it.
-    '''
-    try:
-        (pattern_type, remainder) = pattern_line.split(' ', maxsplit=1)
-    except ValueError:
-        raise ValueError(f'Invalid pattern: {pattern_line}')
-
-    try:
-        (parsed_pattern_style, path) = remainder.split(':', maxsplit=1)
-        pattern_style = borgmatic.borg.pattern.Pattern_style(parsed_pattern_style)
-    except ValueError:
-        pattern_style = default_style
-        path = remainder
-
-    return borgmatic.borg.pattern.Pattern(
-        path,
-        borgmatic.borg.pattern.Pattern_type(pattern_type),
-        borgmatic.borg.pattern.Pattern_style(pattern_style),
-        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
-    )
-
-
-def collect_patterns(config):
-    '''
-    Given a configuration dict, produce a single sequence of patterns comprised of the configured
-    source directories, patterns, excludes, pattern files, and exclude files.
-
-    The idea is that Borg has all these different ways of specifying includes, excludes, source
-    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
-    ease of manipulation within borgmatic.
-    '''
-    try:
-        return (
-            tuple(
-                borgmatic.borg.pattern.Pattern(
-                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
-                )
-                for source_directory in config.get('source_directories', ())
-            )
-            + tuple(
-                parse_pattern(pattern_line.strip())
-                for pattern_line in config.get('patterns', ())
-                if not pattern_line.lstrip().startswith('#')
-                if pattern_line.strip()
-            )
-            + tuple(
-                parse_pattern(
-                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
-                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
-                )
-                for exclude_line in config.get('exclude_patterns', ())
-            )
-            + tuple(
-                parse_pattern(pattern_line.strip())
-                for filename in config.get('patterns_from', ())
-                for pattern_line in open(filename).readlines()
-                if not pattern_line.lstrip().startswith('#')
-                if pattern_line.strip()
-            )
-            + tuple(
-                parse_pattern(
-                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
-                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
-                )
-                for filename in config.get('exclude_from', ())
-                for exclude_line in open(filename).readlines()
-                if not exclude_line.lstrip().startswith('#')
-                if exclude_line.strip()
-            )
-        )
-    except (FileNotFoundError, OSError) as error:
-        logger.debug(error)
-
-        raise ValueError(f'Cannot read patterns_from/exclude_from file: {error.filename}')
-
-
-def expand_directory(directory, working_directory):
-    '''
-    Given a directory path, expand any tilde (representing a user's home directory) and any globs
-    therein. Return a list of one or more resulting paths.
-
-    Take into account the given working directory so that relative paths are supported.
-    '''
-    expanded_directory = os.path.expanduser(directory)
-
-    # This would be a lot easier to do with glob(..., root_dir=working_directory), but root_dir is
-    # only available in Python 3.10+.
-    normalized_directory = os.path.join(working_directory or '', expanded_directory)
-    glob_paths = glob.glob(normalized_directory)
-
-    if not glob_paths:
-        return [expanded_directory]
-
-    working_directory_prefix = os.path.join(working_directory or '', '')
-
-    return [
-        (
-            glob_path
-            # If these are equal, that means we didn't add any working directory prefix above.
-            if normalized_directory == expanded_directory
-            # Remove the working directory prefix that we added above in order to make glob() work.
-            # We can't use os.path.relpath() here because it collapses any use of Borg's slashdot
-            # hack.
-            else glob_path.removeprefix(working_directory_prefix)
-        )
-        for glob_path in glob_paths
-    ]
-
-
-def expand_patterns(patterns, working_directory=None, skip_paths=None):
-    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
-    expand tildes and globs in each root pattern and expand just tildes in each non-root pattern.
-    The idea is that non-root patterns may be regular expressions or other pattern styles containing
-    "*" that borgmatic should not expand as a shell glob.
-
-    Return all the resulting patterns as a tuple.
-
-    If a set of paths are given to skip, then don't expand any patterns matching them.
-    '''
-    if patterns is None:
-        return ()
-
-    return tuple(
-        itertools.chain.from_iterable(
-            (
-                (
-                    borgmatic.borg.pattern.Pattern(
-                        expanded_path,
-                        pattern.type,
-                        pattern.style,
-                        pattern.device,
-                        pattern.source,
-                    )
-                    for expanded_path in expand_directory(pattern.path, working_directory)
-                )
-                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
-                and pattern.path not in (skip_paths or ())
-                else (
-                    borgmatic.borg.pattern.Pattern(
-                        os.path.expanduser(pattern.path),
-                        pattern.type,
-                        pattern.style,
-                        pattern.device,
-                        pattern.source,
-                    ),
-                )
-            )
-            for pattern in patterns
-        )
-    )
-
-
-def device_map_patterns(patterns, working_directory=None):
-    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
-    determine the identifier for the device on which the pattern's path resides—or None if the path
-    doesn't exist or is from a non-root pattern. Return an updated sequence of patterns with the
-    device field populated. But if the device field is already set, don't bother setting it again.
-
-    This is handy for determining whether two different pattern paths are on the same filesystem
-    (have the same device identifier).
-    '''
-    return tuple(
-        borgmatic.borg.pattern.Pattern(
-            pattern.path,
-            pattern.type,
-            pattern.style,
-            device=pattern.device
-            or (
-                os.stat(full_path).st_dev
-                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
-                and os.path.exists(full_path)
-                else None
-            ),
-            source=pattern.source,
-        )
-        for pattern in patterns
-        for full_path in (os.path.join(working_directory or '', pattern.path),)
-    )
-
-
-def deduplicate_patterns(patterns):
-    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances, return them with all duplicate
-    root child patterns removed. For instance, if two root patterns are given with paths "/foo" and
-    "/foo/bar", return just the one with "/foo". Non-root patterns are passed through without
-    modification.
-
-    The one exception to deduplication is two paths are on different filesystems (devices). In that
-    case, they won't get deduplicated, in case they both need to be passed to Borg (e.g. the
-    one_file_system option is true).
-
-    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
-    child patterns, because it will naturally spider the contents of the parent pattern's path. And
-    there are cases where Borg coming across the same file twice will result in duplicate reads and
-    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
-    Borg.
-    '''
-    deduplicated = {}  # Use just the keys as an ordered set.
-
-    for pattern in patterns:
-        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
-            deduplicated[pattern] = True
-            continue
-
-        parents = pathlib.PurePath(pattern.path).parents
-
-        # If another directory in the given list is a parent of current directory (even n levels up)
-        # and both are on the same filesystem, then the current directory is a duplicate.
-        for other_pattern in patterns:
-            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
-                continue
-
-            if any(
-                pathlib.PurePath(other_pattern.path) == parent
-                and pattern.device is not None
-                and other_pattern.device == pattern.device
-                for parent in parents
-            ):
-                break
-        else:
-            deduplicated[pattern] = True
-
-    return tuple(deduplicated.keys())
-
-
-def process_patterns(patterns, working_directory, skip_expand_paths=None):
-    '''
-    Given a sequence of Borg patterns and a configured working directory, expand and deduplicate any
-    "root" patterns, returning the resulting root and non-root patterns as a list.
-
-    If any paths are given to skip, don't expand them.
-    '''
-    skip_paths = set(skip_expand_paths or ())
-
-    return list(
-        deduplicate_patterns(
-            device_map_patterns(
-                expand_patterns(
-                    patterns,
-                    working_directory=working_directory,
-                    skip_paths=skip_paths,
-                )
-            )
-        )
-    )
-
-
 def run_create(
     config_filename,
     repository,
@@ -310,7 +55,7 @@ def run_create(
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
-        patterns = process_patterns(collect_patterns(config), working_directory)
+        patterns = pattern.process_patterns(pattern.collect_patterns(config), working_directory)
         active_dumps = borgmatic.hooks.dispatch.call_hooks(
             'dump_data_sources',
             config,
@@ -324,9 +69,17 @@ def run_create(
         # Process the patterns again in case any data source hooks updated them. Without this step,
         # we could end up with duplicate paths that cause Borg to hang when it tries to read from
         # the same named pipe twice.
-        patterns = process_patterns(patterns, working_directory, skip_expand_paths=config_paths)
+        patterns = pattern.process_patterns(
+            patterns, working_directory, skip_expand_paths=config_paths
+        )
         stream_processes = [process for processes in active_dumps.values() for process in processes]
 
+        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+        # to make sure we only create a real archive if all the streaming processes completed
+        # successfully (create_archive will fail if a streaming process fails, but the archive might
+        # have already been created at this point).
+        use_checkpoint = bool(stream_processes)
+
         json_output = borgmatic.borg.create.create_archive(
             global_arguments.dry_run,
             repository['path'],
@@ -335,14 +88,41 @@ def run_create(
             local_borg_version,
             global_arguments,
             borgmatic_runtime_directory,
+            archive_suffix='.checkpoint' if use_checkpoint else '',
             local_path=local_path,
             remote_path=remote_path,
             json=create_arguments.json,
+            comment=create_arguments.comment,
             stream_processes=stream_processes,
         )
 
+        if use_checkpoint:
+            rename_checkpoint_archive(
+                repository['path'],
+                global_arguments,
+                config,
+                local_borg_version,
+                local_path,
+                remote_path,
+            )
+
         if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+            if use_checkpoint:
+                # Patch archive name and ID
+                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                    repository['path'],
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                )
+
+                output['archive']['name'] = renamed_archive['name']
+                output['archive']['id'] = renamed_archive['id']
+
+            yield output
 
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
@@ -351,3 +131,54 @@ def run_create(
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
+
+
+def rename_checkpoint_archive(
+    repository_path,
+    global_arguments,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    '''
+    Renames the latest archive to not have a '.checkpoint' suffix.
+
+    Raises ValueError if
+    - there is not latest archive
+    - the latest archive does not have a '.checkpoint' suffix
+
+    Implementation note: We cannot reliably get the just created archive name.
+    So we resort to listing the archives and picking the last one.
+
+    A similar comment applies to retrieving the ID of the renamed archive.
+    '''
+    archive = borgmatic.borg.repo_list.get_latest_archive(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+        consider_checkpoints=True,
+    )
+
+    archive_name = archive['name']
+
+    if not archive_name.endswith('.checkpoint'):
+        raise ValueError(f'Latest archive did not have a .checkpoint suffix. Got: {archive_name}')
+
+    new_archive_name = archive_name.removesuffix('.checkpoint')
+
+    logger.info(f'Renaming archive {archive_name} -> {new_archive_name}')
+
+    borgmatic.borg.rename.rename_archive(
+        repository_path,
+        archive_name,
+        new_archive_name,
+        global_arguments.dry_run,
+        config,
+        local_borg_version,
+        local_path,
+        remote_path,
+    )

borgmatic/actions/pattern.py

@@ -0,0 +1,292 @@
+import glob
+import itertools
+import logging
+import os
+import pathlib
+
+import borgmatic.borg.pattern
+
+logger = logging.getLogger(__name__)
+
+
+def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_style.NONE):
+    '''
+    Given a Borg pattern as a string, parse it into a borgmatic.borg.pattern.Pattern instance and
+    return it.
+    '''
+    try:
+        (pattern_type, remainder) = pattern_line.split(' ', maxsplit=1)
+    except ValueError:
+        raise ValueError(f'Invalid pattern: {pattern_line}')
+
+    try:
+        (parsed_pattern_style, path) = remainder.split(':', maxsplit=1)
+        pattern_style = borgmatic.borg.pattern.Pattern_style(parsed_pattern_style)
+    except ValueError:
+        pattern_style = default_style
+        path = remainder
+
+    return borgmatic.borg.pattern.Pattern(
+        path,
+        borgmatic.borg.pattern.Pattern_type(pattern_type),
+        borgmatic.borg.pattern.Pattern_style(pattern_style),
+        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
+    )
+
+
+def collect_patterns(config):
+    '''
+    Given a configuration dict, produce a single sequence of patterns comprised of the configured
+    source directories, patterns, excludes, pattern files, and exclude files.
+
+    The idea is that Borg has all these different ways of specifying includes, excludes, source
+    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
+    ease of manipulation within borgmatic.
+    '''
+    try:
+        return (
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
+                )
+                for source_directory in config.get('source_directories', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for pattern_line in config.get('patterns', ())
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for exclude_line in config.get('exclude_patterns', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for filename in config.get('patterns_from', ())
+                for pattern_line in open(filename).readlines()
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for filename in config.get('exclude_from', ())
+                for exclude_line in open(filename).readlines()
+                if not exclude_line.lstrip().startswith('#')
+                if exclude_line.strip()
+            )
+        )
+    except (FileNotFoundError, OSError) as error:
+        logger.debug(error)
+
+        raise ValueError(f'Cannot read patterns_from/exclude_from file: {error.filename}')
+
+
+def expand_directory(directory, working_directory):
+    '''
+    Given a directory path, expand any tilde (representing a user's home directory) and any globs
+    therein. Return a list of one or more resulting paths.
+
+    Take into account the given working directory so that relative paths are supported.
+    '''
+    expanded_directory = os.path.expanduser(directory)
+
+    # This would be a lot easier to do with glob(..., root_dir=working_directory), but root_dir is
+    # only available in Python 3.10+.
+    normalized_directory = os.path.join(working_directory or '', expanded_directory)
+    glob_paths = glob.glob(normalized_directory)
+
+    if not glob_paths:
+        return [expanded_directory]
+
+    working_directory_prefix = os.path.join(working_directory or '', '')
+
+    return [
+        (
+            glob_path
+            # If these are equal, that means we didn't add any working directory prefix above.
+            if normalized_directory == expanded_directory
+            # Remove the working directory prefix that we added above in order to make glob() work.
+            # We can't use os.path.relpath() here because it collapses any use of Borg's slashdot
+            # hack.
+            else glob_path.removeprefix(working_directory_prefix)
+        )
+        for glob_path in glob_paths
+    ]
+
+
+def expand_patterns(patterns, working_directory=None, skip_paths=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    expand tildes and globs in each root pattern and expand just tildes in each non-root pattern.
+    The idea is that non-root patterns may be regular expressions or other pattern styles containing
+    "*" that borgmatic should not expand as a shell glob.
+
+    Return all the resulting patterns as a tuple.
+
+    If a set of paths are given to skip, then don't expand any patterns matching them.
+    '''
+    if patterns is None:
+        return ()
+
+    return tuple(
+        itertools.chain.from_iterable(
+            (
+                (
+                    borgmatic.borg.pattern.Pattern(
+                        expanded_path,
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    )
+                    for expanded_path in expand_directory(pattern.path, working_directory)
+                )
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.path not in (skip_paths or ())
+                else (
+                    borgmatic.borg.pattern.Pattern(
+                        os.path.expanduser(pattern.path),
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    ),
+                )
+            )
+            for pattern in patterns
+        )
+    )
+
+
+def get_existent_path_or_parent(path):
+    '''
+    Given a path, return it if it exists. Otherwise, return the longest parent directory of the path
+    that exists. Return None if none of these paths exist.
+
+    This is used below for finding an existent path prefix of pattern's path, which is necessary if
+    the path contain globs or other special characters that we don't want to try to interpret
+    (because we want to leave that responsibility to Borg).
+    '''
+    if path.startswith('/e2e/'):
+        return None
+
+    try:
+        return next(
+            candidate_path
+            for candidate_path in (path,)
+            + tuple(str(parent) for parent in pathlib.PurePath(path).parents)
+            if os.path.exists(candidate_path)
+        )
+    except StopIteration:
+        return None
+
+
+def device_map_patterns(patterns, working_directory=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    determine the identifier for the device on which the pattern's path resides—or None if the path
+    doesn't exist or is from a non-root pattern. Return an updated sequence of patterns with the
+    device field populated. But if the device field is already set, don't bother setting it again.
+
+    This is handy for determining whether two different pattern paths are on the same filesystem
+    (have the same device identifier).
+
+    This function only considers the start of a pattern's path—from the start of the path up until
+    there's a path component with a glob or other non-literal character. If there are no such
+    characters, the whole path is considered. The rationale is that it's not feasible for borgmatic
+    to interpret Borg's patterns to see which actual files (and therefore devices) they map to. So
+    for instance, a pattern with a path of "/var/log/*/data" would end up with its device set to the
+    device of "/var/log"—ignoring the "/*/data" part due to that glob.
+
+    The one exception is that if a regular expression pattern path starts with "^", that will get
+    stripped off for purposes of determining its device.
+    '''
+    return tuple(
+        borgmatic.borg.pattern.Pattern(
+            pattern.path,
+            pattern.type,
+            pattern.style,
+            device=pattern.device or (os.stat(existent_path).st_dev if existent_path else None),
+            source=pattern.source,
+        )
+        for pattern in patterns
+        for existent_path in (
+            get_existent_path_or_parent(
+                os.path.join(working_directory or '', pattern.path.lstrip('^'))
+            ),
+        )
+    )
+
+
+def deduplicate_patterns(patterns):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, return them with all duplicate
+    root child patterns removed. For instance, if two root patterns are given with paths "/foo" and
+    "/foo/bar", return just the one with "/foo". Non-root patterns are passed through without
+    modification.
+
+    The one exception to deduplication is two paths are on different filesystems (devices). In that
+    case, they won't get deduplicated, in case they both need to be passed to Borg (e.g. the
+    one_file_system option is true).
+
+    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    child patterns, because it will naturally spider the contents of the parent pattern's path. And
+    there are cases where Borg coming across the same file twice will result in duplicate reads and
+    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
+    Borg.
+    '''
+    deduplicated = {}  # Use just the keys as an ordered set.
+
+    for pattern in patterns:
+        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+            deduplicated[pattern] = True
+            continue
+
+        parents = pathlib.PurePath(pattern.path).parents
+
+        # If another directory in the given list is a parent of current directory (even n levels up)
+        # and both are on the same filesystem, then the current directory is a duplicate.
+        for other_pattern in patterns:
+            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+                continue
+
+            if any(
+                pathlib.PurePath(other_pattern.path) == parent
+                and pattern.device is not None
+                and other_pattern.device == pattern.device
+                for parent in parents
+            ):
+                break
+        else:
+            deduplicated[pattern] = True
+
+    return tuple(deduplicated.keys())
+
+
+def process_patterns(patterns, working_directory, skip_expand_paths=None):
+    '''
+    Given a sequence of Borg patterns and a configured working directory, expand and deduplicate any
+    "root" patterns, returning the resulting root and non-root patterns as a list.
+
+    If any paths are given to skip, don't expand them.
+    '''
+    skip_paths = set(skip_expand_paths or ())
+
+    return list(
+        deduplicate_patterns(
+            device_map_patterns(
+                expand_patterns(
+                    patterns,
+                    working_directory=working_directory,
+                    skip_paths=skip_paths,
+                )
+            )
+        )
+    )

borgmatic/actions/recreate.py

@@ -1,12 +1,18 @@
 import logging
+import subprocess
 
+import borgmatic.borg.info
 import borgmatic.borg.recreate
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
-from borgmatic.actions.create import collect_patterns, process_patterns
+from borgmatic.actions.pattern import collect_patterns, process_patterns
 
 logger = logging.getLogger(__name__)
 
 
+BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS = 30
+
+
 def run_recreate(
     repository,
     config,
@@ -32,22 +38,47 @@ def run_recreate(
             collect_patterns(config), borgmatic.config.paths.get_working_directory(config)
         )
 
-        borgmatic.borg.recreate.recreate_archive(
+        archive = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
-                recreate_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
+            recreate_arguments.archive,
             config,
             local_borg_version,
-            recreate_arguments,
             global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-            patterns=processed_patterns,
+            local_path,
+            remote_path,
         )
+
+        if archive and archive.endswith('.recreate'):
+            if recreate_arguments.archive == 'latest':
+                raise ValueError(
+                    f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.'
+                )
+            else:
+                raise ValueError(
+                    f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.'
+                )
+
+        try:
+            borgmatic.borg.recreate.recreate_archive(
+                repository['path'],
+                archive,
+                config,
+                local_borg_version,
+                recreate_arguments,
+                global_arguments,
+                local_path=local_path,
+                remote_path=remote_path,
+                patterns=processed_patterns,
+            )
+        except subprocess.CalledProcessError as error:
+            if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
+                if recreate_arguments.target:
+                    raise ValueError(
+                        f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.'
+                    )
+                elif archive:
+                    raise ValueError(
+                        f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.'
+                    )
+
+            raise

borgmatic/borg/borg.py

@@ -60,7 +60,7 @@ def run_arbitrary_borg(
     return execute_command(
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
-        shell=True,
+        shell=True,  # noqa: S604
         environment=dict(
             (environment.make_environment(config) or {}),
             **{

borgmatic/borg/check.py

@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
+import shlex
 
 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags, repo_info
@@ -171,7 +172,7 @@ def check_archives(
             + (('--lock-wait', str(lock_wait)) if lock_wait else ())
             + verbosity_flags
             + (('--progress',) if config.get('progress') else ())
-            + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+            + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
             + flags.make_repository_flags(repository_path, local_borg_version)
         )
 

borgmatic/borg/compact.py

@@ -1,7 +1,8 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
-from borgmatic.borg import environment, flags
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
@@ -37,11 +38,16 @@ def compact_segments(
         + (('--threshold', str(threshold)) if threshold else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (
+            ('--dry-run',)
+            if dry_run and feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version)
+            else ()
+        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    if dry_run:
+    if dry_run and not feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version):
         logging.info('Skipping compact (dry run)')
         return
 

borgmatic/borg/create.py

@@ -1,9 +1,8 @@
-import itertools
 import logging
 import os
 import pathlib
+import shlex
 import stat
-import tempfile
 import textwrap
 
 import borgmatic.borg.pattern
@@ -20,76 +19,6 @@ from borgmatic.execute import (
 logger = logging.getLogger(__name__)
 
 
-def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
-    '''
-    Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
-    temporary file in the given borgmatic runtime directory and return the file object so it can
-    continue to exist on disk as long as the caller needs it.
-
-    If an optional open pattern file is given, append to it instead of making a new temporary file.
-    Return None if no patterns are provided.
-    '''
-    if not patterns:
-        return None
-
-    if patterns_file is None:
-        patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
-        operation_name = 'Writing'
-    else:
-        patterns_file.write('\n')
-        operation_name = 'Appending'
-
-    patterns_output = '\n'.join(
-        f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
-        for pattern in patterns
-    )
-    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
-
-    patterns_file.write(patterns_output)
-    patterns_file.flush()
-
-    return patterns_file
-
-
-def make_exclude_flags(config):
-    '''
-    Given a configuration dict with various exclude options, return the corresponding Borg flags as
-    a tuple.
-    '''
-    caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
-    if_present_flags = tuple(
-        itertools.chain.from_iterable(
-            ('--exclude-if-present', if_present)
-            for if_present in config.get('exclude_if_present', ())
-        )
-    )
-    keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
-    exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
-
-    return caches_flag + if_present_flags + keep_exclude_tags_flags + exclude_nodump_flags
-
-
-def make_list_filter_flags(local_borg_version, dry_run):
-    '''
-    Given the local Borg version and whether this is a dry run, return the corresponding flags for
-    passing to "--list --filter". The general idea is that excludes are shown for a dry run or when
-    the verbosity is debug.
-    '''
-    base_flags = 'AME'
-    show_excludes = logger.isEnabledFor(logging.DEBUG)
-
-    if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
-        if show_excludes or dry_run:
-            return f'{base_flags}+-'
-        else:
-            return base_flags
-
-    if show_excludes:
-        return f'{base_flags}x-'
-    else:
-        return f'{base_flags}-'
-
-
 def special_file(path, working_directory=None):
     '''
     Return whether the given path is a special file (character device, block device, or named pipe
@@ -141,7 +70,7 @@ def collect_special_file_paths(
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
     # files including any named pipe we've created. And omit "--filter" because that can break the
-    # paths output parsing below such that path lines no longer start with th expected "- ".
+    # paths output parsing below such that path lines no longer start with the expected "- ".
     paths_output = execute_command_and_capture_output(
         flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
         + ('--dry-run', '--list'),
@@ -182,24 +111,6 @@ def collect_special_file_paths(
     )
 
 
-def check_all_root_patterns_exist(patterns):
-    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances, check that all root pattern
-    paths exist. If any don't, raise an exception.
-    '''
-    missing_paths = [
-        pattern.path
-        for pattern in patterns
-        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
-        if not os.path.exists(pattern.path)
-    ]
-
-    if missing_paths:
-        raise ValueError(
-            f"Source directories or root pattern paths do not exist: {', '.join(missing_paths)}"
-        )
-
-
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
 
 
@@ -211,22 +122,28 @@ def make_base_create_command(
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
     json=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
     Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version,
-    global arguments as an argparse.Namespace instance, and a sequence of borgmatic source
-    directories, return a tuple of (base Borg create command flags, Borg create command positional
-    arguments, open pattern file handle).
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version, global
+    arguments as an argparse.Namespace instance, the borgmatic runtime directory, a string suffix to
+    add to the archive name, the local Borg path, the remote Borg path, whether to output JSON,
+    comment text to add to the created archive, and a sequence of processes streaming data to Borg,
+    return a tuple of (base Borg create command flags, Borg create command positional arguments,
+    open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
-        check_all_root_patterns_exist(patterns)
+        borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
 
-    patterns_file = write_patterns_file(patterns, borgmatic_runtime_directory)
+    patterns_file = borgmatic.borg.pattern.write_patterns_file(
+        patterns, borgmatic_runtime_directory
+    )
     checkpoint_interval = config.get('checkpoint_interval', None)
     checkpoint_volume = config.get('checkpoint_volume', None)
     chunker_params = config.get('chunker_params', None)
@@ -235,10 +152,11 @@ def make_base_create_command(
     upload_buffer_size = config.get('upload_buffer_size', None)
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
-    list_filter_flags = make_list_filter_flags(local_borg_version, dry_run)
+    list_filter_flags = flags.make_list_filter_flags(local_borg_version, dry_run)
     files_cache = config.get('files_cache')
-    archive_name_format = config.get(
-        'archive_name_format', flags.get_default_archive_name_format(local_borg_version)
+    archive_name_format = (
+        config.get('archive_name_format', flags.get_default_archive_name_format(local_borg_version))
+        + archive_suffix
     )
     extra_borg_options = config.get('extra_borg_options', {}).get('create', '')
 
@@ -270,7 +188,8 @@ def make_base_create_command(
         tuple(local_path.split(' '))
         + ('create',)
         + (('--patterns-from', patterns_file.name) if patterns_file else ())
-        + make_exclude_flags(config)
+        + flags.make_exclude_flags(config)
+        + (('--comment', comment) if comment else ())
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
         + (('--chunker-params', chunker_params) if chunker_params else ())
@@ -295,7 +214,7 @@ def make_base_create_command(
             else ()
         )
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
     )
 
     create_positional_arguments = flags.make_repository_archive_flags(
@@ -329,7 +248,7 @@ def make_base_create_command(
             logger.warning(
                 f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
             )
-            patterns_file = write_patterns_file(
+            patterns_file = borgmatic.borg.pattern.write_patterns_file(
                 tuple(
                     borgmatic.borg.pattern.Pattern(
                         special_file_path,
@@ -357,15 +276,20 @@ def create_archive(
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
     json=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
     Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, and global arguments as an
-    argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
+    sequence of loaded configuration paths, the local Borg version, global arguments as an
+    argparse.Namespace instance, the borgmatic runtime directory, a string suffix to add to the
+    archive name, the local Borg path, the remote Borg path, whether to output JSON, and comment
+    text to add to the created archive, and a sequence of processes streaming data to Borg, create a
+    Borg archive and return Borg's JSON output (if any).
 
     If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
     create command while also triggering the given processes to produce output.
@@ -382,9 +306,11 @@ def create_archive(
         local_borg_version,
         global_arguments,
         borgmatic_runtime_directory,
+        archive_suffix,
         local_path,
         remote_path,
         json,
+        comment,
         stream_processes,
     )
 

borgmatic/borg/environment.py

@@ -14,13 +14,16 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'temporary_directory': 'TMPDIR',
 }
 
-DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
-    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
-    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE = {
+    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
 }
 
-DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
-    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
+DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
+    'debug_passphrase': 'BORG_DEBUG_PASSPHRASE',
+    'display_passphrase': 'BORG_DISPLAY_PASSPHRASE',
+    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
+    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
 }
 
 
@@ -82,15 +85,15 @@ def make_environment(config):
     for (
         option_name,
         environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
         if os.environ.get(environment_variable_name) is None:
             value = config.get(option_name)
-            environment[environment_variable_name] = 'yes' if value else 'no'
+            environment[environment_variable_name] = 'YES' if value else 'NO'
 
     for (
         option_name,
         environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
         if value is not None:
             environment[environment_variable_name] = 'YES' if value else 'NO'

borgmatic/borg/feature.py

@@ -18,6 +18,7 @@ class Feature(Enum):
     EXCLUDED_FILES_MINUS = 12
     ARCHIVE_SERIES = 13
     NO_PRUNE_STATS = 14
+    DRY_RUN_COMPACT = 15
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -35,6 +36,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
     Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
     Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
+    Feature.DRY_RUN_COMPACT: parse('1.4.1'),  # borg compact --dry-run support
 }
 
 

borgmatic/borg/flags.py

@@ -197,3 +197,42 @@ def omit_flag_and_value(arguments, flag):
         if flag not in (previous_argument, argument)
         if not argument.startswith(f'{flag}=')
     )
+
+
+def make_exclude_flags(config):
+    '''
+    Given a configuration dict with various exclude options, return the corresponding Borg flags as
+    a tuple.
+    '''
+    caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
+    if_present_flags = tuple(
+        itertools.chain.from_iterable(
+            ('--exclude-if-present', if_present)
+            for if_present in config.get('exclude_if_present', ())
+        )
+    )
+    keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
+    exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
+
+    return caches_flag + if_present_flags + keep_exclude_tags_flags + exclude_nodump_flags
+
+
+def make_list_filter_flags(local_borg_version, dry_run):
+    '''
+    Given the local Borg version and whether this is a dry run, return the corresponding flags for
+    passing to "--list --filter". The general idea is that excludes are shown for a dry run or when
+    the verbosity is debug.
+    '''
+    base_flags = 'AME'
+    show_excludes = logger.isEnabledFor(logging.DEBUG)
+
+    if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
+        if show_excludes or dry_run:
+            return f'{base_flags}+-'
+        else:
+            return base_flags
+
+    if show_excludes:
+        return f'{base_flags}x-'
+    else:
+        return f'{base_flags}-'

borgmatic/borg/pattern.py

@@ -1,5 +1,12 @@
 import collections
 import enum
+import logging
+import os
+import tempfile
+
+import borgmatic.borg.pattern
+
+logger = logging.getLogger(__name__)
 
 
 # See https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
@@ -48,3 +55,52 @@ Pattern = collections.namedtuple(
         Pattern_source.HOOK,
     ),
 )
+
+
+def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
+    '''
+    Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
+    temporary file in the given borgmatic runtime directory and return the file object so it can
+    continue to exist on disk as long as the caller needs it.
+
+    If an optional open pattern file is given, append to it instead of making a new temporary file.
+    Return None if no patterns are provided.
+    '''
+    if not patterns:
+        return None
+
+    if patterns_file is None:
+        patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
+        operation_name = 'Writing'
+    else:
+        patterns_file.write('\n')
+        operation_name = 'Appending'
+
+    patterns_output = '\n'.join(
+        f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
+        for pattern in patterns
+    )
+    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
+
+    patterns_file.write(patterns_output)
+    patterns_file.flush()
+
+    return patterns_file
+
+
+def check_all_root_patterns_exist(patterns):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, check that all root pattern
+    paths exist. If any don't, raise an exception.
+    '''
+    missing_paths = [
+        pattern.path
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        if not os.path.exists(pattern.path)
+    ]
+
+    if missing_paths:
+        raise ValueError(
+            f"Source directories or root pattern paths do not exist: {', '.join(missing_paths)}"
+        )

borgmatic/borg/prune.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -90,7 +91,7 @@ def prune_archives(
         + (('--list',) if config.get('list_details') else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/recreate.py

@@ -6,7 +6,7 @@ import borgmatic.borg.feature
 import borgmatic.config.paths
 import borgmatic.execute
 from borgmatic.borg import flags
-from borgmatic.borg.create import make_exclude_flags, make_list_filter_flags, write_patterns_file
+from borgmatic.borg.pattern import write_patterns_file
 
 logger = logging.getLogger(__name__)
 
@@ -29,9 +29,10 @@ def recreate_archive(
     arguments.
     '''
     lock_wait = config.get('lock_wait', None)
-    exclude_flags = make_exclude_flags(config)
+    exclude_flags = flags.make_exclude_flags(config)
     compression = config.get('compression', None)
     chunker_params = config.get('chunker_params', None)
+
     # Available recompress MODES: "if-different", "always", "never" (default)
     recompress = config.get('recompress', None)
 
@@ -52,7 +53,7 @@ def recreate_archive(
             (
                 '--list',
                 '--filter',
-                make_list_filter_flags(local_borg_version, global_arguments.dry_run),
+                flags.make_list_filter_flags(local_borg_version, global_arguments.dry_run),
             )
             if config.get('list_details')
             else ()

borgmatic/borg/rename.py

@@ -0,0 +1,62 @@
+import logging
+
+import borgmatic.borg.flags
+
+logger = logging.getLogger(__name__)
+
+
+def make_rename_command(
+    dry_run,
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    return (
+        (local_path, 'rename')
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + borgmatic.borg.flags.make_flags('dry-run', dry_run)
+        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
+        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
+        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
+        + borgmatic.borg.flags.make_repository_archive_flags(
+            repository_name, old_archive_name, local_borg_version
+        )
+        + (new_archive_name,)
+    )
+
+
+def rename_archive(
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    dry_run,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    command = make_rename_command(
+        dry_run,
+        repository_name,
+        old_archive_name,
+        new_archive_name,
+        config,
+        local_borg_version,
+        local_path,
+        remote_path,
+    )
+
+    borgmatic.execute.execute_command(
+        command,
+        output_log_level=logging.INFO,
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/repo_create.py

@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
+import shlex
 import subprocess
 
 import borgmatic.config.paths
@@ -86,7 +87,7 @@ def create_repository(
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/repo_list.py

@@ -1,4 +1,5 @@
 import argparse
+import json
 import logging
 
 import borgmatic.config.paths
@@ -29,6 +30,33 @@ def resolve_archive_name(
     if archive != 'latest':
         return archive
 
+    latest_archive = get_latest_archive(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
+
+    return latest_archive['name']
+
+
+def get_latest_archive(
+    repository_path,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+    consider_checkpoints=False,
+):
+    '''
+    Returns a dict with information about the latest archive of a repository.
+
+    Raises ValueError if there are no archives in the repository.
+    '''
+
     full_command = (
         (
             local_path,
@@ -42,24 +70,28 @@ def resolve_archive_name(
         + flags.make_flags('umask', config.get('umask'))
         + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', config.get('lock_wait'))
+        + flags.make_flags('consider-checkpoints', consider_checkpoints)
         + flags.make_flags('last', 1)
-        + ('--short',)
+        + ('--json',)
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    output = execute_command_and_capture_output(
+    json_output = execute_command_and_capture_output(
         full_command,
         environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
+
+    archives = json.loads(json_output)['archives']
+
     try:
-        latest_archive = output.strip().splitlines()[-1]
+        latest_archive = archives[-1]
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive["name"]}')
 
     return latest_archive
 

borgmatic/commands/arguments.py

@@ -357,7 +357,7 @@ def add_array_element_arguments(arguments_group, unparsed_arguments, flag_name):
     if '[0]' not in flag_name or not unparsed_arguments or '--help' in unparsed_arguments:
         return
 
-    pattern = re.compile(fr'^--{flag_name.replace("[0]", r"\[\d+\]").replace(".", r"\.")}$')
+    pattern = re.compile('^--' + flag_name.replace('[0]', r'\[\d+\]').replace('.', r'\.') + '$')
 
     try:
         # Find an existing list index flag (and its action) corresponding to the given flag name.
@@ -587,7 +587,7 @@ def make_parsers(schema, unparsed_arguments):
         '--no-environment-interpolation',
         dest='resolve_env',
         action='store_false',
-        help='Do not resolve environment variables in configuration file',
+        help='Do not resolve environment variables in configuration files',
     )
     global_group.add_argument(
         '--bash-completion',
@@ -667,7 +667,7 @@ def make_parsers(schema, unparsed_arguments):
         dest='make_parent_directories',
         default=None,
         action='store_true',
-        help='Create any missing parent directories of the repository directory',
+        help='Create any missing parent directories of the repository directory [Borg 1.x only]',
     )
     repo_create_group.add_argument(
         '-h', '--help', action='help', help='Show this help message and exit'
@@ -872,6 +872,11 @@ def make_parsers(schema, unparsed_arguments):
     create_group.add_argument(
         '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
     )
+    create_group.add_argument(
+        '--comment',
+        metavar='COMMENT',
+        help='Add a comment text to the archive',
+    )
     create_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
 
     check_parser = action_parsers.add_parser(

borgmatic/commands/borgmatic.py

@@ -95,7 +95,9 @@ class Monitoring_hooks:
         self.config_filename = config_filename
         self.config = config
         self.dry_run = global_arguments.dry_run
-        self.monitoring_log_level = verbosity_to_log_level(config.get('monitoring_verbosity'))
+        self.monitoring_log_level = verbosity_to_log_level(
+            get_verbosity({config_filename: config}, 'monitoring_verbosity')
+        )
         self.monitoring_hooks_are_activated = (
             using_primary_action and self.monitoring_log_level != DISABLED
         )
@@ -226,7 +228,7 @@ def run_configuration(config_filename, config, config_paths, arguments):
                     yield from log_error_records(
                         f'{config_filename}: Error getting local Borg version', error
                     )
-                    return
+                    raise
 
                 for repo in config['repositories']:
                     repo_queue.put(
@@ -279,8 +281,15 @@ def run_configuration(config_filename, config, config_paths, arguments):
                             encountered_error = error
                             error_repository = repository
 
+                # Re-raise any error, so that the Monitoring_hooks context manager wrapping this
+                # code can see the error and act accordingly. Do this here rather than as soon as
+                # the error is encountered so that an error with one repository doesn't prevent
+                # other repositories from running.
+                if encountered_error:
+                    raise encountered_error
+
     except (OSError, CalledProcessError, ValueError) as error:
-        yield from log_error_records('Error running configuration', error)
+        yield from log_error_records('Error running configuration')
 
         encountered_error = error
 
@@ -290,7 +299,10 @@ def run_configuration(config_filename, config, config_paths, arguments):
     try:
         command.execute_hooks(
             command.filter_hooks(
-                config.get('commands'), after='error', action_names=arguments.keys()
+                config.get('commands'),
+                after='error',
+                action_names=arguments.keys(),
+                state_names=['fail'],
             ),
             config.get('umask'),
             borgmatic.config.paths.get_working_directory(config),
@@ -356,7 +368,7 @@ def run_actions(
         **hook_context,
     ):
         for action_name, action_arguments in arguments.items():
-            if action_name == 'global':
+            if action_name == 'global' or action_name in skip_actions:
                 continue
 
             with borgmatic.hooks.command.Before_after_hooks(
@@ -365,10 +377,10 @@ def run_actions(
                 umask=config.get('umask'),
                 working_directory=borgmatic.config.paths.get_working_directory(config),
                 dry_run=global_arguments.dry_run,
-                action_names=arguments.keys(),
+                action_names=(action_name,),
                 **hook_context,
             ):
-                if action_name == 'repo-create' and action_name not in skip_actions:
+                if action_name == 'repo-create':
                     borgmatic.actions.repo_create.run_repo_create(
                         repository,
                         config,
@@ -378,7 +390,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'transfer' and action_name not in skip_actions:
+                elif action_name == 'transfer':
                     borgmatic.actions.transfer.run_transfer(
                         repository,
                         config,
@@ -388,7 +400,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'create' and action_name not in skip_actions:
+                elif action_name == 'create':
                     yield from borgmatic.actions.create.run_create(
                         config_filename,
                         repository,
@@ -401,7 +413,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'recreate' and action_name not in skip_actions:
+                elif action_name == 'recreate':
                     borgmatic.actions.recreate.run_recreate(
                         repository,
                         config,
@@ -411,7 +423,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'prune' and action_name not in skip_actions:
+                elif action_name == 'prune':
                     borgmatic.actions.prune.run_prune(
                         config_filename,
                         repository,
@@ -423,7 +435,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'compact' and action_name not in skip_actions:
+                elif action_name == 'compact':
                     borgmatic.actions.compact.run_compact(
                         config_filename,
                         repository,
@@ -435,7 +447,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'check' and action_name not in skip_actions:
+                elif action_name == 'check':
                     if checks.repository_enabled_for_checks(repository, config):
                         borgmatic.actions.check.run_check(
                             config_filename,
@@ -447,7 +459,7 @@ def run_actions(
                             local_path,
                             remote_path,
                         )
-                elif action_name == 'extract' and action_name not in skip_actions:
+                elif action_name == 'extract':
                     borgmatic.actions.extract.run_extract(
                         config_filename,
                         repository,
@@ -458,7 +470,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'export-tar' and action_name not in skip_actions:
+                elif action_name == 'export-tar':
                     borgmatic.actions.export_tar.run_export_tar(
                         repository,
                         config,
@@ -468,7 +480,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'mount' and action_name not in skip_actions:
+                elif action_name == 'mount':
                     borgmatic.actions.mount.run_mount(
                         repository,
                         config,
@@ -478,7 +490,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'restore' and action_name not in skip_actions:
+                elif action_name == 'restore':
                     borgmatic.actions.restore.run_restore(
                         repository,
                         config,
@@ -488,7 +500,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'repo-list' and action_name not in skip_actions:
+                elif action_name == 'repo-list':
                     yield from borgmatic.actions.repo_list.run_repo_list(
                         repository,
                         config,
@@ -498,7 +510,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'list' and action_name not in skip_actions:
+                elif action_name == 'list':
                     yield from borgmatic.actions.list.run_list(
                         repository,
                         config,
@@ -508,7 +520,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'repo-info' and action_name not in skip_actions:
+                elif action_name == 'repo-info':
                     yield from borgmatic.actions.repo_info.run_repo_info(
                         repository,
                         config,
@@ -518,7 +530,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'info' and action_name not in skip_actions:
+                elif action_name == 'info':
                     yield from borgmatic.actions.info.run_info(
                         repository,
                         config,
@@ -528,7 +540,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'break-lock' and action_name not in skip_actions:
+                elif action_name == 'break-lock':
                     borgmatic.actions.break_lock.run_break_lock(
                         repository,
                         config,
@@ -538,7 +550,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'export' and action_name not in skip_actions:
+                elif action_name == 'export':
                     borgmatic.actions.export_key.run_export_key(
                         repository,
                         config,
@@ -548,7 +560,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'import' and action_name not in skip_actions:
+                elif action_name == 'import':
                     borgmatic.actions.import_key.run_import_key(
                         repository,
                         config,
@@ -558,7 +570,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'change-passphrase' and action_name not in skip_actions:
+                elif action_name == 'change-passphrase':
                     borgmatic.actions.change_passphrase.run_change_passphrase(
                         repository,
                         config,
@@ -568,7 +580,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'delete' and action_name not in skip_actions:
+                elif action_name == 'delete':
                     borgmatic.actions.delete.run_delete(
                         repository,
                         config,
@@ -578,7 +590,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'repo-delete' and action_name not in skip_actions:
+                elif action_name == 'repo-delete':
                     borgmatic.actions.repo_delete.run_repo_delete(
                         repository,
                         config,
@@ -588,7 +600,7 @@ def run_actions(
                         local_path,
                         remote_path,
                     )
-                elif action_name == 'borg' and action_name not in skip_actions:
+                elif action_name == 'borg':
                     borgmatic.actions.borg.run_borg(
                         repository,
                         config,
@@ -738,7 +750,7 @@ def log_error_records(
     except (ValueError, OSError) as error:
         yield log_record(levelno=levelno, levelname=level_name, msg=str(message))
         yield log_record(levelno=levelno, levelname=level_name, msg=str(error))
-    except:  # noqa: E722
+    except:  # noqa: E722, S110
         # Raising above only as a means of determining the error type. Swallow the exception here
         # because we don't want the exception to propagate out of this function.
         pass
@@ -879,23 +891,36 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
         return
 
     try:
+        seen_command_hooks = []
+
         for config_filename, config in configs.items():
-            command.execute_hooks(
-                command.filter_hooks(
-                    config.get('commands'), before='everything', action_names=arguments.keys()
+            command_hooks = command.filter_hooks(
+                tuple(
+                    command_hook
+                    for command_hook in config.get('commands', ())
+                    if command_hook not in seen_command_hooks
                 ),
-                config.get('umask'),
-                borgmatic.config.paths.get_working_directory(config),
-                arguments['global'].dry_run,
-                configuration_filename=config_filename,
-                log_file=log_file_path or '',
+                before='everything',
+                action_names=arguments.keys(),
             )
+
+            if command_hooks:
+                command.execute_hooks(
+                    command_hooks,
+                    config.get('umask'),
+                    borgmatic.config.paths.get_working_directory(config),
+                    arguments['global'].dry_run,
+                    configuration_filename=config_filename,
+                    log_file=log_file_path or '',
+                )
+                seen_command_hooks += list(command_hooks)
     except (CalledProcessError, ValueError, OSError) as error:
         yield from log_error_records('Error running before everything hook', error)
         return
 
     # Execute the actions corresponding to each configuration file.
     json_results = []
+    encountered_error = False
 
     for config_filename, config in configs.items():
         with Log_prefix(config_filename):
@@ -906,6 +931,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
             )
 
             if error_logs:
+                encountered_error = True
                 yield from log_error_records('An error occurred')
                 yield from error_logs
             else:
@@ -928,23 +954,37 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                 local_path=get_local_path(configs),
             )
         except (CalledProcessError, OSError) as error:
+            encountered_error = True
             yield from log_error_records('Error unmounting mount point', error)
 
     if json_results:
         sys.stdout.write(json.dumps(json_results))
 
     try:
+        seen_command_hooks = []
+
         for config_filename, config in configs.items():
-            command.execute_hooks(
-                command.filter_hooks(
-                    config.get('commands'), after='everything', action_names=arguments.keys()
+            command_hooks = command.filter_hooks(
+                tuple(
+                    command_hook
+                    for command_hook in config.get('commands', ())
+                    if command_hook not in seen_command_hooks
                 ),
-                config.get('umask'),
-                borgmatic.config.paths.get_working_directory(config),
-                arguments['global'].dry_run,
-                configuration_filename=config_filename,
-                log_file=log_file_path or '',
+                after='everything',
+                action_names=arguments.keys(),
+                state_names=['fail' if encountered_error else 'finish'],
             )
+
+            if command_hooks:
+                command.execute_hooks(
+                    command_hooks,
+                    config.get('umask'),
+                    borgmatic.config.paths.get_working_directory(config),
+                    arguments['global'].dry_run,
+                    configuration_filename=config_filename,
+                    log_file=log_file_path or '',
+                )
+                seen_command_hooks += list(command_hooks)
     except (CalledProcessError, ValueError, OSError) as error:
         yield from log_error_records('Error running after everything hook', error)
 
@@ -1040,7 +1080,7 @@ def main(extra_summary_logs=[]):  # pragma: no cover
         config_filenames,
         arguments,
         global_arguments.overrides,
-        resolve_env=global_arguments.resolve_env and not validate,
+        resolve_env=global_arguments.resolve_env and not arguments.get('validate'),
     )
 
     # Use the helper function to check and show help on no arguments, passing the preloaded configs

borgmatic/config/generate.py

@@ -51,7 +51,7 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
     elif borgmatic.config.schema.compare_types(schema_type, {'object'}):
         if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
-            source_config = dict(collections.ChainMap(*source_config))
+            source_config = source_config[0]
 
         config = (
             ruamel.yaml.comments.CommentedMap(
@@ -71,7 +71,7 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
         )
         indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
         add_comments_to_configuration_object(
-            config, schema, source_config, indent=indent, skip_first=parent_is_sequence
+            config, schema, source_config, indent=indent, skip_first_field=parent_is_sequence
         )
     elif borgmatic.config.schema.compare_types(schema_type, SCALAR_SCHEMA_TYPES, match=all):
         return example
@@ -108,17 +108,30 @@ def comment_out_optional_configuration(rendered_config):
     '''
     lines = []
     optional = False
+    indent_characters = None
+    indent_characters_at_sentinel = None
 
     for line in rendered_config.split('\n'):
+        indent_characters = len(line) - len(line.lstrip())
+
         # Upon encountering an optional configuration option, comment out lines until the next blank
         # line.
         if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
             optional = True
+            indent_characters_at_sentinel = indent_characters
             continue
 
         # Hit a blank line, so reset commenting.
         if not line.strip():
             optional = False
+            indent_characters_at_sentinel = None
+        # Dedented, so reset commenting.
+        elif (
+            indent_characters_at_sentinel is not None
+            and indent_characters < indent_characters_at_sentinel
+        ):
+            optional = False
+            indent_characters_at_sentinel = None
 
         lines.append(comment_out_line(line) if optional else line)
 
@@ -198,25 +211,28 @@ COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 
 
 def add_comments_to_configuration_object(
-    config, schema, source_config=None, indent=0, skip_first=False
+    config, schema, source_config=None, indent=0, skip_first_field=False
 ):
     '''
     Using descriptions from a schema as a source, add those descriptions as comments to the given
     configuration dict, putting them before each field. Indent the comment the given number of
     characters.
 
+    If skip_first_field is True, omit the comment for the initial field. This is useful for
+    sequences, where the comment for the first field goes before the sequence itself.
+
     And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
     source configuration dict. The idea is that any options used in the source configuration should
     stay active in the generated configuration.
     '''
     for index, field_name in enumerate(config.keys()):
-        if skip_first and index == 0:
+        if skip_first_field and index == 0:
             continue
 
         field_schema = borgmatic.config.schema.get_properties(schema).get(field_name, {})
         description = field_schema.get('description', '').strip()
 
-        # If this isn't a default key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment, flagging it to be commented
         # out from the sample configuration. This sentinel is consumed by downstream processing that
         # does the actual commenting out.
         if field_name not in DEFAULT_KEYS and (
@@ -299,6 +315,12 @@ def generate_sample_configuration(
         source_config = load.load_configuration(source_filename)
         normalize.normalize(source_filename, source_config)
 
+        # The borgmatic.config.normalize.normalize() function tacks on an empty "bootstrap" if
+        # needed, so the hook gets used by default. But we don't want it to end up in the generated
+        # config unless the user has set it explicitly, as an empty "bootstrap:" won't validate.
+        if source_config and source_config.get('bootstrap') == {}:
+            del source_config['bootstrap']
+
     destination_config = merge_source_configuration_into_destination(
         schema_to_sample_configuration(schema, source_config), source_config
     )

borgmatic/config/normalize.py

@@ -159,7 +159,7 @@ def normalize(config_filename, config):
                 dict(
                     levelno=logging.WARNING,
                     levelname='WARNING',
-                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use borgmatic_runtime_directory and borgmatic_state_directory instead.',
+                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use user_runtime_directory and user_state_directory instead.',
                 )
             )
         )
@@ -356,6 +356,8 @@ def normalize(config_filename, config):
                     repository_path.startswith('ssh://')
                     or repository_path.startswith('sftp://')
                     or repository_path.startswith('rclone:')
+                    or repository_path.startswith('s3:')
+                    or repository_path.startswith('b2:')
                 ):
                     config['repositories'].append(repository_dict)
                 else:
@@ -365,7 +367,7 @@ def normalize(config_filename, config):
                             dict(
                                 levelno=logging.WARNING,
                                 levelname='WARNING',
-                                msg=f'{config_filename}: Remote repository paths without ssh:// or rclone: syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
+                                msg=f'{config_filename}: Remote repository paths without ssh://, sftp://, rclone:, s3:, or b2:, syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
                             )
                         )
                     )

borgmatic/config/paths.py

@@ -103,7 +103,9 @@ class Runtime_directory:
 
             self.temporary_directory = None
         else:
-            base_directory = os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'
+            base_directory = (
+                os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'  # noqa: S108
+            )
 
             if not base_directory.startswith(os.path.sep):
                 raise ValueError('The temporary directory must be an absolute path')

borgmatic/config/schema.yaml

@@ -72,7 +72,8 @@ properties:
                     description: |
                         Whether any missing parent directories of the repository
                         path should be created, only used for the repo-create
-                        action. Defaults to false.
+                        action. Defaults to false. (This option is supported 
+                        for Borg 1.x only)
                     example: true
         description: |
             A required list of local or remote repositories with paths and
@@ -389,6 +390,13 @@ properties:
             Path for Borg cache files. Defaults to
             $borg_base_directory/.cache/borg
         example: /path/to/base/cache
+    use_chunks_archive:
+        type: boolean
+        description: |
+            Enables or disables the use of chunks.archive.d for faster cache
+            resyncs in Borg. If true, value is set to "yes" (default) else
+            it's set to "no", reducing disk usage but slowing resyncs.
+        example: true
     borg_files_cache_ttl:
         type: integer
         description: |
@@ -483,6 +491,19 @@ properties:
             Bypass Borg error about a previously unknown unencrypted repository.
             Defaults to false.
         example: true
+    debug_passphrase:
+        type: boolean
+        description: |
+            When set true, display debugging information that includes 
+            passphrases used and passphrase related environment variables set. 
+            Defaults to false.
+        example: true
+    display_passphrase:
+        type: boolean
+        description: |
+            When set true, always shows passphrase and its hex UTF-8 byte
+            sequence. Defaults to false.
+        example: true
     check_i_know_what_i_am_doing:
         type: boolean
         description: |
@@ -558,6 +579,14 @@ properties:
         type: integer
         description: Number of yearly archives to keep.
         example: 1
+    keep_13weekly:
+        type: integer
+        description: Number of quarterly archives to keep (13 week strategy).
+        example: 13
+    keep_3monthly:
+        type: integer
+        description: Number of quarterly archives to keep (3 month strategy).
+        example: 3
     prefix:
         type: string
         description: |
@@ -1225,6 +1254,26 @@ properties:
                               particular actions listed here. Defaults to
                               running for all actions.
                           example: [create, prune, compact, check]
+                      states:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - finish
+                                  - fail
+                          description: |
+                              Only trigger the hook if borgmatic encounters one
+                              of the states (execution results) listed here,
+                              where:
+                               * "finish": No errors occurred.
+                               * "fail": An error occurred.
+                              This state is evaluated only for the scope of the
+                              configured "action", "repository", etc., rather
+                              than for the entire borgmatic run. Only available
+                              for "after" hooks. Defaults to running the hook
+                              for all states.
+                          example:
+                              - finish
                       run:
                           type: array
                           items:
@@ -1525,6 +1574,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MariaDB client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mariadb_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |
@@ -1673,6 +1739,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MySQL client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mysql_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |

borgmatic/execute.py

@@ -288,6 +288,7 @@ def execute_command(
     borg_local_path=None,
     borg_exit_codes=None,
     run_to_completion=True,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -312,11 +313,10 @@ def execute_command(
         stdin=input_file,
         stdout=None if do_not_capture else (output_file or subprocess.PIPE),
         stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
-        shell=shell,
+        shell=shell,  # noqa: S602
         env=environment,
         cwd=working_directory,
-        # Necessary for passing credentials via anonymous pipe.
-        close_fds=False,
+        close_fds=close_fds,
     )
     if not run_to_completion:
         return process
@@ -340,6 +340,7 @@ def execute_command_and_capture_output(
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
@@ -362,11 +363,10 @@ def execute_command_and_capture_output(
             command,
             stdin=input_file,
             stderr=subprocess.STDOUT if capture_stderr else None,
-            shell=shell,
+            shell=shell,  # noqa: S602
             env=environment,
             cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
-            close_fds=False,
+            close_fds=close_fds,
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -390,6 +390,7 @@ def execute_command_with_processes(
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -422,11 +423,10 @@ def execute_command_with_processes(
             stderr=(
                 None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
             ),
-            shell=shell,
+            shell=shell,  # noqa: S602
             env=environment,
             cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
-            close_fds=False,
+            close_fds=close_fds,
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.

borgmatic/hooks/command.py

@@ -1,3 +1,4 @@
+import functools
 import logging
 import os
 import re
@@ -12,6 +13,20 @@ logger = logging.getLogger(__name__)
 
 
 SOFT_FAIL_EXIT_CODE = 75
+BORG_PLACEHOLDERS = {
+    '{hostname}',
+    '{fqdn}',
+    '{reverse-fqdn}',
+    '{now}',
+    '{utcnow}',
+    '{unixtime}',
+    '{user}',
+    '{pid}',
+    '{borgversion}',
+    '{borgmajor}',
+    '{borgminor}',
+    '{borgpatch}',
+}
 
 
 def interpolate_context(hook_description, command, context):
@@ -22,10 +37,13 @@ def interpolate_context(hook_description, command, context):
     for name, value in context.items():
         command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
 
-    for unsupported_variable in re.findall(r'{\w+}', command):
-        logger.warning(
-            f"Variable '{unsupported_variable}' is not supported in {hook_description} hook"
-        )
+    for unsupported_variable in re.findall(r'\{\w+\}', command):
+        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
+        # be a Borg placeholder, as Borg should hopefully consume it.
+        if unsupported_variable not in BORG_PLACEHOLDERS:
+            logger.warning(
+                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook'
+            )
 
     return command
 
@@ -46,21 +64,25 @@ def make_environment(current_environment, sys_module=sys):
     return environment
 
 
-def filter_hooks(command_hooks, before=None, after=None, hook_name=None, action_names=None):
+def filter_hooks(command_hooks, before=None, after=None, action_names=None, state_names=None):
     '''
     Given a sequence of command hook dicts from configuration and one or more filters (before name,
-    after name, calling hook name, or a sequence of action names), filter down the command hooks to
-    just the ones that match the given filters.
+    after name, a sequence of action names, and/or a sequence of execution result state names),
+    filter down the command hooks to just the ones that match the given filters.
     '''
     return tuple(
         hook_config
         for hook_config in command_hooks or ()
         for config_action_names in (hook_config.get('when'),)
+        for config_state_names in (hook_config.get('states'),)
         if before is None or hook_config.get('before') == before
         if after is None or hook_config.get('after') == after
         if action_names is None
         or config_action_names is None
         or set(config_action_names or ()).intersection(set(action_names))
+        if state_names is None
+        or config_state_names is None
+        or set(config_state_names or ()).intersection(set(state_names))
     )
 
 
@@ -81,11 +103,14 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
 
     for hook_config in command_hooks:
         commands = hook_config.get('run')
+        when_description = (
+            f"{'/'.join(hook_config.get('when'))} " if hook_config.get('when') else ''
+        )
 
         if 'before' in hook_config:
-            description = f'before {hook_config.get("before")}'
+            description = f'before {when_description}{hook_config.get("before")}'
         elif 'after' in hook_config:
-            description = f'after {hook_config.get("after")}'
+            description = f'after {when_description}{hook_config.get("after")}'
         else:
             raise ValueError(f'Invalid hook configuration: {hook_config}')
 
@@ -119,7 +144,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                     output_log_level=(
                         logging.ERROR if hook_config.get('after') == 'error' else logging.ANSWER
                     ),
-                    shell=True,
+                    shell=True,  # noqa: S604
                     environment=make_environment(os.environ),
                     working_directory=working_directory,
                 )
@@ -139,7 +164,7 @@ class Before_after_hooks:
            before_after='do_stuff',
            umask=config.get('umask'),
            dry_run=dry_run,
-           hook_name='myhook',
+           action_names=['create'],
        ):
             do()
             some()
@@ -156,22 +181,20 @@ class Before_after_hooks:
         umask,
         working_directory,
         dry_run,
-        hook_name=None,
         action_names=None,
         **context,
     ):
         '''
         Given a sequence of command hook configuration dicts, the before/after name, a umask to run
-        commands with, a working directory to run commands with, a dry run flag, the name of the
-        calling hook, a sequence of action names, and any context for the executed commands, save
-        those data points for use below.
+        commands with, a working directory to run commands with, a dry run flag, a sequence of
+        action names, and any context for the executed commands, save those data points for use
+        below.
         '''
         self.command_hooks = command_hooks
         self.before_after = before_after
         self.umask = umask
         self.working_directory = working_directory
         self.dry_run = dry_run
-        self.hook_name = hook_name
         self.action_names = action_names
         self.context = context
 
@@ -184,7 +207,6 @@ class Before_after_hooks:
                 borgmatic.hooks.command.filter_hooks(
                     self.command_hooks,
                     before=self.before_after,
-                    hook_name=self.hook_name,
                     action_names=self.action_names,
                 ),
                 self.umask,
@@ -194,11 +216,11 @@ class Before_after_hooks:
             )
         except (OSError, subprocess.CalledProcessError) as error:
             if considered_soft_failure(error):
-                return
+                raise
 
             # Trigger the after hook manually, since raising here will prevent it from being run
             # otherwise.
-            self.__exit__(None, None, None)
+            self.__exit__(exception_type=type(error), exception=error, traceback=None)
 
             raise ValueError(f'Error running before {self.before_after} hook: {error}')
 
@@ -211,8 +233,8 @@ class Before_after_hooks:
                 borgmatic.hooks.command.filter_hooks(
                     self.command_hooks,
                     after=self.before_after,
-                    hook_name=self.hook_name,
                     action_names=self.action_names,
+                    state_names=['fail' if exception_type else 'finish'],
                 ),
                 self.umask,
                 self.working_directory,
@@ -221,16 +243,20 @@ class Before_after_hooks:
             )
         except (OSError, subprocess.CalledProcessError) as error:
             if considered_soft_failure(error):
-                return
+                raise
 
             raise ValueError(f'Error running after {self.before_after} hook: {error}')
 
 
+@functools.cache
 def considered_soft_failure(error):
     '''
     Given a configuration filename and an exception object, return whether the exception object
     represents a subprocess.CalledProcessError with a return code of SOFT_FAIL_EXIT_CODE. If so,
     that indicates that the error is a "soft failure", and should not result in an error.
+
+    The results of this function are cached so that it can be called multiple times without logging
+    multiple times.
     '''
     exit_code = getattr(error, 'returncode', None)
     if exit_code is None:

borgmatic/hooks/data_source/btrfs.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import itertools
 import json
 import logging
 import os
@@ -21,9 +22,44 @@ def use_streaming(hook_config, config):  # pragma: no cover
     return False
 
 
-def get_subvolume_mount_points(findmnt_command):
+def get_contained_subvolume_paths(btrfs_command, subvolume_path):
     '''
-    Given a findmnt command to run, get all sorted Btrfs subvolume mount points.
+    Given the path of a Btrfs subvolume, return it in a sequence along with the paths of its
+    contained subvolumes.
+
+    If the btrfs command errors, log that error and return an empty sequence.
+    '''
+    try:
+        btrfs_output = borgmatic.execute.execute_command_and_capture_output(
+            tuple(btrfs_command.split(' '))
+            + (
+                'subvolume',
+                'list',
+                subvolume_path,
+            ),
+            close_fds=True,
+        )
+    except subprocess.CalledProcessError as error:
+        logger.debug(
+            f'Ignoring Btrfs subvolume {subvolume_path} because of error listing its subvolumes: {error}'
+        )
+
+        return ()
+
+    return (subvolume_path,) + tuple(
+        os.path.join(subvolume_path, line.split(' ')[-1])
+        for line in btrfs_output.splitlines()
+        if line.strip()
+    )
+
+
+FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION = 'subvolid=5'
+
+
+def get_all_subvolume_paths(btrfs_command, findmnt_command):
+    '''
+    Given btrfs and findmnt commands to run, get the sorted paths for all Btrfs subvolumes on the
+    system.
     '''
     findmnt_output = borgmatic.execute.execute_command_and_capture_output(
         tuple(findmnt_command.split(' '))
@@ -32,12 +68,28 @@ def get_subvolume_mount_points(findmnt_command):
             'btrfs',
             '--json',
             '--list',  # Request a flat list instead of a nested subvolume hierarchy.
-        )
+        ),
+        close_fds=True,
     )
 
     try:
         return tuple(
-            sorted(filesystem['target'] for filesystem in json.loads(findmnt_output)['filesystems'])
+            sorted(
+                itertools.chain.from_iterable(
+                    # If findmnt gave us a Btrfs root filesystem, list the subvolumes within it.
+                    # This is necessary because findmnt only returns a subvolume's mount point
+                    # rather than its original subvolume path (which can differ). For instance,
+                    # a subvolume might exist at /mnt/subvolume but be mounted at /home/myuser.
+                    # findmnt is still useful though because it's a global way to discover all
+                    # Btrfs subvolumes—even if we have to do some additional legwork ourselves.
+                    (
+                        get_contained_subvolume_paths(btrfs_command, filesystem['target'])
+                        if FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION in filesystem['options'].split(',')
+                        else (filesystem['target'],)
+                    )
+                    for filesystem in json.loads(findmnt_output)['filesystems']
+                )
+            )
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
@@ -49,6 +101,12 @@ Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'),
 
 
 def get_subvolume_property(btrfs_command, subvolume_path, property_name):
+    '''
+    Given a btrfs command, a subvolume path, and a property name to lookup, return the value of the
+    corresponding property.
+
+    Raise subprocess.CalledProcessError if the btrfs command errors.
+    '''
     output = borgmatic.execute.execute_command_and_capture_output(
         tuple(btrfs_command.split(' '))
         + (
@@ -59,6 +117,7 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
             subvolume_path,
             property_name,
         ),
+        close_fds=True,
     )
 
     try:
@@ -72,19 +131,24 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
     }.get(value, value)
 
 
-def omit_read_only_subvolume_mount_points(btrfs_command, subvolume_paths):
+def omit_read_only_subvolume_paths(btrfs_command, subvolume_paths):
     '''
-    Given a Btrfs command to run and a sequence of Btrfs subvolume mount points, filter them down to
-    just those that are read-write. The idea is that Btrfs can't actually snapshot a read-only
-    subvolume, so we should just ignore them.
+    Given a Btrfs command to run and a sequence of Btrfs subvolume paths, filter them down to just
+    those that are read-write. The idea is that Btrfs can't actually snapshot a read-only subvolume,
+    so we should just ignore them.
     '''
     retained_subvolume_paths = []
 
     for subvolume_path in subvolume_paths:
-        if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
-            logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
-        else:
-            retained_subvolume_paths.append(subvolume_path)
+        try:
+            if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
+                logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
+            else:
+                retained_subvolume_paths.append(subvolume_path)
+        except subprocess.CalledProcessError as error:
+            logger.debug(
+                f'Error determining read-only status of Btrfs subvolume {subvolume_path}: {error}'
+            )
 
     return tuple(retained_subvolume_paths)
 
@@ -92,32 +156,32 @@ def omit_read_only_subvolume_mount_points(btrfs_command, subvolume_paths):
 def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     '''
     Given a Btrfs command to run and a sequence of configured patterns, find the intersection
-    between the current Btrfs filesystem and subvolume mount points and the paths of any patterns.
-    The idea is that these pattern paths represent the requested subvolumes to snapshot.
+    between the current Btrfs filesystem and subvolume paths and the paths of any patterns.  The
+    idea is that these pattern paths represent the requested subvolumes to snapshot.
 
     Only include subvolumes that contain at least one root pattern sourced from borgmatic
     configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
     return all subvolumes instead, sorted by path.
 
-    Return the result as a sequence of matching subvolume mount points.
+    Return the result as a sequence of matching Subvolume instances.
     '''
     candidate_patterns = set(patterns or ())
     subvolumes = []
 
-    # For each subvolume mount point, match it against the given patterns to find the subvolumes to
+    # For each subvolume path, match it against the given patterns to find the subvolumes to
     # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
     # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
     # this process, so no two subvolumes end up with the same contained patterns.)
-    for mount_point in reversed(
-        omit_read_only_subvolume_mount_points(
-            btrfs_command, get_subvolume_mount_points(findmnt_command)
+    for subvolume_path in reversed(
+        omit_read_only_subvolume_paths(
+            btrfs_command, get_all_subvolume_paths(btrfs_command, findmnt_command)
         )
     ):
         subvolumes.extend(
-            Subvolume(mount_point, contained_patterns)
+            Subvolume(subvolume_path, contained_patterns)
             for contained_patterns in (
                 borgmatic.hooks.data_source.snapshot.get_contained_patterns(
-                    mount_point, candidate_patterns
+                    subvolume_path, candidate_patterns
                 ),
             )
             if patterns is None
@@ -227,6 +291,7 @@ def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma:
             snapshot_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -241,9 +306,9 @@ def dump_data_sources(
     '''
     Given a Btrfs configuration dict, a configuration dict, the borgmatic configuration file paths,
     the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
-    auto-detect and snapshot any Btrfs subvolume mount points listed in the given patterns. Also
-    update those patterns, replacing subvolume mount points with corresponding snapshot directories
-    so they get stored in the Borg archive instead.
+    auto-detect and snapshot any Btrfs subvolume paths listed in the given patterns. Also update
+    those patterns, replacing subvolume paths with corresponding snapshot directories so they get
+    stored in the Borg archive instead.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
 
@@ -298,6 +363,7 @@ def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
             snapshot_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -324,8 +390,8 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
         logger.debug(error)
         return
 
-    # Reversing the sorted subvolumes ensures that we remove longer mount point paths of child
-    # subvolumes before the shorter mount point paths of parent subvolumes.
+    # Reversing the sorted subvolumes ensures that we remove longer paths of child subvolumes before
+    # the shorter paths of parent subvolumes.
     for subvolume in reversed(all_subvolumes):
         subvolume_snapshots_glob = borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
             os.path.normpath(make_snapshot_path(subvolume.path)),

borgmatic/hooks/data_source/lvm.py

@@ -50,7 +50,8 @@ def get_logical_volumes(lsblk_command, patterns=None):
                     'name,path,mountpoint,type',
                     '--json',
                     '--list',
-                )
+                ),
+                close_fds=True,
             )
         )
     except json.JSONDecodeError as error:
@@ -109,6 +110,7 @@ def snapshot_logical_volume(
             logical_volume_device,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -129,6 +131,7 @@ def mount_snapshot(mount_command, snapshot_device, snapshot_mount_path):  # prag
             snapshot_mount_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -227,12 +230,13 @@ def dump_data_sources(
             )
 
         # Get the device path for the snapshot we just created.
-        try:
-            snapshot = get_snapshots(
-                hook_config.get('lvs_command', 'lvs'), snapshot_name=snapshot_name
-            )[0]
-        except IndexError:
-            raise ValueError(f'Cannot find LVM snapshot {snapshot_name}')
+        if not dry_run:
+            try:
+                snapshot = get_snapshots(
+                    hook_config.get('lvs_command', 'lvs'), snapshot_name=snapshot_name
+                )[0]
+            except IndexError:
+                raise ValueError(f'Cannot find LVM snapshot {snapshot_name}')
 
         # Mount the snapshot into a particular named temporary directory so that the snapshot ends
         # up in the Borg archive at the "original" logical volume mount point path.
@@ -277,6 +281,7 @@ def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
     borgmatic.execute.execute_command(
         tuple(umount_command.split(' ')) + (snapshot_mount_path,),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -291,6 +296,7 @@ def remove_snapshot(lvremove_command, snapshot_device_path):  # pragma: no cover
             snapshot_device_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -318,7 +324,8 @@ def get_snapshots(lvs_command, snapshot_name=None):
                     'lv_name,lv_path',
                     '--select',
                     'lv_attr =~ ^s',  # Filter to just snapshots.
-                )
+                ),
+                close_fds=True,
             )
         )
     except json.JSONDecodeError as error:
@@ -407,7 +414,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
                 continue
 
         if not dry_run:
-            shutil.rmtree(snapshots_directory)
+            shutil.rmtree(snapshots_directory, ignore_errors=True)
 
     # Delete snapshots.
     lvremove_command = hook_config.get('lvremove_command', 'lvremove')

borgmatic/hooks/data_source/mariadb.py

@@ -120,13 +120,19 @@ def database_names_to_dump(database, config, username, password, environment, dr
         shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mariadb_show_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -184,14 +190,20 @@ def execute_dump_command(
         for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mariadb_dump_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -255,7 +267,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -383,18 +402,27 @@ def restore_data_source_dump(
         shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mariadb_restore_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/mongodb.py

@@ -74,11 +74,12 @@ def dump_data_sources(
 
         if dump_format == 'directory':
             dump.create_parent_directory_for_dump(dump_filename)
-            execute_command(command, shell=True)
+            execute_command(command, shell=True)  # noqa: S604
         else:
             dump.create_named_pipe_for_dump(dump_filename)
-            processes.append(execute_command(command, shell=True, run_to_completion=False))
-
+            processes.append(
+                execute_command(command, shell=True, run_to_completion=False)  # noqa: S604
+            )
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(

borgmatic/hooks/data_source/mysql.py

@@ -45,15 +45,21 @@ def database_names_to_dump(database, config, username, password, environment, dr
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('list_options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mysql_show_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -109,16 +115,22 @@ def execute_dump_command(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mysql_dump_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -182,7 +194,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -312,20 +331,29 @@ def restore_data_source_dump(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
     )
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mysql_restore_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/postgresql.py

@@ -224,7 +224,7 @@ def dump_data_sources(
                 dump.create_parent_directory_for_dump(dump_filename)
                 execute_command(
                     command,
-                    shell=True,
+                    shell=True,  # noqa: S604
                     environment=environment,
                 )
             else:
@@ -232,7 +232,7 @@ def dump_data_sources(
                 processes.append(
                     execute_command(
                         command,
-                        shell=True,
+                        shell=True,  # noqa: S604
                         environment=environment,
                         run_to_completion=False,
                     )

borgmatic/hooks/data_source/snapshot.py

@@ -8,9 +8,9 @@ def get_contained_patterns(parent_directory, candidate_patterns):
     '''
     Given a parent directory and a set of candidate patterns potentially inside it, get the subset
     of contained patterns for which the parent directory is actually the parent, a grandparent, the
-    very same directory, etc. The idea is if, say, /var/log and /var/lib are candidate pattern
-    paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at /var, then
-    /var is what we want to snapshot.
+    very same directory, etc. The idea is if, say, "/var/log" and "/var/lib" are candidate pattern
+    paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at "/var", then
+    "/var" is what we want to snapshot.
 
     If a parent directory and a candidate pattern are on different devices, skip the pattern. That's
     because any snapshot of a parent directory won't actually include "contained" directories if
@@ -18,8 +18,8 @@ def get_contained_patterns(parent_directory, candidate_patterns):
 
     For this function to work, a candidate pattern path can't have any globs or other non-literal
     characters in the initial portion of the path that matches the parent directory. For instance, a
-    parent directory of /var would match a candidate pattern path of /var/log/*/data, but not a
-    pattern path like /v*/log/*/data.
+    parent directory of "/var" would match a candidate pattern path of "/var/log/*/data", but not a
+    pattern path like "/v*/log/*/data".
 
     The one exception is that if a regular expression pattern path starts with "^", that will get
     stripped off for purposes of matching against a parent directory.

borgmatic/hooks/data_source/sqlite.py

@@ -88,7 +88,9 @@ def dump_data_sources(
             continue
 
         dump.create_named_pipe_for_dump(dump_filename)
-        processes.append(execute_command(command, shell=True, run_to_completion=False))
+        processes.append(
+            execute_command(command, shell=True, run_to_completion=False)  # noqa: S604
+        )
 
     if not dry_run:
         patterns.append(

borgmatic/hooks/data_source/zfs.py

@@ -53,7 +53,8 @@ def get_datasets_to_backup(zfs_command, patterns):
             'filesystem',
             '-o',
             f'name,mountpoint,canmount,{BORGMATIC_USER_PROPERTY}',
-        )
+        ),
+        close_fds=True,
     )
 
     try:
@@ -131,7 +132,8 @@ def get_all_dataset_mount_points(zfs_command):
             'filesystem',
             '-o',
             'mountpoint',
-        )
+        ),
+        close_fds=True,
     )
 
     return tuple(
@@ -158,6 +160,7 @@ def snapshot_dataset(zfs_command, full_snapshot_name):  # pragma: no cover
             full_snapshot_name,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -180,6 +183,7 @@ def mount_snapshot(mount_command, full_snapshot_name, snapshot_mount_path):  # p
             snapshot_mount_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -310,6 +314,7 @@ def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
     borgmatic.execute.execute_command(
         tuple(umount_command.split(' ')) + (snapshot_mount_path,),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -325,6 +330,7 @@ def destroy_snapshot(zfs_command, full_snapshot_name):  # pragma: no cover
             full_snapshot_name,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -342,7 +348,8 @@ def get_all_snapshots(zfs_command):
             'snapshot',
             '-o',
             'name',
-        )
+        ),
+        close_fds=True,
     )
 
     return tuple(line.rstrip() for line in list_output.splitlines())

borgmatic/hooks/monitoring/cronhub.py

@@ -11,6 +11,7 @@ MONITOR_STATE_TO_CRONHUB = {
     monitor.State.FINISH: 'finish',
     monitor.State.FAIL: 'fail',
 }
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -45,7 +46,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
-            response = requests.get(ping_url)
+            response = requests.get(ping_url, timeout=TIMEOUT_SECONDS)
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/cronitor.py

@@ -11,6 +11,7 @@ MONITOR_STATE_TO_CRONITOR = {
     monitor.State.FINISH: 'complete',
     monitor.State.FAIL: 'fail',
 }
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -40,7 +41,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
-            response = requests.get(ping_url)
+            response = requests.get(ping_url, timeout=TIMEOUT_SECONDS)
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/healthchecks.py

@@ -17,6 +17,7 @@ MONITOR_STATE_TO_HEALTHCHECKS = {
 
 DEFAULT_PING_BODY_LIMIT_BYTES = 100000
 HANDLER_IDENTIFIER = 'healthchecks'
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
@@ -84,7 +85,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
             response = requests.post(
-                ping_url, data=payload.encode('utf-8'), verify=hook_config.get('verify_tls', True)
+                ping_url,
+                data=payload.encode('utf-8'),
+                verify=hook_config.get('verify_tls', True),
+                timeout=TIMEOUT_SECONDS,
             )
             if not response.ok:
                 response.raise_for_status()

borgmatic/hooks/monitoring/loki.py

@@ -15,6 +15,7 @@ MONITOR_STATE_TO_LOKI = {
     monitor.State.FINISH: 'Finished',
     monitor.State.FAIL: 'Failed',
 }
+TIMEOUT_SECONDS = 10
 
 # Threshold at which logs get flushed to loki
 MAX_BUFFER_LINES = 100
@@ -69,7 +70,9 @@ class Loki_log_buffer:
         request_header = {'Content-Type': 'application/json'}
 
         try:
-            result = requests.post(self.url, headers=request_header, data=request_body, timeout=5)
+            result = requests.post(
+                self.url, headers=request_header, data=request_body, timeout=TIMEOUT_SECONDS
+            )
             result.raise_for_status()
         except requests.RequestException:
             logger.warning('Failed to upload logs to loki')

borgmatic/hooks/monitoring/ntfy.py

@@ -7,6 +7,9 @@ import borgmatic.hooks.credential.parse
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -82,7 +85,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not dry_run:
             logging.getLogger('urllib3').setLevel(logging.ERROR)
             try:
-                response = requests.post(f'{base_url}/{topic}', headers=headers, auth=auth)
+                response = requests.post(
+                    f'{base_url}/{topic}', headers=headers, auth=auth, timeout=TIMEOUT_SECONDS
+                )
                 if not response.ok:
                     response.raise_for_status()
             except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/pagerduty.py

@@ -14,6 +14,7 @@ logger = logging.getLogger(__name__)
 EVENTS_API_URL = 'https://events.pagerduty.com/v2/enqueue'
 DEFAULT_LOGS_PAYLOAD_LIMIT_BYTES = 10000
 HANDLER_IDENTIFIER = 'pagerduty'
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
@@ -94,7 +95,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
-        response = requests.post(EVENTS_API_URL, data=payload.encode('utf-8'))
+        response = requests.post(
+            EVENTS_API_URL, data=payload.encode('utf-8'), timeout=TIMEOUT_SECONDS
+        )
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/pushover.py

@@ -8,6 +8,7 @@ logger = logging.getLogger(__name__)
 
 
 EMERGENCY_PRIORITY = 2
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -79,6 +80,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                 'https://api.pushover.net/1/messages.json',
                 headers={'Content-type': 'application/x-www-form-urlencoded'},
                 data=data,
+                timeout=TIMEOUT_SECONDS,
             )
             if not response.ok:
                 response.raise_for_status()

borgmatic/hooks/monitoring/sentry.py

@@ -6,6 +6,9 @@ import requests
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -61,7 +64,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
-        response = requests.post(f'{cron_url}?status={status}')
+        response = requests.post(f'{cron_url}?status={status}', timeout=TIMEOUT_SECONDS)
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/uptime_kuma.py

@@ -5,6 +5,9 @@ import requests
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     push_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -37,7 +40,11 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     logging.getLogger('urllib3').setLevel(logging.ERROR)
 
     try:
-        response = requests.get(f'{push_url}?{query}', verify=hook_config.get('verify_tls', True))
+        response = requests.get(
+            f'{push_url}?{query}',
+            verify=hook_config.get('verify_tls', True),
+            timeout=TIMEOUT_SECONDS,
+        )
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/zabbix.py

@@ -7,6 +7,9 @@ import borgmatic.hooks.credential.parse
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -28,7 +31,7 @@ def send_zabbix_request(server, headers, data):
     logger.debug(f'Sending a "{data["method"]}" request to the Zabbix server')
 
     try:
-        response = requests.post(server, headers=headers, json=data)
+        response = requests.post(server, headers=headers, json=data, timeout=TIMEOUT_SECONDS)
 
         if not response.ok:
             response.raise_for_status()

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -55,6 +55,25 @@ commands:
           - "echo Backup: start"
 ```
 
+By default, an `after` command hook runs even if an error occurs in the
+corresponding `before` hook or between those two hooks. This allows you to
+perform cleanup steps that correspond to `before` preparation commands—even when
+something goes wrong. You may notice that this is a departure from the way that
+the deprecated `after_*` hooks worked in borgmatic prior to version 2.0.0.
+
+<span class="minilink minilink-addedin">New in version 2.0.3</span> You can
+customize this behavior with the `states` option. For instance, here's an
+example of an `after` hook that only triggers on success and not on error:
+
+```yaml
+commands:
+    - after: action
+      when: [create]
+      states: [finish]
+      run:
+          - echo "After successful create!"
+```
+
 Each command in the `commands:` list has the following options:
 
  * `before` or `after`: Name for the point in borgmatic's execution that the commands should be run before or after, one of:
@@ -64,17 +83,21 @@ Each command in the `commands:` list has the following options:
     * `everything` runs before or after all configuration files. Errors here do not trigger `error` hooks or the `fail` state in monitoring hooks. This replaces the deprecated `before_everything` and `after_everything`.
     * `error` runs after an error occurs—and it's only available for `after`. This replaces the deprecated `on_error` hook.
  * `when`: Only trigger the hook when borgmatic is run with particular actions (`create`, `prune`, etc.) listed here. Defaults to running for all actions.
+ * `states`: <span class="minilink minilink-addedin">New in version 2.0.3</span> Only trigger the hook if borgmatic encounters one of the states (execution results) listed here. This state is evaluated only for the scope of the configured `action`, `repository`, etc., rather than for the entire borgmatic run. Only available for `after` hooks. Defaults to running the hook for all states. One or more of:
+    * `finish`: No errors occurred.
+    * `fail`: An error occurred.
  * `run`: List of one or more shell commands or scripts to run when this command hook is triggered.
 
-An `after` command hook runs even if an error occurs in the corresponding
-`before` hook or between those two hooks. This allows you to perform cleanup
-steps that correspond to `before` preparation commands—even when something goes
-wrong. This is a departure from the way that the deprecated `after_*` hooks
-worked in borgmatic prior to version 2.0.0.
+When command hooks run, they respect the `working_directory` option if it is
+configured, meaning that the hook commands are run in that directory.
 
-Additionally, when command hooks run, they respect the `working_directory`
-option if it is configured, meaning that the hook commands are run in that
-directory.
+<span class="minilink minilink-addedin">New in version 2.0.4</span>If the exact
+same `everything` command hook is present in multiple configuration files,
+borgmatic only runs it once.
+
+borgmatic's `--repository` flag does not impact which command hooks get run. But
+you can use the `--config` flag to limit the configuration files (and thus
+command hooks) used.
 
 
 ### Order of execution
@@ -97,11 +120,24 @@ borgmatic for the `create` and `prune` actions. Here's the order of execution:
             * Run `after: action` hooks for `prune`.
         * Run `after: repository` hooks (for the first repository).
     * Run `after: configuration` hooks (from the first configuration file).
+    * Run `after: error` hooks (if an error occurs).
  * Run `after: everything` hooks (from all configuration files).
 
 This same order of execution extends to multiple repositories and/or
 configuration files.
 
+Based on the above, you can see the difference between, say, an `after: action`
+hook with `states: [fail]` and an `after: error` hook. The `after: action hook`
+runs immediately after the create action fails for a particular repository—so
+before any subsequent actions for that repository or other repositories even
+have a chance to run. Whereas the `after: error` hook doesn't run until all
+actions for—and repositories in—a configuration file have had a chance to
+execute.
+
+And if there are multiple hooks defined for a particular step (e.g. `before:
+action` for `create`), then those hooks are run in the order they're defined in
+configuration.
+
 
 ### Deprecated command hooks
 

docs/how-to/backup-your-databases.md

@@ -191,7 +191,7 @@ mysql_databases:
       format: sql
 ```
 
-### Containers
+### Database containers
 
 If your database server is running within a container and borgmatic is too, no
 problem—configure borgmatic to connect to the container's name on its exposed
@@ -209,6 +209,9 @@ postgresql_databases:
 <span class="minilink minilink-addedin">Prior to version 1.8.0</span> Put
 these options in the `hooks:` section of your configuration.
 
+
+#### Database client on the host
+
 But what if borgmatic is running on the host? You can still connect to a
 database server container if its ports are properly exposed to the host. For
 instance, when running the database container, you can specify `--publish
@@ -228,17 +231,19 @@ And then you can configure borgmatic running on the host to connect to the
 database:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: 127.0.0.1
-          port: 5433
-          username: postgres
-          password: trustsome1
+postgresql_databases:
+    - name: users
+      hostname: 127.0.0.1
+      port: 5433
+      username: postgres
+      password: trustsome1
 ```
 
 Alter the ports in these examples to suit your particular database system.
 
+
+#### Database client in a running container
+
 Normally, borgmatic dumps a database by running a database dump command (e.g.
 `pg_dump`) on the host or wherever borgmatic is running, and this command
 connects to your containerized database via the given `hostname` and `port`. But
@@ -250,14 +255,13 @@ to run the dump command.
 For instance, if using Docker and PostgreSQL, something like this might work:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: 127.0.0.1
-          port: 5433
-          username: postgres
-          password: trustsome1
-          pg_dump_command: docker exec my_pg_container pg_dump
+postgresql_databases:
+    - name: users
+      hostname: 127.0.0.1
+      port: 5433
+      username: postgres
+      password: trustsome1
+      pg_dump_command: docker exec my_pg_container pg_dump
 ```
 
 ... where `my_pg_container` is the name of your running database container.
@@ -282,19 +286,43 @@ services:
       - /run/user/1000:/run/user/1000
 ```
 
+And here's an example of using a MariaDB database client within a running Docker
+container:
+
+```yaml
+mariadb_databases:
+    - name: users
+      hostname: 127.0.0.1
+      username: example
+      password: trustsome1
+      password_transport: environment
+      mariadb_dump_command: docker exec --env MYSQL_PWD my_mariadb_container mariadb-dump
+```
+
+The `password_transport: environment` option tells borgmatic to transmit the
+password via environment variable instead of the default behavior of using an
+anonymous pipe. The environment variable transport is potentially less secure
+than the pipe, but it's necessary when the database client is running in a
+container separate from borgmatic.
+
+A similar approach can work with MySQL, using `mysql_dump_command` instead of
+`mariadb_dump_command` to run `mysqldump` in a container.
+
+
+#### Database client in a temporary container
+
 Another variation: If you're running borgmatic on the host but want to spin up a
 temporary `pg_dump` container whenever borgmatic dumps a database, for
 instance to make use of a `pg_dump` version not present on the host, try
 something like this:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: your-database-hostname
-          username: postgres
-          password: trustsome1
-          pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
+postgresql_databases:
+    - name: users
+      hostname: your-database-hostname
+      username: postgres
+      password: trustsome1
+      pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
 ```
 
 The `--env PGPASSWORD` is necessary here for borgmatic to provide your database

docs/how-to/customize-warnings-and-errors.md

@@ -12,52 +12,18 @@ numeric ID indicating success, warning, or error. borgmatic consumes this exit
 code to decide how to respond. Normally, a Borg error results in a borgmatic
 error, while a Borg warning or success doesn't.
 
-But if that default behavior isn't sufficient for your needs, you can
-customize how borgmatic interprets [Borg's exit
+<span class="minilink minilink-addedin">With Borg version 1.4+</span> If the
+default behavior isn't sufficient for your needs, you can customize how
+borgmatic interprets [Borg's exit
 codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
-For instance, to elevate Borg warnings to errors, thereby causing borgmatic to
-error on them, use the following borgmatic configuration:
-
-```yaml
-borg_exit_codes:
-   - exit_code: 1
-     treat_as: error
-```
-
-Be aware though that Borg exits with a warning code for a variety of benign
-situations such as files changing while they're being read, so this example
-may not meet your needs. Keep reading though for more granular exit code
-configuration.
-
-Here's an example that squashes Borg errors to warnings:
-
-```yaml
-borg_exit_codes:
-   - exit_code: 2
-     treat_as: warning
-```
-
-Be careful with this example though, because it prevents borgmatic from
-erroring when Borg errors, which may not be desirable.
-
-
-### More granular configuration
-
-<span class="minilink minilink-addedin">New in Borg version 1.4</span> Borg
-support for [more granular exit
-codes](https://borgbackup.readthedocs.io/en/1.4-maint/usage/general.html#return-codes)
-means that you can configure borgmatic to respond to specific Borg conditions.
-See the full list of [Borg 1.4 error and warning exit
-codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
-The `rc:` numeric value there tells you the exit code for each.
 
 For instance, this borgmatic configuration elevates all Borg backup file
 permission warnings (exit code `105`)—and only those warnings—to errors:
 
 ```yaml
 borg_exit_codes:
-   - exit_code: 105
-     treat_as: error
+    - code: 105
+      treat_as: error
 ```
 
 The following configuration does that *and* elevates backup file not found
@@ -65,16 +31,20 @@ warnings (exit code `107`) to errors as well:
 
 ```yaml
 borg_exit_codes:
-   - exit_code: 105
-     treat_as: error
-   - exit_code: 107
-     treat_as: error
+    - code: 105
+      treat_as: error
+    - code: 107
+      treat_as: error
 ```
 
+See the full list of [Borg 1.4 error and warning exit
+codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
+The `rc:` numeric value there tells you the exit code for each.
+
 If you don't know the exit code for a particular Borg error or warning you're
-experiencing, you can usually find it in your borgmatic output when
-`--verbosity 2` is enabled. For instance, here's a snippet of that output when
-a backup file is not found:
+experiencing, you can usually find it in your borgmatic output when `--verbosity
+2` is enabled. For instance, here's a snippet of that output when a backup file
+is not found:
 
 ```
 /noexist: stat: [Errno 2] No such file or directory: '/noexist'
@@ -84,3 +54,30 @@ terminating with warning status, rc 107
 
 So if you want to configure borgmatic to treat this as an error instead of a
 warning, the exit status to use is `107`.
+
+<span class="minilink minilink-addedin">With Borg version 1.2 and earlier</span>
+Older versions of Borg didn't support granular exit codes, but still
+distinguished between Borg errors and warnings. For instance, to elevate Borg
+warnings to errors, thereby causing borgmatic to error on them, use the
+following borgmatic configuration with Borg 1.2 or below:
+
+```yaml
+borg_exit_codes:
+    - code: 1
+      treat_as: error
+```
+
+Be aware though that Borg exits with a warning code for a variety of benign
+situations such as files changing while they're being read, so this example
+may not meet your needs.
+
+Here's another Borg 1.2 example that squashes Borg errors to warnings:
+
+```yaml
+borg_exit_codes:
+    - code: 2
+      treat_as: warning
+```
+
+Be careful with this example though, because it prevents borgmatic from
+erroring when Borg errors, which may not be desirable.

docs/how-to/make-per-application-backups.md

@@ -141,7 +141,7 @@ for more information.
 
 <span class="minilink minilink-addedin">With Borg version 2.x</span>See the
 [match archives
-documentation](https://borgbackup.readthedocs.io/en/2.0.0b13/usage/help.html#borg-help-match-archives).
+documentation](https://borgbackup.readthedocs.io/en/2.0.0b16/usage/help.html#borg-help-match-archives).
 
 Some borgmatic command-line actions also have a `--match-archives` flag that
 overrides both the auto-matching behavior and the `match_archives`

docs/how-to/snapshot-your-filesystems.md

@@ -116,6 +116,27 @@ you can use the standard
 extract them.
 
 
+#### ZFS performance
+
+<span class="minilink minilink-addedin">With Borg version 1.x</span> Because of
+the way that ZFS snapshot paths can change from one borgmatic invocation to the
+next, the [Borg file
+cache](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#cache)
+may not get cache hits on snapshotted files. This makes backing up ZFS snapshots
+a little slower than non-snapshotted files that have consistent paths. You can
+mitigate this by setting a fixed [runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)
+(that's not located in `/tmp`). This allows borgmatic to use a consistent
+snapshot path from one run to the next, thereby resulting in Borg files cache
+hits.
+
+<span class="minilink minilink-addedin">With Borg version 2.x</span> Snapshotted
+files should get cache hits regardless of whether their paths change, because
+Borg 2.x is smarter about how it looks up file paths in its cache—it constructs
+the cache key with the path *as it's seen in the archive* (which is consistent
+across runs) rather than the full absolute source path (which can change).
+
+
 ### Btrfs
 
 <span class="minilink minilink-addedin">New in version 1.9.4</span> <span
@@ -123,9 +144,8 @@ class="minilink minilink-addedin">Beta feature</span> borgmatic supports taking
 snapshots with the [Btrfs filesystem](https://btrfs.readthedocs.io/) and sending
 those snapshots to Borg for backup.
 
-To use this feature, first you need one or more Btrfs subvolumes on mounted
-filesystems. Then, enable Btrfs within borgmatic by adding the following line to
-your configuration file:
+To use this feature, first you need one or more Btrfs subvolumes. Then, enable
+Btrfs within borgmatic by adding the following line to your configuration file:
 
 ```yaml
 btrfs:
@@ -148,13 +168,18 @@ feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
 
 #### Subvolume discovery
 
-For any read-write subvolume you'd like backed up, add its mount point path to
+For any read-write subvolume you'd like backed up, add its subvolume path to
 borgmatic's `source_directories` option. Btrfs does not support snapshotting
 read-only subvolumes.
 
-<span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
-the mount point as a root pattern with borgmatic's `patterns` or `patterns_from`
-options.
+<span class="minilink minilink-addedin">New in version 2.0.7</span> The path can
+be either the path of the subvolume itself or the mount point where the
+subvolume is mounted. Prior to version 2.0.7, the subvolume path could not be
+used if the subvolume was mounted elsewhere; only the mount point could be used.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> Instead of
+using `source_directories`, you can include the subvolume path as a root pattern
+with borgmatic's `patterns` or `patterns_from` options.
 
 During a backup, borgmatic snapshots these subvolumes (non-recursively) and
 includes the snapshotted files in the paths sent to Borg. borgmatic is also
@@ -163,25 +188,24 @@ responsible for cleaning up (deleting) these snapshots after a backup completes.
 borgmatic is smart enough to look at the parent (and grandparent, etc.)
 directories of each of your `source_directories` to discover any subvolumes. For
 instance, let's say you add `/var/log` and `/var/lib` to your source
-directories, but `/var` is a subvolume mount point. borgmatic will discover that
-and snapshot `/var` accordingly. This also works even with nested subvolumes;
+directories, but `/var` is a subvolume path. borgmatic will discover that and
+snapshot `/var` accordingly. This also works even with nested subvolumes;
 borgmatic selects the subvolume that's the "closest" parent to your source
 directories.
 
 <span class="minilink minilink-addedin">New in version 1.9.6</span> When using
 [patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns),
 the initial portion of a pattern's path that you intend borgmatic to match
-against a subvolume mount point can't have globs or other non-literal characters
-in it—or it won't actually match. For instance, a subvolume mount point of
-`/var` would match a pattern of `+ fm:/var/*/data`, but borgmatic isn't
-currently smart enough to match `/var` to a pattern like `+ fm:/v*/lib/data`.
+against a subvolume path can't have globs or other non-literal characters in
+it—or it won't actually match. For instance, a subvolume path of `/var` would
+match a pattern of `+ fm:/var/*/data`, but borgmatic isn't currently smart
+enough to match `/var` to a pattern like `+ fm:/v*/lib/data`.
 
 Additionally, borgmatic rewrites the snapshot file paths so that they appear at
 their original subvolume locations in a Borg archive. For instance, if your
-subvolume is mounted at `/var/subvolume`, then the snapshotted files will appear
-in an archive at `/var/subvolume` as well—even if borgmatic has to mount the
-snapshot somewhere in `/var/subvolume/.borgmatic-snapshot-1234/` to perform the
-backup.
+subvolume path is `/var/subvolume`, then the snapshotted files will appear in an
+archive at `/var/subvolume` as well—even if borgmatic has to mount the snapshot
+somewhere in `/var/subvolume/.borgmatic-snapshot-1234/` to perform the backup.
 
 <span class="minilink minilink-addedin">With Borg version 1.2 and
 earlier</span>Snapshotted files are instead stored at a path dependent on the
@@ -197,6 +221,22 @@ action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) to extract
 them.
 
 
+#### Btrfs performance
+
+<span class="minilink minilink-addedin">With Borg version 1.x</span> Because of
+the way that Btrfs snapshot paths change from one borgmatic invocation to the
+next, the [Borg file
+cache](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#cache)
+will never get cache hits on snapshotted files. This makes backing up Btrfs
+snapshots a little slower than non-snapshotted files that have consistent paths.
+
+<span class="minilink minilink-addedin">With Borg version 2.x</span> Even
+snapshotted files should get cache hits, because Borg 2.x is smarter about how
+it looks up file paths in its cache—it constructs the cache key with the path
+*as it's seen in the archive* (which is consistent across runs) rather than the
+full absolute source path (which changes).
+
+
 ### LVM
 
 <span class="minilink minilink-addedin">New in version 1.9.4</span> <span
@@ -311,3 +351,24 @@ Logical volume snapshots are stored in a Borg archive as normal files, so
 you can use the standard
 [extract action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) to
 extract them.
+
+
+#### LVM performance
+
+<span class="minilink minilink-addedin">With Borg version 1.x</span> Because of
+the way that LVM snapshot paths can change from one borgmatic invocation to the
+next, the [Borg file
+cache](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#cache)
+may not get cache hits on snapshotted files. This makes backing up LVM snapshots
+a little slower than non-snapshotted files that have consistent paths. You can
+mitigate this by setting a fixed [runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)
+(that's not located in `/tmp`). This allows borgmatic to use a consistent
+snapshot path from one run to the next, thereby resulting in Borg files cache
+hits.
+
+<span class="minilink minilink-addedin">With Borg version 2.x</span> Snapshotted
+files should get cache hits regardless of whether their paths change, because
+Borg 2.x is smarter about how it looks up file paths in its cache—it constructs
+the cache key with the path *as it's seen in the archive* (which is consistent
+across runs) rather than the full absolute source path (which can change).

docs/how-to/upgrade.md

@@ -193,7 +193,7 @@ confirmation of success—or tells you if something hasn't been transferred yet.
 Note that by omitting the `--upgrader` flag, you can also do archive transfers
 between related Borg 2 repositories without upgrading, even down to individual
 archives. For more on that functionality, see the [Borg transfer
-documentation](https://borgbackup.readthedocs.io/en/2.0.0b13/usage/transfer.html).
+documentation](https://borgbackup.readthedocs.io/en/2.0.0b16/usage/transfer.html).
 
 That's it! Now you can use your new Borg 2 repository as normal with
 borgmatic. If you've got multiple repositories, repeat the above process for

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "borgmatic"
-version = "2.0.0"
+version = "2.0.7"
 authors = [
   { name="Dan Helfman", email="witten@torsion.org" },
 ]

test_requirements.txt

@@ -1,32 +1,35 @@
 appdirs==1.4.4
-apprise==1.8.0
-attrs==23.2.0
-black==24.4.2
-certifi==2024.7.4
+apprise==1.9.3
+attrs==25.3.0
+bandit==1.8.5
+black==25.1.0
+certifi==2025.6.15
 chardet==5.2.0
-click==8.1.7
-codespell==2.2.6
-coverage==7.5.1
-flake8==7.0.0
+click==8.2.1
+codespell==2.4.1
+coverage==7.9.1
+flake8==7.3.0
+flake8-bandit==4.1.1
 flake8-quotes==3.4.0
 flake8-use-fstring==1.4
 flake8-variables-names==0.0.6
 flexmock==0.12.1
-idna==3.7
-isort==5.13.2
-jsonschema==4.22.0
-Markdown==3.6
+idna==3.10
+isort==6.0.1
+jsonschema==4.24.0
+Markdown==3.8.2
 mccabe==0.7.0
-packaging==24.0
+packaging==25.0
 pathspec==0.12.1
-pluggy==1.5.0
+pbr==6.1.1
+pluggy==1.6.0
 py==1.11.0
-pycodestyle==2.11.1
-pyflakes==3.2.0
-pytest==8.2.1
-pytest-cov==5.0.0
+pycodestyle==2.14.0
+pyflakes==3.4.0
+pytest==8.4.1
+pytest-cov==6.2.1
 PyYAML>5.0.0
 regex
-requests==2.32.2
+requests==2.32.4
 ruamel.yaml>0.15.0
 toml==0.10.2

tests/end-to-end/commands/fake_btrfs.py

@@ -53,7 +53,7 @@ def save_snapshots(snapshot_paths):
 
 
 def print_subvolume_list(arguments, snapshot_paths):
-    assert arguments.subvolume_path == '/mnt/subvolume'
+    assert arguments.subvolume_path == '/e2e/mnt/subvolume'
 
     if not arguments.snapshots_only:
         for line in BUILTIN_SUBVOLUME_LIST_LINES:

tests/end-to-end/commands/fake_findmnt.py

@@ -14,7 +14,7 @@ def parse_arguments(*unparsed_arguments):
 BUILTIN_FILESYSTEM_MOUNT_OUTPUT = '''{
        "filesystems": [
           {
-             "target": "/mnt/subvolume",
+             "target": "/e2e/mnt/subvolume",
              "source": "/dev/loop0",
              "fstype": "btrfs",
              "options": "rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/"

tests/end-to-end/commands/fake_lsblk.py

@@ -20,7 +20,7 @@ BUILTIN_BLOCK_DEVICES = {
         {
             'name': 'vgroup-lvolume',
             'path': '/dev/mapper/vgroup-lvolume',
-            'mountpoint': '/mnt/lvolume',
+            'mountpoint': '/e2e/mnt/lvolume',
             'type': 'lvm',
         },
         {

tests/end-to-end/commands/fake_zfs.py

@@ -23,20 +23,20 @@ def parse_arguments(*unparsed_arguments):
 
 BUILTIN_DATASETS = (
     {
-        'name': 'pool',
+        'name': 'e2e/pool',
         'used': '256K',
         'avail': '23.7M',
         'refer': '25K',
         'canmount': 'on',
-        'mountpoint': '/pool',
+        'mountpoint': '/e2e/pool',
     },
     {
-        'name': 'pool/dataset',
+        'name': 'e2e/pool/dataset',
         'used': '256K',
         'avail': '23.7M',
         'refer': '25K',
         'canmount': 'on',
-        'mountpoint': '/pool/dataset',
+        'mountpoint': '/e2e/pool/dataset',
     },
 )
 

tests/end-to-end/hooks/data_source/test_btrfs.py

@@ -16,10 +16,10 @@ def generate_configuration(config_path, repository_path):
         open(config_path)
         .read()
         .replace('ssh://user@backupserver/./sourcehostname.borg', repository_path)
-        .replace('- path: /mnt/backup', '')
+        .replace('- path: /e2e/mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /mnt/subvolume/subdir')
+        .replace('- /etc', '- /e2e/mnt/subvolume/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'btrfs:\n'
@@ -51,11 +51,11 @@ def test_btrfs_create_and_list():
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'mnt/subvolume/subdir/file.txt' in output
+        assert 'e2e/mnt/subvolume/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not subprocess.check_output(
-            'python3 /app/tests/end-to-end/commands/fake_btrfs.py subvolume list -s /mnt/subvolume'.split(
+            'python3 /app/tests/end-to-end/commands/fake_btrfs.py subvolume list -s /e2e/mnt/subvolume'.split(
                 ' '
             )
         )

tests/end-to-end/hooks/data_source/test_lvm.py

@@ -17,10 +17,10 @@ def generate_configuration(config_path, repository_path):
         open(config_path)
         .read()
         .replace('ssh://user@backupserver/./sourcehostname.borg', repository_path)
-        .replace('- path: /mnt/backup', '')
+        .replace('- path: /e2e/mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /mnt/lvolume/subdir')
+        .replace('- /etc', '- /e2e/mnt/lvolume/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'lvm:\n'
@@ -56,7 +56,7 @@ def test_lvm_create_and_list():
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'mnt/lvolume/subdir/file.txt' in output
+        assert 'e2e/mnt/lvolume/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not json.loads(

tests/end-to-end/hooks/data_source/test_zfs.py

@@ -19,7 +19,7 @@ def generate_configuration(config_path, repository_path):
         .replace('- path: /mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /pool/dataset/subdir')
+        .replace('- /etc', '- /e2e/pool/dataset/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'zfs:\n'
@@ -45,14 +45,14 @@ def test_zfs_create_and_list():
         )
 
         # Run a create action to exercise ZFS snapshotting and backup.
-        subprocess.check_call(f'borgmatic --config {config_path} create'.split(' '))
+        subprocess.check_call(f'borgmatic -v 2 --config {config_path} create'.split(' '))
 
         # List the resulting archive and assert that the snapshotted files are there.
         output = subprocess.check_output(
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'pool/dataset/subdir/file.txt' in output
+        assert 'e2e/pool/dataset/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not subprocess.check_output(

tests/end-to-end/test_environment_variable.py

@@ -0,0 +1,71 @@
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+
+def generate_configuration(config_path, repository_path):
+    '''
+    Generate borgmatic configuration into a file at the config path, and update the defaults so as
+    to work for testing, including updating the source directories, injecting the given repository
+    path, and tacking on an encryption passphrase environment variable (which will get passed in
+    when the Borg command is called below).
+    '''
+    subprocess.check_call(f'borgmatic config generate --destination {config_path}'.split(' '))
+    config = (
+        open(config_path)
+        .read()
+        .replace('ssh://user@backupserver/./sourcehostname.borg', repository_path)
+        .replace('- path: /mnt/backup', '')
+        .replace('label: local', '')
+        .replace('- /home/user/path with spaces', '')
+        .replace('- /home', f'- {config_path}')
+        .replace('- /etc', '')
+        .replace('- /var/log/syslog*', '')
+        + '\nencryption_passphrase: "${PASSPHRASE}"'
+        # Disable automatic storage of config files so we can test storage and extraction manually.
+        + '\nbootstrap:\n  store_config_files: false'
+    )
+    config_file = open(config_path, 'w')
+    config_file.write(config)
+    config_file.close()
+
+
+def test_borgmatic_command():
+    # Create a Borg repository.
+    temporary_directory = tempfile.mkdtemp()
+    repository_path = os.path.join(temporary_directory, 'test.borg')
+    extract_path = os.path.join(temporary_directory, 'extract')
+
+    original_working_directory = os.getcwd()
+    os.mkdir(extract_path)
+    os.chdir(extract_path)
+    environment = dict(os.environ, **{'PASSPHRASE': 'test'})
+
+    try:
+        config_path = os.path.join(temporary_directory, 'test.yaml')
+        generate_configuration(config_path, repository_path)
+
+        subprocess.check_call(
+            f'borgmatic -v 2 --config {config_path} repo-create --encryption repokey'.split(' '),
+            env=environment,
+        )
+
+        # Run borgmatic to generate a backup archive, and then list it to make sure it exists.
+        subprocess.check_call(
+            f'borgmatic --config {config_path}'.split(' '),
+            env=environment,
+        )
+        output = subprocess.check_output(
+            f'borgmatic --config {config_path} list --json'.split(' '),
+            env=environment,
+        ).decode(sys.stdout.encoding)
+        parsed_output = json.loads(output)
+
+        assert len(parsed_output) == 1
+        assert len(parsed_output[0]['archives']) == 1
+    finally:
+        os.chdir(original_working_directory)
+        shutil.rmtree(temporary_directory)

tests/integration/borg/test_rename.py

@@ -0,0 +1,120 @@
+import logging
+
+from borgmatic.borg import rename as module
+from tests.unit.test_verbosity import insert_logging_mock
+
+
+def test_make_rename_command_includes_log_info():
+    insert_logging_mock(logging.INFO)
+
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--info', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_log_debug():
+    insert_logging_mock(logging.DEBUG)
+
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--debug', '--show-rc', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_dry_run():
+    command = module.make_rename_command(
+        dry_run=True,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--dry-run', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_remote_path():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path='borg1',
+    )
+
+    assert command == (
+        'borg',
+        'rename',
+        '--remote-path',
+        'borg1',
+        'repo::old_archive',
+        'new_archive',
+    )
+
+
+def test_make_rename_command_includes_umask():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'umask': '077'},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--umask', '077', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_log_json():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'log_json': True},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--log-json', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_lock_wait():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'lock_wait': 5},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--lock-wait', '5', 'repo::old_archive', 'new_archive')

tests/integration/commands/test_borgmatic.py

@@ -2,6 +2,7 @@ import subprocess
 
 from flexmock import flexmock
 
+import borgmatic.hooks.command
 from borgmatic.commands import borgmatic as module
 
 
@@ -12,3 +13,81 @@ def test_borgmatic_version_matches_news_version():
     news_version = open('NEWS').readline()
 
     assert borgmatic_version == news_version
+
+
+def test_run_configuration_without_error_pings_monitoring_hooks_start_and_finish():
+    config = {'repositories': [{'path': 'foo'}]}
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
+    flexmock(module).should_receive('run_actions').and_return([])
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.START,
+        object,
+        object,
+    ).once()
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.FINISH,
+        object,
+        object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
+
+
+def test_run_configuration_with_borg_version_error_pings_after_command_hook_with_fail_state():
+    config = {
+        'repositories': [{'path': 'foo'}],
+        'commands': ({'after': 'configuration', 'run': ['echo after'], 'states': ['fail']},),
+    }
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_raise(ValueError)
+    flexmock(module).should_receive('run_actions').and_return([])
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(borgmatic.hooks.command).should_receive('execute_hooks')
+    flexmock(borgmatic.hooks.command).should_receive('execute_hooks').with_args(
+        config['commands'],
+        umask=object,
+        working_directory=object,
+        dry_run=False,
+        log_file=object,
+        configuration_filename=object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
+
+
+def test_run_configuration_with_action_error_pings_monioring_hooks_start_and_fail():
+    config = {'repositories': [{'path': 'foo'}]}
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
+    flexmock(module).should_receive('run_actions').and_raise(OSError)
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.START,
+        object,
+        object,
+    ).once()
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.FAIL,
+        object,
+        object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))

tests/integration/config/test_generate.py

@@ -126,6 +126,45 @@ def test_schema_to_sample_configuration_comments_out_non_source_config_options_i
     assert 'field3' not in config[0].ca.items
 
 
+def test_schema_to_sample_configuration_comments_out_non_source_config_options_in_sequence_of_maps_with_different_subschemas():
+    schema = {
+        'type': 'array',
+        'items': {
+            'type': 'object',
+            'oneOf': [
+                {
+                    'properties': dict(
+                        [
+                            ('field1', {'type': 'string', 'example': 'Example 1'}),
+                            ('field2', {'type': 'string', 'example': 'Example 2'}),
+                        ]
+                    )
+                },
+                {
+                    'properties': dict(
+                        [
+                            ('field2', {'type': 'string', 'example': 'Example 2'}),
+                            ('field3', {'type': 'string', 'example': 'Example 3'}),
+                        ]
+                    )
+                },
+            ],
+        },
+    }
+    source_config = [{'field1': 'value'}, {'field3': 'value'}]
+
+    config = module.schema_to_sample_configuration(schema, source_config)
+
+    assert config == [
+        dict([('field1', 'Example 1'), ('field2', 'Example 2'), ('field3', 'Example 3')])
+    ]
+
+    # The first field in a sequence does not get commented.
+    assert 'field1' not in config[0].ca.items
+    assert 'COMMENT_OUT' in config[0].ca.items['field2'][1][-1]._value
+    assert 'COMMENT_OUT' in config[0].ca.items['field3'][1][-1]._value
+
+
 def test_comment_out_line_skips_blank_line():
     line = '    \n'
 
@@ -169,8 +208,11 @@ foo:
         - quux
 
 repositories:
-    - one
-    - two
+    - path: foo
+      # COMMENT_OUT
+      label: bar
+    - path: baz
+      label: quux
 
 # This comment should be kept.
 # COMMENT_OUT
@@ -185,8 +227,10 @@ other: thing
 #         - quux
 
 repositories:
-    - one
-    - two
+    - path: foo
+#       label: bar
+    - path: baz
+      label: quux
 
 # This comment should be kept.
 # other: thing
@@ -297,14 +341,28 @@ def test_add_comments_to_configuration_comments_out_non_source_config_options():
     assert 'baz' not in config.ca.items
 
 
-def test_add_comments_to_configuration_object_with_skip_first_does_not_comment_out_first_option():
+def test_add_comments_to_configuration_object_with_skip_first_field_does_not_comment_out_first_option():
     config = module.ruamel.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
     schema = {
         'type': 'object',
         'properties': {'foo': {'description': 'Foo'}, 'bar': {'description': 'Bar'}},
     }
 
-    module.add_comments_to_configuration_object(config, schema, skip_first=True)
+    module.add_comments_to_configuration_object(config, schema, skip_first_field=True)
+
+    assert 'foo' not in config.ca.items
+    assert 'COMMENT_OUT' in config.ca.items['bar'][1][-1]._value
+    assert 'baz' not in config.ca.items
+
+
+def test_add_comments_to_configuration_object_with_skip_first_field_does_not_comment_out_first_option():
+    config = module.ruamel.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
+    schema = {
+        'type': 'object',
+        'properties': {'foo': {'description': 'Foo'}, 'bar': {'description': 'Bar'}},
+    }
+
+    module.add_comments_to_configuration_object(config, schema, skip_first_field=True)
 
     assert 'foo' not in config.ca.items
     assert 'COMMENT_OUT' in config.ca.items['bar'][1][-1]._value
@@ -326,15 +384,39 @@ def test_generate_sample_configuration_does_not_raise():
     module.generate_sample_configuration(False, None, 'dest.yaml', 'schema.yaml')
 
 
-def test_generate_sample_configuration_with_source_filename_does_not_raise():
+def test_generate_sample_configuration_with_source_filename_omits_empty_bootstrap_field():
     builtins = flexmock(sys.modules['builtins'])
     builtins.should_receive('open').with_args('schema.yaml').and_return('')
     flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
         flexmock(load=lambda filename: {})
     )
-    flexmock(module.load).should_receive('load_configuration')
+    flexmock(module.load).should_receive('load_configuration').and_return(
+        {'bootstrap': {}, 'foo': 'bar'}
+    )
     flexmock(module.normalize).should_receive('normalize')
-    flexmock(module).should_receive('schema_to_sample_configuration')
+    flexmock(module).should_receive('schema_to_sample_configuration').with_args(
+        object, {'foo': 'bar'}
+    ).once()
+    flexmock(module).should_receive('merge_source_configuration_into_destination')
+    flexmock(module).should_receive('render_configuration')
+    flexmock(module).should_receive('comment_out_optional_configuration')
+    flexmock(module).should_receive('write_configuration')
+
+    module.generate_sample_configuration(False, 'source.yaml', 'dest.yaml', 'schema.yaml')
+
+
+def test_generate_sample_configuration_with_source_filename_keeps_non_empty_bootstrap_field():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_return('')
+    flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
+        flexmock(load=lambda filename: {})
+    )
+    source_config = {'bootstrap': {'stuff': 'here'}, 'foo': 'bar'}
+    flexmock(module.load).should_receive('load_configuration').and_return(source_config)
+    flexmock(module.normalize).should_receive('normalize')
+    flexmock(module).should_receive('schema_to_sample_configuration').with_args(
+        object, source_config
+    ).once()
     flexmock(module).should_receive('merge_source_configuration_into_destination')
     flexmock(module).should_receive('render_configuration')
     flexmock(module).should_receive('comment_out_optional_configuration')

tests/integration/config/test_schema.py

@@ -20,21 +20,28 @@ ACTIONS_MODULE_NAMES_TO_OMIT = {
     'export_key',
     'import_key',
     'json',
+    'pattern',
 }
 ACTIONS_MODULE_NAMES_TO_ADD = {'key', 'umount'}
 
 
-def test_schema_skip_actions_correspond_to_supported_actions():
+def test_schema_actions_correspond_to_supported_actions():
     '''
-    Ensure that the allowed actions in the schema's "skip_actions" option don't drift from
-    borgmatic's actual supported actions.
+    Ensure that the allowed actions in the schema's various options don't drift from borgmatic's
+    actual supported actions.
     '''
     schema = borgmatic.config.load.load_configuration(borgmatic.config.validate.schema_filename())
-    schema_skip_actions = set(schema['properties']['skip_actions']['items']['enum'])
     supported_actions = {
         module.name.replace('_', '-')
         for module in pkgutil.iter_modules(borgmatic.actions.__path__)
         if module.name not in ACTIONS_MODULE_NAMES_TO_OMIT
     }.union(ACTIONS_MODULE_NAMES_TO_ADD)
+    properties = schema['properties']
+    commands_one_of = properties['commands']['items']['oneOf']
 
-    assert schema_skip_actions == supported_actions
+    for schema_actions in (
+        set(properties['skip_actions']['items']['enum']),
+        set(commands_one_of[0]['properties']['when']['items']['enum']),
+        set(commands_one_of[1]['properties']['when']['items']['enum']),
+    ):
+        assert schema_actions == supported_actions

tests/unit/actions/test_check.py

@@ -561,10 +561,10 @@ def test_collect_spot_check_source_paths_parses_borg_output():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('collect_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('process_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
         [Pattern('foo'), Pattern('bar')]
     )
     flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
@@ -601,6 +601,53 @@ def test_collect_spot_check_source_paths_parses_borg_output():
     ) == ('/etc/path', '/etc/other')
 
 
+def test_collect_spot_check_source_paths_omits_progress_from_create_dry_run_command():
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'hook1': False, 'hook2': False}
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
+        [Pattern('foo'), Pattern('bar')]
+    )
+    flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
+        dry_run=True,
+        repository_path='repo',
+        config={'working_directory': '/', 'list_details': True},
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version=object,
+        global_arguments=object,
+        borgmatic_runtime_directory='/run/borgmatic',
+        local_path=object,
+        remote_path=object,
+        stream_processes=False,
+    ).and_return((('borg', 'create'), ('repo::archive',), flexmock()))
+    flexmock(module.borgmatic.borg.environment).should_receive('make_environment').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).and_return(
+        'warning: stuff\n- /etc/path\n+ /etc/other\n? /nope',
+    )
+    flexmock(module.os.path).should_receive('isfile').and_return(True)
+
+    assert module.collect_spot_check_source_paths(
+        repository={'path': 'repo'},
+        config={'working_directory': '/', 'progress': True},
+        local_borg_version=flexmock(),
+        global_arguments=flexmock(),
+        local_path=flexmock(),
+        remote_path=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+    ) == ('/etc/path', '/etc/other')
+
+
 def test_collect_spot_check_source_paths_passes_through_stream_processes_false():
     flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
         {'hook1': False, 'hook2': False}
@@ -608,10 +655,10 @@ def test_collect_spot_check_source_paths_passes_through_stream_processes_false()
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('collect_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('process_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
         [Pattern('foo'), Pattern('bar')]
     )
     flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
@@ -655,10 +702,10 @@ def test_collect_spot_check_source_paths_without_working_directory_parses_borg_o
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('collect_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('process_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
         [Pattern('foo'), Pattern('bar')]
     )
     flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
@@ -702,10 +749,10 @@ def test_collect_spot_check_source_paths_skips_directories():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('collect_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('process_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
         [Pattern('foo'), Pattern('bar')]
     )
     flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
@@ -847,10 +894,10 @@ def test_collect_spot_check_source_paths_uses_working_directory():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('collect_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
         flexmock()
     )
-    flexmock(module.borgmatic.actions.create).should_receive('process_patterns').and_return(
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
         [Pattern('foo'), Pattern('bar')]
     )
     flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
@@ -891,8 +938,8 @@ def test_collect_spot_check_source_paths_uses_working_directory():
 
 
 def test_compare_spot_check_hashes_returns_paths_having_failing_hashes():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -932,8 +979,8 @@ def test_compare_spot_check_hashes_returns_paths_having_failing_hashes():
 
 
 def test_compare_spot_check_hashes_returns_relative_paths_having_failing_hashes():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -973,8 +1020,8 @@ def test_compare_spot_check_hashes_returns_relative_paths_having_failing_hashes(
 
 
 def test_compare_spot_check_hashes_handles_data_sample_percentage_above_100():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1014,8 +1061,8 @@ def test_compare_spot_check_hashes_handles_data_sample_percentage_above_100():
 
 
 def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1024,7 +1071,9 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
     flexmock(module.os.path).should_receive('islink').and_return(False)
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
-    ).with_args(('/usr/local/bin/xxh64sum', '/foo', '/bar'), working_directory=None).and_return(
+    ).with_args(
+        ('/usr/local/bin/xxhsum', '-H64', '/foo', '/bar'), working_directory=None
+    ).and_return(
         'hash1  /foo\nhash2  /bar'
     )
     flexmock(module.borgmatic.borg.list).should_receive('capture_archive_listing').and_return(
@@ -1039,7 +1088,7 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
                 {
                     'name': 'spot',
                     'data_sample_percentage': 50,
-                    'xxh64sum_command': '/usr/local/bin/xxh64sum',
+                    'xxh64sum_command': '/usr/local/bin/xxhsum -H64',
                 },
             ]
         },
@@ -1052,8 +1101,8 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
 
 
 def test_compare_spot_check_hashes_considers_path_missing_from_archive_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1089,8 +1138,8 @@ def test_compare_spot_check_hashes_considers_path_missing_from_archive_as_not_ma
 
 
 def test_compare_spot_check_hashes_considers_symlink_path_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1125,8 +1174,8 @@ def test_compare_spot_check_hashes_considers_symlink_path_as_not_matching():
 
 
 def test_compare_spot_check_hashes_considers_non_existent_path_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1162,8 +1211,8 @@ def test_compare_spot_check_hashes_considers_non_existent_path_as_not_matching()
 
 def test_compare_spot_check_hashes_with_too_many_paths_feeds_them_to_commands_in_chunks():
     flexmock(module).SAMPLE_PATHS_SUBSET_COUNT = 2
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1208,8 +1257,8 @@ def test_compare_spot_check_hashes_with_too_many_paths_feeds_them_to_commands_in
 
 
 def test_compare_spot_check_hashes_uses_working_directory_to_access_source_paths():
-    flexmock(module.random).should_receive('sample').replace_with(
-        lambda population, count: population[:count]
+    flexmock(module.random).should_receive('SystemRandom').and_return(
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         '/working/dir',

tests/unit/actions/test_create.py

@@ -1,429 +1,10 @@
-import io
-import sys
+import json
+import os
 
 import pytest
 from flexmock import flexmock
 
 from borgmatic.actions import create as module
-from borgmatic.borg.pattern import Pattern, Pattern_source, Pattern_style, Pattern_type
-
-
-@pytest.mark.parametrize(
-    'pattern_line,expected_pattern',
-    (
-        ('R /foo', Pattern('/foo', source=Pattern_source.CONFIG)),
-        ('P sh', Pattern('sh', Pattern_type.PATTERN_STYLE, source=Pattern_source.CONFIG)),
-        ('+ /foo*', Pattern('/foo*', Pattern_type.INCLUDE, source=Pattern_source.CONFIG)),
-        (
-            '+ sh:/foo*',
-            Pattern(
-                '/foo*', Pattern_type.INCLUDE, Pattern_style.SHELL, source=Pattern_source.CONFIG
-            ),
-        ),
-    ),
-)
-def test_parse_pattern_transforms_pattern_line_to_instance(pattern_line, expected_pattern):
-    module.parse_pattern(pattern_line) == expected_pattern
-
-
-def test_parse_pattern_with_invalid_pattern_line_errors():
-    with pytest.raises(ValueError):
-        module.parse_pattern('/foo')
-
-
-def test_collect_patterns_converts_source_directories():
-    assert module.collect_patterns({'source_directories': ['/foo', '/bar']}) == (
-        Pattern('/foo', source=Pattern_source.CONFIG),
-        Pattern('/bar', source=Pattern_source.CONFIG),
-    )
-
-
-def test_collect_patterns_parses_config_patterns():
-    flexmock(module).should_receive('parse_pattern').with_args('R /foo').and_return(Pattern('/foo'))
-    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
-    flexmock(module).should_receive('parse_pattern').with_args('').never()
-    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
-    flexmock(module).should_receive('parse_pattern').with_args('R /bar').and_return(Pattern('/bar'))
-
-    assert module.collect_patterns({'patterns': ['R /foo', '# comment', '', '   ', 'R /bar']}) == (
-        Pattern('/foo'),
-        Pattern('/bar'),
-    )
-
-
-def test_collect_patterns_converts_exclude_patterns():
-    assert module.collect_patterns({'exclude_patterns': ['/foo', '/bar', 'sh:**/baz']}) == (
-        Pattern(
-            '/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH, source=Pattern_source.CONFIG
-        ),
-        Pattern(
-            '/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH, source=Pattern_source.CONFIG
-        ),
-        Pattern(
-            '**/baz', Pattern_type.NO_RECURSE, Pattern_style.SHELL, source=Pattern_source.CONFIG
-        ),
-    )
-
-
-def test_collect_patterns_reads_config_patterns_from_file():
-    builtins = flexmock(sys.modules['builtins'])
-    builtins.should_receive('open').with_args('file1.txt').and_return(io.StringIO('R /foo'))
-    builtins.should_receive('open').with_args('file2.txt').and_return(
-        io.StringIO('R /bar\n# comment\n\n   \nR /baz')
-    )
-    flexmock(module).should_receive('parse_pattern').with_args('R /foo').and_return(Pattern('/foo'))
-    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
-    flexmock(module).should_receive('parse_pattern').with_args('').never()
-    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
-    flexmock(module).should_receive('parse_pattern').with_args('R /bar').and_return(Pattern('/bar'))
-    flexmock(module).should_receive('parse_pattern').with_args('R /baz').and_return(Pattern('/baz'))
-
-    assert module.collect_patterns({'patterns_from': ['file1.txt', 'file2.txt']}) == (
-        Pattern('/foo'),
-        Pattern('/bar'),
-        Pattern('/baz'),
-    )
-
-
-def test_collect_patterns_errors_on_missing_config_patterns_from_file():
-    builtins = flexmock(sys.modules['builtins'])
-    builtins.should_receive('open').with_args('file1.txt').and_raise(FileNotFoundError)
-    flexmock(module).should_receive('parse_pattern').never()
-
-    with pytest.raises(ValueError):
-        module.collect_patterns({'patterns_from': ['file1.txt', 'file2.txt']})
-
-
-def test_collect_patterns_reads_config_exclude_from_file():
-    builtins = flexmock(sys.modules['builtins'])
-    builtins.should_receive('open').with_args('file1.txt').and_return(io.StringIO('/foo'))
-    builtins.should_receive('open').with_args('file2.txt').and_return(
-        io.StringIO('/bar\n# comment\n\n   \n/baz')
-    )
-    flexmock(module).should_receive('parse_pattern').with_args(
-        '! /foo', default_style=Pattern_style.FNMATCH
-    ).and_return(Pattern('/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
-    flexmock(module).should_receive('parse_pattern').with_args(
-        '! /bar', default_style=Pattern_style.FNMATCH
-    ).and_return(Pattern('/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
-    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
-    flexmock(module).should_receive('parse_pattern').with_args('').never()
-    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
-    flexmock(module).should_receive('parse_pattern').with_args(
-        '! /baz', default_style=Pattern_style.FNMATCH
-    ).and_return(Pattern('/baz', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
-
-    assert module.collect_patterns({'exclude_from': ['file1.txt', 'file2.txt']}) == (
-        Pattern('/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
-        Pattern('/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
-        Pattern('/baz', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
-    )
-
-
-def test_collect_patterns_errors_on_missing_config_exclude_from_file():
-    builtins = flexmock(sys.modules['builtins'])
-    builtins.should_receive('open').with_args('file1.txt').and_raise(OSError)
-    flexmock(module).should_receive('parse_pattern').never()
-
-    with pytest.raises(ValueError):
-        module.collect_patterns({'exclude_from': ['file1.txt', 'file2.txt']})
-
-
-def test_expand_directory_with_basic_path_passes_it_through():
-    flexmock(module.os.path).should_receive('expanduser').and_return('foo')
-    flexmock(module.glob).should_receive('glob').and_return([])
-
-    paths = module.expand_directory('foo', None)
-
-    assert paths == ['foo']
-
-
-def test_expand_directory_with_glob_expands():
-    flexmock(module.os.path).should_receive('expanduser').and_return('foo*')
-    flexmock(module.glob).should_receive('glob').and_return(['foo', 'food'])
-
-    paths = module.expand_directory('foo*', None)
-
-    assert paths == ['foo', 'food']
-
-
-def test_expand_directory_strips_off_working_directory():
-    flexmock(module.os.path).should_receive('expanduser').and_return('foo')
-    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo').and_return([]).once()
-
-    paths = module.expand_directory('foo', working_directory='/working/dir')
-
-    assert paths == ['foo']
-
-
-def test_expand_directory_globs_working_directory_and_strips_it_off():
-    flexmock(module.os.path).should_receive('expanduser').and_return('foo*')
-    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo*').and_return(
-        ['/working/dir/foo', '/working/dir/food']
-    ).once()
-
-    paths = module.expand_directory('foo*', working_directory='/working/dir')
-
-    assert paths == ['foo', 'food']
-
-
-def test_expand_directory_with_slashdot_hack_globs_working_directory_and_strips_it_off():
-    flexmock(module.os.path).should_receive('expanduser').and_return('./foo*')
-    flexmock(module.glob).should_receive('glob').with_args('/working/dir/./foo*').and_return(
-        ['/working/dir/./foo', '/working/dir/./food']
-    ).once()
-
-    paths = module.expand_directory('./foo*', working_directory='/working/dir')
-
-    assert paths == ['./foo', './food']
-
-
-def test_expand_directory_with_working_directory_matching_start_of_directory_does_not_strip_it_off():
-    flexmock(module.os.path).should_receive('expanduser').and_return('/working/dir/foo')
-    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo').and_return(
-        ['/working/dir/foo']
-    ).once()
-
-    paths = module.expand_directory('/working/dir/foo', working_directory='/working/dir')
-
-    assert paths == ['/working/dir/foo']
-
-
-def test_expand_patterns_flattens_expanded_directories():
-    flexmock(module).should_receive('expand_directory').with_args('~/foo', None).and_return(
-        ['/root/foo']
-    )
-    flexmock(module).should_receive('expand_directory').with_args('bar*', None).and_return(
-        ['bar', 'barf']
-    )
-
-    paths = module.expand_patterns((Pattern('~/foo'), Pattern('bar*')))
-
-    assert paths == (Pattern('/root/foo'), Pattern('bar'), Pattern('barf'))
-
-
-def test_expand_patterns_with_working_directory_passes_it_through():
-    flexmock(module).should_receive('expand_directory').with_args('foo', '/working/dir').and_return(
-        ['/working/dir/foo']
-    )
-
-    patterns = module.expand_patterns((Pattern('foo'),), working_directory='/working/dir')
-
-    assert patterns == (Pattern('/working/dir/foo'),)
-
-
-def test_expand_patterns_does_not_expand_skip_paths():
-    flexmock(module).should_receive('expand_directory').with_args('/foo', None).and_return(['/foo'])
-    flexmock(module).should_receive('expand_directory').with_args('/bar*', None).never()
-
-    patterns = module.expand_patterns((Pattern('/foo'), Pattern('/bar*')), skip_paths=('/bar*',))
-
-    assert patterns == (Pattern('/foo'), Pattern('/bar*'))
-
-
-def test_expand_patterns_considers_none_as_no_patterns():
-    assert module.expand_patterns(None) == ()
-
-
-def test_expand_patterns_expands_tildes_and_globs_in_root_patterns():
-    flexmock(module.os.path).should_receive('expanduser').never()
-    flexmock(module).should_receive('expand_directory').and_return(
-        ['/root/foo/one', '/root/foo/two']
-    )
-
-    paths = module.expand_patterns((Pattern('~/foo/*'),))
-
-    assert paths == (Pattern('/root/foo/one'), Pattern('/root/foo/two'))
-
-
-def test_expand_patterns_expands_only_tildes_in_non_root_patterns():
-    flexmock(module).should_receive('expand_directory').never()
-    flexmock(module.os.path).should_receive('expanduser').and_return('/root/bar/*')
-
-    paths = module.expand_patterns((Pattern('~/bar/*', Pattern_type.INCLUDE),))
-
-    assert paths == (Pattern('/root/bar/*', Pattern_type.INCLUDE),)
-
-
-def test_device_map_patterns_gives_device_id_per_path():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
-
-    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar')))
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('/bar', device=66),
-    )
-
-
-def test_device_map_patterns_only_considers_root_patterns():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/bar*').never()
-
-    device_map = module.device_map_patterns(
-        (Pattern('/foo'), Pattern('/bar*', Pattern_type.INCLUDE))
-    )
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('/bar*', Pattern_type.INCLUDE),
-    )
-
-
-def test_device_map_patterns_with_missing_path_does_not_error():
-    flexmock(module.os.path).should_receive('exists').and_return(True).and_return(False)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/bar').never()
-
-    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar')))
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('/bar'),
-    )
-
-
-def test_device_map_patterns_uses_working_directory_to_construct_path():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/working/dir/bar').and_return(
-        flexmock(st_dev=66)
-    )
-
-    device_map = module.device_map_patterns(
-        (Pattern('/foo'), Pattern('bar')), working_directory='/working/dir'
-    )
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('bar', device=66),
-    )
-
-
-def test_device_map_patterns_with_existing_device_id_does_not_overwrite_it():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=100))
-
-    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar', device=66)))
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('/bar', device=66),
-    )
-
-
-@pytest.mark.parametrize(
-    'patterns,expected_patterns',
-    (
-        ((Pattern('/', device=1), Pattern('/root', device=1)), (Pattern('/', device=1),)),
-        ((Pattern('/', device=1), Pattern('/root/', device=1)), (Pattern('/', device=1),)),
-        (
-            (Pattern('/', device=1), Pattern('/root', device=2)),
-            (Pattern('/', device=1), Pattern('/root', device=2)),
-        ),
-        ((Pattern('/root', device=1), Pattern('/', device=1)), (Pattern('/', device=1),)),
-        (
-            (Pattern('/root', device=1), Pattern('/root/foo', device=1)),
-            (Pattern('/root', device=1),),
-        ),
-        (
-            (Pattern('/root/', device=1), Pattern('/root/foo', device=1)),
-            (Pattern('/root/', device=1),),
-        ),
-        (
-            (Pattern('/root', device=1), Pattern('/root/foo/', device=1)),
-            (Pattern('/root', device=1),),
-        ),
-        (
-            (Pattern('/root', device=1), Pattern('/root/foo', device=2)),
-            (Pattern('/root', device=1), Pattern('/root/foo', device=2)),
-        ),
-        (
-            (Pattern('/root/foo', device=1), Pattern('/root', device=1)),
-            (Pattern('/root', device=1),),
-        ),
-        (
-            (Pattern('/root', device=None), Pattern('/root/foo', device=None)),
-            (Pattern('/root'), Pattern('/root/foo')),
-        ),
-        (
-            (
-                Pattern('/root', device=1),
-                Pattern('/etc', device=1),
-                Pattern('/root/foo/bar', device=1),
-            ),
-            (Pattern('/root', device=1), Pattern('/etc', device=1)),
-        ),
-        (
-            (
-                Pattern('/root', device=1),
-                Pattern('/root/foo', device=1),
-                Pattern('/root/foo/bar', device=1),
-            ),
-            (Pattern('/root', device=1),),
-        ),
-        ((Pattern('/dup', device=1), Pattern('/dup', device=1)), (Pattern('/dup', device=1),)),
-        (
-            (Pattern('/foo', device=1), Pattern('/bar', device=1)),
-            (Pattern('/foo', device=1), Pattern('/bar', device=1)),
-        ),
-        (
-            (Pattern('/foo', device=1), Pattern('/bar', device=2)),
-            (Pattern('/foo', device=1), Pattern('/bar', device=2)),
-        ),
-        ((Pattern('/root/foo', device=1),), (Pattern('/root/foo', device=1),)),
-        (
-            (Pattern('/', device=1), Pattern('/root', Pattern_type.INCLUDE, device=1)),
-            (Pattern('/', device=1), Pattern('/root', Pattern_type.INCLUDE, device=1)),
-        ),
-        (
-            (Pattern('/root', Pattern_type.INCLUDE, device=1), Pattern('/', device=1)),
-            (Pattern('/root', Pattern_type.INCLUDE, device=1), Pattern('/', device=1)),
-        ),
-    ),
-)
-def test_deduplicate_patterns_omits_child_paths_on_the_same_filesystem(patterns, expected_patterns):
-    assert module.deduplicate_patterns(patterns) == expected_patterns
-
-
-def test_process_patterns_includes_patterns():
-    flexmock(module).should_receive('deduplicate_patterns').and_return(
-        (Pattern('foo'), Pattern('bar'))
-    )
-    flexmock(module).should_receive('device_map_patterns').and_return({})
-    flexmock(module).should_receive('expand_patterns').with_args(
-        (Pattern('foo'), Pattern('bar')),
-        working_directory='/working',
-        skip_paths=set(),
-    ).and_return(()).once()
-
-    assert module.process_patterns(
-        (Pattern('foo'), Pattern('bar')),
-        working_directory='/working',
-    ) == [Pattern('foo'), Pattern('bar')]
-
-
-def test_process_patterns_skips_expand_for_requested_paths():
-    skip_paths = {flexmock()}
-    flexmock(module).should_receive('deduplicate_patterns').and_return(
-        (Pattern('foo'), Pattern('bar'))
-    )
-    flexmock(module).should_receive('device_map_patterns').and_return({})
-    flexmock(module).should_receive('expand_patterns').with_args(
-        (Pattern('foo'), Pattern('bar')),
-        working_directory='/working',
-        skip_paths=skip_paths,
-    ).and_return(()).once()
-
-    assert module.process_patterns(
-        (Pattern('foo'), Pattern('bar')),
-        working_directory='/working',
-        skip_expand_paths=skip_paths,
-    ) == [Pattern('foo'), Pattern('bar')]
 
 
 def test_run_create_executes_and_calls_hooks_for_configured_repository():
@@ -437,14 +18,15 @@ def test_run_create_executes_and_calls_hooks_for_configured_repository():
     flexmock(module.borgmatic.hooks.dispatch).should_receive(
         'call_hooks_even_if_unconfigured'
     ).and_return({})
-    flexmock(module).should_receive('collect_patterns').and_return(())
-    flexmock(module).should_receive('process_patterns').and_return([])
-    flexmock(module.os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
     create_arguments = flexmock(
         repository=None,
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -478,14 +60,15 @@ def test_run_create_runs_with_selected_repository():
     flexmock(module.borgmatic.hooks.dispatch).should_receive(
         'call_hooks_even_if_unconfigured'
     ).and_return({})
-    flexmock(module).should_receive('collect_patterns').and_return(())
-    flexmock(module).should_receive('process_patterns').and_return([])
-    flexmock(module.os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
     create_arguments = flexmock(
         repository=flexmock(),
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -518,6 +101,7 @@ def test_run_create_bails_if_repository_does_not_match():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -550,6 +134,7 @@ def test_run_create_with_both_list_and_json_errors():
         progress=flexmock(),
         statistics=flexmock(),
         json=True,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -583,6 +168,7 @@ def test_run_create_with_both_list_and_progress_errors():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -621,14 +207,15 @@ def test_run_create_produces_json():
     flexmock(module.borgmatic.hooks.dispatch).should_receive(
         'call_hooks_even_if_unconfigured'
     ).and_return({})
-    flexmock(module).should_receive('collect_patterns').and_return(())
-    flexmock(module).should_receive('process_patterns').and_return([])
-    flexmock(module.os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
     create_arguments = flexmock(
         repository=flexmock(),
         progress=flexmock(),
         statistics=flexmock(),
         json=True,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -647,3 +234,193 @@ def test_run_create_produces_json():
             remote_path=None,
         )
     ) == [parsed_json]
+
+
+def test_run_create_with_active_dumps_roundtrips_via_checkpoint_archive():
+    mock_dump_process = flexmock()
+    mock_dump_process.should_receive('poll').and_return(None).and_return(0)
+
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').never()
+    flexmock(module.borgmatic.config.paths).should_receive('Runtime_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.create).should_receive('create_archive').once()
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'dump': mock_dump_process}
+    )
+    flexmock(module.borgmatic.hooks.dispatch).should_receive(
+        'call_hooks_even_if_unconfigured'
+    ).and_return({})
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'archive.checkpoint'}
+    )
+
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+
+    flexmock(module).should_receive('rename_checkpoint_archive').with_args(
+        repository_path='repo',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    ).once()
+    create_arguments = flexmock(
+        repository=None,
+        progress=flexmock(),
+        statistics=flexmock(),
+        json=False,
+        comment=None,
+        list_details=flexmock(),
+    )
+
+    list(
+        module.run_create(
+            config_filename='test.yaml',
+            repository={'path': 'repo'},
+            config={},
+            config_paths=['/tmp/test.yaml'],
+            local_borg_version=None,
+            create_arguments=create_arguments,
+            global_arguments=global_arguments,
+            dry_run_label='',
+            local_path=None,
+            remote_path=None,
+        )
+    )
+
+
+def test_run_create_with_active_dumps_json_updates_archive_info():
+    mock_dump_process = flexmock()
+    mock_dump_process.should_receive('poll').and_return(None).and_return(0)
+
+    borg_create_result = {
+        'archive': {
+            'command_line': ['foo'],
+            'name': 'archive.checkpoint',
+            'id': 'id1',
+        },
+        'cache': {},
+        'repository': {
+            'id': 'repo-id',
+        },
+    }
+
+    expected_create_result = {
+        'archive': {
+            'command_line': ['foo'],
+            'name': 'archive',
+            'id': 'id2',
+        },
+        'cache': {},
+        'repository': {'id': 'repo-id', 'label': ''},
+    }
+
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').never()
+    flexmock(module.borgmatic.config.paths).should_receive('Runtime_directory').and_return(
+        flexmock()
+    )
+
+    flexmock(module.borgmatic.borg.create).should_receive('create_archive').and_return(
+        json.dumps(borg_create_result)
+    ).once()
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'dump': mock_dump_process}
+    )
+    flexmock(module.borgmatic.hooks.dispatch).should_receive(
+        'call_hooks_even_if_unconfigured'
+    ).and_return({})
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+
+    flexmock(module).should_receive('rename_checkpoint_archive').with_args(
+        repository_path='repo',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    ).once()
+
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id2', 'name': 'archive'},
+    )
+
+    create_arguments = flexmock(
+        repository=None,
+        progress=flexmock(),
+        statistics=flexmock(),
+        json=True,
+        comment=None,
+        list_details=flexmock(),
+    )
+
+    assert list(
+        module.run_create(
+            config_filename='test.yaml',
+            repository={'path': 'repo'},
+            config={},
+            config_paths=['/tmp/test.yaml'],
+            local_borg_version=None,
+            create_arguments=create_arguments,
+            global_arguments=global_arguments,
+            dry_run_label='',
+            local_path=None,
+            remote_path=None,
+        )
+    ) == [expected_create_result]
+
+
+def test_rename_checkpoint_archive_renames_archive():
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'archive.checkpoint'}
+    )
+
+    flexmock(module.borgmatic.borg.rename).should_receive('rename_archive').with_args(
+        repository_name='path',
+        old_archive_name='archive.checkpoint',
+        new_archive_name='archive',
+        dry_run=False,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    )
+
+    module.rename_checkpoint_archive(
+        repository_path='path',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    )
+
+
+def test_rename_checkpoint_archive_checks_suffix():
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'unexpected-archive'}
+    )
+
+    with pytest.raises(
+        ValueError,
+        match='Latest archive did not have a .checkpoint suffix. Got: unexpected-archive',
+    ):
+        module.rename_checkpoint_archive(
+            repository_path='path',
+            global_arguments=global_arguments,
+            config={},
+            local_borg_version=None,
+            local_path=None,
+            remote_path=None,
+        )

tests/unit/actions/test_pattern.py

@@ -0,0 +1,459 @@
+import io
+import sys
+
+import pytest
+from flexmock import flexmock
+
+from borgmatic.actions import pattern as module
+from borgmatic.borg.pattern import Pattern, Pattern_source, Pattern_style, Pattern_type
+
+
+@pytest.mark.parametrize(
+    'pattern_line,expected_pattern',
+    (
+        ('R /foo', Pattern('/foo', source=Pattern_source.CONFIG)),
+        ('P sh', Pattern('sh', Pattern_type.PATTERN_STYLE, source=Pattern_source.CONFIG)),
+        ('+ /foo*', Pattern('/foo*', Pattern_type.INCLUDE, source=Pattern_source.CONFIG)),
+        (
+            '+ sh:/foo*',
+            Pattern(
+                '/foo*', Pattern_type.INCLUDE, Pattern_style.SHELL, source=Pattern_source.CONFIG
+            ),
+        ),
+    ),
+)
+def test_parse_pattern_transforms_pattern_line_to_instance(pattern_line, expected_pattern):
+    module.parse_pattern(pattern_line) == expected_pattern
+
+
+def test_parse_pattern_with_invalid_pattern_line_errors():
+    with pytest.raises(ValueError):
+        module.parse_pattern('/foo')
+
+
+def test_collect_patterns_converts_source_directories():
+    assert module.collect_patterns({'source_directories': ['/foo', '/bar']}) == (
+        Pattern('/foo', source=Pattern_source.CONFIG),
+        Pattern('/bar', source=Pattern_source.CONFIG),
+    )
+
+
+def test_collect_patterns_parses_config_patterns():
+    flexmock(module).should_receive('parse_pattern').with_args('R /foo').and_return(Pattern('/foo'))
+    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
+    flexmock(module).should_receive('parse_pattern').with_args('').never()
+    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
+    flexmock(module).should_receive('parse_pattern').with_args('R /bar').and_return(Pattern('/bar'))
+
+    assert module.collect_patterns({'patterns': ['R /foo', '# comment', '', '   ', 'R /bar']}) == (
+        Pattern('/foo'),
+        Pattern('/bar'),
+    )
+
+
+def test_collect_patterns_converts_exclude_patterns():
+    assert module.collect_patterns({'exclude_patterns': ['/foo', '/bar', 'sh:**/baz']}) == (
+        Pattern(
+            '/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH, source=Pattern_source.CONFIG
+        ),
+        Pattern(
+            '/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH, source=Pattern_source.CONFIG
+        ),
+        Pattern(
+            '**/baz', Pattern_type.NO_RECURSE, Pattern_style.SHELL, source=Pattern_source.CONFIG
+        ),
+    )
+
+
+def test_collect_patterns_reads_config_patterns_from_file():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('file1.txt').and_return(io.StringIO('R /foo'))
+    builtins.should_receive('open').with_args('file2.txt').and_return(
+        io.StringIO('R /bar\n# comment\n\n   \nR /baz')
+    )
+    flexmock(module).should_receive('parse_pattern').with_args('R /foo').and_return(Pattern('/foo'))
+    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
+    flexmock(module).should_receive('parse_pattern').with_args('').never()
+    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
+    flexmock(module).should_receive('parse_pattern').with_args('R /bar').and_return(Pattern('/bar'))
+    flexmock(module).should_receive('parse_pattern').with_args('R /baz').and_return(Pattern('/baz'))
+
+    assert module.collect_patterns({'patterns_from': ['file1.txt', 'file2.txt']}) == (
+        Pattern('/foo'),
+        Pattern('/bar'),
+        Pattern('/baz'),
+    )
+
+
+def test_collect_patterns_errors_on_missing_config_patterns_from_file():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('file1.txt').and_raise(FileNotFoundError)
+    flexmock(module).should_receive('parse_pattern').never()
+
+    with pytest.raises(ValueError):
+        module.collect_patterns({'patterns_from': ['file1.txt', 'file2.txt']})
+
+
+def test_collect_patterns_reads_config_exclude_from_file():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('file1.txt').and_return(io.StringIO('/foo'))
+    builtins.should_receive('open').with_args('file2.txt').and_return(
+        io.StringIO('/bar\n# comment\n\n   \n/baz')
+    )
+    flexmock(module).should_receive('parse_pattern').with_args(
+        '! /foo', default_style=Pattern_style.FNMATCH
+    ).and_return(Pattern('/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
+    flexmock(module).should_receive('parse_pattern').with_args(
+        '! /bar', default_style=Pattern_style.FNMATCH
+    ).and_return(Pattern('/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
+    flexmock(module).should_receive('parse_pattern').with_args('# comment').never()
+    flexmock(module).should_receive('parse_pattern').with_args('').never()
+    flexmock(module).should_receive('parse_pattern').with_args('   ').never()
+    flexmock(module).should_receive('parse_pattern').with_args(
+        '! /baz', default_style=Pattern_style.FNMATCH
+    ).and_return(Pattern('/baz', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH))
+
+    assert module.collect_patterns({'exclude_from': ['file1.txt', 'file2.txt']}) == (
+        Pattern('/foo', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
+        Pattern('/bar', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
+        Pattern('/baz', Pattern_type.NO_RECURSE, Pattern_style.FNMATCH),
+    )
+
+
+def test_collect_patterns_errors_on_missing_config_exclude_from_file():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('file1.txt').and_raise(OSError)
+    flexmock(module).should_receive('parse_pattern').never()
+
+    with pytest.raises(ValueError):
+        module.collect_patterns({'exclude_from': ['file1.txt', 'file2.txt']})
+
+
+def test_expand_directory_with_basic_path_passes_it_through():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo')
+    flexmock(module.glob).should_receive('glob').and_return([])
+
+    paths = module.expand_directory('foo', None)
+
+    assert paths == ['foo']
+
+
+def test_expand_directory_with_glob_expands():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo*')
+    flexmock(module.glob).should_receive('glob').and_return(['foo', 'food'])
+
+    paths = module.expand_directory('foo*', None)
+
+    assert paths == ['foo', 'food']
+
+
+def test_expand_directory_strips_off_working_directory():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo')
+    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo').and_return([]).once()
+
+    paths = module.expand_directory('foo', working_directory='/working/dir')
+
+    assert paths == ['foo']
+
+
+def test_expand_directory_globs_working_directory_and_strips_it_off():
+    flexmock(module.os.path).should_receive('expanduser').and_return('foo*')
+    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo*').and_return(
+        ['/working/dir/foo', '/working/dir/food']
+    ).once()
+
+    paths = module.expand_directory('foo*', working_directory='/working/dir')
+
+    assert paths == ['foo', 'food']
+
+
+def test_expand_directory_with_slashdot_hack_globs_working_directory_and_strips_it_off():
+    flexmock(module.os.path).should_receive('expanduser').and_return('./foo*')
+    flexmock(module.glob).should_receive('glob').with_args('/working/dir/./foo*').and_return(
+        ['/working/dir/./foo', '/working/dir/./food']
+    ).once()
+
+    paths = module.expand_directory('./foo*', working_directory='/working/dir')
+
+    assert paths == ['./foo', './food']
+
+
+def test_expand_directory_with_working_directory_matching_start_of_directory_does_not_strip_it_off():
+    flexmock(module.os.path).should_receive('expanduser').and_return('/working/dir/foo')
+    flexmock(module.glob).should_receive('glob').with_args('/working/dir/foo').and_return(
+        ['/working/dir/foo']
+    ).once()
+
+    paths = module.expand_directory('/working/dir/foo', working_directory='/working/dir')
+
+    assert paths == ['/working/dir/foo']
+
+
+def test_expand_patterns_flattens_expanded_directories():
+    flexmock(module).should_receive('expand_directory').with_args('~/foo', None).and_return(
+        ['/root/foo']
+    )
+    flexmock(module).should_receive('expand_directory').with_args('bar*', None).and_return(
+        ['bar', 'barf']
+    )
+
+    paths = module.expand_patterns((Pattern('~/foo'), Pattern('bar*')))
+
+    assert paths == (Pattern('/root/foo'), Pattern('bar'), Pattern('barf'))
+
+
+def test_expand_patterns_with_working_directory_passes_it_through():
+    flexmock(module).should_receive('expand_directory').with_args('foo', '/working/dir').and_return(
+        ['/working/dir/foo']
+    )
+
+    patterns = module.expand_patterns((Pattern('foo'),), working_directory='/working/dir')
+
+    assert patterns == (Pattern('/working/dir/foo'),)
+
+
+def test_expand_patterns_does_not_expand_skip_paths():
+    flexmock(module).should_receive('expand_directory').with_args('/foo', None).and_return(['/foo'])
+    flexmock(module).should_receive('expand_directory').with_args('/bar*', None).never()
+
+    patterns = module.expand_patterns((Pattern('/foo'), Pattern('/bar*')), skip_paths=('/bar*',))
+
+    assert patterns == (Pattern('/foo'), Pattern('/bar*'))
+
+
+def test_expand_patterns_considers_none_as_no_patterns():
+    assert module.expand_patterns(None) == ()
+
+
+def test_expand_patterns_expands_tildes_and_globs_in_root_patterns():
+    flexmock(module.os.path).should_receive('expanduser').never()
+    flexmock(module).should_receive('expand_directory').and_return(
+        ['/root/foo/one', '/root/foo/two']
+    )
+
+    paths = module.expand_patterns((Pattern('~/foo/*'),))
+
+    assert paths == (Pattern('/root/foo/one'), Pattern('/root/foo/two'))
+
+
+def test_expand_patterns_expands_only_tildes_in_non_root_patterns():
+    flexmock(module).should_receive('expand_directory').never()
+    flexmock(module.os.path).should_receive('expanduser').and_return('/root/bar/*')
+
+    paths = module.expand_patterns((Pattern('~/bar/*', Pattern_type.INCLUDE),))
+
+    assert paths == (Pattern('/root/bar/*', Pattern_type.INCLUDE),)
+
+
+def test_get_existent_path_or_parent_passes_through_existent_path():
+    flexmock(module.os.path).should_receive('exists').and_return(True)
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz') == '/foo/bar/baz'
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_none():
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz') is None
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_existent_parent():
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar/baz*').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar').and_return(True)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo').never()
+    flexmock(module.os.path).should_receive('exists').with_args('/').never()
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz*') == '/foo/bar'
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_existent_grandparent():
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar/baz*').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo').and_return(True)
+    flexmock(module.os.path).should_receive('exists').with_args('/').never()
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz*') == '/foo'
+
+
+def test_get_existent_path_or_parent_with_end_to_end_test_prefix_returns_none():
+    flexmock(module.os.path).should_receive('exists').never()
+
+    assert module.get_existent_path_or_parent('/e2e/foo/bar/baz') is None
+
+
+def test_device_map_patterns_gives_device_id_per_path():
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
+    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
+    flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
+
+    device_map = module.device_map_patterns(
+        (
+            Pattern('/foo'),
+            Pattern('^/bar', type=Pattern_type.INCLUDE, style=Pattern_style.REGULAR_EXPRESSION),
+        )
+    )
+
+    assert device_map == (
+        Pattern('/foo', device=55),
+        Pattern(
+            '^/bar', type=Pattern_type.INCLUDE, style=Pattern_style.REGULAR_EXPRESSION, device=66
+        ),
+    )
+
+
+def test_device_map_patterns_with_missing_path_does_not_error():
+    flexmock(module).should_receive('get_existent_path_or_parent').with_args('/foo').and_return(
+        '/foo'
+    )
+    flexmock(module).should_receive('get_existent_path_or_parent').with_args('/bar').and_return(
+        None
+    )
+    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
+    flexmock(module.os).should_receive('stat').with_args('/bar').never()
+
+    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar')))
+
+    assert device_map == (
+        Pattern('/foo', device=55),
+        Pattern('/bar'),
+    )
+
+
+def test_device_map_patterns_uses_working_directory_to_construct_path():
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
+    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
+    flexmock(module.os).should_receive('stat').with_args('/working/dir/bar').and_return(
+        flexmock(st_dev=66)
+    )
+
+    device_map = module.device_map_patterns(
+        (Pattern('/foo'), Pattern('bar')), working_directory='/working/dir'
+    )
+
+    assert device_map == (
+        Pattern('/foo', device=55),
+        Pattern('bar', device=66),
+    )
+
+
+def test_device_map_patterns_with_existing_device_id_does_not_overwrite_it():
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
+    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
+    flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=100))
+
+    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar', device=66)))
+
+    assert device_map == (
+        Pattern('/foo', device=55),
+        Pattern('/bar', device=66),
+    )
+
+
+@pytest.mark.parametrize(
+    'patterns,expected_patterns',
+    (
+        ((Pattern('/', device=1), Pattern('/root', device=1)), (Pattern('/', device=1),)),
+        ((Pattern('/', device=1), Pattern('/root/', device=1)), (Pattern('/', device=1),)),
+        (
+            (Pattern('/', device=1), Pattern('/root', device=2)),
+            (Pattern('/', device=1), Pattern('/root', device=2)),
+        ),
+        ((Pattern('/root', device=1), Pattern('/', device=1)), (Pattern('/', device=1),)),
+        (
+            (Pattern('/root', device=1), Pattern('/root/foo', device=1)),
+            (Pattern('/root', device=1),),
+        ),
+        (
+            (Pattern('/root/', device=1), Pattern('/root/foo', device=1)),
+            (Pattern('/root/', device=1),),
+        ),
+        (
+            (Pattern('/root', device=1), Pattern('/root/foo/', device=1)),
+            (Pattern('/root', device=1),),
+        ),
+        (
+            (Pattern('/root', device=1), Pattern('/root/foo', device=2)),
+            (Pattern('/root', device=1), Pattern('/root/foo', device=2)),
+        ),
+        (
+            (Pattern('/root/foo', device=1), Pattern('/root', device=1)),
+            (Pattern('/root', device=1),),
+        ),
+        (
+            (Pattern('/root', device=None), Pattern('/root/foo', device=None)),
+            (Pattern('/root'), Pattern('/root/foo')),
+        ),
+        (
+            (
+                Pattern('/root', device=1),
+                Pattern('/etc', device=1),
+                Pattern('/root/foo/bar', device=1),
+            ),
+            (Pattern('/root', device=1), Pattern('/etc', device=1)),
+        ),
+        (
+            (
+                Pattern('/root', device=1),
+                Pattern('/root/foo', device=1),
+                Pattern('/root/foo/bar', device=1),
+            ),
+            (Pattern('/root', device=1),),
+        ),
+        ((Pattern('/dup', device=1), Pattern('/dup', device=1)), (Pattern('/dup', device=1),)),
+        (
+            (Pattern('/foo', device=1), Pattern('/bar', device=1)),
+            (Pattern('/foo', device=1), Pattern('/bar', device=1)),
+        ),
+        (
+            (Pattern('/foo', device=1), Pattern('/bar', device=2)),
+            (Pattern('/foo', device=1), Pattern('/bar', device=2)),
+        ),
+        ((Pattern('/root/foo', device=1),), (Pattern('/root/foo', device=1),)),
+        (
+            (Pattern('/', device=1), Pattern('/root', Pattern_type.INCLUDE, device=1)),
+            (Pattern('/', device=1), Pattern('/root', Pattern_type.INCLUDE, device=1)),
+        ),
+        (
+            (Pattern('/root', Pattern_type.INCLUDE, device=1), Pattern('/', device=1)),
+            (Pattern('/root', Pattern_type.INCLUDE, device=1), Pattern('/', device=1)),
+        ),
+    ),
+)
+def test_deduplicate_patterns_omits_child_paths_on_the_same_filesystem(patterns, expected_patterns):
+    assert module.deduplicate_patterns(patterns) == expected_patterns
+
+
+def test_process_patterns_includes_patterns():
+    flexmock(module).should_receive('deduplicate_patterns').and_return(
+        (Pattern('foo'), Pattern('bar'))
+    )
+    flexmock(module).should_receive('device_map_patterns').and_return({})
+    flexmock(module).should_receive('expand_patterns').with_args(
+        (Pattern('foo'), Pattern('bar')),
+        working_directory='/working',
+        skip_paths=set(),
+    ).and_return(()).once()
+
+    assert module.process_patterns(
+        (Pattern('foo'), Pattern('bar')),
+        working_directory='/working',
+    ) == [Pattern('foo'), Pattern('bar')]
+
+
+def test_process_patterns_skips_expand_for_requested_paths():
+    skip_paths = {flexmock()}
+    flexmock(module).should_receive('deduplicate_patterns').and_return(
+        (Pattern('foo'), Pattern('bar'))
+    )
+    flexmock(module).should_receive('device_map_patterns').and_return({})
+    flexmock(module).should_receive('expand_patterns').with_args(
+        (Pattern('foo'), Pattern('bar')),
+        working_directory='/working',
+        skip_paths=skip_paths,
+    ).and_return(()).once()
+
+    assert module.process_patterns(
+        (Pattern('foo'), Pattern('bar')),
+        working_directory='/working',
+        skip_expand_paths=skip_paths,
+    ) == [Pattern('foo'), Pattern('bar')]

tests/unit/actions/test_recreate.py

@@ -1,3 +1,4 @@
+import pytest
 from flexmock import flexmock
 
 from borgmatic.actions import recreate as module
@@ -6,15 +7,19 @@ from borgmatic.actions import recreate as module
 def test_run_recreate_does_not_raise():
     flexmock(module.logger).answer = lambda message: None
     flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        None
+    )
     flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive')
 
-    recreate_arguments = flexmock(repository=flexmock(), archive=None)
-
     module.run_recreate(
         repository={'path': 'repo'},
         config={},
         local_borg_version=None,
-        recreate_arguments=recreate_arguments,
+        recreate_arguments=flexmock(repository=flexmock(), archive=None),
         global_arguments=flexmock(),
         local_path=None,
         remote_path=None,
@@ -24,16 +29,154 @@ def test_run_recreate_does_not_raise():
 def test_run_recreate_with_archive_does_not_raise():
     flexmock(module.logger).answer = lambda message: None
     flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive'
+    )
     flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive')
 
-    recreate_arguments = flexmock(repository=flexmock(), archive='test-archive')
-
     module.run_recreate(
         repository={'path': 'repo'},
         config={},
         local_borg_version=None,
-        recreate_arguments=recreate_arguments,
+        recreate_arguments=flexmock(repository=flexmock(), archive='test-archive'),
         global_arguments=flexmock(),
         local_path=None,
         remote_path=None,
     )
+
+
+def test_run_recreate_with_leftover_recreate_archive_raises():
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive.recreate'
+    )
+    flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive')
+
+    with pytest.raises(ValueError):
+        module.run_recreate(
+            repository={'path': 'repo'},
+            config={},
+            local_borg_version=None,
+            recreate_arguments=flexmock(repository=flexmock(), archive='test-archive.recreate'),
+            global_arguments=flexmock(),
+            local_path=None,
+            remote_path=None,
+        )
+
+
+def test_run_recreate_with_latest_archive_resolving_to_leftover_recreate_archive_raises():
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive.recreate'
+    )
+    flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive')
+
+    with pytest.raises(ValueError):
+        module.run_recreate(
+            repository={'path': 'repo'},
+            config={},
+            local_borg_version=None,
+            recreate_arguments=flexmock(repository=flexmock(), archive='latest'),
+            global_arguments=flexmock(),
+            local_path=None,
+            remote_path=None,
+        )
+
+
+def test_run_recreate_with_archive_already_exists_error_raises():
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive'
+    )
+    flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive').and_raise(
+        module.subprocess.CalledProcessError(
+            returncode=module.BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS,
+            cmd='borg recreate or whatever',
+        )
+    )
+
+    with pytest.raises(ValueError):
+        module.run_recreate(
+            repository={'path': 'repo'},
+            config={},
+            local_borg_version=None,
+            recreate_arguments=flexmock(repository=flexmock(), archive='test-archive', target=None),
+            global_arguments=flexmock(),
+            local_path=None,
+            remote_path=None,
+        )
+
+
+def test_run_recreate_with_target_and_archive_already_exists_error_raises():
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive'
+    )
+    flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive').and_raise(
+        module.subprocess.CalledProcessError(
+            returncode=module.BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS,
+            cmd='borg recreate or whatever',
+        )
+    )
+
+    with pytest.raises(ValueError):
+        module.run_recreate(
+            repository={'path': 'repo'},
+            config={},
+            local_borg_version=None,
+            recreate_arguments=flexmock(
+                repository=flexmock(), archive='test-archive', target='target-archive'
+            ),
+            global_arguments=flexmock(),
+            local_path=None,
+            remote_path=None,
+        )
+
+
+def test_run_recreate_with_other_called_process_error_passes_it_through():
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.repo_list).should_receive('resolve_archive_name').and_return(
+        'test-archive'
+    )
+    flexmock(module.borgmatic.borg.recreate).should_receive('recreate_archive').and_raise(
+        module.subprocess.CalledProcessError(
+            returncode=1,
+            cmd='borg recreate or whatever',
+        )
+    )
+
+    with pytest.raises(module.subprocess.CalledProcessError):
+        module.run_recreate(
+            repository={'path': 'repo'},
+            config={},
+            local_borg_version=None,
+            recreate_arguments=flexmock(
+                repository=flexmock(), archive='test-archive', target='target-archive'
+            ),
+            global_arguments=flexmock(),
+            local_path=None,
+            remote_path=None,
+        )

tests/unit/borg/test_check.py

@@ -911,12 +911,14 @@ def test_check_archives_with_retention_prefix():
 
 
 def test_check_archives_with_extra_borg_options_passes_through_to_borg():
-    config = {'extra_borg_options': {'check': '--extra --options'}}
+    config = {'extra_borg_options': {'check': '--extra --options "value with space"'}}
     flexmock(module).should_receive('make_check_name_flags').with_args(
         {'repository'}, ()
     ).and_return(())
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
-    insert_execute_command_mock(('borg', 'check', '--extra', '--options', 'repo'))
+    insert_execute_command_mock(
+        ('borg', 'check', '--extra', '--options', 'value with space', 'repo')
+    )
 
     module.check_archives(
         repository_path='repo',

tests/unit/borg/test_compact.py

@@ -68,8 +68,15 @@ def test_compact_segments_with_log_debug_calls_borg_with_debug_flag():
     )
 
 
-def test_compact_segments_with_dry_run_skips_borg_call():
+def test_compact_segments_with_dry_run_skips_borg_call_when_feature_unavailable():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.2.3'
+    ).and_return(False)
+    flexmock(module.environment).should_receive('make_environment').never()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').never()
     flexmock(module).should_receive('execute_command').never()
+    flexmock(logging).should_receive('info').with_args('Skipping compact (dry run)').once()
 
     module.compact_segments(
         repository_path='repo',
@@ -80,6 +87,24 @@ def test_compact_segments_with_dry_run_skips_borg_call():
     )
 
 
+def test_compact_segments_with_dry_run_executes_borg_call_when_feature_available():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.4.1'
+    ).and_return(True)
+    flexmock(module.environment).should_receive('make_environment').once()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').once()
+    flexmock(module).should_receive('execute_command').once()
+
+    module.compact_segments(
+        repository_path='repo',
+        config={},
+        local_borg_version='1.4.1',
+        global_arguments=flexmock(),
+        dry_run=True,
+    )
+
+
 def test_compact_segments_with_local_path_calls_borg_via_local_path():
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
     insert_execute_command_mock(('borg1',) + COMPACT_COMMAND[1:] + ('repo',), logging.INFO)
@@ -207,12 +232,14 @@ def test_compact_segments_with_lock_wait_calls_borg_with_lock_wait_flags():
 
 def test_compact_segments_with_extra_borg_options_calls_borg_with_extra_options():
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
-    insert_execute_command_mock(COMPACT_COMMAND + ('--extra', '--options', 'repo'), logging.INFO)
+    insert_execute_command_mock(
+        COMPACT_COMMAND + ('--extra', '--options', 'value with space', 'repo'), logging.INFO
+    )
 
     module.compact_segments(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'compact': '--extra --options'}},
+        config={'extra_borg_options': {'compact': '--extra --options "value with space"'}},
         local_borg_version='1.2.3',
         global_arguments=flexmock(),
     )

tests/unit/borg/test_create.py

@@ -9,131 +9,6 @@ from borgmatic.borg.pattern import Pattern, Pattern_source, Pattern_style, Patte
 from ..test_verbosity import insert_logging_mock
 
 
-def test_write_patterns_file_writes_pattern_lines():
-    temporary_file = flexmock(name='filename', flush=lambda: None)
-    temporary_file.should_receive('write').with_args('R /foo\n+ sh:/foo/bar')
-    flexmock(module.tempfile).should_receive('NamedTemporaryFile').and_return(temporary_file)
-
-    module.write_patterns_file(
-        [Pattern('/foo'), Pattern('/foo/bar', Pattern_type.INCLUDE, Pattern_style.SHELL)],
-        borgmatic_runtime_directory='/run/user/0',
-    )
-
-
-def test_write_patterns_file_with_empty_exclude_patterns_does_not_raise():
-    module.write_patterns_file([], borgmatic_runtime_directory='/run/user/0')
-
-
-def test_write_patterns_file_appends_to_existing():
-    patterns_file = flexmock(name='filename', flush=lambda: None)
-    patterns_file.should_receive('write').with_args('\n')
-    patterns_file.should_receive('write').with_args('R /foo\n+ /foo/bar')
-    flexmock(module.tempfile).should_receive('NamedTemporaryFile').never()
-
-    module.write_patterns_file(
-        [Pattern('/foo'), Pattern('/foo/bar', Pattern_type.INCLUDE)],
-        borgmatic_runtime_directory='/run/user/0',
-        patterns_file=patterns_file,
-    )
-
-
-def test_make_exclude_flags_includes_exclude_caches_when_true_in_config():
-    exclude_flags = module.make_exclude_flags(config={'exclude_caches': True})
-
-    assert exclude_flags == ('--exclude-caches',)
-
-
-def test_make_exclude_flags_does_not_include_exclude_caches_when_false_in_config():
-    exclude_flags = module.make_exclude_flags(config={'exclude_caches': False})
-
-    assert exclude_flags == ()
-
-
-def test_make_exclude_flags_includes_exclude_if_present_when_in_config():
-    exclude_flags = module.make_exclude_flags(
-        config={'exclude_if_present': ['exclude_me', 'also_me']}
-    )
-
-    assert exclude_flags == (
-        '--exclude-if-present',
-        'exclude_me',
-        '--exclude-if-present',
-        'also_me',
-    )
-
-
-def test_make_exclude_flags_includes_keep_exclude_tags_when_true_in_config():
-    exclude_flags = module.make_exclude_flags(config={'keep_exclude_tags': True})
-
-    assert exclude_flags == ('--keep-exclude-tags',)
-
-
-def test_make_exclude_flags_does_not_include_keep_exclude_tags_when_false_in_config():
-    exclude_flags = module.make_exclude_flags(config={'keep_exclude_tags': False})
-
-    assert exclude_flags == ()
-
-
-def test_make_exclude_flags_includes_exclude_nodump_when_true_in_config():
-    exclude_flags = module.make_exclude_flags(config={'exclude_nodump': True})
-
-    assert exclude_flags == ('--exclude-nodump',)
-
-
-def test_make_exclude_flags_does_not_include_exclude_nodump_when_false_in_config():
-    exclude_flags = module.make_exclude_flags(config={'exclude_nodump': False})
-
-    assert exclude_flags == ()
-
-
-def test_make_exclude_flags_is_empty_when_config_has_no_excludes():
-    exclude_flags = module.make_exclude_flags(config={})
-
-    assert exclude_flags == ()
-
-
-def test_make_list_filter_flags_with_debug_and_feature_available_includes_plus_and_minus():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
-    flexmock(module.feature).should_receive('available').and_return(True)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME+-'
-
-
-def test_make_list_filter_flags_with_info_and_feature_available_omits_plus_and_minus():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
-    flexmock(module.feature).should_receive('available').and_return(True)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME'
-
-
-def test_make_list_filter_flags_with_debug_and_feature_available_and_dry_run_includes_plus_and_minus():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
-    flexmock(module.feature).should_receive('available').and_return(True)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=True) == 'AME+-'
-
-
-def test_make_list_filter_flags_with_info_and_feature_available_and_dry_run_includes_plus_and_minus():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
-    flexmock(module.feature).should_receive('available').and_return(True)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=True) == 'AME+-'
-
-
-def test_make_list_filter_flags_with_debug_and_feature_not_available_includes_x():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
-    flexmock(module.feature).should_receive('available').and_return(False)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AMEx-'
-
-
-def test_make_list_filter_flags_with_info_and_feature_not_available_omits_x():
-    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
-    flexmock(module.feature).should_receive('available').and_return(False)
-
-    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME-'
-
-
 @pytest.mark.parametrize(
     'character_device,block_device,fifo,expected_result',
     (
@@ -326,10 +201,10 @@ REPO_ARCHIVE = (f'repo::{DEFAULT_ARCHIVE_NAME}',)
 
 def test_make_base_create_produces_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -355,16 +230,16 @@ def test_make_base_create_produces_borg_command():
 def test_make_base_create_command_includes_patterns_file_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     mock_pattern_file = flexmock(name='/tmp/patterns')
-    flexmock(module).should_receive('write_patterns_file').and_return(mock_pattern_file).and_return(
-        None
-    )
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(
+        mock_pattern_file
+    ).and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
     pattern_flags = ('--patterns-from', mock_pattern_file.name)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -390,13 +265,13 @@ def test_make_base_create_command_includes_patterns_file_in_borg_command():
 
 def test_make_base_create_command_with_store_config_false_omits_config_files():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -455,13 +330,13 @@ def test_make_base_create_command_includes_configuration_option_as_command_flag(
     option_name, option_value, feature_available, option_flags
 ):
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(feature_available)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -487,13 +362,13 @@ def test_make_base_create_command_includes_configuration_option_as_command_flag(
 
 def test_make_base_create_command_includes_dry_run_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -517,15 +392,48 @@ def test_make_base_create_command_includes_dry_run_in_borg_command():
     assert not pattern_file
 
 
-def test_make_base_create_command_includes_local_path_in_borg_command():
+def test_make_base_create_command_includes_comment_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+        (f'repo::{DEFAULT_ARCHIVE_NAME}',)
+    )
+
+    (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': ['exclude'],
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+        comment='a comment',
+    )
+
+    assert create_flags == ('borg', 'create', '--comment', 'a comment')
+    assert create_positional_arguments == REPO_ARCHIVE
+    assert not pattern_file
+
+
+def test_make_base_create_command_includes_local_path_in_borg_command():
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
+        '{hostname}'
+    )
+    flexmock(module.feature).should_receive('available').and_return(True)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -551,13 +459,13 @@ def test_make_base_create_command_includes_local_path_in_borg_command():
 
 def test_make_base_create_command_includes_remote_path_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -583,13 +491,13 @@ def test_make_base_create_command_includes_remote_path_in_borg_command():
 
 def test_make_base_create_command_includes_log_json_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -615,13 +523,13 @@ def test_make_base_create_command_includes_log_json_in_borg_command():
 
 def test_make_base_create_command_includes_list_flags_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -649,22 +557,22 @@ def test_make_base_create_command_with_stream_processes_ignores_read_special_fal
     patterns = [Pattern('foo'), Pattern('bar')]
     patterns_file = flexmock(name='patterns')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').with_args(
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').with_args(
         patterns, '/run/borgmatic'
     ).and_return(patterns_file)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
     flexmock(module.logger).should_receive('warning').twice()
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module).should_receive('collect_special_file_paths').and_return(('/dev/null',)).once()
-    flexmock(module).should_receive('write_patterns_file').with_args(
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').with_args(
         (
             Pattern(
                 '/dev/null',
@@ -676,7 +584,7 @@ def test_make_base_create_command_with_stream_processes_ignores_read_special_fal
         '/run/borgmatic',
         patterns_file=patterns_file,
     ).and_return(patterns_file).once()
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
 
     (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
         dry_run=False,
@@ -700,22 +608,22 @@ def test_make_base_create_command_with_stream_processes_ignores_read_special_fal
 
 def test_make_base_create_command_without_patterns_and_with_stream_processes_ignores_read_special_false_and_excludes_special_files():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').with_args(
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').with_args(
         [], '/run/borgmatic'
     ).and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
     flexmock(module.logger).should_receive('warning').twice()
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module).should_receive('collect_special_file_paths').and_return(('/dev/null',)).once()
-    flexmock(module).should_receive('write_patterns_file').with_args(
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').with_args(
         (
             Pattern(
                 '/dev/null',
@@ -727,7 +635,7 @@ def test_make_base_create_command_without_patterns_and_with_stream_processes_ign
         '/run/borgmatic',
         patterns_file=None,
     ).and_return(flexmock(name='patterns')).once()
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
 
     (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
         dry_run=False,
@@ -751,13 +659,13 @@ def test_make_base_create_command_without_patterns_and_with_stream_processes_ign
 
 def test_make_base_create_command_with_stream_processes_and_read_special_true_skips_special_files_excludes():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -786,13 +694,13 @@ def test_make_base_create_command_with_stream_processes_and_read_special_true_sk
 
 def test_make_base_create_command_includes_archive_name_format_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         ('repo::ARCHIVE_NAME',)
     )
@@ -818,13 +726,13 @@ def test_make_base_create_command_includes_archive_name_format_in_borg_command()
 
 def test_make_base_create_command_includes_default_archive_name_format_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         ('repo::{hostname}',)
     )
@@ -849,13 +757,13 @@ def test_make_base_create_command_includes_default_archive_name_format_in_borg_c
 
 def test_make_base_create_command_includes_archive_name_format_with_placeholders_in_borg_command():
     repository_archive_pattern = 'repo::Documents_{hostname}-{now}'  # noqa: FS003
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (repository_archive_pattern,)
     )
@@ -881,13 +789,13 @@ def test_make_base_create_command_includes_archive_name_format_with_placeholders
 
 def test_make_base_create_command_includes_repository_and_archive_name_format_with_placeholders_in_borg_command():
     repository_archive_pattern = '{fqdn}::Documents_{hostname}-{now}'  # noqa: FS003
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (repository_archive_pattern,)
     )
@@ -911,15 +819,43 @@ def test_make_base_create_command_includes_repository_and_archive_name_format_wi
     assert not pattern_file
 
 
+def test_make_base_create_command_includes_archive_suffix_in_borg_command():
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
+        DEFAULT_ARCHIVE_NAME
+    )
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+
+    (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+        archive_suffix='.checkpoint',
+    )
+
+    assert create_flags == ('borg', 'create')
+    assert create_positional_arguments == (f'repo::{DEFAULT_ARCHIVE_NAME}.checkpoint',)
+    assert not pattern_file
+
+
 def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('write_patterns_file').and_return(None)
-    flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
     flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
         '{hostname}'
     )
     flexmock(module.feature).should_receive('available').and_return(True)
-    flexmock(module).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
     flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
         (f'repo::{DEFAULT_ARCHIVE_NAME}',)
     )
@@ -930,7 +866,7 @@ def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
         config={
             'source_directories': ['foo', 'bar'],
             'repositories': ['repo'],
-            'extra_borg_options': {'create': '--extra --options'},
+            'extra_borg_options': {'create': '--extra --options "value with space"'},
         },
         patterns=[Pattern('foo'), Pattern('bar')],
         local_borg_version='1.2.3',
@@ -938,14 +874,16 @@ def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
         borgmatic_runtime_directory='/run/borgmatic',
     )
 
-    assert create_flags == ('borg', 'create', '--extra', '--options')
+    assert create_flags == ('borg', 'create', '--extra', '--options', 'value with space')
     assert create_positional_arguments == REPO_ARCHIVE
     assert not pattern_file
 
 
 def test_make_base_create_command_with_non_existent_directory_and_source_directories_must_exist_raises():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
-    flexmock(module).should_receive('check_all_root_patterns_exist').and_raise(ValueError)
+    flexmock(module.borgmatic.borg.pattern).should_receive(
+        'check_all_root_patterns_exist'
+    ).and_raise(ValueError)
 
     with pytest.raises(ValueError):
         module.make_base_create_command(
@@ -1537,6 +1475,41 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_flag():
     assert json_output == '[]'
 
 
+def test_create_archive_with_comment_calls_borg_with_comment_flag():
+    flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
+    flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
+    flexmock(module).should_receive('make_base_create_command').and_return(
+        (('borg', 'create', '--comment', 'a comment'), REPO_ARCHIVE, flexmock())
+    )
+    flexmock(module.environment).should_receive('make_environment')
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module).should_receive('execute_command').with_args(
+        ('borg', 'create', '--comment', 'a comment') + REPO_ARCHIVE,
+        output_log_level=logging.INFO,
+        output_file=None,
+        borg_local_path='borg',
+        borg_exit_codes=None,
+        working_directory=None,
+        environment=None,
+    )
+
+    module.create_archive(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/borgmatic/run',
+        json=False,
+        comment='a comment',
+    )
+
+
 def test_create_archive_calls_borg_with_working_directory():
     flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
     flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -1571,22 +1544,3 @@ def test_create_archive_calls_borg_with_working_directory():
         global_arguments=flexmock(),
         borgmatic_runtime_directory='/borgmatic/run',
     )
-
-
-def test_check_all_root_patterns_exist_with_existent_pattern_path_does_not_raise():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-
-    module.check_all_root_patterns_exist([Pattern('foo')])
-
-
-def test_check_all_root_patterns_exist_with_non_root_pattern_skips_existence_check():
-    flexmock(module.os.path).should_receive('exists').never()
-
-    module.check_all_root_patterns_exist([Pattern('foo', Pattern_type.INCLUDE)])
-
-
-def test_check_all_root_patterns_exist_with_non_existent_pattern_path_raises():
-    flexmock(module.os.path).should_receive('exists').and_return(False)
-
-    with pytest.raises(ValueError):
-        module.check_all_root_patterns_exist([Pattern('foo')])

tests/unit/borg/test_environment.py

@@ -90,8 +90,11 @@ def test_make_environment_without_configuration_sets_certain_environment_variabl
     assert environment == {
         'USER': 'root',
         'BORG_EXIT_CODES': 'modern',
-        'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'no',
-        'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'no',
+        'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'NO',
+        'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'NO',
+        'BORG_USE_CHUNKS_ARCHIVE': 'NO',
+        'BORG_DEBUG_PASSPHRASE': 'NO',
+        'BORG_DISPLAY_PASSPHRASE': 'NO',
     }
 
 
@@ -101,6 +104,9 @@ def test_make_environment_without_configuration_passes_through_default_environme
             'USER': 'root',
             'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
             'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+            'BORG_USE_CHUNKS_ARCHIVE': 'yup',
+            'BORG_DEBUG_PASSPHRASE': 'nah',
+            'BORG_DISPLAY_PASSPHRASE': 'yup',
         }
     )
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -113,11 +119,14 @@ def test_make_environment_without_configuration_passes_through_default_environme
         'USER': 'root',
         'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
         'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+        'BORG_USE_CHUNKS_ARCHIVE': 'yup',
+        'BORG_DEBUG_PASSPHRASE': 'nah',
+        'BORG_DISPLAY_PASSPHRASE': 'yup',
         'BORG_EXIT_CODES': 'modern',
     }
 
 
-def test_make_environment_with_relocated_repo_access_true_should_set_environment_yes():
+def test_make_environment_with_relocated_repo_access_true_should_set_environment_YES():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
@@ -125,10 +134,10 @@ def test_make_environment_with_relocated_repo_access_true_should_set_environment
     flexmock(module.os).should_receive('pipe').never()
     environment = module.make_environment({'relocated_repo_access_is_ok': True})
 
-    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'yes'
+    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'YES'
 
 
-def test_make_environment_with_relocated_repo_access_false_should_set_environment_no():
+def test_make_environment_with_relocated_repo_access_false_should_set_environment_NO():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
@@ -136,7 +145,7 @@ def test_make_environment_with_relocated_repo_access_false_should_set_environmen
     flexmock(module.os).should_receive('pipe').never()
     environment = module.make_environment({'relocated_repo_access_is_ok': False})
 
-    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'no'
+    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'NO'
 
 
 def test_make_environment_check_i_know_what_i_am_doing_true_should_set_environment_YES():
@@ -161,6 +170,50 @@ def test_make_environment_check_i_know_what_i_am_doing_false_should_set_environm
     assert environment.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING') == 'NO'
 
 
+def test_make_environment_debug_passphrase_true_should_set_environment_YES():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'debug_passphrase': True})
+
+    assert environment.get('BORG_DEBUG_PASSPHRASE') == 'YES'
+
+
+def test_make_environment_debug_passphrase_false_should_set_environment_NO():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'debug_passphrase': False})
+
+    assert environment.get('BORG_DEBUG_PASSPHRASE') == 'NO'
+
+
+def test_make_environment_display_passphrase_true_should_set_environment_YES():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'display_passphrase': True})
+
+    assert environment.get('BORG_DISPLAY_PASSPHRASE') == 'YES'
+
+
+def test_make_environment_display_passphrase_false_should_set_environment_NO():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'display_passphrase': False})
+
+    assert environment.get('BORG_DISPLAY_PASSPHRASE') == 'NO'
+
+
 def test_make_environment_with_integer_variable_value():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -170,3 +223,17 @@ def test_make_environment_with_integer_variable_value():
     environment = module.make_environment({'borg_files_cache_ttl': 40})
 
     assert environment.get('BORG_FILES_CACHE_TTL') == '40'
+
+
+def test_make_environment_with_use_chunks_archive_should_set_correct_environment_value():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+
+    environment = module.make_environment({'use_chunks_archive': True})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'YES'
+
+    environment = module.make_environment({'use_chunks_archive': False})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'NO'

tests/unit/borg/test_flags.py

@@ -327,3 +327,100 @@ def test_omit_flag_and_value_without_flag_present_passes_through_arguments():
         'create',
         '--other',
     )
+
+
+def test_make_exclude_flags_includes_exclude_caches_when_true_in_config():
+    exclude_flags = module.make_exclude_flags(config={'exclude_caches': True})
+
+    assert exclude_flags == ('--exclude-caches',)
+
+
+def test_make_exclude_flags_does_not_include_exclude_caches_when_false_in_config():
+    exclude_flags = module.make_exclude_flags(config={'exclude_caches': False})
+
+    assert exclude_flags == ()
+
+
+def test_make_exclude_flags_includes_exclude_if_present_when_in_config():
+    exclude_flags = module.make_exclude_flags(
+        config={'exclude_if_present': ['exclude_me', 'also_me']}
+    )
+
+    assert exclude_flags == (
+        '--exclude-if-present',
+        'exclude_me',
+        '--exclude-if-present',
+        'also_me',
+    )
+
+
+def test_make_exclude_flags_includes_keep_exclude_tags_when_true_in_config():
+    exclude_flags = module.make_exclude_flags(config={'keep_exclude_tags': True})
+
+    assert exclude_flags == ('--keep-exclude-tags',)
+
+
+def test_make_exclude_flags_does_not_include_keep_exclude_tags_when_false_in_config():
+    exclude_flags = module.make_exclude_flags(config={'keep_exclude_tags': False})
+
+    assert exclude_flags == ()
+
+
+def test_make_exclude_flags_includes_exclude_nodump_when_true_in_config():
+    exclude_flags = module.make_exclude_flags(config={'exclude_nodump': True})
+
+    assert exclude_flags == ('--exclude-nodump',)
+
+
+def test_make_exclude_flags_does_not_include_exclude_nodump_when_false_in_config():
+    exclude_flags = module.make_exclude_flags(config={'exclude_nodump': False})
+
+    assert exclude_flags == ()
+
+
+def test_make_exclude_flags_is_empty_when_config_has_no_excludes():
+    exclude_flags = module.make_exclude_flags(config={})
+
+    assert exclude_flags == ()
+
+
+def test_make_list_filter_flags_with_debug_and_feature_available_includes_plus_and_minus():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
+    flexmock(module.feature).should_receive('available').and_return(True)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME+-'
+
+
+def test_make_list_filter_flags_with_info_and_feature_available_omits_plus_and_minus():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
+    flexmock(module.feature).should_receive('available').and_return(True)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME'
+
+
+def test_make_list_filter_flags_with_debug_and_feature_available_and_dry_run_includes_plus_and_minus():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
+    flexmock(module.feature).should_receive('available').and_return(True)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=True) == 'AME+-'
+
+
+def test_make_list_filter_flags_with_info_and_feature_available_and_dry_run_includes_plus_and_minus():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
+    flexmock(module.feature).should_receive('available').and_return(True)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=True) == 'AME+-'
+
+
+def test_make_list_filter_flags_with_debug_and_feature_not_available_includes_x():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(True)
+    flexmock(module.feature).should_receive('available').and_return(False)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AMEx-'
+
+
+def test_make_list_filter_flags_with_info_and_feature_not_available_omits_x():
+    flexmock(module.logger).should_receive('isEnabledFor').and_return(False)
+    flexmock(module.feature).should_receive('available').and_return(False)
+
+    assert module.make_list_filter_flags(local_borg_version=flexmock(), dry_run=False) == 'AME-'

tests/unit/borg/test_pattern.py

@@ -0,0 +1,52 @@
+import pytest
+from flexmock import flexmock
+
+from borgmatic.borg import pattern as module
+from borgmatic.borg.pattern import Pattern, Pattern_style, Pattern_type
+
+
+def test_write_patterns_file_writes_pattern_lines():
+    temporary_file = flexmock(name='filename', flush=lambda: None)
+    temporary_file.should_receive('write').with_args('R /foo\n+ sh:/foo/bar')
+    flexmock(module.tempfile).should_receive('NamedTemporaryFile').and_return(temporary_file)
+
+    module.write_patterns_file(
+        [Pattern('/foo'), Pattern('/foo/bar', Pattern_type.INCLUDE, Pattern_style.SHELL)],
+        borgmatic_runtime_directory='/run/user/0',
+    )
+
+
+def test_write_patterns_file_with_empty_exclude_patterns_does_not_raise():
+    module.write_patterns_file([], borgmatic_runtime_directory='/run/user/0')
+
+
+def test_write_patterns_file_appends_to_existing():
+    patterns_file = flexmock(name='filename', flush=lambda: None)
+    patterns_file.should_receive('write').with_args('\n')
+    patterns_file.should_receive('write').with_args('R /foo\n+ /foo/bar')
+    flexmock(module.tempfile).should_receive('NamedTemporaryFile').never()
+
+    module.write_patterns_file(
+        [Pattern('/foo'), Pattern('/foo/bar', Pattern_type.INCLUDE)],
+        borgmatic_runtime_directory='/run/user/0',
+        patterns_file=patterns_file,
+    )
+
+
+def test_check_all_root_patterns_exist_with_existent_pattern_path_does_not_raise():
+    flexmock(module.os.path).should_receive('exists').and_return(True)
+
+    module.check_all_root_patterns_exist([Pattern('foo')])
+
+
+def test_check_all_root_patterns_exist_with_non_root_pattern_skips_existence_check():
+    flexmock(module.os.path).should_receive('exists').never()
+
+    module.check_all_root_patterns_exist([Pattern('foo', Pattern_type.INCLUDE)])
+
+
+def test_check_all_root_patterns_exist_with_non_existent_pattern_path_raises():
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+
+    with pytest.raises(ValueError):
+        module.check_all_root_patterns_exist([Pattern('foo')])

tests/unit/borg/test_prune.py

@@ -43,6 +43,28 @@ def test_make_prune_flags_returns_flags_from_config():
     assert result == BASE_PRUNE_FLAGS
 
 
+def test_make_prune_flags_with_keep_13weekly_and_keep_3monthly():
+    config = {
+        'keep_13weekly': 4,
+        'keep_3monthly': 5,
+    }
+    flexmock(module.feature).should_receive('available').and_return(True)
+    flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
+
+    result = module.make_prune_flags(
+        config, flexmock(match_archives=None), local_borg_version='1.2.3'
+    )
+
+    expected = (
+        '--keep-13weekly',
+        '4',
+        '--keep-3monthly',
+        '5',
+    )
+
+    assert result == expected
+
+
 def test_make_prune_flags_accepts_prefix_with_placeholders():
     config = {
         'keep_daily': 1,
@@ -450,13 +472,15 @@ def test_prune_archives_with_extra_borg_options_calls_borg_with_extra_options():
     flexmock(module.feature).should_receive('available').with_args(
         module.feature.Feature.NO_PRUNE_STATS, '1.2.3'
     ).and_return(False)
-    insert_execute_command_mock(PRUNE_COMMAND + ('--extra', '--options', 'repo'), logging.INFO)
+    insert_execute_command_mock(
+        PRUNE_COMMAND + ('--extra', '--options', 'value with space', 'repo'), logging.INFO
+    )
 
     prune_arguments = flexmock(statistics=False, list_details=False)
     module.prune_archives(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'prune': '--extra --options'}},
+        config={'extra_borg_options': {'prune': '--extra --options "value with space"'}},
         local_borg_version='1.2.3',
         global_arguments=flexmock(),
         prune_arguments=prune_arguments,

tests/unit/borg/test_recreate.py

@@ -21,9 +21,9 @@ def insert_execute_command_mock(command, working_directory=None, borg_exit_codes
 
 
 def test_recreate_archive_dry_run_skips_execution():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -59,9 +59,9 @@ def test_recreate_archive_dry_run_skips_execution():
 
 
 def test_recreate_calls_borg_with_required_flags():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -94,9 +94,9 @@ def test_recreate_calls_borg_with_required_flags():
 
 
 def test_recreate_with_remote_path():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -129,9 +129,9 @@ def test_recreate_with_remote_path():
 
 
 def test_recreate_with_lock_wait():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -163,9 +163,9 @@ def test_recreate_with_lock_wait():
 
 
 def test_recreate_with_log_info():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -199,9 +199,9 @@ def test_recreate_with_log_info():
 
 
 def test_recreate_with_log_debug():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -234,9 +234,9 @@ def test_recreate_with_log_debug():
 
 
 def test_recreate_with_log_json():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -268,8 +268,8 @@ def test_recreate_with_log_json():
 
 
 def test_recreate_with_list_config_calls_borg_with_list_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -280,7 +280,9 @@ def test_recreate_with_list_config_calls_borg_with_list_flag():
             'repo',
         )
     )
-    flexmock(module).should_receive('make_list_filter_flags').and_return('AME+-')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return(
+        'AME+-'
+    )
     insert_execute_command_mock(
         ('borg', 'recreate', '--list', '--filter', 'AME+-', '--repo', 'repo')
     )
@@ -304,8 +306,8 @@ def test_recreate_with_list_config_calls_borg_with_list_flag():
 
 
 def test_recreate_with_patterns_from_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -341,8 +343,8 @@ def test_recreate_with_patterns_from_flag():
 
 
 def test_recreate_with_exclude_flags():
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -353,7 +355,9 @@ def test_recreate_with_exclude_flags():
             'repo',
         )
     )
-    flexmock(module).should_receive('make_exclude_flags').and_return(('--exclude', 'pattern'))
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(
+        ('--exclude', 'pattern')
+    )
     insert_execute_command_mock(('borg', 'recreate', '--exclude', 'pattern', '--repo', 'repo'))
 
     module.recreate_archive(
@@ -375,9 +379,9 @@ def test_recreate_with_exclude_flags():
 
 
 def test_recreate_with_target_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -409,9 +413,9 @@ def test_recreate_with_target_flag():
 
 
 def test_recreate_with_comment_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -445,9 +449,9 @@ def test_recreate_with_comment_flag():
 
 
 def test_recreate_with_timestamp_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -481,9 +485,9 @@ def test_recreate_with_timestamp_flag():
 
 
 def test_recreate_with_compression_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -515,9 +519,9 @@ def test_recreate_with_compression_flag():
 
 
 def test_recreate_with_chunker_params_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -551,9 +555,9 @@ def test_recreate_with_chunker_params_flag():
 
 
 def test_recreate_with_recompress_flag():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -585,9 +589,9 @@ def test_recreate_with_recompress_flag():
 
 
 def test_recreate_with_match_archives_star():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -619,9 +623,9 @@ def test_recreate_with_match_archives_star():
 
 
 def test_recreate_with_match_archives_regex():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -653,9 +657,9 @@ def test_recreate_with_match_archives_regex():
 
 
 def test_recreate_with_match_archives_shell():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').and_return(())
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(True)
     flexmock(module.borgmatic.borg.flags).should_receive(
@@ -687,9 +691,9 @@ def test_recreate_with_match_archives_shell():
 
 
 def test_recreate_with_match_archives_and_feature_available_calls_borg_with_match_archives():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').with_args(
         'foo-*', None, '1.2.3'
     ).and_return(('--match-archives', 'foo-*'))
@@ -719,9 +723,9 @@ def test_recreate_with_match_archives_and_feature_available_calls_borg_with_matc
 
 
 def test_recreate_with_archives_flag_and_feature_available_calls_borg_with_match_archives():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').with_args(
         'archive', None, '1.2.3'
     ).and_return(('--match-archives', 'archive'))
@@ -753,9 +757,9 @@ def test_recreate_with_archives_flag_and_feature_available_calls_borg_with_match
 
 
 def test_recreate_with_match_archives_and_feature_not_available_calls_borg_without_match_archives():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').never()
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(False)
     flexmock(module.borgmatic.borg.flags).should_receive('make_repository_flags').and_return(
@@ -783,9 +787,9 @@ def test_recreate_with_match_archives_and_feature_not_available_calls_borg_witho
 
 
 def test_recreate_with_archives_flags_and_feature_not_available_calls_borg_with_combined_repo_and_archive():
-    flexmock(module.borgmatic.borg.create).should_receive('make_exclude_flags').and_return(())
-    flexmock(module.borgmatic.borg.create).should_receive('write_patterns_file').and_return(None)
-    flexmock(module.borgmatic.borg.create).should_receive('make_list_filter_flags').and_return('')
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('')
     flexmock(module.borgmatic.borg.flags).should_receive('make_match_archives_flags').never()
     flexmock(module.borgmatic.borg.feature).should_receive('available').and_return(False)
     flexmock(module.borgmatic.borg.flags).should_receive(

tests/unit/borg/test_rename.py

@@ -0,0 +1,37 @@
+import logging
+
+from flexmock import flexmock
+
+from borgmatic.borg import rename as module
+
+
+def test_rename_archive_calls_borg_rename():
+    environment = flexmock()
+
+    # Note: make_rename_command is tested as integration test.
+    flexmock(module).should_receive('make_rename_command').and_return(('borg', 'fake-command'))
+    flexmock(module.borgmatic.borg.environment).should_receive('make_environment').and_return(
+        environment
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        '/working/dir'
+    )
+    flexmock(module.borgmatic.execute).should_receive('execute_command').with_args(
+        ('borg', 'fake-command'),
+        output_log_level=logging.INFO,
+        environment=environment,
+        working_directory='/working/dir',
+        borg_local_path='borg',
+        borg_exit_codes=None,
+    ).once()
+
+    module.rename_archive(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )

tests/unit/borg/test_repo_create.py

@@ -484,7 +484,7 @@ def test_create_repository_with_umask_calls_borg_with_umask_flag():
 def test_create_repository_with_extra_borg_options_calls_borg_with_extra_options():
     insert_repo_info_command_not_found_mock()
     insert_repo_create_command_mock(
-        REPO_CREATE_COMMAND + ('--extra', '--options', '--repo', 'repo')
+        REPO_CREATE_COMMAND + ('--extra', '--options', 'value with space', '--repo', 'repo')
     )
     flexmock(module.feature).should_receive('available').and_return(True)
     flexmock(module.flags).should_receive('make_repository_flags').and_return(
@@ -497,7 +497,7 @@ def test_create_repository_with_extra_borg_options_calls_borg_with_extra_options
     module.create_repository(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'repo-create': '--extra --options'}},
+        config={'extra_borg_options': {'repo-create': '--extra --options "value with space"'}},
         local_borg_version='2.3.4',
         global_arguments=flexmock(),
         encryption_mode='repokey',

tests/unit/borg/test_repo_list.py

@@ -1,4 +1,5 @@
 import argparse
+import json
 import logging
 
 import pytest
@@ -11,7 +12,7 @@ from ..test_verbosity import insert_logging_mock
 BORG_LIST_LATEST_ARGUMENTS = (
     '--last',
     '1',
-    '--short',
+    '--json',
     'repo',
 )
 
@@ -31,8 +32,43 @@ def test_resolve_archive_name_passes_through_non_latest_archive_name():
     )
 
 
-def test_resolve_archive_name_calls_borg_with_flags():
+def test_resolve_archive_name_calls_get_latest_archive():
     expected_archive = 'archive-name'
+
+    repository_path = flexmock()
+    config = flexmock()
+    local_borg_version = flexmock()
+    global_arguments = flexmock()
+    local_path = flexmock()
+    remote_path = flexmock()
+
+    flexmock(module.environment).should_receive('make_environment')
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module).should_receive('get_latest_archive').with_args(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    ).and_return({'name': expected_archive})
+
+    assert (
+        module.resolve_archive_name(
+            repository_path,
+            'latest',
+            config,
+            local_borg_version,
+            global_arguments,
+            local_path,
+            remote_path,
+        )
+        == expected_archive
+    )
+
+
+def test_get_latest_archive_calls_borg_with_flags():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -41,12 +77,11 @@ def test_resolve_archive_name_calls_borg_with_flags():
         borg_exit_codes=None,
         environment=None,
         working_directory=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -55,8 +90,8 @@ def test_resolve_archive_name_calls_borg_with_flags():
     )
 
 
-def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_log_info_calls_borg_without_info_flag():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -65,13 +100,12 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
     insert_logging_mock(logging.INFO)
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -80,8 +114,8 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
     )
 
 
-def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_log_debug_calls_borg_without_debug_flag():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -90,13 +124,12 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
     insert_logging_mock(logging.DEBUG)
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -105,8 +138,8 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
     )
 
 
-def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_local_path_calls_borg_via_local_path():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -115,12 +148,11 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
         working_directory=None,
         borg_local_path='borg1',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -130,8 +162,8 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
     )
 
 
-def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_exit_codes_calls_borg_using_them():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     borg_exit_codes = flexmock()
@@ -141,12 +173,11 @@ def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=borg_exit_codes,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'borg_exit_codes': borg_exit_codes},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -155,8 +186,8 @@ def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
     )
 
 
-def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_remote_path_calls_borg_with_remote_path_flags():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -165,12 +196,11 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -180,8 +210,8 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
     )
 
 
-def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_umask_calls_borg_with_umask_flags():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -190,12 +220,11 @@ def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'umask': '077'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -204,7 +233,7 @@ def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
     )
 
 
-def test_resolve_archive_name_without_archives_raises():
+def test_get_latest_archive_without_archives_raises():
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -213,20 +242,19 @@ def test_resolve_archive_name_without_archives_raises():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return('')
+    ).and_return(json.dumps({'archives': []}))
 
     with pytest.raises(ValueError):
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
         )
 
 
-def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_log_json_calls_borg_with_log_json_flags():
+    expected_archive = {'name': 'archive-name'}
 
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
@@ -236,12 +264,11 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'log_json': True},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -250,8 +277,8 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
     )
 
 
-def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_with_lock_wait_calls_borg_with_lock_wait_flags():
+    expected_archive = {'name': 'archive-name'}
 
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
@@ -261,12 +288,11 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'lock_wait': 'okay'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -275,8 +301,8 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
     )
 
 
-def test_resolve_archive_name_calls_borg_with_working_directory():
-    expected_archive = 'archive-name'
+def test_get_latest_archive_calls_borg_with_working_directory():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         '/working/dir',
@@ -287,12 +313,11 @@ def test_resolve_archive_name_calls_borg_with_working_directory():
         borg_exit_codes=None,
         environment=None,
         working_directory='/working/dir',
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'working_directory': '/working/dir'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),

tests/unit/commands/test_borgmatic.py

@@ -28,6 +28,7 @@ def test_get_skip_actions_uses_config_and_arguments(config, arguments, expected_
 
 
 def test_monitoring_hooks_with_monioring_disabled_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(module.DISABLED)
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -41,6 +42,7 @@ def test_monitoring_hooks_with_monioring_disabled_bails():
 
 
 def test_monitoring_hooks_with_non_primary_action_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -54,6 +56,7 @@ def test_monitoring_hooks_with_non_primary_action_bails():
 
 
 def test_monitoring_hooks_pings_monitors():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -117,6 +120,7 @@ def test_monitoring_hooks_pings_monitors():
 
 
 def test_monitoring_hooks_with_start_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -172,6 +176,7 @@ def test_monitoring_hooks_with_start_ping_error_raises():
 
 
 def test_monitoring_hooks_with_log_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -227,6 +232,7 @@ def test_monitoring_hooks_with_log_ping_error_raises():
 
 
 def test_monitoring_hooks_with_finish_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -281,7 +287,8 @@ def test_monitoring_hooks_with_finish_ping_error_raises():
             pass
 
 
-def test_monitoring_with_wrapped_code_error_pings_fail():
+def test_monitoring_hooks_with_wrapped_code_error_pings_fail():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -345,7 +352,8 @@ def test_monitoring_with_wrapped_code_error_pings_fail():
             raise OSError()
 
 
-def test_monitoring_with_fail_ping_error_raise_original_error():
+def test_monitoring_hooks_with_fail_ping_error_raise_original_error():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -465,8 +473,10 @@ def test_run_configuration_logs_actions_error():
     flexmock(module).should_receive('Monitoring_hooks').and_return(flexmock())
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
-    expected_results = [flexmock()]
-    flexmock(module).should_receive('log_error_records').and_return(expected_results)
+    expected_results = [flexmock(), flexmock()]
+    flexmock(module).should_receive('log_error_records').and_return(
+        expected_results[:1]
+    ).and_return(expected_results[1:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     flexmock(module.command).should_receive('filter_hooks')
@@ -532,7 +542,7 @@ def test_run_configuration_logs_on_error_hook_error():
     expected_results = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').and_return(
         expected_results[:1]
-    ).and_return(expected_results[1:])
+    ).and_return(expected_results[1:2]).and_return(expected_results[2:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     config = {'repositories': [{'path': 'foo'}]}
@@ -601,8 +611,10 @@ def test_run_configuration_bails_for_on_error_hook_soft_failure():
     error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks').and_raise(error)
-    expected_results = [flexmock()]
-    flexmock(module).should_receive('log_error_records').and_return(expected_results)
+    expected_results = [flexmock(), flexmock()]
+    flexmock(module).should_receive('log_error_records').and_return(
+        expected_results[:1]
+    ).and_return(expected_results[1:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     config = {'repositories': [{'path': 'foo'}]}
@@ -654,11 +666,14 @@ def test_run_configuration_retries_hard_error():
         levelno=logging.WARNING,
         log_command_error_output=True,
     ).and_return([flexmock()])
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository',
         OSError,
-    ).and_return(error_logs)
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {'repositories': [{'path': 'foo'}], 'retries': 1}
@@ -680,13 +695,16 @@ def test_run_configuration_retries_repositories_in_order():
     flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError).times(2)
-    expected_results = [flexmock(), flexmock()]
+    expected_results = [flexmock(), flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
     ).and_return(expected_results[:1]).ordered()
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(expected_results[1:]).ordered()
+    ).and_return(expected_results[1:2]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(expected_results[2:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
@@ -728,6 +746,10 @@ def test_run_configuration_retries_round_robin():
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
     ).and_return(bar_error_logs).ordered()
+    config_error_logs = [flexmock()]
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(config_error_logs).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -741,7 +763,7 @@ def test_run_configuration_retries_round_robin():
 
     results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
 
-    assert results == foo_error_logs + bar_error_logs
+    assert results == foo_error_logs + bar_error_logs + config_error_logs
 
 
 def test_run_configuration_with_one_retry():
@@ -766,10 +788,13 @@ def test_run_configuration_with_one_retry():
         levelno=logging.WARNING,
         log_command_error_output=True,
     ).and_return(flexmock()).ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -818,10 +843,13 @@ def test_run_configuration_with_retry_wait_does_backoff_after_each_retry():
     ).and_return([flexmock()]).ordered()
 
     flexmock(time).should_receive('sleep').with_args(30).and_return().ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -867,10 +895,13 @@ def test_run_configuration_with_multiple_repositories_retries_with_timeout():
 
     # Sleep before retrying bar (and failing)
     flexmock(time).should_receive('sleep').with_args(10).and_return().ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -891,6 +922,9 @@ def test_run_configuration_with_multiple_repositories_retries_with_timeout():
 def test_run_actions_runs_repo_create():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.repo_create).should_receive('run_repo_create').once()
 
@@ -914,6 +948,9 @@ def test_run_actions_runs_repo_create():
 def test_run_actions_adds_label_file_to_hook_context():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.create).should_receive('run_create').with_args(
@@ -947,6 +984,9 @@ def test_run_actions_adds_label_file_to_hook_context():
 def test_run_actions_adds_log_file_to_hook_context():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.create).should_receive('run_create').with_args(
@@ -980,6 +1020,9 @@ def test_run_actions_adds_log_file_to_hook_context():
 def test_run_actions_runs_transfer():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.transfer).should_receive('run_transfer').once()
 
@@ -1000,6 +1043,9 @@ def test_run_actions_runs_transfer():
 def test_run_actions_runs_create():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.create).should_receive('run_create').and_yield(expected).once()
@@ -1019,10 +1065,26 @@ def test_run_actions_runs_create():
     assert result == (expected,)
 
 
-def test_run_actions_with_skip_actions_skips_create():
+def test_run_actions_with_skip_actions_does_not_run_action_or_action_command_hooks():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return(['create'])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
+    flexmock(module.command).should_receive('Before_after_hooks').with_args(
+        command_hooks=object,
+        before_after='action',
+        umask=object,
+        working_directory=object,
+        dry_run=object,
+        action_names=object,
+        configuration_filename=object,
+        repository_label=object,
+        log_file=object,
+        repositories=object,
+        repository=object,
+    ).never()
     flexmock(borgmatic.actions.create).should_receive('run_create').never()
 
     tuple(
@@ -1042,6 +1104,9 @@ def test_run_actions_with_skip_actions_skips_create():
 def test_run_actions_runs_recreate():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
 
     flexmock(borgmatic.actions.recreate).should_receive('run_recreate').once()
@@ -1060,29 +1125,12 @@ def test_run_actions_runs_recreate():
     )
 
 
-def test_run_actions_with_skip_actions_skips_recreate():
-    flexmock(module).should_receive('add_custom_log_levels')
-    flexmock(module).should_receive('get_skip_actions').and_return(['recreate'])
-    flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
-    flexmock(borgmatic.actions.recreate).should_receive('run_recreate').never()
-
-    tuple(
-        module.run_actions(
-            arguments={'global': flexmock(dry_run=False), 'recreate': flexmock()},
-            config_filename=flexmock(),
-            config={'repositories': [], 'skip_actions': ['recreate']},
-            config_paths=[],
-            local_path=flexmock(),
-            remote_path=flexmock(),
-            local_borg_version=flexmock(),
-            repository={'path': 'repo'},
-        )
-    )
-
-
 def test_run_actions_runs_prune():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.prune).should_receive('run_prune').once()
 
@@ -1100,29 +1148,12 @@ def test_run_actions_runs_prune():
     )
 
 
-def test_run_actions_with_skip_actions_skips_prune():
-    flexmock(module).should_receive('add_custom_log_levels')
-    flexmock(module).should_receive('get_skip_actions').and_return(['prune'])
-    flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
-    flexmock(borgmatic.actions.prune).should_receive('run_prune').never()
-
-    tuple(
-        module.run_actions(
-            arguments={'global': flexmock(dry_run=False), 'prune': flexmock()},
-            config_filename=flexmock(),
-            config={'repositories': [], 'skip_actions': ['prune']},
-            config_paths=[],
-            local_path=flexmock(),
-            remote_path=flexmock(),
-            local_borg_version=flexmock(),
-            repository={'path': 'repo'},
-        )
-    )
-
-
 def test_run_actions_runs_compact():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.compact).should_receive('run_compact').once()
 
@@ -1140,29 +1171,12 @@ def test_run_actions_runs_compact():
     )
 
 
-def test_run_actions_with_skip_actions_skips_compact():
-    flexmock(module).should_receive('add_custom_log_levels')
-    flexmock(module).should_receive('get_skip_actions').and_return(['compact'])
-    flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
-    flexmock(borgmatic.actions.compact).should_receive('run_compact').never()
-
-    tuple(
-        module.run_actions(
-            arguments={'global': flexmock(dry_run=False), 'compact': flexmock()},
-            config_filename=flexmock(),
-            config={'repositories': [], 'skip_actions': ['compact']},
-            config_paths=[],
-            local_path=flexmock(),
-            remote_path=flexmock(),
-            local_borg_version=flexmock(),
-            repository={'path': 'repo'},
-        )
-    )
-
-
 def test_run_actions_runs_check_when_repository_enabled_for_checks():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(True)
     flexmock(borgmatic.actions.check).should_receive('run_check').once()
@@ -1184,6 +1198,9 @@ def test_run_actions_runs_check_when_repository_enabled_for_checks():
 def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(False)
     flexmock(borgmatic.actions.check).should_receive('run_check').never()
@@ -1202,30 +1219,12 @@ def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
     )
 
 
-def test_run_actions_with_skip_actions_skips_check():
-    flexmock(module).should_receive('add_custom_log_levels')
-    flexmock(module).should_receive('get_skip_actions').and_return(['check'])
-    flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
-    flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(True)
-    flexmock(borgmatic.actions.check).should_receive('run_check').never()
-
-    tuple(
-        module.run_actions(
-            arguments={'global': flexmock(dry_run=False), 'check': flexmock()},
-            config_filename=flexmock(),
-            config={'repositories': [], 'skip_actions': ['check']},
-            config_paths=[],
-            local_path=flexmock(),
-            remote_path=flexmock(),
-            local_borg_version=flexmock(),
-            repository={'path': 'repo'},
-        )
-    )
-
-
 def test_run_actions_runs_extract():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.extract).should_receive('run_extract').once()
 
@@ -1246,6 +1245,9 @@ def test_run_actions_runs_extract():
 def test_run_actions_runs_export_tar():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.export_tar).should_receive('run_export_tar').once()
 
@@ -1266,6 +1268,9 @@ def test_run_actions_runs_export_tar():
 def test_run_actions_runs_mount():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.mount).should_receive('run_mount').once()
 
@@ -1286,6 +1291,9 @@ def test_run_actions_runs_mount():
 def test_run_actions_runs_restore():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.restore).should_receive('run_restore').once()
 
@@ -1306,6 +1314,9 @@ def test_run_actions_runs_restore():
 def test_run_actions_runs_repo_list():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.repo_list).should_receive('run_repo_list').and_yield(expected).once()
@@ -1328,6 +1339,9 @@ def test_run_actions_runs_repo_list():
 def test_run_actions_runs_list():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.list).should_receive('run_list').and_yield(expected).once()
@@ -1350,6 +1364,9 @@ def test_run_actions_runs_list():
 def test_run_actions_runs_repo_info():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.repo_info).should_receive('run_repo_info').and_yield(expected).once()
@@ -1372,6 +1389,9 @@ def test_run_actions_runs_repo_info():
 def test_run_actions_runs_info():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     expected = flexmock()
     flexmock(borgmatic.actions.info).should_receive('run_info').and_yield(expected).once()
@@ -1394,6 +1414,9 @@ def test_run_actions_runs_info():
 def test_run_actions_runs_break_lock():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.break_lock).should_receive('run_break_lock').once()
 
@@ -1414,6 +1437,9 @@ def test_run_actions_runs_break_lock():
 def test_run_actions_runs_export_key():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.export_key).should_receive('run_export_key').once()
 
@@ -1434,6 +1460,9 @@ def test_run_actions_runs_export_key():
 def test_run_actions_runs_import_key():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.import_key).should_receive('run_import_key').once()
 
@@ -1454,6 +1483,9 @@ def test_run_actions_runs_import_key():
 def test_run_actions_runs_change_passphrase():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.change_passphrase).should_receive('run_change_passphrase').once()
 
@@ -1477,6 +1509,9 @@ def test_run_actions_runs_change_passphrase():
 def test_run_actions_runs_delete():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.delete).should_receive('run_delete').once()
 
@@ -1497,6 +1532,9 @@ def test_run_actions_runs_delete():
 def test_run_actions_runs_repo_delete():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.repo_delete).should_receive('run_repo_delete').once()
 
@@ -1520,6 +1558,9 @@ def test_run_actions_runs_repo_delete():
 def test_run_actions_runs_borg():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.borg).should_receive('run_borg').once()
 
@@ -1540,6 +1581,9 @@ def test_run_actions_runs_borg():
 def test_run_actions_runs_multiple_actions_in_argument_order():
     flexmock(module).should_receive('add_custom_log_levels')
     flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(borgmatic.actions.borg).should_receive('run_borg').once().ordered()
     flexmock(borgmatic.actions.restore).should_receive('run_restore').once().ordered()
@@ -1562,6 +1606,50 @@ def test_run_actions_runs_multiple_actions_in_argument_order():
     )
 
 
+def test_run_actions_runs_action_hooks_for_one_action_at_a_time():
+    flexmock(module).should_receive('add_custom_log_levels')
+    flexmock(module).should_receive('get_skip_actions').and_return([])
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
+
+    for action_name in ('borg', 'restore'):
+        flexmock(module.command).should_receive('Before_after_hooks').with_args(
+            command_hooks=object,
+            before_after='action',
+            umask=object,
+            working_directory=object,
+            dry_run=object,
+            action_names=(action_name,),
+            configuration_filename=object,
+            repository_label=object,
+            log_file=object,
+            repositories=object,
+            repository=object,
+        ).and_return(flexmock()).once()
+
+    flexmock(borgmatic.actions.borg).should_receive('run_borg')
+    flexmock(borgmatic.actions.restore).should_receive('run_restore')
+
+    tuple(
+        module.run_actions(
+            arguments={
+                'global': flexmock(dry_run=False),
+                'borg': flexmock(),
+                'restore': flexmock(),
+            },
+            config_filename=flexmock(),
+            config={'repositories': []},
+            config_paths=[],
+            local_path=flexmock(),
+            remote_path=flexmock(),
+            local_borg_version=flexmock(),
+            repository={'path': 'repo'},
+        )
+    )
+
+
 @pytest.mark.parametrize(
     'resolve_env',
     ((True, False),),
@@ -1852,7 +1940,12 @@ def test_collect_highlander_action_summary_logs_error_on_run_validate_failure():
 
 def test_collect_configuration_run_summary_logs_info_for_success():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -1871,9 +1964,17 @@ def test_collect_configuration_run_summary_logs_info_for_success():
 
 
 def test_collect_configuration_run_summary_executes_hooks_for_create():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
-    flexmock(module.command).should_receive('execute_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook])
+    flexmock(module.command).should_receive('execute_hooks').twice()
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
     arguments = {
@@ -1883,7 +1984,7 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -1893,9 +1994,51 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
     assert {log.levelno for log in logs} == {logging.INFO}
 
 
+def test_collect_configuration_run_summary_deduplicates_everything_hooks_across_config_files():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, before='everything', action_names=object
+    ).and_return([before_everything_hook]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        (after_everything_hook,), before='everything', action_names=object
+    ).and_return([]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        (before_everything_hook,), after='everything', action_names=object, state_names=['finish']
+    ).and_return([]).once()
+    flexmock(module.command).should_receive('execute_hooks').twice()
+    flexmock(module).should_receive('Log_prefix').and_return(flexmock())
+    flexmock(module).should_receive('run_configuration').and_return([])
+    arguments = {
+        'create': flexmock(),
+        'global': flexmock(monitoring_verbosity=1, dry_run=False),
+    }
+
+    logs = tuple(
+        module.collect_configuration_run_summary_logs(
+            {'test.yaml': {'commands': command_hooks}, 'other.yaml': {'commands': command_hooks}},
+            config_paths=['/tmp/test.yaml', '/tmp/other.yaml'],
+            arguments=arguments,
+            log_file_path=None,
+        )
+    )
+
+    assert {log.levelno for log in logs} == {logging.INFO}
+
+
 def test_collect_configuration_run_summary_logs_info_for_success_with_extract():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -1938,7 +2081,12 @@ def test_collect_configuration_run_summary_logs_extract_with_repository_error():
 
 def test_collect_configuration_run_summary_logs_info_for_success_with_mount():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -1984,7 +2132,12 @@ def test_collect_configuration_run_summary_logs_mount_with_repository_error():
 
 def test_collect_configuration_run_summary_logs_missing_configs_error():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     arguments = {'global': flexmock(config_paths=[])}
     expected_logs = (flexmock(),)
@@ -1999,9 +2152,17 @@ def test_collect_configuration_run_summary_logs_missing_configs_error():
     assert logs == expected_logs
 
 
-def test_collect_configuration_run_summary_logs_pre_hook_error():
+def test_collect_configuration_run_summary_logs_before_hook_error():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    ).and_return([after_everything_hook])
     flexmock(module.command).should_receive('execute_hooks').and_raise(ValueError)
     expected_logs = (flexmock(),)
     flexmock(module).should_receive('log_error_records').and_return(expected_logs)
@@ -2012,7 +2173,7 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -2022,9 +2183,17 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
     assert logs == expected_logs
 
 
-def test_collect_configuration_run_summary_logs_post_hook_error():
+def test_collect_configuration_run_summary_logs_after_hook_error():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook])
     flexmock(module.command).should_receive('execute_hooks').and_return(None).and_raise(ValueError)
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2037,7 +2206,7 @@ def test_collect_configuration_run_summary_logs_post_hook_error():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -2072,7 +2241,12 @@ def test_collect_configuration_run_summary_logs_for_list_with_archive_and_reposi
 
 def test_collect_configuration_run_summary_logs_info_for_success_with_list():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2095,7 +2269,12 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_list():
 
 def test_collect_configuration_run_summary_logs_run_configuration_error_logs():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return(
@@ -2118,7 +2297,12 @@ def test_collect_configuration_run_summary_logs_run_configuration_error_logs():
 
 def test_collect_configuration_run_summary_logs_run_umount_error():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2145,7 +2329,12 @@ def test_collect_configuration_run_summary_logs_run_umount_error():
 
 def test_collect_configuration_run_summary_logs_outputs_merged_json_results():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return(['foo', 'bar']).and_return(

tests/unit/config/test_normalize.py

@@ -334,6 +334,16 @@ def test_normalize_commands_moves_individual_command_hooks_to_unified_commands(
             {'repositories': [{'path': 'rclone:host:repo'}]},
             True,
         ),
+        (
+            {'repositories': ['s3:stuff']},
+            {'repositories': [{'path': 's3:stuff'}]},
+            True,
+        ),
+        (
+            {'repositories': ['b2:stuff']},
+            {'repositories': [{'path': 'b2:stuff'}]},
+            True,
+        ),
         (
             {'repositories': ['file:///repo']},
             {'repositories': [{'path': '/repo'}]},

tests/unit/hooks/data_source/test_btrfs.py

@@ -5,7 +5,31 @@ from borgmatic.borg.pattern import Pattern, Pattern_source, Pattern_style, Patte
 from borgmatic.hooks.data_source import btrfs as module
 
 
-def test_get_subvolume_mount_points_parses_findmnt_output():
+def test_get_contained_subvolume_paths_parses_btrfs_output():
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).with_args(('btrfs', 'subvolume', 'list', '/mnt0'), close_fds=True).and_return(
+        'ID 256 gen 28 top level 5 path @sub\nID 258 gen 17 top level 5 path snap\n\n'
+    )
+
+    assert module.get_contained_subvolume_paths('btrfs', '/mnt0') == (
+        '/mnt0',
+        '/mnt0/@sub',
+        '/mnt0/snap',
+    )
+
+
+def test_get_contained_subvolume_paths_swallows_called_process_error():
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).with_args(('btrfs', 'subvolume', 'list', '/mnt0'), close_fds=True).and_raise(
+        module.subprocess.CalledProcessError(1, 'btrfs')
+    )
+
+    assert module.get_contained_subvolume_paths('btrfs', '/mnt0') == ()
+
+
+def test_get_all_subvolume_paths_parses_findmnt_output():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return(
@@ -22,31 +46,53 @@ def test_get_subvolume_mount_points_parses_findmnt_output():
                  "source": "/dev/loop0",
                  "fstype": "btrfs",
                  "options": "rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/"
+              },
+              {
+                 "target": "/mnt2",
+                 "source": "/dev/loop0",
+                 "fstype": "btrfs",
+                 "options": "rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/"
               }
            ]
         }
         '''
     )
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt0'
+    ).and_return(('/mnt0',))
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt1'
+    ).and_return(('/mnt1', '/mnt1/sub'))
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt2'
+    ).never()
 
-    assert module.get_subvolume_mount_points('findmnt') == ('/mnt0', '/mnt1')
+    assert module.get_all_subvolume_paths('btrfs', 'findmnt') == (
+        '/mnt0',
+        '/mnt1',
+        '/mnt1/sub',
+        '/mnt2',
+    )
 
 
-def test_get_subvolume_mount_points_with_invalid_findmnt_json_errors():
+def test_get_all_subvolume_paths_with_invalid_findmnt_json_errors():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return('{')
+    flexmock(module).should_receive('get_contained_subvolume_paths').never()
 
     with pytest.raises(ValueError):
-        module.get_subvolume_mount_points('findmnt')
+        module.get_all_subvolume_paths('btrfs', 'findmnt')
 
 
-def test_get_subvolume_mount_points_with_findmnt_json_missing_filesystems_errors():
+def test_get_all_subvolume_paths_with_findmnt_json_missing_filesystems_errors():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return('{"wtf": "something is wrong here"}')
+    flexmock(module).should_receive('get_contained_subvolume_paths').never()
 
     with pytest.raises(ValueError):
-        module.get_subvolume_mount_points('findmnt')
+        module.get_all_subvolume_paths('btrfs', 'findmnt')
 
 
 def test_get_subvolume_property_with_invalid_btrfs_output_errors():
@@ -82,7 +128,7 @@ def test_get_subvolume_property_passes_through_general_value():
     assert module.get_subvolume_property('btrfs', '/foo', 'thing') == 'value'
 
 
-def test_omit_read_only_subvolume_mount_points_filters_out_read_only():
+def test_omit_read_only_subvolume_paths_filters_out_read_only_subvolumes():
     flexmock(module).should_receive('get_subvolume_property').with_args(
         'btrfs', '/foo', 'ro'
     ).and_return(False)
@@ -93,17 +139,29 @@ def test_omit_read_only_subvolume_mount_points_filters_out_read_only():
         'btrfs', '/baz', 'ro'
     ).and_return(False)
 
-    assert module.omit_read_only_subvolume_mount_points('btrfs', ('/foo', '/bar', '/baz')) == (
+    assert module.omit_read_only_subvolume_paths('btrfs', ('/foo', '/bar', '/baz')) == (
         '/foo',
         '/baz',
     )
 
 
+def test_omit_read_only_subvolume_paths_filters_out_erroring_subvolumes():
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/foo', 'ro'
+    ).and_raise(module.subprocess.CalledProcessError(1, 'btrfs'))
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/bar', 'ro'
+    ).and_return(True)
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/baz', 'ro'
+    ).and_return(False)
+
+    assert module.omit_read_only_subvolume_paths('btrfs', ('/foo', '/bar', '/baz')) == ('/baz',)
+
+
 def test_get_subvolumes_collects_subvolumes_matching_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
-        ('/mnt1', '/mnt2')
-    )
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
 
     contained_pattern = Pattern(
         '/mnt1',
@@ -128,10 +186,8 @@ def test_get_subvolumes_collects_subvolumes_matching_patterns():
 
 
 def test_get_subvolumes_skips_non_root_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
-        ('/mnt1', '/mnt2')
-    )
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'
@@ -162,10 +218,8 @@ def test_get_subvolumes_skips_non_root_patterns():
 
 
 def test_get_subvolumes_skips_non_config_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
-        ('/mnt1', '/mnt2')
-    )
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'
@@ -196,10 +250,8 @@ def test_get_subvolumes_skips_non_config_patterns():
 
 
 def test_get_subvolumes_without_patterns_collects_all_subvolumes():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
-        ('/mnt1', '/mnt2')
-    )
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'

tests/unit/hooks/data_source/test_lvm.py

@@ -214,6 +214,7 @@ def test_snapshot_logical_volume_with_percentage_snapshot_name_uses_lvcreate_ext
             '/dev/snap',
         ),
         output_log_level=object,
+        close_fds=True,
     )
 
     module.snapshot_logical_volume('lvcreate', 'snap', '/dev/snap', '10%ORIGIN')
@@ -233,6 +234,7 @@ def test_snapshot_logical_volume_with_non_percentage_snapshot_name_uses_lvcreate
             '/dev/snap',
         ),
         output_log_level=object,
+        close_fds=True,
     )
 
     module.snapshot_logical_volume('lvcreate', 'snap', '/dev/snap', '10TB')
@@ -554,16 +556,7 @@ def test_dump_data_sources_with_dry_run_skips_snapshots_and_does_not_touch_patte
     )
     flexmock(module.os).should_receive('getpid').and_return(1234)
     flexmock(module).should_receive('snapshot_logical_volume').never()
-    flexmock(module).should_receive('get_snapshots').with_args(
-        'lvs', snapshot_name='lvolume1_borgmatic-1234'
-    ).and_return(
-        (module.Snapshot(name='lvolume1_borgmatic-1234', device_path='/dev/lvolume1_snap'),)
-    )
-    flexmock(module).should_receive('get_snapshots').with_args(
-        'lvs', snapshot_name='lvolume2_borgmatic-1234'
-    ).and_return(
-        (module.Snapshot(name='lvolume2_borgmatic-1234', device_path='/dev/lvolume2_snap'),)
-    )
+    flexmock(module).should_receive('get_snapshots').never()
     flexmock(module).should_receive('mount_snapshot').never()
 
     assert (

tests/unit/hooks/data_source/test_mariadb.py

@@ -167,6 +167,38 @@ def test_database_names_to_dump_queries_mariadb_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mariadb',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mariadb_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -247,9 +279,22 @@ def test_dump_data_sources_dumps_each_database():
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
     ).replace_with(lambda value, config: value)
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[0],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -285,9 +330,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -311,6 +361,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -378,6 +472,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mariadb_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=mariadb.cnf --skip-ssl'}
     flexmock(module).should_receive('parse_extra_options').and_return(
@@ -479,6 +621,48 @@ def test_execute_dump_command_runs_mariadb_dump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mariadb-dump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mariadb_dump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -849,54 +1033,6 @@ def test_execute_dump_command_with_dry_run_skips_mariadb_dump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mariadb_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1168,6 +1304,49 @@ def test_restore_data_source_dump_runs_mariadb_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mariadb', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/data_source/test_mysql.py

@@ -60,6 +60,42 @@ def test_database_names_to_dump_queries_mysql_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mysql',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mysql_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -141,9 +177,22 @@ def test_dump_data_sources_dumps_each_database():
         'resolve_credential'
     ).and_return(None)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[0],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -179,9 +228,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -205,6 +259,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -272,6 +370,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mysql_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=my.cnf --skip-ssl'}
     flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
@@ -375,6 +521,52 @@ def test_execute_dump_command_runs_mysqldump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mysqldump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mysqldump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -761,54 +953,6 @@ def test_execute_dump_command_with_dry_run_skips_mysqldump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mysql_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1092,6 +1236,53 @@ def test_restore_data_source_dump_runs_mysql_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mysql', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/monitoring/test_cronhub.py

@@ -6,7 +6,7 @@ from borgmatic.hooks.monitoring import cronhub as module
 def test_ping_monitor_rewrites_ping_url_for_start_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -22,7 +22,7 @@ def test_ping_monitor_rewrites_ping_url_for_start_state():
 def test_ping_monitor_rewrites_ping_url_and_state_for_start_state():
     hook_config = {'ping_url': 'https://example.com/ping/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -38,7 +38,7 @@ def test_ping_monitor_rewrites_ping_url_and_state_for_start_state():
 def test_ping_monitor_rewrites_ping_url_for_finish_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/finish/abcdef'
+        'https://example.com/finish/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -54,7 +54,7 @@ def test_ping_monitor_rewrites_ping_url_for_finish_state():
 def test_ping_monitor_rewrites_ping_url_for_fail_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/fail/abcdef'
+        'https://example.com/fail/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -105,7 +105,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_cronitor.py

@@ -5,9 +5,9 @@ from borgmatic.hooks.monitoring import cronitor as module
 
 def test_ping_monitor_hits_ping_url_for_start_state():
     hook_config = {'ping_url': 'https://example.com'}
-    flexmock(module.requests).should_receive('get').with_args('https://example.com/run').and_return(
-        flexmock(ok=True)
-    )
+    flexmock(module.requests).should_receive('get').with_args(
+        'https://example.com/run', timeout=int
+    ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
         hook_config,
@@ -22,7 +22,7 @@ def test_ping_monitor_hits_ping_url_for_start_state():
 def test_ping_monitor_hits_ping_url_for_finish_state():
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/complete'
+        'https://example.com/complete', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -38,7 +38,7 @@ def test_ping_monitor_hits_ping_url_for_finish_state():
 def test_ping_monitor_hits_ping_url_for_fail_state():
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/fail'
+        'https://example.com/fail', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -88,9 +88,9 @@ def test_ping_monitor_with_other_error_logs_warning():
     response.should_receive('raise_for_status').and_raise(
         module.requests.exceptions.RequestException
     )
-    flexmock(module.requests).should_receive('get').with_args('https://example.com/run').and_return(
-        response
-    )
+    flexmock(module.requests).should_receive('get').with_args(
+        'https://example.com/run', timeout=int
+    ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_healthchecks.py

@@ -79,7 +79,7 @@ def test_ping_monitor_hits_ping_url_for_start_state():
     ).never()
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -100,7 +100,7 @@ def test_ping_monitor_hits_ping_url_for_finish_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=True
+        'https://example.com', data=payload.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -121,7 +121,7 @@ def test_ping_monitor_hits_ping_url_for_fail_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/fail', data=payload.encode('utf'), verify=True
+        'https://example.com/fail', data=payload.encode('utf'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -142,7 +142,7 @@ def test_ping_monitor_hits_ping_url_for_log_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/log', data=payload.encode('utf'), verify=True
+        'https://example.com/log', data=payload.encode('utf'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -166,6 +166,7 @@ def test_ping_monitor_with_ping_uuid_hits_corresponding_url():
         f"https://hc-ping.com/{hook_config['ping_url']}",
         data=payload.encode('utf-8'),
         verify=True,
+        timeout=int,
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -186,7 +187,7 @@ def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=False
+        'https://example.com', data=payload.encode('utf-8'), verify=False, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -207,7 +208,7 @@ def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=True
+        'https://example.com', data=payload.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -260,7 +261,7 @@ def test_ping_monitor_hits_ping_url_when_states_matching():
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'states': ['start', 'finish']}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -279,7 +280,7 @@ def test_ping_monitor_adds_create_query_parameter_when_create_slug_true():
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'create_slug': True}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start?create=1', data=''.encode('utf-8'), verify=True
+        'https://example.com/start?create=1', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -298,7 +299,7 @@ def test_ping_monitor_does_not_add_create_query_parameter_when_create_slug_false
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'create_slug': False}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -317,6 +318,7 @@ def test_ping_monitor_does_not_add_create_query_parameter_when_ping_url_is_uuid(
         f"https://hc-ping.com/{hook_config['ping_url']}",
         data=''.encode('utf-8'),
         verify=True,
+        timeout=int,
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -352,7 +354,7 @@ def test_ping_monitor_with_connection_error_logs_warning():
     ).never()
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -376,7 +378,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_ntfy.py

@@ -43,6 +43,7 @@ def test_ping_monitor_minimal_config_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -67,6 +68,7 @@ def test_ping_monitor_with_access_token_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -93,6 +95,7 @@ def test_ping_monitor_with_username_password_and_access_token_ignores_username_p
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -119,6 +122,7 @@ def test_ping_monitor_with_username_password_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('testuser', 'fakepassword'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -140,6 +144,7 @@ def test_ping_monitor_with_password_but_no_username_warns():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -162,6 +167,7 @@ def test_ping_monitor_with_username_but_no_password_warns():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -218,6 +224,7 @@ def test_ping_monitor_minimal_config_hits_selfhosted_ntfy_on_fail():
         f'{custom_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -253,7 +260,7 @@ def test_ping_monitor_custom_message_hits_hosted_ntfy_on_fail():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.requests).should_receive('post').with_args(
-        f'{default_base_url}/{topic}', headers=custom_message_headers, auth=None
+        f'{default_base_url}/{topic}', headers=custom_message_headers, auth=None, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -275,6 +282,7 @@ def test_ping_monitor_custom_state_hits_hosted_ntfy_on_start():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.START),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -296,6 +304,7 @@ def test_ping_monitor_with_connection_error_logs_warning():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -340,6 +349,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_pushover.py

@@ -23,6 +23,7 @@ def test_ping_monitor_config_with_minimum_config_fail_state_backup_successfully_
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'fail',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -81,6 +82,7 @@ def test_ping_monitor_start_state_backup_default_message_successfully_send_to_pu
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'start',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -117,6 +119,7 @@ def test_ping_monitor_start_state_backup_custom_message_successfully_send_to_pus
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'custom start message',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -155,6 +158,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -193,6 +197,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -231,6 +236,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -328,6 +334,7 @@ def test_ping_monitor_start_state_backup_based_on_documentation_advanced_example
             'html': 1,
             'ttl': 10,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -399,6 +406,7 @@ def test_ping_monitor_fail_state_backup_based_on_documentation_advanced_example_
             'url': 'https://ticketing-system.example.com/login',
             'url_title': 'Login to ticketing system',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -467,6 +475,7 @@ def test_ping_monitor_finish_state_backup_based_on_documentation_advanced_exampl
             'url': 'https://ticketing-system.example.com/login',
             'url_title': 'Login to ticketing system',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -550,6 +559,7 @@ def test_ping_monitor_push_post_error_bails():
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'fail',
         },
+        timeout=int,
     ).and_return(push_response).once()
 
     flexmock(module.logger).should_receive('warning').once()

tests/unit/hooks/monitoring/test_sentry.py

@@ -29,7 +29,8 @@ def test_ping_monitor_constructs_cron_url_and_pings_it(state, configured_states,
         hook_config['states'] = configured_states
 
     flexmock(module.requests).should_receive('post').with_args(
-        f'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status={expected_status}'
+        f'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status={expected_status}',
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -134,7 +135,8 @@ def test_ping_monitor_with_network_error_does_not_raise():
         module.requests.exceptions.ConnectionError
     )
     flexmock(module.requests).should_receive('post').with_args(
-        'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status=in_progress'
+        'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status=in_progress',
+        timeout=int,
     ).and_return(response).once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_uptimekuma.py

@@ -10,7 +10,7 @@ CUSTOM_PUSH_URL = 'https://uptime.example.com/api/push/efgh5678'
 def test_ping_monitor_hits_default_uptimekuma_on_fail():
     hook_config = {}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{DEFAULT_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{DEFAULT_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -26,7 +26,7 @@ def test_ping_monitor_hits_default_uptimekuma_on_fail():
 def test_ping_monitor_hits_custom_uptimekuma_on_fail():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -42,7 +42,7 @@ def test_ping_monitor_hits_custom_uptimekuma_on_fail():
 def test_ping_monitor_custom_uptimekuma_on_start():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=up&msg=start', verify=True
+        f'{CUSTOM_PUSH_URL}?status=up&msg=start', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -58,7 +58,7 @@ def test_ping_monitor_custom_uptimekuma_on_start():
 def test_ping_monitor_custom_uptimekuma_on_finish():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=up&msg=finish', verify=True
+        f'{CUSTOM_PUSH_URL}?status=up&msg=finish', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -116,7 +116,7 @@ def test_ping_monitor_does_not_hit_custom_uptimekuma_on_finish_dry_run():
 def test_ping_monitor_with_connection_error_logs_warning():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -137,7 +137,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -168,7 +168,7 @@ def test_ping_monitor_with_invalid_run_state():
 def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
     hook_config = {'push_url': CUSTOM_PUSH_URL, 'verify_tls': False}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=False
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=False, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -184,7 +184,7 @@ def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
 def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
     hook_config = {'push_url': CUSTOM_PUSH_URL, 'verify_tls': True}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_zabbix.py

@@ -74,7 +74,7 @@ def test_send_zabbix_request_with_post_error_bails():
     )
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -89,7 +89,7 @@ def test_send_zabbix_request_with_invalid_json_response_bails():
     response.should_receive('json').and_raise(module.requests.exceptions.JSONDecodeError)
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -103,7 +103,7 @@ def test_send_zabbix_request_with_success_returns_response_result():
     response.should_receive('json').and_return({'result': {'foo': 'bar'}})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) == {'foo': 'bar'}
@@ -117,7 +117,7 @@ def test_send_zabbix_request_with_success_passes_through_missing_result():
     response.should_receive('json').and_return({})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -131,7 +131,7 @@ def test_send_zabbix_request_with_error_bails():
     response.should_receive('json').and_return({'result': {'data': [{'error': 'oops'}]}})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None

tests/unit/hooks/test_command.py

@@ -11,8 +11,16 @@ def test_interpolate_context_passes_through_command_without_variable():
     assert module.interpolate_context('pre-backup', 'ls', {'foo': 'bar'}) == 'ls'
 
 
-def test_interpolate_context_passes_through_command_with_unknown_variable():
+def test_interpolate_context_warns_and_passes_through_command_with_unknown_variable():
     command = 'ls {baz}'  # noqa: FS003
+    flexmock(module.logger).should_receive('warning').once()
+
+    assert module.interpolate_context('pre-backup', command, {'foo': 'bar'}) == command
+
+
+def test_interpolate_context_does_not_warn_and_passes_through_command_with_unknown_variable_matching_borg_placeholder():
+    command = 'ls {hostname}'  # noqa: FS003
+    flexmock(module.logger).should_receive('warning').never()
 
     assert module.interpolate_context('pre-backup', command, {'foo': 'bar'}) == command
 
@@ -167,6 +175,44 @@ def test_make_environment_with_pyinstaller_and_LD_LIBRARY_PATH_ORIG_copies_it_in
                 },
             ),
         ),
+        (
+            (
+                {
+                    'before': 'action',
+                    'states': ['finish'],  # Not actually valid; only valid for "after".
+                    'run': ['foo'],
+                },
+                {
+                    'after': 'action',
+                    'run': ['bar'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['finish'],
+                    'run': ['baz'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['fail'],
+                    'run': ['quux'],
+                },
+            ),
+            {
+                'after': 'action',
+                'state_names': ['finish'],
+            },
+            (
+                {
+                    'after': 'action',
+                    'run': ['bar'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['finish'],
+                    'run': ['baz'],
+                },
+            ),
+        ),
     ),
 )
 def test_filter_hooks(hooks, filters, expected_hooks):
@@ -336,14 +382,13 @@ def test_before_after_hooks_calls_command_hooks():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').twice()
 
@@ -353,7 +398,6 @@ def test_before_after_hooks_calls_command_hooks():
         umask=1234,
         working_directory='/working',
         dry_run=False,
-        hook_name='myhook',
         action_names=['create'],
         context1='stuff',
         context2='such',
@@ -369,14 +413,13 @@ def test_before_after_hooks_with_before_error_runs_after_hook_and_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['fail'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_raise(OSError).and_return(None)
     flexmock(module).should_receive('considered_soft_failure').and_return(False)
@@ -388,7 +431,6 @@ def test_before_after_hooks_with_before_error_runs_after_hook_and_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',
@@ -396,7 +438,7 @@ def test_before_after_hooks_with_before_error_runs_after_hook_and_raises():
             assert False  # This should never get called.
 
 
-def test_before_after_hooks_with_before_soft_failure_does_not_raise():
+def test_before_after_hooks_with_before_soft_failure_raises():
     commands = [
         {'before': 'repository', 'run': ['foo', 'bar']},
         {'after': 'repository', 'run': ['baz']},
@@ -404,30 +446,61 @@ def test_before_after_hooks_with_before_soft_failure_does_not_raise():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
-    ).and_return(flexmock()).once()
+        state_names=['finish'],
+    ).never()
     flexmock(module).should_receive('execute_hooks').and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(True)
 
-    with module.Before_after_hooks(
-        command_hooks=commands,
-        before_after='action',
-        umask=1234,
-        working_directory='/working',
-        dry_run=False,
-        hook_name='myhook',
+    with pytest.raises(OSError):
+        with module.Before_after_hooks(
+            command_hooks=commands,
+            before_after='action',
+            umask=1234,
+            working_directory='/working',
+            dry_run=False,
+            action_names=['create'],
+            context1='stuff',
+            context2='such',
+        ):
+            pass
+
+
+def test_before_after_hooks_with_wrapped_code_error_runs_after_hook_and_raises():
+    commands = [
+        {'before': 'repository', 'run': ['foo', 'bar']},
+        {'after': 'repository', 'run': ['baz']},
+    ]
+    flexmock(module).should_receive('filter_hooks').with_args(
+        commands,
+        before='action',
         action_names=['create'],
-        context1='stuff',
-        context2='such',
-    ):
-        pass
+    ).and_return(flexmock()).once()
+    flexmock(module).should_receive('filter_hooks').with_args(
+        commands,
+        after='action',
+        action_names=['create'],
+        state_names=['fail'],
+    ).and_return(flexmock()).once()
+    flexmock(module).should_receive('execute_hooks').twice()
+
+    with pytest.raises(ValueError):
+        with module.Before_after_hooks(
+            command_hooks=commands,
+            before_after='action',
+            umask=1234,
+            working_directory='/working',
+            dry_run=False,
+            action_names=['create'],
+            context1='stuff',
+            context2='such',
+        ):
+            raise ValueError()
 
 
 def test_before_after_hooks_with_after_error_raises():
@@ -438,14 +511,13 @@ def test_before_after_hooks_with_after_error_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_return(None).and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(False)
@@ -457,7 +529,6 @@ def test_before_after_hooks_with_after_error_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',
@@ -465,7 +536,7 @@ def test_before_after_hooks_with_after_error_raises():
             pass
 
 
-def test_before_after_hooks_with_after_soft_failure_does_not_raise():
+def test_before_after_hooks_with_after_soft_failure_raises():
     commands = [
         {'before': 'repository', 'run': ['foo', 'bar']},
         {'after': 'repository', 'run': ['baz']},
@@ -473,30 +544,29 @@ def test_before_after_hooks_with_after_soft_failure_does_not_raise():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_return(None).and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(True)
 
-    with module.Before_after_hooks(
-        command_hooks=commands,
-        before_after='action',
-        umask=1234,
-        working_directory='/working',
-        dry_run=False,
-        hook_name='myhook',
-        action_names=['create'],
-        context1='stuff',
-        context2='such',
-    ):
-        pass
+    with pytest.raises(OSError):
+        with module.Before_after_hooks(
+            command_hooks=commands,
+            before_after='action',
+            umask=1234,
+            working_directory='/working',
+            dry_run=False,
+            action_names=['create'],
+            context1='stuff',
+            context2='such',
+        ):
+            pass
 
 
 def test_considered_soft_failure_treats_soft_fail_exit_code_as_soft_fail():
@@ -513,3 +583,11 @@ def test_considered_soft_failure_does_not_treat_other_exit_code_as_soft_fail():
 
 def test_considered_soft_failure_does_not_treat_other_exception_type_as_soft_fail():
     assert not module.considered_soft_failure(Exception())
+
+
+def test_considered_soft_failure_caches_results_and_only_logs_once():
+    error = subprocess.CalledProcessError(module.SOFT_FAIL_EXIT_CODE, 'try again')
+    flexmock(module.logger).should_receive('info').once()
+
+    assert module.considered_soft_failure(error)
+    assert module.considered_soft_failure(error)