NEWS wording tweaks for clarity.

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/984

NEWS

@@ -1,3 +1,89 @@
+1.9.9
+ * #635: Log the repository path or label on every relevant log message, not just some logs.
+ * #961: When the "encryption_passcommand" option is set, call the command once from borgmatic to
+   collect the encryption passphrase and then pass it to Borg multiple times. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #981: Fix a "spot" check file count delta error.
+ * #982: Fix for borgmatic "exclude_patterns" and "exclude_from" recursing into excluded
+   subdirectories.
+ * #983: Fix the Btrfs hook to support subvolumes with names like "@home" different from their
+   mount points.
+ * #985: Change the default value for the "--original-hostname" flag from "localhost" to no host
+   specified. This way, the "restore" action works without a hostname if there's a single matching
+   database dump.
+
+1.9.8
+ * #979: Fix root patterns so they don't have an invalid "sh:" prefix before getting passed to Borg.
+ * Expand the recent contributors documentation section to include ticket submitters—not just code
+   contributors—because there are multiple ways to contribute to the project! See:
+   https://torsion.org/borgmatic/#recent-contributors
+
+1.9.7
+ * #855: Add a Sentry monitoring hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook
+ * #968: Fix for a "spot" check error when a filename in the most recent archive contains a newline.
+ * #970: Fix for an error when there's a blank line in the configured patterns or excludes.
+ * #971: Fix for "exclude_from" files being completely ignored.
+ * #977: Fix for "exclude_patterns" and "exclude_from" not supporting explicit pattern styles (e.g.,
+   "sh:" or "re:").
+
+1.9.6
+ * #959: Fix an error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+ * #960: Fix for archives storing relative source directory paths such that they contain the working
+   directory.
+ * #960: Fix the "spot" check to support relative source directory paths.
+ * #962: For the ZFS, Btrfs, and LVM hooks, perform path rewriting for excludes and patterns in
+   addition to the existing source directories rewriting.
+ * #962: Under the hood, merge all configured source directories, excludes, and patterns into a
+   unified temporary patterns file for passing to Borg. The borgmatic configuration options remain
+   unchanged.
+ * #962: For the LVM hook, add support for nested logical volumes.
+ * #965: Fix a borgmatic runtime directory error when running the "spot" check with a database hook
+   enabled.
+ * #969: Fix the "restore" action to work on database dumps without a port when a default port is
+   present in configuration.
+ * Fix the "spot" check to no longer consider pipe files within an archive for file comparisons.
+ * Fix the "spot" check to have a nicer error when there are no source paths to compare.
+ * Fix auto-excluding of special files (when databases are configured) to support relative source
+   directory paths.
+ * Drop support for Python 3.8, which has been end-of-lifed.
+
+1.9.5
+ * #418: Backup and restore databases that have the same name but with different ports, hostnames,
+   or hooks.
+ * #947: To avoid a hang in the database hooks, error and exit when the borgmatic runtime
+   directory overlaps with the configured excludes.
+ * #954: Fix a findmnt command error in the Btrfs hook by switching to parsing JSON output.
+ * #956: Fix the printing of a color reset code even when color is disabled.
+ * #958: Drop colorama as a library dependency.
+ * When the ZFS, Btrfs, or LVM hooks aren't configured, don't try to cleanup snapshots for them.
+
+1.9.4
+ * #80 (beta): Add an LVM hook for snapshotting and backing up LVM logical volumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #251 (beta): Add a Btrfs hook for snapshotting and backing up Btrfs subvolumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #926: Fix a library error when running within a PyInstaller bundle.
+ * #950: Fix a snapshot unmount error in the ZFS hook when using nested datasets.
+ * Update the ZFS hook to discover and snapshot ZFS datasets even if they are parent/grandparent
+   directories of your source directories.
+ * Reorganize data source and monitoring hooks to make developing new hooks easier.
+
+1.9.3
+ * #261 (beta): Add a ZFS hook for snapshotting and backing up ZFS datasets. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * Remove any temporary copies of the manifest file created in support of the "bootstrap" action.
+ * Deprecate the "store_config_files" option at the global scope and move it under the "bootstrap"
+   hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive
+ * Require the runtime directory to be an absolute path.
+ * Add a "--deleted" flag to the "repo-list" action for listing deleted archives that haven't
+   yet been compacted (Borg 2 only).
+ * Promote the "spot" check from a beta feature to stable.
+
 1.9.2
  * #441: Apply the "umask" option to all relevant actions, not just some of them.
  * #722: Remove the restriction that the "extract" and "mount" actions must match a single

README.md

@@ -61,6 +61,10 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 <a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
@@ -71,6 +75,7 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 <a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
@@ -160,4 +165,8 @@ info on cloning source code, running tests, etc.
 
 ### Recent contributors
 
+Thanks to all borgmatic contributors! There are multiple ways to contribute to
+this project, so the following includes those who have fixed bugs, contributed
+features, *or* filed tickets.
+
 {% include borgmatic/contributors.html %}

borgmatic/actions/borg.py

@@ -22,9 +22,7 @@ def run_borg(
     if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, borg_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Running arbitrary Borg command'
-        )
+        logger.info('Running arbitrary Borg command')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             borg_arguments.archive,

borgmatic/actions/break_lock.py

@@ -21,9 +21,7 @@ def run_break_lock(
     if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, break_lock_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Breaking repository and cache locks'
-        )
+        logger.info('Breaking repository and cache locks')
         borgmatic.borg.break_lock.break_lock(
             repository['path'],
             config,

borgmatic/actions/change_passphrase.py

@@ -24,9 +24,7 @@ def run_change_passphrase(
             repository, change_passphrase_arguments.repository
         )
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Changing repository passphrase'
-        )
+        logger.info('Changing repository passphrase')
         borgmatic.borg.change_passphrase.change_passphrase(
             repository['path'],
             config,

borgmatic/actions/check.py

@@ -8,6 +8,7 @@ import pathlib
 import random
 import shutil
 
+import borgmatic.actions.create
 import borgmatic.borg.check
 import borgmatic.borg.create
 import borgmatic.borg.environment
@@ -345,7 +346,13 @@ def upgrade_check_times(config, borg_repository_id):
 
 
 def collect_spot_check_source_paths(
-    repository, config, local_borg_version, global_arguments, local_path, remote_path
+    repository,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path,
+    remote_path,
+    borgmatic_runtime_directory,
 ):
     '''
     Given a repository configuration dict, a configuration dict, the local Borg version, global
@@ -356,20 +363,23 @@ def collect_spot_check_source_paths(
         borgmatic.hooks.dispatch.call_hooks(
             'use_streaming',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         ).values()
     )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
+    (create_flags, create_positional_arguments, pattern_file) = (
         borgmatic.borg.create.make_base_create_command(
             dry_run=True,
             repository_path=repository['path'],
             config=config,
-            config_paths=(),
+            patterns=borgmatic.actions.create.process_patterns(
+                borgmatic.actions.create.collect_patterns(config),
+                working_directory,
+            ),
             local_borg_version=local_borg_version,
             global_arguments=global_arguments,
-            borgmatic_runtime_directories=(),
+            borgmatic_runtime_directory=borgmatic_runtime_directory,
             local_path=local_path,
             remote_path=remote_path,
             list_files=True,
@@ -390,7 +400,7 @@ def collect_spot_check_source_paths(
 
     paths = tuple(
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in paths_output.splitlines()
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
 
@@ -400,6 +410,7 @@ def collect_spot_check_source_paths(
 
 
 BORG_DIRECTORY_FILE_TYPE = 'd'
+BORG_PIPE_FILE_TYPE = 'p'
 
 
 def collect_spot_check_archive_paths(
@@ -417,6 +428,9 @@ def collect_spot_check_archive_paths(
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
     remote Borg path, and the borgmatic runtime directory, collect the paths from the given archive
     (but only include files and symlinks and exclude borgmatic runtime directories).
+
+    These paths do not have a leading slash, as that's how Borg stores them. As a result, we don't
+    know whether they came from absolute or relative source directories.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -428,15 +442,17 @@ def collect_spot_check_archive_paths(
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} /{path}{NL}',  # noqa: FS003
+            path_format='{type} {path}{NUL}',  # noqa: FS003
             local_path=local_path,
             remote_path=remote_path,
         )
         for (file_type, path) in (line.split(' ', 1),)
-        if file_type != BORG_DIRECTORY_FILE_TYPE
-        if pathlib.Path('/borgmatic') not in pathlib.Path(path).parents
-        if pathlib.Path(borgmatic_source_directory) not in pathlib.Path(path).parents
-        if pathlib.Path(borgmatic_runtime_directory) not in pathlib.Path(path).parents
+        if file_type not in (BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE)
+        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
     )
 
 
@@ -451,15 +467,14 @@ def compare_spot_check_hashes(
     global_arguments,
     local_path,
     remote_path,
-    log_prefix,
     source_paths,
 ):
     '''
     Given a repository configuration dict, the name of the latest archive, a configuration dict, the
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
-    remote Borg path, a log label, and spot check source paths, compare the hashes for a sampling of
-    the source paths with hashes from corresponding paths in the given archive. Return a sequence of
-    the paths that fail that hash comparison.
+    remote Borg path, and spot check source paths, compare the hashes for a sampling of the source
+    paths with hashes from corresponding paths in the given archive. Return a sequence of the paths
+    that fail that hash comparison.
     '''
     # Based on the configured sample percentage, come up with a list of random sample files from the
     # source directories.
@@ -475,7 +490,7 @@ def compare_spot_check_hashes(
         if os.path.exists(os.path.join(working_directory or '', source_path))
     }
     logger.debug(
-        f'{log_prefix}: Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
+        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
     )
 
     source_sample_paths_iterator = iter(source_sample_paths)
@@ -523,7 +538,7 @@ def compare_spot_check_hashes(
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} /{path}{NL}',  # noqa: FS003
+                    path_format='{xxh64} {path}{NUL}',  # noqa: FS003
                     local_path=local_path,
                     remote_path=remote_path,
                 )
@@ -535,7 +550,7 @@ def compare_spot_check_hashes(
     failing_paths = []
 
     for path, source_hash in source_hashes.items():
-        archive_hash = archive_hashes.get(path)
+        archive_hash = archive_hashes.get(path.lstrip(os.path.sep))
 
         if archive_hash is not None and archive_hash == source_hash:
             continue
@@ -563,8 +578,7 @@ def spot_check(
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    log_prefix = f'{repository.get("label", repository["path"])}'
-    logger.debug(f'{log_prefix}: Running spot check')
+    logger.debug('Running spot check')
 
     try:
         spot_check_config = next(
@@ -585,8 +599,9 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
+        borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(source_paths)} total source paths for spot check')
+    logger.debug(f'{len(source_paths)} total source paths for spot check')
 
     archive = borgmatic.borg.repo_list.resolve_archive_name(
         repository['path'],
@@ -597,7 +612,7 @@ def spot_check(
         local_path,
         remote_path,
     )
-    logger.debug(f'{log_prefix}: Using archive {archive} for spot check')
+    logger.debug(f'Using archive {archive} for spot check')
 
     archive_paths = collect_spot_check_archive_paths(
         repository,
@@ -609,18 +624,27 @@ def spot_check(
         remote_path,
         borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(archive_paths)} total archive paths for spot check')
+    logger.debug(f'{len(archive_paths)} total archive paths for spot check')
+
+    if len(source_paths) == 0:
+        logger.debug(
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+        )
+        raise ValueError(
+            'Spot check failed: There are no source paths to compare against the archive'
+        )
 
     # Calculate the percentage delta between the source paths count and the archive paths count, and
     # compare that delta to the configured count tolerance percentage.
     count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100
 
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
+        rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
         logger.debug(
-            f'{log_prefix}: Paths in source paths but not latest archive: {", ".join(set(source_paths) - set(archive_paths)) or "none"}'
+            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
         )
         logger.debug(
-            f'{log_prefix}: Paths in latest archive but not source paths: {", ".join(set(archive_paths) - set(source_paths)) or "none"}'
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
         )
         raise ValueError(
             f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
@@ -634,25 +658,24 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
-        log_prefix,
         source_paths,
     )
 
     # Error if the percentage of failing hashes exceeds the configured tolerance percentage.
-    logger.debug(f'{log_prefix}: {len(failing_paths)} non-matching spot check hashes')
+    logger.debug(f'{len(failing_paths)} non-matching spot check hashes')
     data_tolerance_percentage = spot_check_config['data_tolerance_percentage']
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
         logger.debug(
-            f'{log_prefix}: Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
         )
         raise ValueError(
             f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'
         )
 
     logger.info(
-        f'{log_prefix}: Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
+        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
     )
 
 
@@ -686,8 +709,7 @@ def run_check(
         **hook_context,
     )
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Running consistency checks')
+    logger.info('Running consistency checks')
 
     repository_id = borgmatic.borg.check.get_repository_id(
         repository['path'],
@@ -740,9 +762,7 @@ def run_check(
         write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
-        with borgmatic.config.paths.Runtime_directory(
-            config, log_prefix
-        ) as borgmatic_runtime_directory:
+        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
             spot_check(
                 repository,
                 config,

borgmatic/actions/compact.py

@@ -37,9 +37,7 @@ def run_compact(
         **hook_context,
     )
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Compacting segments{dry_run_label}'
-        )
+        logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(
             global_arguments.dry_run,
             repository['path'],
@@ -53,9 +51,8 @@ def run_compact(
             threshold=compact_arguments.threshold,
         )
     else:  # pragma: nocover
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Skipping compact (only available/needed in Borg 1.2+)'
-        )
+        logger.info('Skipping compact (only available/needed in Borg 1.2+)')
+
     borgmatic.hooks.command.execute_hook(
         config.get('after_compact'),
         config.get('umask'),

borgmatic/actions/config/bootstrap.py

@@ -41,11 +41,10 @@ def get_config_paths(archive_name, bootstrap_arguments, global_arguments, local_
     config = make_bootstrap_config(bootstrap_arguments)
 
     # Probe for the manifest file in multiple locations, as the default location has moved to the
-    # borgmatic runtime directory (which get stored as just "/borgmatic" with Borg 1.4+). But we
+    # borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But we
     # still want to support reading the manifest from previously created archives as well.
     with borgmatic.config.paths.Runtime_directory(
         {'user_runtime_directory': bootstrap_arguments.user_runtime_directory},
-        bootstrap_arguments.repository,
     ) as borgmatic_runtime_directory:
         for base_directory in (
             'borgmatic',

borgmatic/actions/create.py

@@ -1,43 +1,254 @@
-import importlib.metadata
-import json
+import glob
+import itertools
 import logging
 import os
+import pathlib
 
 import borgmatic.actions.json
 import borgmatic.borg.create
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.command
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
 
 logger = logging.getLogger(__name__)
 
 
-def create_borgmatic_manifest(config, config_paths, borgmatic_runtime_directory, dry_run):
+def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_style.NONE):
     '''
-    Given a configuration dict, a sequence of config file paths, the borgmatic runtime directory,
-    and whether this is a dry run, create a borgmatic manifest file to store the paths to the
-    configuration files used to create the archive.
+    Given a Borg pattern as a string, parse it into a borgmatic.borg.pattern.Pattern instance and
+    return it.
     '''
-    if dry_run:
-        return
+    try:
+        (pattern_type, remainder) = pattern_line.split(' ', maxsplit=1)
+    except ValueError:
+        raise ValueError(f'Invalid pattern: {pattern_line}')
 
-    borgmatic_manifest_path = os.path.join(
-        borgmatic_runtime_directory, 'bootstrap', 'manifest.json'
+    try:
+        (parsed_pattern_style, path) = remainder.split(':', maxsplit=1)
+        pattern_style = borgmatic.borg.pattern.Pattern_style(parsed_pattern_style)
+    except ValueError:
+        pattern_style = default_style
+        path = remainder
+
+    return borgmatic.borg.pattern.Pattern(
+        path,
+        borgmatic.borg.pattern.Pattern_type(pattern_type),
+        borgmatic.borg.pattern.Pattern_style(pattern_style),
     )
 
-    if not os.path.exists(borgmatic_manifest_path):
-        os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
 
-    with open(borgmatic_manifest_path, 'w') as config_list_file:
-        json.dump(
-            {
-                'borgmatic_version': importlib.metadata.version('borgmatic'),
-                'config_paths': config_paths,
-            },
-            config_list_file,
+def collect_patterns(config):
+    '''
+    Given a configuration dict, produce a single sequence of patterns comprised of the configured
+    source directories, patterns, excludes, pattern files, and exclude files.
+
+    The idea is that Borg has all these different ways of specifying includes, excludes, source
+    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
+    ease of manipulation within borgmatic.
+    '''
+    try:
+        return (
+            tuple(
+                borgmatic.borg.pattern.Pattern(source_directory)
+                for source_directory in config.get('source_directories', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for pattern_line in config.get('patterns', ())
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for exclude_line in config.get('exclude_patterns', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for filename in config.get('patterns_from', ())
+                for pattern_line in open(filename).readlines()
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for filename in config.get('exclude_from', ())
+                for exclude_line in open(filename).readlines()
+                if not exclude_line.lstrip().startswith('#')
+                if exclude_line.strip()
+            )
         )
+    except (FileNotFoundError, OSError) as error:
+        logger.debug(error)
+
+        raise ValueError(f'Cannot read patterns_from/exclude_from file: {error.filename}')
+
+
+def expand_directory(directory, working_directory):
+    '''
+    Given a directory path, expand any tilde (representing a user's home directory) and any globs
+    therein. Return a list of one or more resulting paths.
+
+    Take into account the given working directory so that relative paths are supported.
+    '''
+    expanded_directory = os.path.expanduser(directory)
+
+    # This would be a lot easier to do with glob(..., root_dir=working_directory), but root_dir is
+    # only available in Python 3.10+.
+    normalized_directory = os.path.join(working_directory or '', expanded_directory)
+    glob_paths = glob.glob(normalized_directory)
+
+    if not glob_paths:
+        return [expanded_directory]
+
+    working_directory_prefix = os.path.join(working_directory or '', '')
+
+    return [
+        (
+            glob_path
+            # If these are equal, that means we didn't add any working directory prefix above.
+            if normalized_directory == expanded_directory
+            # Remove the working directory prefix that we added above in order to make glob() work.
+            # We can't use os.path.relpath() here because it collapses any use of Borg's slashdot
+            # hack.
+            else glob_path.removeprefix(working_directory_prefix)
+        )
+        for glob_path in glob_paths
+    ]
+
+
+def expand_patterns(patterns, working_directory=None, skip_paths=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    expand tildes and globs in each root pattern. Return all the resulting patterns (not just the
+    root patterns) as a tuple.
+
+    If a set of paths are given to skip, then don't expand any patterns matching them.
+    '''
+    if patterns is None:
+        return ()
+
+    return tuple(
+        itertools.chain.from_iterable(
+            (
+                (
+                    borgmatic.borg.pattern.Pattern(
+                        expanded_path,
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                    )
+                    for expanded_path in expand_directory(pattern.path, working_directory)
+                )
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.path not in (skip_paths or ())
+                else (pattern,)
+            )
+            for pattern in patterns
+        )
+    )
+
+
+def device_map_patterns(patterns, working_directory=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    determine the identifier for the device on which the pattern's path resides—or None if the path
+    doesn't exist or is from a non-root pattern. Return an updated sequence of patterns with the
+    device field populated. But if the device field is already set, don't bother setting it again.
+
+    This is handy for determining whether two different pattern paths are on the same filesystem
+    (have the same device identifier).
+    '''
+    return tuple(
+        borgmatic.borg.pattern.Pattern(
+            pattern.path,
+            pattern.type,
+            pattern.style,
+            device=pattern.device
+            or (
+                os.stat(full_path).st_dev
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and os.path.exists(full_path)
+                else None
+            ),
+        )
+        for pattern in patterns
+        for full_path in (os.path.join(working_directory or '', pattern.path),)
+    )
+
+
+def deduplicate_patterns(patterns):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, return them with all duplicate
+    root child patterns removed. For instance, if two root patterns are given with paths "/foo" and
+    "/foo/bar", return just the one with "/foo". Non-root patterns are passed through without
+    modification.
+
+    The one exception to deduplication is two paths are on different filesystems (devices). In that
+    case, they won't get deduplicated, in case they both need to be passed to Borg (e.g. the
+    one_file_system option is true).
+
+    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    child patterns, because it will naturally spider the contents of the parent pattern's path. And
+    there are cases where Borg coming across the same file twice will result in duplicate reads and
+    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
+    Borg.
+    '''
+    deduplicated = {}  # Use just the keys as an ordered set.
+
+    for pattern in patterns:
+        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+            deduplicated[pattern] = True
+            continue
+
+        parents = pathlib.PurePath(pattern.path).parents
+
+        # If another directory in the given list is a parent of current directory (even n levels up)
+        # and both are on the same filesystem, then the current directory is a duplicate.
+        for other_pattern in patterns:
+            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+                continue
+
+            if any(
+                pathlib.PurePath(other_pattern.path) == parent
+                and pattern.device is not None
+                and other_pattern.device == pattern.device
+                for parent in parents
+            ):
+                break
+        else:
+            deduplicated[pattern] = True
+
+    return tuple(deduplicated.keys())
+
+
+def process_patterns(patterns, working_directory, skip_expand_paths=None):
+    '''
+    Given a sequence of Borg patterns and a configured working directory, expand and deduplicate any
+    "root" patterns, returning the resulting root and non-root patterns as a list.
+
+    If any paths are given to skip, don't expand them.
+    '''
+    skip_paths = set(skip_expand_paths or ())
+
+    return list(
+        deduplicate_patterns(
+            device_map_patterns(
+                expand_patterns(
+                    patterns,
+                    working_directory=working_directory,
+                    skip_paths=skip_paths,
+                )
+            )
+        )
+    )
 
 
 def run_create(
@@ -72,43 +283,39 @@ def run_create(
         **hook_context,
     )
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Creating archive{dry_run_label}')
+    logger.info(f'Creating archive{dry_run_label}')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
+        patterns = process_patterns(collect_patterns(config), working_directory)
         active_dumps = borgmatic.hooks.dispatch.call_hooks(
             'dump_data_sources',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            config_paths,
             borgmatic_runtime_directory,
+            patterns,
             global_arguments.dry_run,
         )
-        stream_processes = [process for processes in active_dumps.values() for process in processes]
 
-        if config.get('store_config_files', True):
-            create_borgmatic_manifest(
-                config,
-                config_paths,
-                borgmatic_runtime_directory,
-                global_arguments.dry_run,
-            )
+        # Process the patterns again in case any data source hooks updated them. Without this step,
+        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+        # the same named pipe twice.
+        patterns = process_patterns(patterns, working_directory, skip_expand_paths=config_paths)
+        stream_processes = [process for processes in active_dumps.values() for process in processes]
 
         json_output = borgmatic.borg.create.create_archive(
             global_arguments.dry_run,
             repository['path'],
             config,
-            config_paths,
+            patterns,
             local_borg_version,
             global_arguments,
             borgmatic_runtime_directory,
@@ -127,8 +334,7 @@ def run_create(
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            config_filename,
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )

borgmatic/actions/delete.py

@@ -23,7 +23,7 @@ def run_delete(
     if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, delete_arguments.repository
     ):
-        logger.answer(f'{repository.get("label", repository["path"])}: Deleting archives')
+        logger.answer('Deleting archives')
 
         archive_name = (
             borgmatic.borg.repo_list.resolve_archive_name(

borgmatic/actions/export_key.py

@@ -21,7 +21,7 @@ def run_export_key(
     if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_arguments.repository
     ):
-        logger.info(f'{repository.get("label", repository["path"])}: Exporting repository key')
+        logger.info('Exporting repository key')
         borgmatic.borg.export_key.export_key(
             repository['path'],
             config,

borgmatic/actions/export_tar.py

@@ -22,9 +22,7 @@ def run_export_tar(
     if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_tar_arguments.repository
     ):
-        logger.info(
-            f'{repository["path"]}: Exporting archive {export_tar_arguments.archive} as tar file'
-        )
+        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
         borgmatic.borg.export_tar.export_tar_archive(
             global_arguments.dry_run,
             repository['path'],

borgmatic/actions/extract.py

@@ -33,9 +33,7 @@ def run_extract(
     if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, extract_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Extracting archive {extract_arguments.archive}'
-        )
+        logger.info(f'Extracting archive {extract_arguments.archive}')
         borgmatic.borg.extract.extract_archive(
             global_arguments.dry_run,
             repository['path'],

borgmatic/actions/info.py

@@ -27,9 +27,7 @@ def run_info(
         repository, info_arguments.repository
     ):
         if not info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying archive summary information'
-            )
+            logger.answer('Displaying archive summary information')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             info_arguments.archive,

borgmatic/actions/list.py

@@ -27,9 +27,9 @@ def run_list(
     ):
         if not list_arguments.json:
             if list_arguments.find_paths:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
+                logger.answer('Searching archives')
             elif not list_arguments.archive:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
+                logger.answer('Listing archives')
 
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],

borgmatic/actions/mount.py

@@ -23,11 +23,9 @@ def run_mount(
         repository, mount_arguments.repository
     ):
         if mount_arguments.archive:
-            logger.info(
-                f'{repository.get("label", repository["path"])}: Mounting archive {mount_arguments.archive}'
-            )
+            logger.info(f'Mounting archive {mount_arguments.archive}')
         else:  # pragma: nocover
-            logger.info(f'{repository.get("label", repository["path"])}: Mounting repository')
+            logger.info('Mounting repository')
 
         borgmatic.borg.mount.mount_archive(
             repository['path'],

borgmatic/actions/prune.py

@@ -35,7 +35,7 @@ def run_prune(
         global_arguments.dry_run,
         **hook_context,
     )
-    logger.info(f'{repository.get("label", repository["path"])}: Pruning archives{dry_run_label}')
+    logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/actions/repo_create.py

@@ -23,7 +23,7 @@ def run_repo_create(
     ):
         return
 
-    logger.info(f'{repository.get("label", repository["path"])}: Creating repository')
+    logger.info('Creating repository')
     borgmatic.borg.repo_create.create_repository(
         global_arguments.dry_run,
         repository['path'],

borgmatic/actions/repo_delete.py

@@ -21,8 +21,7 @@ def run_repo_delete(
         repository, repo_delete_arguments.repository
     ):
         logger.answer(
-            f'{repository.get("label", repository["path"])}: Deleting repository'
-            + (' cache' if repo_delete_arguments.cache_only else '')
+            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else '')
         )
 
         borgmatic.borg.repo_delete.delete_repository(

borgmatic/actions/repo_info.py

@@ -25,9 +25,7 @@ def run_repo_info(
         repository, repo_info_arguments.repository
     ):
         if not repo_info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying repository summary information'
-            )
+            logger.answer('Displaying repository summary information')
 
         json_output = borgmatic.borg.repo_info.display_repository_info(
             repository['path'],

borgmatic/actions/repo_list.py

@@ -25,7 +25,7 @@ def run_repo_list(
         repository, repo_list_arguments.repository
     ):
         if not repo_list_arguments.json:
-            logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
+            logger.answer('Listing repository')
 
         json_output = borgmatic.borg.repo_list.list_repository(
             repository['path'],

borgmatic/actions/restore.py

@@ -1,4 +1,4 @@
-import copy
+import collections
 import logging
 import os
 import pathlib
@@ -11,58 +11,111 @@ import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
+import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
 
 logger = logging.getLogger(__name__)
 
 
-UNSPECIFIED_HOOK = object()
+UNSPECIFIED = object()
 
 
-def get_configured_data_source(
-    config,
-    archive_data_source_names,
-    hook_name,
-    data_source_name,
-    configuration_data_source_name=None,
-):
+Dump = collections.namedtuple(
+    'Dump',
+    ('hook_name', 'data_source_name', 'hostname', 'port'),
+    defaults=('localhost', None),
+)
+
+
+def dumps_match(first, second, default_port=None):
     '''
-    Find the first data source with the given hook name and data source name in the configuration
-    dict and the given archive data source names dict (from hook name to data source names contained
-    in a particular backup archive). If UNSPECIFIED_HOOK is given as the hook name, search all data
-    source hooks for the named data source. If a configuration data source name is given, use that
-    instead of the data source name to lookup the data source in the given hooks configuration.
-
-    Return the found data source as a tuple of (found hook name, data source configuration dict) or
-    (None, None) if not found.
+    Compare two Dump instances for equality while supporting a field value of UNSPECIFIED, which
+    indicates that the field should match any value. If a default port is given, then consider any
+    dump having that port to match with a dump having a None port.
     '''
-    if not configuration_data_source_name:
-        configuration_data_source_name = data_source_name
+    for field_name in first._fields:
+        first_value = getattr(first, field_name)
+        second_value = getattr(second, field_name)
 
-    if hook_name == UNSPECIFIED_HOOK:
-        hooks_to_search = {
-            hook_name: value
-            for (hook_name, value) in config.items()
-            if hook_name in borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES
-        }
+        if default_port is not None and field_name == 'port':
+            if first_value == default_port and second_value is None:
+                continue
+            if second_value == default_port and first_value is None:
+                continue
+
+        if first_value == UNSPECIFIED or second_value == UNSPECIFIED:
+            continue
+
+        if first_value != second_value:
+            return False
+
+    return True
+
+
+def render_dump_metadata(dump):
+    '''
+    Given a Dump instance, make a display string describing it for use in log messages.
+    '''
+    name = 'unspecified' if dump.data_source_name is UNSPECIFIED else dump.data_source_name
+    hostname = dump.hostname or UNSPECIFIED
+    port = None if dump.port is UNSPECIFIED else dump.port
+
+    if port:
+        metadata = f'{name}@:{port}' if hostname is UNSPECIFIED else f'{name}@{hostname}:{port}'
     else:
-        try:
-            hooks_to_search = {hook_name: config[hook_name]}
-        except KeyError:
-            return (None, None)
+        metadata = f'{name}' if hostname is UNSPECIFIED else f'{name}@{hostname}'
 
-    return next(
-        (
-            (name, hook_data_source)
-            for (name, hook) in hooks_to_search.items()
-            for hook_data_source in hook
-            if hook_data_source['name'] == configuration_data_source_name
-            and data_source_name in archive_data_source_names.get(name, [])
-        ),
-        (None, None),
+    if dump.hook_name not in (None, UNSPECIFIED):
+        return f'{metadata} ({dump.hook_name})'
+
+    return metadata
+
+
+def get_configured_data_source(config, restore_dump):
+    '''
+    Search in the given configuration dict for dumps corresponding to the given dump to restore. If
+    there are multiple matches, error.
+
+    Return the found data source as a data source configuration dict or None if not found.
+    '''
+    try:
+        hooks_to_search = {restore_dump.hook_name: config[restore_dump.hook_name]}
+    except KeyError:
+        return None
+
+    matching_dumps = tuple(
+        hook_data_source
+        for (hook_name, hook_config) in hooks_to_search.items()
+        for hook_data_source in hook_config
+        for default_port in (
+            borgmatic.hooks.dispatch.call_hook(
+                function_name='get_default_port',
+                config=config,
+                hook_name=hook_name,
+            ),
+        )
+        if dumps_match(
+            Dump(
+                hook_name,
+                hook_data_source.get('name'),
+                hook_data_source.get('hostname', 'localhost'),
+                hook_data_source.get('port'),
+            ),
+            restore_dump,
+            default_port,
+        )
     )
 
+    if not matching_dumps:
+        return None
+
+    if len(matching_dumps) > 1:
+        raise ValueError(
+            f'Cannot restore data source {render_dump_metadata(restore_dump)} because there are multiple matching data sources configured'
+        )
+
+    return matching_dumps[0]
+
 
 def strip_path_prefix_from_extracted_dump_destination(
     destination_path, borgmatic_runtime_directory
@@ -97,7 +150,7 @@ def strip_path_prefix_from_extracted_dump_destination(
         break
 
 
-def restore_single_data_source(
+def restore_single_dump(
     repository,
     config,
     local_borg_version,
@@ -115,18 +168,19 @@ def restore_single_data_source(
     username/password as connection params, and a configured data source configuration dict, restore
     that data source from the archive.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Restoring data source {data_source["name"]}'
+    dump_metadata = render_dump_metadata(
+        Dump(hook_name, data_source['name'], data_source.get('hostname'), data_source.get('port'))
     )
 
+    logger.info(f'Restoring data source {dump_metadata}')
+
     dump_patterns = borgmatic.hooks.dispatch.call_hooks(
         'make_data_source_dump_patterns',
         config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+        borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         borgmatic_runtime_directory,
         data_source['name'],
-    )[hook_name]
+    )[hook_name.split('_databases', 1)[0]]
 
     destination_path = (
         tempfile.mkdtemp(dir=borgmatic_runtime_directory)
@@ -141,7 +195,11 @@ def restore_single_data_source(
             dry_run=global_arguments.dry_run,
             repository=repository['path'],
             archive=archive_name,
-            paths=[borgmatic.hooks.dump.convert_glob_patterns_to_borg_pattern(dump_patterns)],
+            paths=[
+                borgmatic.hooks.data_source.dump.convert_glob_patterns_to_borg_pattern(
+                    dump_patterns
+                )
+            ],
             config=config,
             local_borg_version=local_borg_version,
             global_arguments=global_arguments,
@@ -162,11 +220,10 @@ def restore_single_data_source(
             shutil.rmtree(destination_path, ignore_errors=True)
 
     # Run a single data source restore, consuming the extract stdout (if any).
-    borgmatic.hooks.dispatch.call_hooks(
+    borgmatic.hooks.dispatch.call_hook(
         function_name='restore_data_source_dump',
         config=config,
-        log_prefix=repository['path'],
-        hook_names=[hook_name],
+        hook_name=hook_name,
         data_source=data_source,
         dry_run=global_arguments.dry_run,
         extract_process=extract_process,
@@ -175,7 +232,7 @@ def restore_single_data_source(
     )
 
 
-def collect_archive_data_source_names(
+def collect_dumps_from_archive(
     repository,
     archive,
     config,
@@ -187,17 +244,17 @@ def collect_archive_data_source_names(
 ):
     '''
     Given a local or remote repository path, a resolved archive name, a configuration dict, the
-    local Borg version, global_arguments an argparse.Namespace, local and remote Borg paths, and the
-    borgmatic runtime directory, query the archive for the names of data sources it contains as
-    dumps and return them as a dict from hook name to a sequence of data source names.
+    local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
+    borgmatic runtime directory, query the archive for the names of data sources dumps it contains
+    and return them as a set of Dump instances.
     '''
     borgmatic_source_directory = str(
         pathlib.Path(borgmatic.config.paths.get_borgmatic_source_directory(config))
     )
 
     # Probe for the data source dumps in multiple locations, as the default location has moved to
-    # the borgmatic runtime directory (which get stored as just "/borgmatic" with Borg 1.4+). But we
-    # still want to support reading dumps from previously created archives as well.
+    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
+    # we still want to support reading dumps from previously created archives as well.
     dump_paths = borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
@@ -206,7 +263,9 @@ def collect_archive_data_source_names(
         global_arguments,
         list_paths=[
             'sh:'
-            + borgmatic.hooks.dump.make_data_source_dump_path(base_directory, '*_databases/*/*')
+            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
+                base_directory, '*_databases/*/*'
+            )
             for base_directory in (
                 'borgmatic',
                 borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
@@ -217,110 +276,148 @@ def collect_archive_data_source_names(
         remote_path=remote_path,
     )
 
-    # Determine the data source names corresponding to the dumps found in the archive and
-    # add them to restore_names.
-    archive_data_source_names = {}
+    # Parse the paths of dumps found in the archive to get their respective dump metadata.
+    dumps_from_archive = set()
 
     for dump_path in dump_paths:
         if not dump_path:
             continue
 
+        # Probe to find the base directory that's at the start of the dump path.
         for base_directory in (
             'borgmatic',
             borgmatic_runtime_directory,
             borgmatic_source_directory,
         ):
             try:
-                (hook_name, _, data_source_name) = dump_path.split(base_directory + os.path.sep, 1)[
-                    1
-                ].split(os.path.sep)[0:3]
+                (hook_name, host_and_port, data_source_name) = dump_path.split(
+                    base_directory + os.path.sep, 1
+                )[1].split(os.path.sep)[0:3]
             except (ValueError, IndexError):
-                pass
-            else:
-                if data_source_name not in archive_data_source_names.get(hook_name, []):
-                    archive_data_source_names.setdefault(hook_name, []).extend([data_source_name])
-                    break
+                continue
+
+            parts = host_and_port.split(':', 1)
+
+            if len(parts) == 1:
+                parts += (None,)
+
+            (hostname, port) = parts
+
+            try:
+                port = int(port)
+            except (ValueError, TypeError):
+                port = None
+
+            dumps_from_archive.add(Dump(hook_name, data_source_name, hostname, port))
+
+            # We've successfully parsed the dump path, so need to probe any further.
+            break
         else:
             logger.warning(
-                f'{repository}: Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
+                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
             )
 
-    return archive_data_source_names
+    return dumps_from_archive
 
 
-def find_data_sources_to_restore(requested_data_source_names, archive_data_source_names):
+def get_dumps_to_restore(restore_arguments, dumps_from_archive):
     '''
-    Given a sequence of requested data source names to restore and a dict of hook name to the names
-    of data sources found in an archive, return an expanded sequence of data source names to
-    restore, replacing "all" with actual data source names as appropriate.
+    Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
+    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    Dump instances from the archive to restore. As part of this, replace any Dump having a data
+    source name of "all" with multiple named Dump instances as appropriate.
 
-    Raise ValueError if any of the requested data source names cannot be found in the archive.
+    Raise ValueError if any of the requested data source names cannot be found in the archive or if
+    there are multiple archive dump matches for a given requested dump.
     '''
-    # A map from data source hook name to the data source names to restore for that hook.
-    restore_names = (
-        {UNSPECIFIED_HOOK: requested_data_source_names}
-        if requested_data_source_names
-        else {UNSPECIFIED_HOOK: ['all']}
+    requested_dumps = (
+        {
+            Dump(
+                hook_name=(
+                    (
+                        restore_arguments.hook
+                        if restore_arguments.hook.endswith('_databases')
+                        else f'{restore_arguments.hook}_databases'
+                    )
+                    if restore_arguments.hook
+                    else UNSPECIFIED
+                ),
+                data_source_name=name,
+                hostname=restore_arguments.original_hostname or UNSPECIFIED,
+                port=restore_arguments.original_port,
+            )
+            for name in restore_arguments.data_sources or (UNSPECIFIED,)
+        }
+        if restore_arguments.hook
+        or restore_arguments.data_sources
+        or restore_arguments.original_hostname
+        or restore_arguments.original_port
+        else {
+            Dump(
+                hook_name=UNSPECIFIED,
+                data_source_name='all',
+                hostname=UNSPECIFIED,
+                port=UNSPECIFIED,
+            )
+        }
     )
+    missing_dumps = set()
+    dumps_to_restore = set()
 
-    # If "all" is in restore_names, then replace it with the names of dumps found within the
-    # archive.
-    if 'all' in restore_names[UNSPECIFIED_HOOK]:
-        restore_names[UNSPECIFIED_HOOK].remove('all')
+    # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
+    if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
+        dumps_to_restore.update(dumps_from_archive)
 
-        for hook_name, data_source_names in archive_data_source_names.items():
-            restore_names.setdefault(hook_name, []).extend(data_source_names)
+    # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
+    for requested_dump in requested_dumps:
+        if requested_dump.data_source_name == 'all':
+            continue
 
-            # If a data source is to be restored as part of "all", then remove it from restore names
-            # so it doesn't get restored twice.
-            for data_source_name in data_source_names:
-                if data_source_name in restore_names[UNSPECIFIED_HOOK]:
-                    restore_names[UNSPECIFIED_HOOK].remove(data_source_name)
-
-    if not restore_names[UNSPECIFIED_HOOK]:
-        restore_names.pop(UNSPECIFIED_HOOK)
-
-    combined_restore_names = set(
-        name for data_source_names in restore_names.values() for name in data_source_names
-    )
-    combined_archive_data_source_names = set(
-        name
-        for data_source_names in archive_data_source_names.values()
-        for name in data_source_names
-    )
-
-    missing_names = sorted(set(combined_restore_names) - combined_archive_data_source_names)
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
-        raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from archive"
+        matching_dumps = tuple(
+            archive_dump
+            for archive_dump in dumps_from_archive
+            if dumps_match(requested_dump, archive_dump)
         )
 
-    return restore_names
+        if len(matching_dumps) == 0:
+            missing_dumps.add(requested_dump)
+        elif len(matching_dumps) == 1:
+            dumps_to_restore.add(matching_dumps[0])
+        else:
+            raise ValueError(
+                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.'
+            )
+
+    if missing_dumps:
+        rendered_dumps = ', '.join(
+            f'{render_dump_metadata(dump)}' for dump in sorted(missing_dumps)
+        )
+
+        raise ValueError(
+            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive"
+        )
+
+    return dumps_to_restore
 
 
-def ensure_data_sources_found(restore_names, remaining_restore_names, found_names):
+def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
     '''
-    Given a dict from hook name to data source names to restore, a dict from hook name to remaining
-    data source names to restore, and a sequence of found (actually restored) data source names,
-    raise ValueError if requested data source to restore were missing from the archive and/or
+    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    if any requested dumps to restore weren't restored, indicating that they were missing from the
     configuration.
     '''
-    combined_restore_names = set(
-        name
-        for data_source_names in tuple(restore_names.values())
-        + tuple(remaining_restore_names.values())
-        for name in data_source_names
-    )
-
-    if not combined_restore_names and not found_names:
+    if not dumps_actually_restored:
         raise ValueError('No data source dumps were found to restore')
 
-    missing_names = sorted(set(combined_restore_names) - set(found_names))
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
+    missing_dumps = sorted(
+        dumps_to_restore - dumps_actually_restored, key=lambda dump: dump.data_source_name
+    )
+
+    if missing_dumps:
+        rendered_dumps = ', '.join(f'{render_dump_metadata(dump)}' for dump in missing_dumps)
+
         raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from borgmatic's configuration"
+            f"Cannot restore data source{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from borgmatic's configuration"
         )
 
 
@@ -337,24 +434,21 @@ def run_restore(
     Run the "restore" action for the given repository, but only if the repository matches the
     requested repository in restore arguments.
 
-    Raise ValueError if a configured data source could not be found to restore.
+    Raise ValueError if a configured data source could not be found to restore or there's no
+    matching dump in the archive.
     '''
     if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
         repository, restore_arguments.repository
     ):
         return
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Restoring data sources from archive {restore_arguments.archive}')
+    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
 
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
@@ -368,7 +462,7 @@ def run_restore(
             local_path,
             remote_path,
         )
-        archive_data_source_names = collect_archive_data_source_names(
+        dumps_from_archive = collect_dumps_from_archive(
             repository['path'],
             archive_name,
             config,
@@ -378,11 +472,9 @@ def run_restore(
             remote_path,
             borgmatic_runtime_directory,
         )
-        restore_names = find_data_sources_to_restore(
-            restore_arguments.data_sources, archive_data_source_names
-        )
-        found_names = set()
-        remaining_restore_names = {}
+        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
+
+        dumps_actually_restored = set()
         connection_params = {
             'hostname': restore_arguments.hostname,
             'port': restore_arguments.port,
@@ -391,69 +483,49 @@ def run_restore(
             'restore_path': restore_arguments.restore_path,
         }
 
-        for hook_name, data_source_names in restore_names.items():
-            for data_source_name in data_source_names:
-                found_hook_name, found_data_source = get_configured_data_source(
-                    config, archive_data_source_names, hook_name, data_source_name
-                )
+        # Restore each dump.
+        for restore_dump in dumps_to_restore:
+            found_data_source = get_configured_data_source(
+                config,
+                restore_dump,
+            )
 
-                if not found_data_source:
-                    remaining_restore_names.setdefault(found_hook_name or hook_name, []).append(
-                        data_source_name
-                    )
-                    continue
-
-                found_names.add(data_source_name)
-                restore_single_data_source(
-                    repository,
+            # For a dump that wasn't found via an exact match in the configuration, try to fallback
+            # to an "all" data source.
+            if not found_data_source:
+                found_data_source = get_configured_data_source(
                     config,
-                    local_borg_version,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                    archive_name,
-                    found_hook_name or hook_name,
-                    dict(found_data_source, **{'schemas': restore_arguments.schemas}),
-                    connection_params,
-                    borgmatic_runtime_directory,
-                )
-
-        # For any data sources that weren't found via exact matches in the configuration, try to
-        # fallback to "all" entries.
-        for hook_name, data_source_names in remaining_restore_names.items():
-            for data_source_name in data_source_names:
-                found_hook_name, found_data_source = get_configured_data_source(
-                    config, archive_data_source_names, hook_name, data_source_name, 'all'
+                    Dump(restore_dump.hook_name, 'all', restore_dump.hostname, restore_dump.port),
                 )
 
                 if not found_data_source:
                     continue
 
-                found_names.add(data_source_name)
-                data_source = copy.copy(found_data_source)
-                data_source['name'] = data_source_name
+                found_data_source = dict(found_data_source)
+                found_data_source['name'] = restore_dump.data_source_name
 
-                restore_single_data_source(
-                    repository,
-                    config,
-                    local_borg_version,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                    archive_name,
-                    found_hook_name or hook_name,
-                    dict(data_source, **{'schemas': restore_arguments.schemas}),
-                    connection_params,
-                    borgmatic_runtime_directory,
-                )
+            dumps_actually_restored.add(restore_dump)
+
+            restore_single_dump(
+                repository,
+                config,
+                local_borg_version,
+                global_arguments,
+                local_path,
+                remote_path,
+                archive_name,
+                restore_dump.hook_name,
+                dict(found_data_source, **{'schemas': restore_arguments.schemas}),
+                connection_params,
+                borgmatic_runtime_directory,
+            )
 
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
 
-    ensure_data_sources_found(restore_names, remaining_restore_names, found_names)
+    ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)

borgmatic/actions/transfer.py

@@ -17,9 +17,7 @@ def run_transfer(
     '''
     Run the "transfer" action for the given repository.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Transferring archives to repository'
-    )
+    logger.info('Transferring archives to repository')
     borgmatic.borg.transfer.transfer_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/borg/change_passphrase.py

@@ -41,7 +41,7 @@ def change_passphrase(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping change password (dry run)')
+        logger.info('Skipping change password (dry run)')
         return
 
     # If the original passphrase is set programmatically, then Borg won't prompt for a new one! So

borgmatic/borg/compact.py

@@ -43,7 +43,7 @@ def compact_segments(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping compact (dry run)')
+        logging.info('Skipping compact (dry run)')
         return
 
     execute_command(

borgmatic/borg/create.py

@@ -1,4 +1,3 @@
-import glob
 import itertools
 import logging
 import os
@@ -7,6 +6,7 @@ import stat
 import tempfile
 import textwrap
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
@@ -20,167 +20,42 @@ from borgmatic.execute import (
 logger = logging.getLogger(__name__)
 
 
-def expand_directory(directory, working_directory):
+def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
     '''
-    Given a directory path, expand any tilde (representing a user's home directory) and any globs
-    therein. Return a list of one or more resulting paths.
-    '''
-    expanded_directory = os.path.join(working_directory or '', os.path.expanduser(directory))
+    Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
+    temporary file in the given borgmatic runtime directory and return the file object so it can
+    continue to exist on disk as long as the caller needs it.
 
-    return glob.glob(expanded_directory) or [expanded_directory]
-
-
-def expand_directories(directories, working_directory=None):
-    '''
-    Given a sequence of directory paths and an optional working directory, expand tildes and globs
-    in each one. Return all the resulting directories as a single flattened tuple.
-    '''
-    if directories is None:
-        return ()
-
-    return tuple(
-        itertools.chain.from_iterable(
-            expand_directory(directory, working_directory) for directory in directories
-        )
-    )
-
-
-def expand_home_directories(directories):
-    '''
-    Given a sequence of directory paths, expand tildes in each one. Do not perform any globbing.
-    Return the results as a tuple.
-    '''
-    if directories is None:
-        return ()
-
-    return tuple(os.path.expanduser(directory) for directory in directories)
-
-
-def map_directories_to_devices(directories, working_directory=None):
-    '''
-    Given a sequence of directories and an optional working directory, return a map from directory
-    to an identifier for the device on which that directory resides or None if the path doesn't
-    exist.
-
-    This is handy for determining whether two different directories are on the same filesystem (have
-    the same device identifier).
-    '''
-    return {
-        directory: os.stat(full_directory).st_dev if os.path.exists(full_directory) else None
-        for directory in directories
-        for full_directory in (os.path.join(working_directory or '', directory),)
-    }
-
-
-def deduplicate_directories(directory_devices, additional_directory_devices):
-    '''
-    Given a map from directory to the identifier for the device on which that directory resides,
-    return the directories as a sorted tuple with all duplicate child directories removed. For
-    instance, if paths is ('/foo', '/foo/bar'), return just: ('/foo',)
-
-    The one exception to this rule is if two paths are on different filesystems (devices). In that
-    case, they won't get de-duplicated in case they both need to be passed to Borg (e.g. the
-    location.one_file_system option is true).
-
-    The idea is that if Borg is given a parent directory, then it doesn't also need to be given
-    child directories, because it will naturally spider the contents of the parent directory. And
-    there are cases where Borg coming across the same file twice will result in duplicate reads and
-    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
-    Borg.
-
-    If any additional directory devices are given, also deduplicate against them, but don't include
-    them in the returned directories.
-    '''
-    deduplicated = set()
-    directories = sorted(directory_devices.keys())
-    additional_directories = sorted(additional_directory_devices.keys())
-    all_devices = {**directory_devices, **additional_directory_devices}
-
-    for directory in directories:
-        deduplicated.add(directory)
-        parents = pathlib.PurePath(directory).parents
-
-        # If another directory in the given list (or the additional list) is a parent of current
-        # directory (even n levels up) and both are on the same filesystem, then the current
-        # directory is a duplicate.
-        for other_directory in directories + additional_directories:
-            for parent in parents:
-                if (
-                    pathlib.PurePath(other_directory) == parent
-                    and all_devices[directory] is not None
-                    and all_devices[other_directory] == all_devices[directory]
-                ):
-                    if directory in deduplicated:
-                        deduplicated.remove(directory)
-                    break
-
-    return tuple(sorted(deduplicated))
-
-
-def write_pattern_file(patterns=None, sources=None, pattern_file=None):
-    '''
-    Given a sequence of patterns and an optional sequence of source directories, write them to a
-    named temporary file (with the source directories as additional roots) and return the file.
-    If an optional open pattern file is given, overwrite it instead of making a new temporary file.
+    If an optional open pattern file is given, append to it instead of making a new temporary file.
     Return None if no patterns are provided.
     '''
-    if not patterns and not sources:
+    if not patterns:
         return None
 
-    if pattern_file is None:
-        pattern_file = tempfile.NamedTemporaryFile('w')
+    if patterns_file is None:
+        patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
+        operation_name = 'Writing'
     else:
-        pattern_file.seek(0)
+        patterns_file.write('\n')
+        operation_name = 'Appending'
 
-    pattern_file.write(
-        '\n'.join(tuple(patterns or ()) + tuple(f'R {source}' for source in (sources or [])))
+    patterns_output = '\n'.join(
+        f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
+        for pattern in patterns
     )
-    pattern_file.flush()
+    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
 
-    return pattern_file
+    patterns_file.write(patterns_output)
+    patterns_file.flush()
+
+    return patterns_file
 
 
-def ensure_files_readable(*filename_lists):
+def make_exclude_flags(config):
     '''
-    Given a sequence of filename sequences, ensure that each filename is openable. This prevents
-    unreadable files from being passed to Borg, which in certain situations only warns instead of
-    erroring.
+    Given a configuration dict with various exclude options, return the corresponding Borg flags as
+    a tuple.
     '''
-    for file_object in itertools.chain.from_iterable(
-        filename_list for filename_list in filename_lists if filename_list
-    ):
-        open(file_object).close()
-
-
-def make_pattern_flags(config, pattern_filename=None):
-    '''
-    Given a configuration dict with a potential patterns_from option, and a filename containing any
-    additional patterns, return the corresponding Borg flags for those files as a tuple.
-    '''
-    pattern_filenames = tuple(config.get('patterns_from') or ()) + (
-        (pattern_filename,) if pattern_filename else ()
-    )
-
-    return tuple(
-        itertools.chain.from_iterable(
-            ('--patterns-from', pattern_filename) for pattern_filename in pattern_filenames
-        )
-    )
-
-
-def make_exclude_flags(config, exclude_filename=None):
-    '''
-    Given a configuration dict with various exclude options, and a filename containing any exclude
-    patterns, return the corresponding Borg flags as a tuple.
-    '''
-    exclude_filenames = tuple(config.get('exclude_from') or ()) + (
-        (exclude_filename,) if exclude_filename else ()
-    )
-    exclude_from_flags = tuple(
-        itertools.chain.from_iterable(
-            ('--exclude-from', exclude_filename) for exclude_filename in exclude_filenames
-        )
-    )
     caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
     if_present_flags = tuple(
         itertools.chain.from_iterable(
@@ -191,13 +66,7 @@ def make_exclude_flags(config, exclude_filename=None):
     keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
     exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
 
-    return (
-        exclude_from_flags
-        + caches_flag
-        + if_present_flags
-        + keep_exclude_tags_flags
-        + exclude_nodump_flags
-    )
+    return caches_flag + if_present_flags + keep_exclude_tags_flags + exclude_nodump_flags
 
 
 def make_list_filter_flags(local_borg_version, dry_run):
@@ -221,39 +90,14 @@ def make_list_filter_flags(local_borg_version, dry_run):
         return f'{base_flags}-'
 
 
-def collect_borgmatic_runtime_directories(borgmatic_runtime_directory):
-    '''
-    Return a list of borgmatic-specific runtime directories used for temporary runtime data like
-    streaming database dumps and bootstrap metadata. If no such directories exist, return an empty
-    list.
-    '''
-    return [borgmatic_runtime_directory] if os.path.exists(borgmatic_runtime_directory) else []
-
-
-ROOT_PATTERN_PREFIX = 'R '
-
-
-def pattern_root_directories(patterns=None):
-    '''
-    Given a sequence of patterns, parse out and return just the root directories.
-    '''
-    if not patterns:
-        return []
-
-    return [
-        pattern.split(ROOT_PATTERN_PREFIX, maxsplit=1)[1]
-        for pattern in patterns
-        if pattern.startswith(ROOT_PATTERN_PREFIX)
-    ]
-
-
-def special_file(path):
+def special_file(path, working_directory=None):
     '''
     Return whether the given path is a special file (character device, block device, or named pipe
-    / FIFO).
+    / FIFO). If a working directory is given, take it into account when making the full path to
+    check.
     '''
     try:
-        mode = os.stat(path).st_mode
+        mode = os.stat(os.path.join(working_directory or '', path)).st_mode
     except (FileNotFoundError, OSError):
         return False
 
@@ -273,14 +117,25 @@ def any_parent_directories(path, candidate_parents):
 
 
 def collect_special_file_paths(
-    create_command, config, local_path, working_directory, borg_environment, skip_directories
+    dry_run,
+    create_command,
+    config,
+    local_path,
+    working_directory,
+    borg_environment,
+    borgmatic_runtime_directory,
 ):
     '''
-    Given a Borg create command as a tuple, a configuration dict, a local Borg path, a working
-    directory, a dict of environment variables to pass to Borg, and a sequence of parent directories
-    to skip, collect the paths for any special files (character devices, block devices, and named
+    Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
+    a working directory, a dict of environment variables to pass to Borg, and the borgmatic runtime
+    directory, collect the paths for any special files (character devices, block devices, and named
     pipes / FIFOs) that Borg would encounter during a create. These are all paths that could cause
     Borg to hang if its --read-special flag is used.
+
+    Skip looking for special files in the given borgmatic runtime directory, as borgmatic creates
+    its own special files there for database dumps. And if the borgmatic runtime directory is
+    configured to be excluded from the files Borg backs up, error, because this means Borg won't be
+    able to consume any database dumps and therefore borgmatic will hang.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
     # files including any named pipe we've created.
@@ -299,32 +154,39 @@ def collect_special_file_paths(
         for path_line in paths_output.split('\n')
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
+    skip_paths = {}
+
+    if os.path.exists(borgmatic_runtime_directory):
+        skip_paths = {
+            path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+        }
+
+        if not skip_paths and not dry_run:
+            raise ValueError(
+                f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.'
+            )
 
     return tuple(
-        path
-        for path in paths
-        if special_file(path) and not any_parent_directories(path, skip_directories)
+        path for path in paths if special_file(path, working_directory) if path not in skip_paths
     )
 
 
-def check_all_source_directories_exist(source_directories, working_directory=None):
+def check_all_root_patterns_exist(patterns):
     '''
-    Given a sequence of source directories and an optional working directory to serve as a prefix
-    for each (if it's a relative directory), check that the source directories all exist. If any do
-    not, raise an exception.
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, check that all root pattern
+    paths exist. If any don't, raise an exception.
     '''
-    missing_directories = [
-        source_directory
-        for source_directory in source_directories
-        if not all(
-            [
-                os.path.exists(os.path.join(working_directory or '', directory))
-                for directory in expand_directory(source_directory, working_directory)
-            ]
-        )
+    missing_paths = [
+        pattern.path
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        if not os.path.exists(pattern.path)
     ]
-    if missing_directories:
-        raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}")
+
+    if missing_paths:
+        raise ValueError(
+            f"Source directories / root pattern paths do not exist: {', '.join(missing_paths)}"
+        )
 
 
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
@@ -334,10 +196,10 @@ def make_base_create_command(
     dry_run,
     repository_path,
     config,
-    config_paths,
+    patterns,
     local_borg_version,
     global_arguments,
-    borgmatic_runtime_directories,
+    borgmatic_runtime_directory,
     local_path='borg',
     remote_path=None,
     progress=False,
@@ -346,44 +208,16 @@ def make_base_create_command(
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, global arguments as an
-    argparse.Namespace instance, and a sequence of borgmatic source directories, return a tuple of
-    (base Borg create command flags, Borg create command positional arguments, open pattern file
-    handle, open exclude file handle).
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version,
+    global arguments as an argparse.Namespace instance, and a sequence of borgmatic source
+    directories, return a tuple of (base Borg create command flags, Borg create command positional
+    arguments, open pattern file handle).
     '''
-    working_directory = borgmatic.config.paths.get_working_directory(config)
-
     if config.get('source_directories_must_exist', False):
-        check_all_source_directories_exist(
-            config.get('source_directories'), working_directory=working_directory
-        )
+        check_all_root_patterns_exist(patterns)
 
-    sources = deduplicate_directories(
-        map_directories_to_devices(
-            expand_directories(
-                tuple(config.get('source_directories', ()))
-                + borgmatic_runtime_directories
-                + tuple(config_paths if config.get('store_config_files', True) else ()),
-                working_directory=working_directory,
-            )
-        ),
-        additional_directory_devices=map_directories_to_devices(
-            expand_directories(
-                pattern_root_directories(config.get('patterns')),
-                working_directory=working_directory,
-            )
-        ),
-    )
-
-    ensure_files_readable(config.get('patterns_from'), config.get('exclude_from'))
-
-    pattern_file = (
-        write_pattern_file(config.get('patterns'), sources)
-        if config.get('patterns') or config.get('patterns_from')
-        else None
-    )
-    exclude_file = write_pattern_file(expand_home_directories(config.get('exclude_patterns')))
+    patterns_file = write_patterns_file(patterns, borgmatic_runtime_directory)
     checkpoint_interval = config.get('checkpoint_interval', None)
     checkpoint_volume = config.get('checkpoint_volume', None)
     chunker_params = config.get('chunker_params', None)
@@ -426,8 +260,8 @@ def make_base_create_command(
     create_flags = (
         tuple(local_path.split(' '))
         + ('create',)
-        + make_pattern_flags(config, pattern_file.name if pattern_file else None)
-        + make_exclude_flags(config, exclude_file.name if exclude_file else None)
+        + (('--patterns-from', patterns_file.name) if patterns_file else ())
+        + make_exclude_flags(config)
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
         + (('--chunker-params', chunker_params) if chunker_params else ())
@@ -457,24 +291,26 @@ def make_base_create_command(
 
     create_positional_arguments = flags.make_repository_archive_flags(
         repository_path, archive_name_format, local_borg_version
-    ) + (sources if not pattern_file else ())
+    )
 
     # If database hooks are enabled (as indicated by streaming processes), exclude files that might
     # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
     if stream_processes and not config.get('read_special'):
         logger.warning(
-            f'{repository_path}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
+            'Ignoring configured "read_special" value of false, as true is needed for database hooks.'
         )
         borg_environment = environment.make_environment(config)
+        working_directory = borgmatic.config.paths.get_working_directory(config)
 
-        logger.debug(f'{repository_path}: Collecting special file paths')
+        logger.debug('Collecting special file paths')
         special_file_paths = collect_special_file_paths(
+            dry_run,
             create_flags + create_positional_arguments,
             config,
             local_path,
             working_directory,
             borg_environment,
-            skip_directories=borgmatic_runtime_directories,
+            borgmatic_runtime_directory=borgmatic_runtime_directory,
         )
 
         if special_file_paths:
@@ -484,24 +320,32 @@ def make_base_create_command(
                 placeholder=' ...',
             )
             logger.warning(
-                f'{repository_path}: Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
+                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
             )
-            exclude_file = write_pattern_file(
-                expand_home_directories(
-                    tuple(config.get('exclude_patterns') or ()) + special_file_paths
+            patterns_file = write_patterns_file(
+                tuple(
+                    borgmatic.borg.pattern.Pattern(
+                        special_file_path,
+                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                    )
+                    for special_file_path in special_file_paths
                 ),
-                pattern_file=exclude_file,
+                borgmatic_runtime_directory,
+                patterns_file=patterns_file,
             )
-            create_flags += make_exclude_flags(config, exclude_file.name)
 
-    return (create_flags, create_positional_arguments, pattern_file, exclude_file)
+            if '--patterns-from' not in create_flags:
+                create_flags += ('--patterns-from', patterns_file.name)
+
+    return (create_flags, create_positional_arguments, patterns_file)
 
 
 def create_archive(
     dry_run,
     repository_path,
     config,
-    config_paths,
+    patterns,
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
@@ -514,7 +358,7 @@ def create_archive(
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
     sequence of loaded configuration paths, the local Borg version, and global arguments as an
     argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
 
@@ -524,27 +368,21 @@ def create_archive(
     borgmatic.logger.add_custom_log_levels()
 
     working_directory = borgmatic.config.paths.get_working_directory(config)
-    borgmatic_runtime_directories = expand_directories(
-        collect_borgmatic_runtime_directories(borgmatic_runtime_directory),
-        working_directory=working_directory,
-    )
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
-        make_base_create_command(
-            dry_run,
-            repository_path,
-            config,
-            config_paths,
-            local_borg_version,
-            global_arguments,
-            borgmatic_runtime_directories,
-            local_path,
-            remote_path,
-            progress,
-            json,
-            list_files,
-            stream_processes,
-        )
+    (create_flags, create_positional_arguments, patterns_file) = make_base_create_command(
+        dry_run,
+        repository_path,
+        config,
+        patterns,
+        local_borg_version,
+        global_arguments,
+        borgmatic_runtime_directory,
+        local_path,
+        remote_path,
+        progress,
+        json,
+        list_files,
+        stream_processes,
     )
 
     if json:

borgmatic/borg/environment.py

@@ -1,5 +1,7 @@
 import os
 
+import borgmatic.borg.passcommand
+
 OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_base_directory': 'BORG_BASE_DIR',
     'borg_config_directory': 'BORG_CONFIG_DIR',
@@ -7,7 +9,6 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
-    'encryption_passcommand': 'BORG_PASSCOMMAND',
     'encryption_passphrase': 'BORG_PASSPHRASE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
@@ -36,6 +37,19 @@ def make_environment(config):
         if value:
             environment[environment_variable_name] = str(value)
 
+    passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+
+    # If the passcommand produced a passphrase, send it to Borg via an anonymous pipe.
+    if passphrase:
+        read_file_descriptor, write_file_descriptor = os.pipe()
+        os.write(write_file_descriptor, passphrase.encode('utf-8'))
+        os.close(write_file_descriptor)
+
+        # This, plus subprocess.Popen(..., close_fds=False) in execute.py, is necessary for the Borg
+        # child process to inherit the file descriptor.
+        os.set_inheritable(read_file_descriptor, True)
+        environment['BORG_PASSPHRASE_FD'] = str(read_file_descriptor)
+
     for (
         option_name,
         environment_variable_name,

borgmatic/borg/export_key.py

@@ -60,7 +60,7 @@ def export_key(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping key export (dry run)')
+        logger.info('Skipping key export (dry run)')
         return
 
     execute_command(

borgmatic/borg/export_tar.py

@@ -63,7 +63,7 @@ def export_tar_archive(
         output_log_level = logging.INFO
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping export to tar file (dry run)')
+        logging.info('Skipping export to tar file (dry run)')
         return
 
     execute_command(

borgmatic/borg/list.py

@@ -120,7 +120,7 @@ def capture_archive_listing(
                     paths=[path for path in list_paths] if list_paths else None,
                     find_paths=None,
                     json=None,
-                    format=path_format or '{path}{NL}',  # noqa: FS003
+                    format=path_format or '{path}{NUL}',  # noqa: FS003
                 ),
                 global_arguments,
                 local_path,
@@ -131,8 +131,8 @@ def capture_archive_listing(
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\n')
-        .split('\n')
+        .strip('\0')
+        .split('\0')
     )
 
 
@@ -230,14 +230,14 @@ def list_archive(
                 borg_exit_codes=borg_exit_codes,
             )
             .strip('\n')
-            .split('\n')
+            .splitlines()
         )
     else:
         archive_lines = (list_arguments.archive,)
 
     # For each archive listed by Borg, run list on the contents of that archive.
     for archive in archive_lines:
-        logger.answer(f'{repository_path}: Listing archive {archive}')
+        logger.answer(f'Listing archive {archive}')
 
         archive_arguments = copy.copy(list_arguments)
         archive_arguments.archive = archive

borgmatic/borg/passcommand.py

@@ -0,0 +1,50 @@
+import functools
+import logging
+import shlex
+
+import borgmatic.config.paths
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+@functools.cache
+def run_passcommand(passcommand, passphrase_configured, working_directory):
+    '''
+    Run the given passcommand using the given working directory and return the passphrase produced
+    by the command. But bail first if a passphrase is already configured; this mimics Borg's
+    behavior.
+
+    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
+    per borgmatic invocation.
+    '''
+    if passcommand and passphrase_configured:
+        logger.warning(
+            'Ignoring the "encryption_passcommand" option because "encryption_passphrase" is set'
+        )
+        return None
+
+    return borgmatic.execute.execute_command_and_capture_output(
+        shlex.split(passcommand),
+        working_directory=working_directory,
+    )
+
+
+def get_passphrase_from_passcommand(config):
+    '''
+    Given the configuration dict, call the configured passcommand to produce and return an
+    encryption passphrase. In effect, we're doing an end-run around Borg by invoking its passcommand
+    ourselves. This allows us to pass the resulting passphrase to multiple different Borg
+    invocations without the user having to be prompted multiple times.
+
+    If no passcommand is configured, then return None.
+    '''
+    passcommand = config.get('encryption_passcommand')
+
+    if not passcommand:
+        return None
+
+    passphrase = config.get('encryption_passphrase')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    return run_passcommand(passcommand, bool(passphrase), working_directory)

borgmatic/borg/pattern.py

@@ -0,0 +1,31 @@
+import collections
+import enum
+
+
+# See https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
+class Pattern_type(enum.Enum):
+    ROOT = 'R'  # A ROOT pattern always has a NONE pattern style.
+    PATTERN_STYLE = 'P'
+    EXCLUDE = '-'
+    NO_RECURSE = '!'
+    INCLUDE = '+'
+
+
+class Pattern_style(enum.Enum):
+    NONE = ''
+    FNMATCH = 'fm'
+    SHELL = 'sh'
+    REGULAR_EXPRESSION = 're'
+    PATH_PREFIX = 'pp'
+    PATH_FULL_MATCH = 'pf'
+
+
+Pattern = collections.namedtuple(
+    'Pattern',
+    ('path', 'type', 'style', 'device'),
+    defaults=(
+        Pattern_type.ROOT,
+        Pattern_style.NONE,
+        None,
+    ),
+)

borgmatic/borg/repo_create.py

@@ -57,7 +57,7 @@ def create_repository(
                 f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
             )
 
-        logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
+        logger.info('Repository already exists. Skipping creation.')
         return
     except subprocess.CalledProcessError as error:
         if error.returncode not in REPO_INFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
@@ -91,7 +91,7 @@ def create_repository(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping repository creation (dry run)')
+        logging.info('Skipping repository creation (dry run)')
         return
 
     # Do not capture output here, so as to support interactive prompts.

borgmatic/borg/repo_list.py

@@ -59,7 +59,7 @@ def resolve_archive_name(
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'{repository_path}: Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive}')
 
     return latest_archive
 

borgmatic/commands/arguments.py

@@ -1153,7 +1153,7 @@ def make_parsers():
         metavar='NAME',
         dest='data_sources',
         action='append',
-        help="Name of data source (e.g. database) to restore from archive, must be defined in borgmatic's configuration, can specify flag multiple times, defaults to all data sources in the archive",
+        help="Name of data source (e.g. database) to restore from the archive, must be defined in borgmatic's configuration, can specify the flag multiple times, defaults to all data sources in the archive",
     )
     restore_group.add_argument(
         '--schema',
@@ -1182,6 +1182,19 @@ def make_parsers():
         '--restore-path',
         help='Path to restore SQLite database dumps to. Defaults to the "restore_path" option in borgmatic\'s configuration',
     )
+    restore_group.add_argument(
+        '--original-hostname',
+        help='The hostname where the dump to restore came from, only necessary if you need to disambiguate dumps',
+    )
+    restore_group.add_argument(
+        '--original-port',
+        type=int,
+        help="The port where the dump to restore came from (if that port is in borgmatic's configuration), only necessary if you need to disambiguate dumps",
+    )
+    restore_group.add_argument(
+        '--hook',
+        help='The name of the data source hook for the dump to restore, only necessary if you need to disambiguate dumps',
+    )
     restore_group.add_argument(
         '-h', '--help', action='help', help='Show this help message and exit'
     )
@@ -1244,6 +1257,12 @@ def make_parsers():
         metavar='TIMESPAN',
         help='List archives that are newer than the specified time range (e.g. 7d or 12m) from the current time [Borg 2.x+ only]',
     )
+    repo_list_group.add_argument(
+        '--deleted',
+        default=False,
+        action='store_true',
+        help="List only deleted archives that haven't yet been compacted [Borg 2.x+ only]",
+    )
     repo_list_group.add_argument(
         '-h', '--help', action='help', help='Show this help message and exit'
     )

borgmatic/commands/borgmatic.py

@@ -8,8 +8,6 @@ import time
 from queue import Queue
 from subprocess import CalledProcessError
 
-import colorama
-
 import borgmatic.actions.borg
 import borgmatic.actions.break_lock
 import borgmatic.actions.change_passphrase
@@ -39,8 +37,15 @@ from borgmatic.borg import umount as borg_umount
 from borgmatic.borg import version as borg_version
 from borgmatic.commands.arguments import parse_arguments
 from borgmatic.config import checks, collect, validate
-from borgmatic.hooks import command, dispatch, monitor
-from borgmatic.logger import DISABLED, add_custom_log_levels, configure_logging, should_do_markup
+from borgmatic.hooks import command, dispatch
+from borgmatic.hooks.monitoring import monitor
+from borgmatic.logger import (
+    DISABLED,
+    Log_prefix,
+    add_custom_log_levels,
+    configure_logging,
+    should_do_markup,
+)
 from borgmatic.signals import configure_signals
 from borgmatic.verbosity import verbosity_to_log_level
 
@@ -87,12 +92,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
 
     if skip_actions:
         logger.debug(
-            f"{config_filename}: Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
+            f"Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
         )
 
     try:
         local_borg_version = borg_version.local_borg_version(config, local_path)
-        logger.debug(f'{config_filename}: Borg {local_borg_version}')
+        logger.debug(f'Borg {local_borg_version}')
     except (OSError, CalledProcessError, ValueError) as error:
         yield from log_error_records(f'{config_filename}: Error getting local Borg version', error)
         return
@@ -102,8 +107,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'initialize_monitor',
                 config,
+                dispatch.Hook_type.MONITORING,
                 config_filename,
-                monitor.MONITOR_HOOK_NAMES,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
@@ -111,14 +116,14 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
+                dispatch.Hook_type.MONITORING,
                 config_filename,
-                monitor.MONITOR_HOOK_NAMES,
                 monitor.State.START,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
     except (OSError, CalledProcessError) as error:
-        if command.considered_soft_failure(config_filename, error):
+        if command.considered_soft_failure(error):
             return
 
         encountered_error = error
@@ -133,53 +138,49 @@ def run_configuration(config_filename, config, config_paths, arguments):
 
         while not repo_queue.empty():
             repository, retry_num = repo_queue.get()
-            logger.debug(
-                f'{repository.get("label", repository["path"])}: Running actions for repository'
-            )
-            timeout = retry_num * retry_wait
-            if timeout:
-                logger.warning(
-                    f'{repository.get("label", repository["path"])}: Sleeping {timeout}s before next retry'
-                )
-                time.sleep(timeout)
-            try:
-                yield from run_actions(
-                    arguments=arguments,
-                    config_filename=config_filename,
-                    config=config,
-                    config_paths=config_paths,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                    local_borg_version=local_borg_version,
-                    repository=repository,
-                )
-            except (OSError, CalledProcessError, ValueError) as error:
-                if retry_num < retries:
-                    repo_queue.put(
-                        (repository, retry_num + 1),
+
+            with Log_prefix(repository.get('label', repository['path'])):
+                logger.debug('Running actions for repository')
+                timeout = retry_num * retry_wait
+                if timeout:
+                    logger.warning(f'Sleeping {timeout}s before next retry')
+                    time.sleep(timeout)
+                try:
+                    yield from run_actions(
+                        arguments=arguments,
+                        config_filename=config_filename,
+                        config=config,
+                        config_paths=config_paths,
+                        local_path=local_path,
+                        remote_path=remote_path,
+                        local_borg_version=local_borg_version,
+                        repository=repository,
                     )
-                    tuple(  # Consume the generator so as to trigger logging.
-                        log_error_records(
-                            f'{repository.get("label", repository["path"])}: Error running actions for repository',
-                            error,
-                            levelno=logging.WARNING,
-                            log_command_error_output=True,
+                except (OSError, CalledProcessError, ValueError) as error:
+                    if retry_num < retries:
+                        repo_queue.put(
+                            (repository, retry_num + 1),
                         )
-                    )
-                    logger.warning(
-                        f'{repository.get("label", repository["path"])}: Retrying... attempt {retry_num + 1}/{retries}'
-                    )
-                    continue
+                        tuple(  # Consume the generator so as to trigger logging.
+                            log_error_records(
+                                'Error running actions for repository',
+                                error,
+                                levelno=logging.WARNING,
+                                log_command_error_output=True,
+                            )
+                        )
+                        logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
+                        continue
 
-                if command.considered_soft_failure(config_filename, error):
-                    continue
+                    if command.considered_soft_failure(error):
+                        continue
 
-                yield from log_error_records(
-                    f'{repository.get("label", repository["path"])}: Error running actions for repository',
-                    error,
-                )
-                encountered_error = error
-                error_repository = repository['path']
+                    yield from log_error_records(
+                        'Error running actions for repository',
+                        error,
+                    )
+                    encountered_error = error
+                    error_repository = repository['path']
 
     try:
         if monitoring_hooks_are_activated:
@@ -187,16 +188,16 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
+                dispatch.Hook_type.MONITORING,
                 config_filename,
-                monitor.MONITOR_HOOK_NAMES,
                 monitor.State.LOG,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
     except (OSError, CalledProcessError) as error:
-        if not command.considered_soft_failure(config_filename, error):
+        if not command.considered_soft_failure(error):
             encountered_error = error
-            yield from log_error_records(f'{repository["path"]}: Error pinging monitor', error)
+            yield from log_error_records('Error pinging monitor', error)
 
     if not encountered_error:
         try:
@@ -204,8 +205,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
                 dispatch.call_hooks(
                     'ping_monitor',
                     config,
+                    dispatch.Hook_type.MONITORING,
                     config_filename,
-                    monitor.MONITOR_HOOK_NAMES,
                     monitor.State.FINISH,
                     monitoring_log_level,
                     global_arguments.dry_run,
@@ -213,13 +214,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
                 dispatch.call_hooks(
                     'destroy_monitor',
                     config,
-                    config_filename,
-                    monitor.MONITOR_HOOK_NAMES,
+                    dispatch.Hook_type.MONITORING,
                     monitoring_log_level,
                     global_arguments.dry_run,
                 )
         except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(config_filename, error):
+            if command.considered_soft_failure(error):
                 return
 
             encountered_error = error
@@ -240,8 +240,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
+                dispatch.Hook_type.MONITORING,
                 config_filename,
-                monitor.MONITOR_HOOK_NAMES,
                 monitor.State.FAIL,
                 monitoring_log_level,
                 global_arguments.dry_run,
@@ -249,13 +249,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'destroy_monitor',
                 config,
-                config_filename,
-                monitor.MONITOR_HOOK_NAMES,
+                dispatch.Hook_type.MONITORING,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
         except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(config_filename, error):
+            if command.considered_soft_failure(error):
                 return
 
             yield from log_error_records(f'{config_filename}: Error running on-error hook', error)
@@ -820,23 +819,27 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments):
 
     # Execute the actions corresponding to each configuration file.
     json_results = []
-    for config_filename, config in configs.items():
-        results = list(run_configuration(config_filename, config, config_paths, arguments))
-        error_logs = tuple(result for result in results if isinstance(result, logging.LogRecord))
 
-        if error_logs:
-            yield from log_error_records(f'{config_filename}: An error occurred')
-            yield from error_logs
-        else:
-            yield logging.makeLogRecord(
-                dict(
-                    levelno=logging.INFO,
-                    levelname='INFO',
-                    msg=f'{config_filename}: Successfully ran configuration file',
-                )
+    for config_filename, config in configs.items():
+        with Log_prefix(config_filename):
+            results = list(run_configuration(config_filename, config, config_paths, arguments))
+            error_logs = tuple(
+                result for result in results if isinstance(result, logging.LogRecord)
             )
-            if results:
-                json_results.extend(results)
+
+            if error_logs:
+                yield from log_error_records('An error occurred')
+                yield from error_logs
+            else:
+                yield logging.makeLogRecord(
+                    dict(
+                        levelno=logging.INFO,
+                        levelname='INFO',
+                        msg='Successfully ran configuration file',
+                    )
+                )
+                if results:
+                    json_results.extend(results)
 
     if 'umount' in arguments:
         logger.info(f"Unmounting mount point {arguments['umount'].mount_point}")
@@ -914,7 +917,7 @@ def main(extra_summary_logs=[]):  # pragma: no cover
         getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
     )
     color_enabled = should_do_markup(global_arguments.no_color or any_json_flags, configs)
-    colorama.init(autoreset=color_enabled, strip=not color_enabled)
+
     try:
         configure_logging(
             verbosity_to_log_level(global_arguments.verbosity),

borgmatic/config/generate.py

@@ -44,12 +44,12 @@ def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
     if example is not None:
         return example
 
-    if schema_type == 'array':
+    if schema_type == 'array' or (isinstance(schema_type, list) and 'array' in schema_type):
         config = ruamel.yaml.comments.CommentedSeq(
             [schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
         )
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
-    elif schema_type == 'object':
+    elif schema_type == 'object' or (isinstance(schema_type, list) and 'object' in schema_type):
         config = ruamel.yaml.comments.CommentedMap(
             [
                 (field_name, schema_to_sample_configuration(sub_schema, level + 1))

borgmatic/config/normalize.py

@@ -93,6 +93,25 @@ def normalize(config_filename, config):
         )
         config['exclude_if_present'] = [exclude_if_present]
 
+    # Unconditionally set the bootstrap hook so that it's enabled by default and config files get
+    # stored in each Borg archive.
+    config.setdefault('bootstrap', {})
+
+    # Move store_config_files from the global scope to the bootstrap hook.
+    store_config_files = config.get('store_config_files')
+    if store_config_files is not None:
+        logs.append(
+            logging.makeLogRecord(
+                dict(
+                    levelno=logging.WARNING,
+                    levelname='WARNING',
+                    msg=f'{config_filename}: The store_config_files option has moved under the bootstrap hook. Specifying store_config_files at the global scope is deprecated and support will be removed from a future release.',
+                )
+            )
+        )
+        del config['store_config_files']
+        config['bootstrap']['store_config_files'] = store_config_files
+
     # Upgrade various monitoring hooks from a string to a dict.
     healthchecks = config.get('healthchecks')
     if isinstance(healthchecks, str):

borgmatic/config/paths.py

@@ -33,6 +33,35 @@ def get_borgmatic_source_directory(config):
 TEMPORARY_DIRECTORY_PREFIX = 'borgmatic-'
 
 
+def replace_temporary_subdirectory_with_glob(
+    path, temporary_directory_prefix=TEMPORARY_DIRECTORY_PREFIX
+):
+    '''
+    Given an absolute temporary directory path and an optional temporary directory prefix, look for
+    a subdirectory within it starting with the temporary directory prefix (or a default) and replace
+    it with an appropriate glob. For instance, given:
+
+        /tmp/borgmatic-aet8kn93/borgmatic
+
+    ... replace it with:
+
+        /tmp/borgmatic-*/borgmatic
+
+    This is useful for finding previous temporary directories from prior borgmatic runs.
+    '''
+    return os.path.join(
+        '/',
+        *(
+            (
+                f'{temporary_directory_prefix}*'
+                if subdirectory.startswith(temporary_directory_prefix)
+                else subdirectory
+            )
+            for subdirectory in path.split(os.path.sep)
+        ),
+    )
+
+
 class Runtime_directory:
     '''
     A Python context manager for creating and cleaning up the borgmatic runtime directory used for
@@ -47,14 +76,13 @@ class Runtime_directory:
     automatically gets cleaned up as necessary.
     '''
 
-    def __init__(self, config, log_prefix):
+    def __init__(self, config):
         '''
-        Given a configuration dict and a log prefix, determine the borgmatic runtime directory,
-        creating a secure, temporary directory within it if necessary. Defaults to
-        $XDG_RUNTIME_DIR/./borgmatic or $RUNTIME_DIRECTORY/./borgmatic or
-        $TMPDIR/borgmatic-[random]/./borgmatic or $TEMP/borgmatic-[random]/./borgmatic or
-        /tmp/borgmatic-[random]/./borgmatic where "[random]" is a randomly generated string intended
-        to avoid path collisions.
+        Given a configuration dict determine the borgmatic runtime directory, creating a secure,
+        temporary directory within it if necessary. Defaults to $XDG_RUNTIME_DIR/./borgmatic or
+        $RUNTIME_DIRECTORY/./borgmatic or $TMPDIR/borgmatic-[random]/./borgmatic or
+        $TEMP/borgmatic-[random]/./borgmatic or /tmp/borgmatic-[random]/./borgmatic where "[random]"
+        is a randomly generated string intended to avoid path collisions.
 
         If XDG_RUNTIME_DIR or RUNTIME_DIRECTORY is set and already ends in "/borgmatic", then don't
         tack on a second "/borgmatic" path component.
@@ -70,9 +98,16 @@ class Runtime_directory:
         )
 
         if runtime_directory:
+            if not runtime_directory.startswith(os.path.sep):
+                raise ValueError('The runtime directory must be an absolute path')
+
             self.temporary_directory = None
         else:
             base_directory = os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'
+
+            if not base_directory.startswith(os.path.sep):
+                raise ValueError('The temporary directory must be an absolute path')
+
             os.makedirs(base_directory, mode=0o700, exist_ok=True)
             self.temporary_directory = tempfile.TemporaryDirectory(
                 prefix=TEMPORARY_DIRECTORY_PREFIX,
@@ -84,12 +119,14 @@ class Runtime_directory:
 
         self.runtime_path = expand_user_in_path(
             os.path.join(
-                base_path if final_directory == 'borgmatic' else runtime_directory, '.', 'borgmatic'
+                base_path if final_directory == 'borgmatic' else runtime_directory,
+                '.',  # Borg 1.4+ "slashdot" hack.
+                'borgmatic',
             )
         )
         os.makedirs(self.runtime_path, mode=0o700, exist_ok=True)
 
-        logger.debug(f'{log_prefix}: Using runtime directory {os.path.normpath(self.runtime_path)}')
+        logger.debug(f'Using runtime directory {os.path.normpath(self.runtime_path)}')
 
     def __enter__(self):
         '''
@@ -102,7 +139,13 @@ class Runtime_directory:
         Delete any temporary directory that was created as part of initialization.
         '''
         if self.temporary_directory:
-            self.temporary_directory.cleanup()
+            try:
+                self.temporary_directory.cleanup()
+            # The cleanup() call errors if, for instance, there's still a
+            # mounted filesystem within the temporary directory. There's
+            # nothing we can do about that here, so swallow the error.
+            except OSError:
+                pass
 
 
 def make_runtime_directory_glob(borgmatic_runtime_directory):

borgmatic/config/schema.yaml

@@ -68,9 +68,7 @@ properties:
         type: boolean
         description: |
             Stay in same file system; do not cross mount points beyond the given
-            source directories. Defaults to false. But when a database hook is
-            used, the setting here is ignored and one_file_system is considered
-            true.
+            source directories. Defaults to false.
         example: true
     numeric_ids:
         type: boolean
@@ -133,8 +131,7 @@ properties:
             Any paths matching these patterns are included/excluded from
             backups. Globs are expanded. (Tildes are not.) See the output of
             "borg help patterns" for more details. Quote any value if it
-            contains leading punctuation, so it parses correctly. Note that only
-            one of "patterns" and "source_directories" may be used.
+            contains leading punctuation, so it parses correctly.
         example:
             - 'R /'
             - '- /home/*/.cache'
@@ -146,9 +143,8 @@ properties:
             type: string
         description: |
             Read include/exclude patterns from one or more separate named files,
-            one pattern per line. Note that Borg considers this option
-            experimental. See the output of "borg help patterns" for more
-            details.
+            one pattern per line. See the output of "borg help patterns" for
+            more details.
         example:
             - /etc/borgmatic/patterns
     exclude_patterns:
@@ -209,8 +205,8 @@ properties:
         description: |
             Deprecated. Only used for locating database dumps and bootstrap
             metadata within backup archives created prior to deprecation.
-            Replaced by borgmatic_runtime_directory and
-            borgmatic_state_directory. Defaults to ~/.borgmatic
+            Replaced by user_runtime_directory and user_state_directory.
+            Defaults to ~/.borgmatic
         example: /tmp/borgmatic
     user_runtime_directory:
         type: string
@@ -229,18 +225,11 @@ properties:
             create the check records again (and therefore re-run checks).
             Defaults to $XDG_STATE_HOME or ~/.local/state.
         example: /var/lib/borgmatic
-    store_config_files:
-        type: boolean
-        description: |
-            Store configuration files used to create a backup in the backup
-            itself. Defaults to true. Changing this to false prevents "borgmatic
-            bootstrap" from extracting configuration files from the backup.
-        example: false
     source_directories_must_exist:
         type: boolean
         description: |
-            If true, then source directories must exist, otherwise an error is
-            raised. Defaults to false.
+            If true, then source directories (and root pattern paths) must
+            exist. If they don't, an error is raised. Defaults to false.
         example: true
     encryption_passcommand:
         type: string
@@ -942,6 +931,20 @@ properties:
             of them (after any action).
         example:
             - "echo Completed actions."
+    bootstrap:
+        type: object
+        properties:
+            store_config_files:
+                type: boolean
+                description: |
+                    Store configuration files used to create a backup inside the
+                    backup itself. Defaults to true. Changing this to false
+                    prevents "borgmatic bootstrap" from extracting configuration
+                    files from the backup.
+                example: false
+        description: |
+            Support for the "borgmatic bootstrap" action, used to extract
+            borgmatic configuration files from a backup archive.
     postgresql_databases:
         type: array
         items:
@@ -956,8 +959,8 @@ properties:
                         dump all databases on the host. (Also set the "format"
                         to dump each database to a separate file instead of one
                         combined file.) Note that using this database hook
-                        implicitly enables both read_special and one_file_system
-                        (see above) to support dump and restore streaming.
+                        implicitly enables read_special (see above) to support
+                        dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1138,9 +1141,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1265,9 +1267,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -1400,9 +1401,9 @@ properties:
                     description: |
                         Path to the SQLite database file to dump. If relative,
                         it is relative to the current working directory. Note
-                        that using this database hook implicitly enables both
-                        read_special and one_file_system (see above) to support
-                        dump and restore streaming.
+                        that using this database hook implicitly enables
+                        read_special (see above) to support dump and restore
+                        streaming.
                     example: /var/lib/sqlite/users.db
                 restore_path:
                     type: string
@@ -1422,9 +1423,8 @@ properties:
                     description: |
                         Database name (required if using this hook). Or "all" to
                         dump all databases on the host. Note that using this
-                        database hook implicitly enables both read_special and
-                        one_file_system (see above) to support dump and restore
-                        streaming.
+                        database hook implicitly enables read_special (see
+                        above) to support dump and restore streaming.
                     example: users
                 hostname:
                     type: string
@@ -2255,7 +2255,130 @@ properties:
                     config: "__config"
                     hostname: "__hostname"
         description: |
-            Configuration for a monitoring integration with Grafana loki. You
+            Configuration for a monitoring integration with Grafana Loki. You
             can send the logs to a self-hosted instance or create an account at
             https://grafana.com/auth/sign-up/create-user. See borgmatic
             monitoring documentation for details.
+    sentry:
+        type: object
+        required: ['data_source_name_url', 'monitor_slug']
+        additionalProperties: false
+        properties:
+            data_source_name_url:
+                type: string
+                description: |
+                    Sentry Data Source Name (DSN) URL, associated with a
+                    particular Sentry project. Used to construct a cron URL,
+                    notified when a backup begins, ends, or errors.
+                example: https://5f80ec@o294220.ingest.us.sentry.io/203069
+            monitor_slug:
+                type: string
+                description: |
+                    Sentry monitor slug, associated with a particular Sentry
+                    project monitor. Used along with the data source name URL to
+                    construct a cron URL.
+                example: mymonitor
+            states:
+                type: array
+                items:
+                    type: string
+                    enum:
+                        - start
+                        - finish
+                        - fail
+                    uniqueItems: true
+                description: |
+                    List of one or more monitoring states to ping for: "start",
+                    "finish", and/or "fail". Defaults to pinging for all states.
+                example:
+                    - start
+                    - finish
+        description: |
+            Configuration for a monitoring integration with Sentry. You can use
+            a self-hosted instance via https://develop.sentry.dev/self-hosted/
+            or create a cloud-hosted account at https://sentry.io. See borgmatic
+            monitoring documentation for details.
+    zfs:
+        type: ["object", "null"]
+        additionalProperties: false
+        properties:
+            zfs_command:
+                type: string
+                description: |
+                    Command to use instead of "zfs".
+                example: /usr/local/bin/zfs
+            mount_command:
+                type: string
+                description: |
+                    Command to use instead of "mount".
+                example: /usr/local/bin/mount
+            umount_command:
+                type: string
+                description: |
+                    Command to use instead of "umount".
+                example: /usr/local/bin/umount
+        description: |
+            Configuration for integration with the ZFS filesystem.
+    btrfs:
+        type: ["object", "null"]
+        additionalProperties: false
+        properties:
+            btrfs_command:
+                type: string
+                description: |
+                    Command to use instead of "btrfs".
+                example: /usr/local/bin/btrfs
+            findmnt_command:
+                type: string
+                description: |
+                    Command to use instead of "findmnt".
+                example: /usr/local/bin/findmnt
+        description: |
+            Configuration for integration with the Btrfs filesystem.
+    lvm:
+        type: ["object", "null"]
+        additionalProperties: false
+        properties:
+            snapshot_size:
+                type: string
+                description: |
+                    Size to allocate for each snapshot taken, including the
+                    units to use for that size. Defaults to "10%ORIGIN" (10%
+                    of the size of logical volume being snapshotted). See the
+                    lvcreate "--size" and "--extents" documentation for more
+                    information:
+                    https://www.man7.org/linux/man-pages/man8/lvcreate.8.html
+                example: 5GB
+            lvcreate_command:
+                type: string
+                description: |
+                    Command to use instead of "lvcreate".
+                example: /usr/local/bin/lvcreate
+            lvremove_command:
+                type: string
+                description: |
+                    Command to use instead of "lvremove".
+                example: /usr/local/bin/lvremove
+            lvs_command:
+                type: string
+                description: |
+                    Command to use instead of "lvs".
+                example: /usr/local/bin/lvs
+            lsblk_command:
+                type: string
+                description: |
+                    Command to use instead of "lsblk".
+                example: /usr/local/bin/lsblk
+            mount_command:
+                type: string
+                description: |
+                    Command to use instead of "mount".
+                example: /usr/local/bin/mount
+            umount_command:
+                type: string
+                description: |
+                    Command to use instead of "umount".
+                example: /usr/local/bin/umount
+        description: |
+            Configuration for integration with Linux LVM (Logical Volume
+            Manager).

borgmatic/execute.py

@@ -6,6 +6,8 @@ import select
 import subprocess
 import textwrap
 
+import borgmatic.logger
+
 logger = logging.getLogger(__name__)
 
 
@@ -305,17 +307,20 @@ def execute_command(
         shell=shell,
         env=environment,
         cwd=working_directory,
+        # Necessary for the passcommand credential hook to work.
+        close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
     )
     if not run_to_completion:
         return process
 
-    log_outputs(
-        (process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        log_outputs(
+            (process,),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
 
 
 def execute_command_and_capture_output(
@@ -351,6 +356,8 @@ def execute_command_and_capture_output(
             shell=shell,
             env=environment,
             cwd=working_directory,
+            # Necessary for the passcommand credential hook to work.
+            close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -411,6 +418,8 @@ def execute_command_with_processes(
             shell=shell,
             env=environment,
             cwd=working_directory,
+            # Necessary for the passcommand credential hook to work.
+            close_fds=not bool((extra_environment or {}).get('BORG_PASSPHRASE_FD')),
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.

borgmatic/hooks/command.py

@@ -2,8 +2,9 @@ import logging
 import os
 import re
 import shlex
+import sys
 
-from borgmatic import execute
+import borgmatic.execute
 
 logger = logging.getLogger(__name__)
 
@@ -11,7 +12,7 @@ logger = logging.getLogger(__name__)
 SOFT_FAIL_EXIT_CODE = 75
 
 
-def interpolate_context(config_filename, hook_description, command, context):
+def interpolate_context(hook_description, command, context):
     '''
     Given a config filename, a hook description, a single hook command, and a dict of context
     names/values, interpolate the values by "{name}" into the command and return the result.
@@ -21,12 +22,26 @@ def interpolate_context(config_filename, hook_description, command, context):
 
     for unsupported_variable in re.findall(r'{\w+}', command):
         logger.warning(
-            f"{config_filename}: Variable '{unsupported_variable}' is not supported in {hook_description} hook"
+            f"Variable '{unsupported_variable}' is not supported in {hook_description} hook"
         )
 
     return command
 
 
+def make_environment(current_environment, sys_module=sys):
+    '''
+    Given the existing system environment as a map from environment variable name to value, return
+    (in the same form) any extra environment variables that should be used when running command
+    hooks.
+    '''
+    # Detect whether we're running within a PyInstaller bundle. If so, set or clear LD_LIBRARY_PATH
+    # based on the value of LD_LIBRARY_PATH_ORIG. This prevents library version information errors.
+    if getattr(sys_module, 'frozen', False) and hasattr(sys_module, '_MEIPASS'):
+        return {'LD_LIBRARY_PATH': current_environment.get('LD_LIBRARY_PATH_ORIG', '')}
+
+    return {}
+
+
 def execute_hook(commands, umask, config_filename, description, dry_run, **context):
     '''
     Given a list of hook commands to execute, a umask to execute with (or None), a config filename,
@@ -39,46 +54,45 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
     Raise subprocesses.CalledProcessError if an error occurs in a hook.
     '''
     if not commands:
-        logger.debug(f'{config_filename}: No commands to run for {description} hook')
+        logger.debug(f'No commands to run for {description} hook')
         return
 
     dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
 
     context['configuration_filename'] = config_filename
-    commands = [
-        interpolate_context(config_filename, description, command, context) for command in commands
-    ]
+    commands = [interpolate_context(description, command, context) for command in commands]
 
     if len(commands) == 1:
-        logger.info(f'{config_filename}: Running command for {description} hook{dry_run_label}')
+        logger.info(f'Running command for {description} hook{dry_run_label}')
     else:
         logger.info(
-            f'{config_filename}: Running {len(commands)} commands for {description} hook{dry_run_label}',
+            f'Running {len(commands)} commands for {description} hook{dry_run_label}',
         )
 
     if umask:
         parsed_umask = int(str(umask), 8)
-        logger.debug(f'{config_filename}: Set hook umask to {oct(parsed_umask)}')
+        logger.debug(f'Set hook umask to {oct(parsed_umask)}')
         original_umask = os.umask(parsed_umask)
     else:
         original_umask = None
 
     try:
         for command in commands:
-            if not dry_run:
-                execute.execute_command(
-                    [command],
-                    output_log_level=(
-                        logging.ERROR if description == 'on-error' else logging.WARNING
-                    ),
-                    shell=True,
-                )
+            if dry_run:
+                continue
+
+            borgmatic.execute.execute_command(
+                [command],
+                output_log_level=(logging.ERROR if description == 'on-error' else logging.WARNING),
+                shell=True,
+                extra_environment=make_environment(os.environ),
+            )
     finally:
         if original_umask:
             os.umask(original_umask)
 
 
-def considered_soft_failure(config_filename, error):
+def considered_soft_failure(error):
     '''
     Given a configuration filename and an exception object, return whether the exception object
     represents a subprocess.CalledProcessError with a return code of SOFT_FAIL_EXIT_CODE. If so,
@@ -90,7 +104,7 @@ def considered_soft_failure(config_filename, error):
 
     if exit_code == SOFT_FAIL_EXIT_CODE:
         logger.info(
-            f'{config_filename}: Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining repository actions',
+            f'Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining repository actions',
         )
         return True
 

borgmatic/hooks/data_source/bootstrap.py

@@ -0,0 +1,125 @@
+import glob
+import importlib
+import json
+import logging
+import os
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic configuration file
+    paths, the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    create a borgmatic manifest file to store the paths of the configuration files used to create
+    the archive. But skip this if the bootstrap store_config_files option is False or if this is a
+    dry run.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+    '''
+    if hook_config and hook_config.get('store_config_files') is False:
+        return []
+
+    borgmatic_manifest_path = os.path.join(
+        borgmatic_runtime_directory, 'bootstrap', 'manifest.json'
+    )
+
+    if dry_run:
+        return []
+
+    os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
+
+    with open(borgmatic_manifest_path, 'w') as manifest_file:
+        json.dump(
+            {
+                'borgmatic_version': importlib.metadata.version('borgmatic'),
+                'config_paths': config_paths,
+            },
+            manifest_file,
+        )
+
+    patterns.extend(borgmatic.borg.pattern.Pattern(config_path) for config_path in config_paths)
+    patterns.append(
+        borgmatic.borg.pattern.Pattern(os.path.join(borgmatic_runtime_directory, 'bootstrap'))
+    )
+
+    return []
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, then remove the manifest file created above. If this is a dry run,
+    then don't actually remove anything.
+    '''
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    manifest_glob = os.path.join(
+        borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(borgmatic_runtime_directory),
+        ),
+        'bootstrap',
+    )
+    logger.debug(
+        f'Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}'
+    )
+
+    for manifest_directory in glob.glob(manifest_glob):
+        manifest_file_path = os.path.join(manifest_directory, 'manifest.json')
+        logger.debug(f'Removing bootstrap manifest at {manifest_file_path}{dry_run_label}')
+
+        if dry_run:
+            continue
+
+        try:
+            os.remove(manifest_file_path)
+        except FileNotFoundError:
+            pass
+
+        try:
+            os.rmdir(manifest_directory)
+        except FileNotFoundError:
+            pass
+
+
+def make_data_source_dump_patterns(
+    hook_config, config, borgmatic_runtime_directory, name=None
+):  # pragma: no cover
+    '''
+    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
+    generic "restore".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
+    generic "restore".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/data_source/btrfs.py

@@ -0,0 +1,328 @@
+import collections
+import glob
+import json
+import logging
+import os
+import shutil
+import subprocess
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+import borgmatic.execute
+import borgmatic.hooks.data_source.snapshot
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+def get_subvolume_mount_points(findmnt_command):
+    '''
+    Given a findmnt command to run, get all sorted Btrfs subvolume mount points.
+    '''
+    findmnt_output = borgmatic.execute.execute_command_and_capture_output(
+        tuple(findmnt_command.split(' '))
+        + (
+            '-t',  # Filesystem type.
+            'btrfs',
+            '--json',
+            '--list',  # Request a flat list instead of a nested subvolume hierarchy.
+        )
+    )
+
+    try:
+        return tuple(
+            sorted(filesystem['target'] for filesystem in json.loads(findmnt_output)['filesystems'])
+        )
+    except json.JSONDecodeError as error:
+        raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
+    except KeyError as error:
+        raise ValueError(f'Invalid {findmnt_command} output: Missing key "{error}"')
+
+
+Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
+
+
+def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
+    '''
+    Given a Btrfs command to run and a sequence of configured patterns, find the intersection
+    between the current Btrfs filesystem and subvolume mount points and the paths of any patterns.
+    The idea is that these pattern paths represent the requested subvolumes to snapshot.
+
+    If patterns is None, then return all subvolumes, sorted by path.
+
+    Return the result as a sequence of matching subvolume mount points.
+    '''
+    candidate_patterns = set(patterns or ())
+    subvolumes = []
+
+    # For each subvolume mount point, match it against the given patterns to find the subvolumes to
+    # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
+    # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
+    # this process, so no two subvolumes end up with the same contained patterns.)
+    for mount_point in reversed(get_subvolume_mount_points(findmnt_command)):
+        subvolumes.extend(
+            Subvolume(mount_point, contained_patterns)
+            for contained_patterns in (
+                borgmatic.hooks.data_source.snapshot.get_contained_patterns(
+                    mount_point, candidate_patterns
+                ),
+            )
+            if patterns is None or contained_patterns
+        )
+
+    return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))
+
+
+BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot-'
+
+
+def make_snapshot_path(subvolume_path):
+    '''
+    Given the path to a subvolume, make a corresponding snapshot path for it.
+    '''
+    return os.path.join(
+        subvolume_path,
+        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        # Included so that the snapshot ends up in the Borg archive at the "original" subvolume path.
+    ) + subvolume_path.rstrip(os.path.sep)
+
+
+def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
+    '''
+    Given the path to a subvolume, make a corresponding exclude pattern for its embedded snapshot
+    path. This is to work around a quirk of Btrfs: If you make a snapshot path as a child directory
+    of a subvolume, then the snapshot's own initial directory component shows up as an empty
+    directory within the snapshot itself. For instance, if you have a Btrfs subvolume at /mnt and
+    make a snapshot of it at:
+
+        /mnt/.borgmatic-snapshot-1234/mnt
+
+    ... then the snapshot itself will have an empty directory at:
+
+        /mnt/.borgmatic-snapshot-1234/mnt/.borgmatic-snapshot-1234
+
+    So to prevent that from ending up in the Borg archive, this function produces an exclude pattern
+    to exclude that path.
+    '''
+    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+
+    return borgmatic.borg.pattern.Pattern(
+        os.path.join(
+            subvolume_path,
+            snapshot_directory,
+            subvolume_path.lstrip(os.path.sep),
+            snapshot_directory,
+        ),
+        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+        borgmatic.borg.pattern.Pattern_style.FNMATCH,
+    )
+
+
+def make_borg_snapshot_pattern(subvolume_path, pattern):
+    '''
+    Given the path to a subvolume and a pattern as a borgmatic.borg.pattern.Pattern instance whose
+    path is inside the subvolume, return a new Pattern with its path rewritten to be in a snapshot
+    path intended for giving to Borg.
+
+    Move any initial caret in a regular expression pattern path to the beginning, so as not to break
+    the regular expression.
+    '''
+    initial_caret = (
+        '^'
+        if pattern.style == borgmatic.borg.pattern.Pattern_style.REGULAR_EXPRESSION
+        and pattern.path.startswith('^')
+        else ''
+    )
+
+    rewritten_path = initial_caret + os.path.join(
+        subvolume_path,
+        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        '.',  # Borg 1.4+ "slashdot" hack.
+        # Included so that the source directory ends up in the Borg archive at its "original" path.
+        pattern.path.lstrip('^').lstrip(os.path.sep),
+    )
+
+    return borgmatic.borg.pattern.Pattern(
+        rewritten_path,
+        pattern.type,
+        pattern.style,
+        pattern.device,
+    )
+
+
+def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma: no cover
+    '''
+    Given a Btrfs command to run, the path to a subvolume, and the path for a snapshot, create a new
+    Btrfs snapshot of the subvolume.
+    '''
+    os.makedirs(os.path.dirname(snapshot_path), mode=0o700, exist_ok=True)
+
+    borgmatic.execute.execute_command(
+        tuple(btrfs_command.split(' '))
+        + (
+            'subvolume',
+            'snapshot',
+            '-r',  # Read-only.
+            subvolume_path,
+            snapshot_path,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any Btrfs subvolume mount points listed in the given patterns. Also
+    update those patterns, replacing subvolume mount points with corresponding snapshot directories
+    so they get stored in the Borg archive instead.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+
+    If this is a dry run, then don't actually snapshot anything.
+    '''
+    dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
+    logger.info(f'Snapshotting Btrfs subvolumes{dry_run_label}')
+
+    # Based on the configured patterns, determine Btrfs subvolumes to backup.
+    btrfs_command = hook_config.get('btrfs_command', 'btrfs')
+    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
+    subvolumes = get_subvolumes(btrfs_command, findmnt_command, patterns)
+
+    if not subvolumes:
+        logger.warning(f'No Btrfs subvolumes found to snapshot{dry_run_label}')
+
+    # Snapshot each subvolume, rewriting patterns to use their snapshot paths.
+    for subvolume in subvolumes:
+        logger.debug(f'Creating Btrfs snapshot for {subvolume.path} subvolume')
+
+        snapshot_path = make_snapshot_path(subvolume.path)
+
+        if dry_run:
+            continue
+
+        snapshot_subvolume(btrfs_command, subvolume.path, snapshot_path)
+
+        for pattern in subvolume.contained_patterns:
+            snapshot_pattern = make_borg_snapshot_pattern(subvolume.path, pattern)
+
+            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            try:
+                patterns[patterns.index(pattern)] = snapshot_pattern
+            except ValueError:
+                patterns.append(snapshot_pattern)
+
+        patterns.append(make_snapshot_exclude_pattern(subvolume.path))
+
+    return []
+
+
+def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
+    '''
+    Given a Btrfs command to run and the name of a snapshot path, delete it.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(btrfs_command.split(' '))
+        + (
+            'subvolume',
+            'delete',
+            snapshot_path,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, delete any Btrfs snapshots created by borgmatic. If this is a dry run
+    or Btrfs isn't configured in borgmatic's configuration, then don't actually remove anything.
+    '''
+    if hook_config is None:
+        return
+
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    btrfs_command = hook_config.get('btrfs_command', 'btrfs')
+    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
+
+    try:
+        all_subvolumes = get_subvolumes(btrfs_command, findmnt_command)
+    except FileNotFoundError as error:
+        logger.debug(f'Could not find "{error.filename}" command')
+        return
+    except subprocess.CalledProcessError as error:
+        logger.debug(error)
+        return
+
+    # Reversing the sorted subvolumes ensures that we remove longer mount point paths of child
+    # subvolumes before the shorter mount point paths of parent subvolumes.
+    for subvolume in reversed(all_subvolumes):
+        subvolume_snapshots_glob = borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(make_snapshot_path(subvolume.path)),
+            temporary_directory_prefix=BORGMATIC_SNAPSHOT_PREFIX,
+        )
+
+        logger.debug(
+            f'Looking for snapshots to remove in {subvolume_snapshots_glob}{dry_run_label}'
+        )
+
+        for snapshot_path in glob.glob(subvolume_snapshots_glob):
+            if not os.path.isdir(snapshot_path):
+                continue
+
+            logger.debug(f'Deleting Btrfs snapshot {snapshot_path}{dry_run_label}')
+
+            if dry_run:
+                continue
+
+            try:
+                delete_snapshot(btrfs_command, snapshot_path)
+            except FileNotFoundError:
+                logger.debug(f'Could not find "{btrfs_command}" command')
+                return
+            except subprocess.CalledProcessError as error:
+                logger.debug(error)
+                return
+
+            # Strip off the subvolume path from the end of the snapshot path and then delete the
+            # resulting directory.
+            shutil.rmtree(snapshot_path.rsplit(subvolume.path, 1)[0])
+
+
+def make_data_source_dump_patterns(
+    hook_config, config, borgmatic_runtime_directory, name=None
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/data_source/dump.py

@@ -5,13 +5,7 @@ import shutil
 
 logger = logging.getLogger(__name__)
 
-DATA_SOURCE_HOOK_NAMES = (
-    'mariadb_databases',
-    'mysql_databases',
-    'mongodb_databases',
-    'postgresql_databases',
-    'sqlite_databases',
-)
+IS_A_HOOK = False
 
 
 def make_data_source_dump_path(borgmatic_runtime_directory, data_source_hook_name):
@@ -22,17 +16,19 @@ def make_data_source_dump_path(borgmatic_runtime_directory, data_source_hook_nam
     return os.path.join(borgmatic_runtime_directory, data_source_hook_name)
 
 
-def make_data_source_dump_filename(dump_path, name, hostname=None):
+def make_data_source_dump_filename(dump_path, name, hostname=None, port=None):
     '''
-    Based on the given dump directory path, data source name, and hostname, return a filename to use
-    for the data source dump. The hostname defaults to localhost.
+    Based on the given dump directory path, data source name, hostname, and port, return a filename
+    to use for the data source dump. The hostname defaults to localhost.
 
     Raise ValueError if the data source name is invalid.
     '''
     if os.path.sep in name:
         raise ValueError(f'Invalid data source name {name}')
 
-    return os.path.join(dump_path, hostname or 'localhost', name)
+    return os.path.join(
+        dump_path, (hostname or 'localhost') + ('' if port is None else f':{port}'), name
+    )
 
 
 def create_parent_directory_for_dump(dump_path):
@@ -50,14 +46,14 @@ def create_named_pipe_for_dump(dump_path):
     os.mkfifo(dump_path, mode=0o600)
 
 
-def remove_data_source_dumps(dump_path, data_source_type_name, log_prefix, dry_run):
+def remove_data_source_dumps(dump_path, data_source_type_name, dry_run):
     '''
     Remove all data source dumps in the given dump directory path (including the directory itself).
     If this is a dry run, then don't actually remove anything.
     '''
     dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
 
-    logger.debug(f'{log_prefix}: Removing {data_source_type_name} data source dumps{dry_run_label}')
+    logger.debug(f'Removing {data_source_type_name} data source dumps{dry_run_label}')
 
     if dry_run:
         return

borgmatic/hooks/data_source/lvm.py

@@ -0,0 +1,426 @@
+import collections
+import glob
+import json
+import logging
+import os
+import shutil
+import subprocess
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+import borgmatic.execute
+import borgmatic.hooks.data_source.snapshot
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+BORGMATIC_SNAPSHOT_PREFIX = 'borgmatic-'
+Logical_volume = collections.namedtuple(
+    'Logical_volume', ('name', 'device_path', 'mount_point', 'contained_patterns')
+)
+
+
+def get_logical_volumes(lsblk_command, patterns=None):
+    '''
+    Given an lsblk command to run and a sequence of configured patterns, find the intersection
+    between the current LVM logical volume mount points and the paths of any patterns. The idea is
+    that these pattern paths represent the requested logical volumes to snapshot.
+
+    If patterns is None, include all logical volume mounts points, not just those in patterns.
+
+    Return the result as a sequence of Logical_volume instances.
+    '''
+    try:
+        devices_info = json.loads(
+            borgmatic.execute.execute_command_and_capture_output(
+                # Use lsblk instead of lvs here because lvs can't show active mounts.
+                tuple(lsblk_command.split(' '))
+                + (
+                    '--output',
+                    'name,path,mountpoint,type',
+                    '--json',
+                    '--list',
+                )
+            )
+        )
+    except json.JSONDecodeError as error:
+        raise ValueError(f'Invalid {lsblk_command} JSON output: {error}')
+
+    candidate_patterns = set(patterns or ())
+
+    try:
+        # Sort from longest to shortest mount points, so longer mount points get a whack at the
+        # candidate pattern piñata before their parents do. (Patterns are consumed below, so no two
+        # logical volumes end up with the same contained patterns.)
+        return tuple(
+            Logical_volume(device['name'], device['path'], device['mountpoint'], contained_patterns)
+            for device in sorted(
+                devices_info['blockdevices'],
+                key=lambda device: device['mountpoint'] or '',
+                reverse=True,
+            )
+            if device['mountpoint'] and device['type'] == 'lvm'
+            for contained_patterns in (
+                borgmatic.hooks.data_source.snapshot.get_contained_patterns(
+                    device['mountpoint'], candidate_patterns
+                ),
+            )
+            if not patterns or contained_patterns
+        )
+    except KeyError as error:
+        raise ValueError(f'Invalid {lsblk_command} output: Missing key "{error}"')
+
+
+def snapshot_logical_volume(
+    lvcreate_command,
+    snapshot_name,
+    logical_volume_device,
+    snapshot_size,
+):
+    '''
+    Given an lvcreate command to run, a snapshot name, the path to the logical volume device to
+    snapshot, and a snapshot size string, create a new LVM snapshot.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(lvcreate_command.split(' '))
+        + (
+            '--snapshot',
+            ('--extents' if '%' in snapshot_size else '--size'),
+            snapshot_size,
+            '--permission',
+            'r',  # Read-only.
+            '--name',
+            snapshot_name,
+            logical_volume_device,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def mount_snapshot(mount_command, snapshot_device, snapshot_mount_path):  # pragma: no cover
+    '''
+    Given a mount command to run, the device path for an existing snapshot, and the path where the
+    snapshot should be mounted, mount the snapshot as read-only (making any necessary directories
+    first).
+    '''
+    os.makedirs(snapshot_mount_path, mode=0o700, exist_ok=True)
+
+    borgmatic.execute.execute_command(
+        tuple(mount_command.split(' '))
+        + (
+            '-o',
+            'ro',
+            snapshot_device,
+            snapshot_mount_path,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+    '''
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
+    path rewritten to be in a snapshot directory based on the given runtime directory.
+
+    Move any initial caret in a regular expression pattern path to the beginning, so as not to break
+    the regular expression.
+    '''
+    initial_caret = (
+        '^'
+        if pattern.style == borgmatic.borg.pattern.Pattern_style.REGULAR_EXPRESSION
+        and pattern.path.startswith('^')
+        else ''
+    )
+
+    rewritten_path = initial_caret + os.path.join(
+        normalized_runtime_directory,
+        'lvm_snapshots',
+        '.',  # Borg 1.4+ "slashdot" hack.
+        # Included so that the source directory ends up in the Borg archive at its "original" path.
+        pattern.path.lstrip('^').lstrip(os.path.sep),
+    )
+
+    return borgmatic.borg.pattern.Pattern(
+        rewritten_path,
+        pattern.type,
+        pattern.style,
+        pattern.device,
+    )
+
+
+DEFAULT_SNAPSHOT_SIZE = '10%ORIGIN'
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given an LVM configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any LVM logical volume mount points listed in the given patterns. Also
+    update those patterns, replacing logical volume mount points with corresponding snapshot
+    directories so they get stored in the Borg archive instead.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+
+    If this is a dry run, then don't actually snapshot anything.
+    '''
+    dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
+    logger.info(f'Snapshotting LVM logical volumes{dry_run_label}')
+
+    # List logical volumes to get their mount points.
+    lsblk_command = hook_config.get('lsblk_command', 'lsblk')
+    requested_logical_volumes = get_logical_volumes(lsblk_command, patterns)
+
+    # Snapshot each logical volume, rewriting source directories to use the snapshot paths.
+    snapshot_suffix = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+    normalized_runtime_directory = os.path.normpath(borgmatic_runtime_directory)
+
+    if not requested_logical_volumes:
+        logger.warning(f'No LVM logical volumes found to snapshot{dry_run_label}')
+
+    for logical_volume in requested_logical_volumes:
+        snapshot_name = f'{logical_volume.name}_{snapshot_suffix}'
+        logger.debug(
+            f'Creating LVM snapshot {snapshot_name} of {logical_volume.mount_point}{dry_run_label}'
+        )
+
+        if not dry_run:
+            snapshot_logical_volume(
+                hook_config.get('lvcreate_command', 'lvcreate'),
+                snapshot_name,
+                logical_volume.device_path,
+                hook_config.get('snapshot_size', DEFAULT_SNAPSHOT_SIZE),
+            )
+
+        # Get the device path for the snapshot we just created.
+        try:
+            snapshot = get_snapshots(
+                hook_config.get('lvs_command', 'lvs'), snapshot_name=snapshot_name
+            )[0]
+        except IndexError:
+            raise ValueError(f'Cannot find LVM snapshot {snapshot_name}')
+
+        # Mount the snapshot into a particular named temporary directory so that the snapshot ends
+        # up in the Borg archive at the "original" logical volume mount point path.
+        snapshot_mount_path = os.path.join(
+            normalized_runtime_directory,
+            'lvm_snapshots',
+            logical_volume.mount_point.lstrip(os.path.sep),
+        )
+
+        logger.debug(
+            f'Mounting LVM snapshot {snapshot_name} at {snapshot_mount_path}{dry_run_label}'
+        )
+
+        if dry_run:
+            continue
+
+        mount_snapshot(
+            hook_config.get('mount_command', 'mount'), snapshot.device_path, snapshot_mount_path
+        )
+
+        for pattern in logical_volume.contained_patterns:
+            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+
+            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            try:
+                patterns[patterns.index(pattern)] = snapshot_pattern
+            except ValueError:
+                patterns.append(snapshot_pattern)
+
+    return []
+
+
+def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
+    '''
+    Given a umount command to run and the mount path of a snapshot, unmount it.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(umount_command.split(' ')) + (snapshot_mount_path,),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def remove_snapshot(lvremove_command, snapshot_device_path):  # pragma: no cover
+    '''
+    Given an lvremove command to run and the device path of a snapshot, remove it it.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(lvremove_command.split(' '))
+        + (
+            '--force',  # Suppress an interactive "are you sure?" type prompt.
+            snapshot_device_path,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+Snapshot = collections.namedtuple(
+    'Snapshot',
+    ('name', 'device_path'),
+)
+
+
+def get_snapshots(lvs_command, snapshot_name=None):
+    '''
+    Given an lvs command to run, return all LVM snapshots as a sequence of Snapshot instances.
+
+    If a snapshot name is given, filter the results to that snapshot.
+    '''
+    try:
+        snapshot_info = json.loads(
+            borgmatic.execute.execute_command_and_capture_output(
+                # Use lvs instead of lsblk here because lsblk can't filter to just snapshots.
+                tuple(lvs_command.split(' '))
+                + (
+                    '--report-format',
+                    'json',
+                    '--options',
+                    'lv_name,lv_path',
+                    '--select',
+                    'lv_attr =~ ^s',  # Filter to just snapshots.
+                )
+            )
+        )
+    except json.JSONDecodeError as error:
+        raise ValueError(f'Invalid {lvs_command} JSON output: {error}')
+
+    try:
+        return tuple(
+            Snapshot(snapshot['lv_name'], snapshot['lv_path'])
+            for snapshot in snapshot_info['report'][0]['lv']
+            if snapshot_name is None or snapshot['lv_name'] == snapshot_name
+        )
+    except IndexError:
+        raise ValueError(f'Invalid {lvs_command} output: Missing report data')
+    except KeyError as error:
+        raise ValueError(f'Invalid {lvs_command} output: Missing key "{error}"')
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given an LVM configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, unmount and delete any LVM snapshots created by borgmatic. If this is
+    a dry run or LVM isn't configured in borgmatic's configuration, then don't actually remove
+    anything.
+    '''
+    if hook_config is None:
+        return
+
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    # Unmount snapshots.
+    try:
+        logical_volumes = get_logical_volumes(hook_config.get('lsblk_command', 'lsblk'))
+    except FileNotFoundError as error:
+        logger.debug(f'Could not find "{error.filename}" command')
+        return
+    except subprocess.CalledProcessError as error:
+        logger.debug(error)
+        return
+
+    snapshots_glob = os.path.join(
+        borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(borgmatic_runtime_directory),
+        ),
+        'lvm_snapshots',
+    )
+    logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
+    umount_command = hook_config.get('umount_command', 'umount')
+
+    for snapshots_directory in glob.glob(snapshots_glob):
+        if not os.path.isdir(snapshots_directory):
+            continue
+
+        for logical_volume in logical_volumes:
+            snapshot_mount_path = os.path.join(
+                snapshots_directory, logical_volume.mount_point.lstrip(os.path.sep)
+            )
+            if not os.path.isdir(snapshot_mount_path):
+                continue
+
+            # This might fail if the directory is already mounted, but we swallow errors here since
+            # we'll do another recursive delete below. The point of doing it here is that we don't
+            # want to try to unmount a non-mounted directory (which *will* fail).
+            if not dry_run:
+                shutil.rmtree(snapshot_mount_path, ignore_errors=True)
+
+                # If the delete was successful, that means there's nothing to unmount.
+                if not os.path.isdir(snapshot_mount_path):
+                    continue
+
+            logger.debug(f'Unmounting LVM snapshot at {snapshot_mount_path}{dry_run_label}')
+
+            if dry_run:
+                continue
+
+            try:
+                unmount_snapshot(umount_command, snapshot_mount_path)
+            except FileNotFoundError:
+                logger.debug(f'Could not find "{umount_command}" command')
+                return
+            except subprocess.CalledProcessError as error:
+                logger.debug(error)
+                return
+
+        if not dry_run:
+            shutil.rmtree(snapshots_directory)
+
+    # Delete snapshots.
+    lvremove_command = hook_config.get('lvremove_command', 'lvremove')
+
+    try:
+        snapshots = get_snapshots(hook_config.get('lvs_command', 'lvs'))
+    except FileNotFoundError as error:
+        logger.debug(f'Could not find "{error.filename}" command')
+        return
+    except subprocess.CalledProcessError as error:
+        logger.debug(error)
+        return
+
+    for snapshot in snapshots:
+        # Only delete snapshots that borgmatic actually created!
+        if not snapshot.name.split('_')[-1].startswith(BORGMATIC_SNAPSHOT_PREFIX):
+            continue
+
+        logger.debug(f'Deleting LVM snapshot {snapshot.name}{dry_run_label}')
+
+        if not dry_run:
+            remove_snapshot(lvremove_command, snapshot.device_path)
+
+
+def make_data_source_dump_patterns(
+    hook_config, config, borgmatic_runtime_directory, name=None
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/data_source/mariadb.py

@@ -3,13 +3,14 @@ import logging
 import os
 import shlex
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
     execute_command_with_processes,
 )
-from borgmatic.hooks import dump
+from borgmatic.hooks.data_source import dump
 
 logger = logging.getLogger(__name__)
 
@@ -24,7 +25,7 @@ def make_dump_path(base_directory):  # pragma: no cover
 SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def database_names_to_dump(database, extra_environment, dry_run):
     '''
     Given a requested database config, return the corresponding sequence of database names to dump.
     In the case of "all", query for the names of databases on the configured host and return them,
@@ -48,7 +49,7 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
-    logger.debug(f'{log_prefix}: Querying for "all" MariaDB databases to dump')
+    logger.debug('Querying for "all" MariaDB databases to dump')
     show_output = execute_command_and_capture_output(
         show_command, extra_environment=extra_environment
     )
@@ -61,24 +62,26 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
 
 
 def execute_dump_command(
-    database, log_prefix, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database, dump_path, database_names, extra_environment, dry_run, dry_run_label
 ):
     '''
     Kick off a dump for the given MariaDB database (provided as a configuration dict) to a named
-    pipe constructed from the given dump path and database name. Use the given log prefix in any
-    log entries.
+    pipe constructed from the given dump path and database name.
 
     Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
     this is a dry run, then don't actually dump anything and return None.
     '''
     database_name = database['name']
     dump_filename = dump.make_data_source_dump_filename(
-        dump_path, database['name'], database.get('hostname')
+        dump_path,
+        database['name'],
+        database.get('hostname'),
+        database.get('port'),
     )
 
     if os.path.exists(dump_filename):
         logger.warning(
-            f'{log_prefix}: Skipping duplicate dump of MariaDB database "{database_name}" to {dump_filename}'
+            f'Skipping duplicate dump of MariaDB database "{database_name}" to {dump_filename}'
         )
         return None
 
@@ -99,9 +102,7 @@ def execute_dump_command(
         + ('--result-file', dump_filename)
     )
 
-    logger.debug(
-        f'{log_prefix}: Dumping MariaDB database "{database_name}" to {dump_filename}{dry_run_label}'
-    )
+    logger.debug(f'Dumping MariaDB database "{database_name}" to {dump_filename}{dry_run_label}')
     if dry_run:
         return None
 
@@ -114,35 +115,45 @@ def execute_dump_command(
     )
 
 
-def use_streaming(databases, config, log_prefix):
+def get_default_port(databases, config):  # pragma: no cover
+    return 3306
+
+
+def use_streaming(databases, config):
     '''
-    Given a sequence of MariaDB database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MariaDB database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
 
-def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def dump_data_sources(
+    databases,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
     '''
     Dump the given MariaDB databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
+    Also append the the parent directory of the database dumps to the given patterns list, so the
+    dumps actually get backed up.
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping MariaDB databases{dry_run_label}')
+    logger.info(f'Dumping MariaDB databases{dry_run_label}')
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
         extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
-        dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
-        )
+        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -157,7 +168,6 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                 processes.append(
                     execute_dump_command(
                         renamed_database,
-                        log_prefix,
                         dump_path,
                         (dump_name,),
                         extra_environment,
@@ -169,7 +179,6 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             processes.append(
                 execute_dump_command(
                     database,
-                    log_prefix,
                     dump_path,
                     dump_database_names,
                     extra_environment,
@@ -178,29 +187,34 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                 )
             )
 
+    if not dry_run:
+        patterns.append(
+            borgmatic.borg.pattern.Pattern(
+                os.path.join(borgmatic_runtime_directory, 'mariadb_databases')
+            )
+        )
+
     return [process for process in processes if process]
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic_runtime_directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic_runtime_directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MariaDB', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MariaDB', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -218,7 +232,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -227,9 +240,9 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     hostname = connection_params['hostname'] or data_source.get(
@@ -263,7 +276,7 @@ def restore_data_source_dump(
     )
     extra_environment = {'MYSQL_PWD': password} if password else None
 
-    logger.debug(f"{log_prefix}: Restoring MariaDB database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 

borgmatic/hooks/data_source/mongodb.py

@@ -1,9 +1,11 @@
 import logging
+import os
 import shlex
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 from borgmatic.execute import execute_command, execute_command_with_processes
-from borgmatic.hooks import dump
+from borgmatic.hooks.data_source import dump
 
 logger = logging.getLogger(__name__)
 
@@ -15,38 +17,53 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'mongodb_databases')
 
 
-def use_streaming(databases, config, log_prefix):
+def get_default_port(databases, config):  # pragma: no cover
+    return 27017
+
+
+def use_streaming(databases, config):
     '''
-    Given a sequence of MongoDB database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MongoDB database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(database.get('format') != 'directory' for database in databases)
 
 
-def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def dump_data_sources(
+    databases,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
     '''
     Dump the given MongoDB databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the borgmatic
-    runtime directory to construct the destination path (used for the directory format and the given
-    log prefix in any log entries.
+    runtime directory to construct the destination path (used for the directory format.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
+    Also append the the parent directory of the database dumps to the given patterns list, so the
+    dumps actually get backed up.
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
 
-    logger.info(f'{log_prefix}: Dumping MongoDB databases{dry_run_label}')
+    logger.info(f'Dumping MongoDB databases{dry_run_label}')
 
     processes = []
     for database in databases:
         name = database['name']
         dump_filename = dump.make_data_source_dump_filename(
-            make_dump_path(borgmatic_runtime_directory), name, database.get('hostname')
+            make_dump_path(borgmatic_runtime_directory),
+            name,
+            database.get('hostname'),
+            database.get('port'),
         )
         dump_format = database.get('format', 'archive')
 
         logger.debug(
-            f'{log_prefix}: Dumping MongoDB database {name} to {dump_filename}{dry_run_label}',
+            f'Dumping MongoDB database {name} to {dump_filename}{dry_run_label}',
         )
         if dry_run:
             continue
@@ -60,6 +77,13 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             dump.create_named_pipe_for_dump(dump_filename)
             processes.append(execute_command(command, shell=True, run_to_completion=False))
 
+    if not dry_run:
+        patterns.append(
+            borgmatic.borg.pattern.Pattern(
+                os.path.join(borgmatic_runtime_directory, 'mongodb_databases')
+            )
+        )
+
     return processes
 
 
@@ -92,25 +116,23 @@ def build_dump_command(database, dump_filename, dump_format):
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic_runtime_directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic_runtime_directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MongoDB', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MongoDB', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -128,7 +150,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -138,9 +159,9 @@ def restore_data_source_dump(
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
     configuration dict, but the given hook configuration is ignored. The given configuration dict is
-    used to construct the destination path, and the given log prefix is used for any log entries. If
-    this is a dry run, then don't actually restore anything. Trigger the given active extract
-    process (an instance of subprocess.Popen) to produce output to consume.
+    used to construct the destination path. If this is a dry run, then don't actually restore
+    anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
+    output to consume.
 
     If the extract process is None, then restore the dump from the filesystem rather than from an
     extract stream.
@@ -155,7 +176,7 @@ def restore_data_source_dump(
         extract_process, data_source, dump_filename, connection_params
     )
 
-    logger.debug(f"{log_prefix}: Restoring MongoDB database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MongoDB database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 

borgmatic/hooks/data_source/mysql.py

@@ -3,13 +3,14 @@ import logging
 import os
 import shlex
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
     execute_command_with_processes,
 )
-from borgmatic.hooks import dump
+from borgmatic.hooks.data_source import dump
 
 logger = logging.getLogger(__name__)
 
@@ -24,7 +25,7 @@ def make_dump_path(base_directory):  # pragma: no cover
 SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def database_names_to_dump(database, extra_environment, dry_run):
     '''
     Given a requested database config, return the corresponding sequence of database names to dump.
     In the case of "all", query for the names of databases on the configured host and return them,
@@ -48,7 +49,7 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
-    logger.debug(f'{log_prefix}: Querying for "all" MySQL databases to dump')
+    logger.debug('Querying for "all" MySQL databases to dump')
     show_output = execute_command_and_capture_output(
         show_command, extra_environment=extra_environment
     )
@@ -61,24 +62,26 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
 
 
 def execute_dump_command(
-    database, log_prefix, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database, dump_path, database_names, extra_environment, dry_run, dry_run_label
 ):
     '''
     Kick off a dump for the given MySQL/MariaDB database (provided as a configuration dict) to a
-    named pipe constructed from the given dump path and database name. Use the given log prefix in
-    any log entries.
+    named pipe constructed from the given dump path and database name.
 
     Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
     this is a dry run, then don't actually dump anything and return None.
     '''
     database_name = database['name']
     dump_filename = dump.make_data_source_dump_filename(
-        dump_path, database['name'], database.get('hostname')
+        dump_path,
+        database['name'],
+        database.get('hostname'),
+        database.get('port'),
     )
 
     if os.path.exists(dump_filename):
         logger.warning(
-            f'{log_prefix}: Skipping duplicate dump of MySQL database "{database_name}" to {dump_filename}'
+            f'Skipping duplicate dump of MySQL database "{database_name}" to {dump_filename}'
         )
         return None
 
@@ -98,9 +101,7 @@ def execute_dump_command(
         + ('--result-file', dump_filename)
     )
 
-    logger.debug(
-        f'{log_prefix}: Dumping MySQL database "{database_name}" to {dump_filename}{dry_run_label}'
-    )
+    logger.debug(f'Dumping MySQL database "{database_name}" to {dump_filename}{dry_run_label}')
     if dry_run:
         return None
 
@@ -113,35 +114,45 @@ def execute_dump_command(
     )
 
 
-def use_streaming(databases, config, log_prefix):
+def get_default_port(databases, config):  # pragma: no cover
+    return 3306
+
+
+def use_streaming(databases, config):
     '''
-    Given a sequence of MySQL database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MySQL database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
 
-def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def dump_data_sources(
+    databases,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
     '''
     Dump the given MySQL/MariaDB databases to a named pipe. The databases are supplied as a sequence
     of dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
+    Also append the the parent directory of the database dumps to the given patterns list, so the
+    dumps actually get backed up.
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping MySQL databases{dry_run_label}')
+    logger.info(f'Dumping MySQL databases{dry_run_label}')
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
         extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
-        dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
-        )
+        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -156,7 +167,6 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                 processes.append(
                     execute_dump_command(
                         renamed_database,
-                        log_prefix,
                         dump_path,
                         (dump_name,),
                         extra_environment,
@@ -168,7 +178,6 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             processes.append(
                 execute_dump_command(
                     database,
-                    log_prefix,
                     dump_path,
                     dump_database_names,
                     extra_environment,
@@ -177,29 +186,34 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                 )
             )
 
+    if not dry_run:
+        patterns.append(
+            borgmatic.borg.pattern.Pattern(
+                os.path.join(borgmatic_runtime_directory, 'mysql_databases')
+            )
+        )
+
     return [process for process in processes if process]
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MySQL', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MySQL', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -217,7 +231,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -226,9 +239,9 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     hostname = connection_params['hostname'] or data_source.get(
@@ -262,7 +275,7 @@ def restore_data_source_dump(
     )
     extra_environment = {'MYSQL_PWD': password} if password else None
 
-    logger.debug(f"{log_prefix}: Restoring MySQL database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 

borgmatic/hooks/data_source/postgresql.py

@@ -5,13 +5,14 @@ import os
 import pathlib
 import shlex
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
     execute_command_with_processes,
 )
-from borgmatic.hooks import dump
+from borgmatic.hooks.data_source import dump
 
 logger = logging.getLogger(__name__)
 
@@ -57,7 +58,7 @@ def make_extra_environment(database, restore_connection_params=None):
 EXCLUDED_DATABASE_NAMES = ('template0', 'template1')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def database_names_to_dump(database, extra_environment, dry_run):
     '''
     Given a requested database config, return the corresponding sequence of database names to dump.
     In the case of "all" when a database format is given, query for the names of databases on the
@@ -84,7 +85,7 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
         + (('--username', database['username']) if 'username' in database else ())
         + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
     )
-    logger.debug(f'{log_prefix}: Querying for "all" PostgreSQL databases to dump')
+    logger.debug('Querying for "all" PostgreSQL databases to dump')
     list_output = execute_command_and_capture_output(
         list_command, extra_environment=extra_environment
     )
@@ -96,37 +97,47 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
     )
 
 
-def use_streaming(databases, config, log_prefix):
+def get_default_port(databases, config):  # pragma: no cover
+    return 5432
+
+
+def use_streaming(databases, config):
     '''
-    Given a sequence of PostgreSQL database configuration dicts, a configuration dict (ignored), and
-    a log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of PostgreSQL database configuration dicts, a configuration dict (ignored),
+    return whether streaming will be using during dumps.
     '''
     return any(database.get('format') != 'directory' for database in databases)
 
 
-def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def dump_data_sources(
+    databases,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
     '''
     Dump the given PostgreSQL databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
+    Also append the the parent directory of the database dumps to the given patterns list, so the
+    dumps actually get backed up.
 
     Raise ValueError if the databases to dump cannot be determined.
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping PostgreSQL databases{dry_run_label}')
+    logger.info(f'Dumping PostgreSQL databases{dry_run_label}')
 
     for database in databases:
         extra_environment = make_extra_environment(database)
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
-        )
+        dump_database_names = database_names_to_dump(database, extra_environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -142,11 +153,14 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                 for part in shlex.split(database.get('pg_dump_command') or default_dump_command)
             )
             dump_filename = dump.make_data_source_dump_filename(
-                dump_path, database_name, database.get('hostname')
+                dump_path,
+                database_name,
+                database.get('hostname'),
+                database.get('port'),
             )
             if os.path.exists(dump_filename):
                 logger.warning(
-                    f'{log_prefix}: Skipping duplicate dump of PostgreSQL database "{database_name}" to {dump_filename}'
+                    f'Skipping duplicate dump of PostgreSQL database "{database_name}" to {dump_filename}'
                 )
                 continue
 
@@ -180,7 +194,7 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             )
 
             logger.debug(
-                f'{log_prefix}: Dumping PostgreSQL database "{database_name}" to {dump_filename}{dry_run_label}'
+                f'Dumping PostgreSQL database "{database_name}" to {dump_filename}{dry_run_label}'
             )
             if dry_run:
                 continue
@@ -203,29 +217,36 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
                     )
                 )
 
+    if not dry_run:
+        patterns.append(
+            borgmatic.borg.pattern.Pattern(
+                os.path.join(borgmatic_runtime_directory, 'postgresql_databases')
+            )
+        )
+
     return processes
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
     dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'PostgreSQL', log_prefix, dry_run
+        make_dump_path(borgmatic_runtime_directory), 'PostgreSQL', dry_run
     )
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -243,7 +264,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -253,10 +273,9 @@ def restore_data_source_dump(
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
     configuration dict, but the given hook configuration is ignored. The given borgmatic runtime
-    directory is used to construct the destination path (used for the directory format), and the
-    given log prefix is used for any log entries. If this is a dry run, then don't actually restore
-    anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
-    output to consume.
+    directory is used to construct the destination path (used for the directory format). If this is
+    a dry run, then don't actually restore anything. Trigger the given active extract process (an
+    instance of subprocess.Popen) to produce output to consume.
 
     If the extract process is None, then restore the dump from the filesystem rather than from an
     extract stream.
@@ -329,9 +348,7 @@ def restore_data_source_dump(
         data_source, restore_connection_params=connection_params
     )
 
-    logger.debug(
-        f"{log_prefix}: Restoring PostgreSQL database {data_source['name']}{dry_run_label}"
-    )
+    logger.debug(f"Restoring PostgreSQL database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 

borgmatic/hooks/data_source/snapshot.py

@@ -0,0 +1,39 @@
+import pathlib
+
+IS_A_HOOK = False
+
+
+def get_contained_patterns(parent_directory, candidate_patterns):
+    '''
+    Given a parent directory and a set of candidate patterns potentially inside it, get the subset
+    of contained patterns for which the parent directory is actually the parent, a grandparent, the
+    very same directory, etc. The idea is if, say, /var/log and /var/lib are candidate pattern
+    paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at /var, then
+    /var is what we want to snapshot.
+
+    For this to work, a candidate pattern path can't have any globs or other non-literal characters
+    in the initial portion of the path that matches the parent directory. For instance, a parent
+    directory of /var would match a candidate pattern path of /var/log/*/data, but not a pattern
+    path like /v*/log/*/data.
+
+    The one exception is that if a regular expression pattern path starts with "^", that will get
+    stripped off for purposes of matching against a parent directory.
+
+    As part of this, also mutate the given set of candidate patterns to remove any actually
+    contained patterns from it. That way, this function can be called multiple times, successively
+    processing candidate patterns until none are left—and avoiding assigning any candidate pattern
+    to more than one parent directory.
+    '''
+    if not candidate_patterns:
+        return ()
+
+    contained_patterns = tuple(
+        candidate
+        for candidate in candidate_patterns
+        for candidate_path in (pathlib.PurePath(candidate.path.lstrip('^')),)
+        if pathlib.PurePath(parent_directory) == candidate_path
+        or pathlib.PurePath(parent_directory) in candidate_path.parents
+    )
+    candidate_patterns -= set(contained_patterns)
+
+    return contained_patterns

borgmatic/hooks/data_source/sqlite.py

@@ -2,9 +2,10 @@ import logging
 import os
 import shlex
 
+import borgmatic.borg.pattern
 import borgmatic.config.paths
 from borgmatic.execute import execute_command, execute_command_with_processes
-from borgmatic.hooks import dump
+from borgmatic.hooks.data_source import dump
 
 logger = logging.getLogger(__name__)
 
@@ -16,27 +17,40 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'sqlite_databases')
 
 
-def use_streaming(databases, config, log_prefix):
+def get_default_port(databases, config):  # pragma: no cover
+    return None  # SQLite doesn't use a port.
+
+
+def use_streaming(databases, config):
     '''
-    Given a sequence of SQLite database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of SQLite database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
 
-def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def dump_data_sources(
+    databases,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
     '''
     Dump the given SQLite databases to a named pipe. The databases are supplied as a sequence of
     configuration dicts, as per the configuration schema. Use the given borgmatic runtime directory
-    to construct the destination path and the given log prefix in any log entries.
+    to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
+    Also append the the parent directory of the database dumps to the given patterns list, so the
+    dumps actually get backed up.
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping SQLite databases{dry_run_label}')
+    logger.info(f'Dumping SQLite databases{dry_run_label}')
 
     for database in databases:
         database_path = database['path']
@@ -45,7 +59,7 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             logger.warning('The "all" database name has no meaning for SQLite databases')
         if not os.path.exists(database_path):
             logger.warning(
-                f'{log_prefix}: No SQLite database at {database_path}; an empty database will be created and dumped'
+                f'No SQLite database at {database_path}; an empty database will be created and dumped'
             )
 
         dump_path = make_dump_path(borgmatic_runtime_directory)
@@ -53,7 +67,7 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
 
         if os.path.exists(dump_filename):
             logger.warning(
-                f'{log_prefix}: Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
+                f'Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
             )
             continue
 
@@ -65,7 +79,7 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
             shlex.quote(dump_filename),
         )
         logger.debug(
-            f'{log_prefix}: Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
+            f'Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
         )
         if dry_run:
             continue
@@ -73,29 +87,34 @@ def dump_data_sources(databases, config, log_prefix, borgmatic_runtime_directory
         dump.create_named_pipe_for_dump(dump_filename)
         processes.append(execute_command(command, shell=True, run_to_completion=False))
 
+    if not dry_run:
+        patterns.append(
+            borgmatic.borg.pattern.Pattern(
+                os.path.join(borgmatic_runtime_directory, 'sqlite_databases')
+            )
+        )
+
     return processes
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'SQLite', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'SQLite', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -113,7 +132,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -122,22 +140,22 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     database_path = connection_params['restore_path'] or data_source.get(
         'restore_path', data_source.get('path')
     )
 
-    logger.debug(f'{log_prefix}: Restoring SQLite database at {database_path}{dry_run_label}')
+    logger.debug(f'Restoring SQLite database at {database_path}{dry_run_label}')
     if dry_run:
         return
 
     try:
         os.remove(database_path)
-        logger.warning(f'{log_prefix}: Removed existing SQLite database at {database_path}')
+        logger.warning(f'Removed existing SQLite database at {database_path}')
     except FileNotFoundError:  # pragma: no cover
         pass
 

borgmatic/hooks/data_source/zfs.py

@@ -0,0 +1,413 @@
+import collections
+import glob
+import logging
+import os
+import shutil
+import subprocess
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+import borgmatic.execute
+import borgmatic.hooks.data_source.snapshot
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+BORGMATIC_SNAPSHOT_PREFIX = 'borgmatic-'
+BORGMATIC_USER_PROPERTY = 'org.torsion.borgmatic:backup'
+
+
+Dataset = collections.namedtuple(
+    'Dataset',
+    ('name', 'mount_point', 'auto_backup', 'contained_patterns'),
+    defaults=(False, ()),
+)
+
+
+def get_datasets_to_backup(zfs_command, patterns):
+    '''
+    Given a ZFS command to run and a sequence of configured patterns, find the intersection between
+    the current ZFS dataset mount points and the paths of any patterns. The idea is that these
+    pattern paths represent the requested datasets to snapshot. But also include any datasets tagged
+    with a borgmatic-specific user property, whether or not they appear in the patterns.
+
+    Return the result as a sequence of Dataset instances, sorted by mount point.
+    '''
+    list_output = borgmatic.execute.execute_command_and_capture_output(
+        tuple(zfs_command.split(' '))
+        + (
+            'list',
+            '-H',
+            '-t',
+            'filesystem',
+            '-o',
+            f'name,mountpoint,{BORGMATIC_USER_PROPERTY}',
+        )
+    )
+
+    try:
+        # Sort from longest to shortest mount points, so longer mount points get a whack at the
+        # candidate pattern piñata before their parents do. (Patterns are consumed during the second
+        # loop below, so no two datasets end up with the same contained patterns.)
+        datasets = sorted(
+            (
+                Dataset(dataset_name, mount_point, (user_property_value == 'auto'), ())
+                for line in list_output.splitlines()
+                for (dataset_name, mount_point, user_property_value) in (line.rstrip().split('\t'),)
+            ),
+            key=lambda dataset: dataset.mount_point,
+            reverse=True,
+        )
+    except ValueError:
+        raise ValueError(f'Invalid {zfs_command} list output')
+
+    candidate_patterns = set(patterns)
+
+    return tuple(
+        sorted(
+            (
+                Dataset(
+                    dataset.name,
+                    dataset.mount_point,
+                    dataset.auto_backup,
+                    contained_patterns,
+                )
+                for dataset in datasets
+                for contained_patterns in (
+                    (
+                        (
+                            (borgmatic.borg.pattern.Pattern(dataset.mount_point),)
+                            if dataset.auto_backup
+                            else ()
+                        )
+                        + borgmatic.hooks.data_source.snapshot.get_contained_patterns(
+                            dataset.mount_point, candidate_patterns
+                        )
+                    ),
+                )
+                if contained_patterns
+            ),
+            key=lambda dataset: dataset.mount_point,
+        )
+    )
+
+
+def get_all_dataset_mount_points(zfs_command):
+    '''
+    Given a ZFS command to run, return all ZFS datasets as a sequence of sorted mount points.
+    '''
+    list_output = borgmatic.execute.execute_command_and_capture_output(
+        tuple(zfs_command.split(' '))
+        + (
+            'list',
+            '-H',
+            '-t',
+            'filesystem',
+            '-o',
+            'mountpoint',
+        )
+    )
+
+    return tuple(sorted(line.rstrip() for line in list_output.splitlines()))
+
+
+def snapshot_dataset(zfs_command, full_snapshot_name):  # pragma: no cover
+    '''
+    Given a ZFS command to run and a snapshot name of the form "dataset@snapshot", create a new ZFS
+    snapshot.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(zfs_command.split(' '))
+        + (
+            'snapshot',
+            full_snapshot_name,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def mount_snapshot(mount_command, full_snapshot_name, snapshot_mount_path):  # pragma: no cover
+    '''
+    Given a mount command to run, an existing snapshot name of the form "dataset@snapshot", and the
+    path where the snapshot should be mounted, mount the snapshot (making any necessary directories
+    first).
+    '''
+    os.makedirs(snapshot_mount_path, mode=0o700, exist_ok=True)
+
+    borgmatic.execute.execute_command(
+        tuple(mount_command.split(' '))
+        + (
+            '-t',
+            'zfs',
+            '-o',
+            'ro',
+            full_snapshot_name,
+            snapshot_mount_path,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+    '''
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
+    path rewritten to be in a snapshot directory based on the given runtime directory.
+
+    Move any initial caret in a regular expression pattern path to the beginning, so as not to break
+    the regular expression.
+    '''
+    initial_caret = (
+        '^'
+        if pattern.style == borgmatic.borg.pattern.Pattern_style.REGULAR_EXPRESSION
+        and pattern.path.startswith('^')
+        else ''
+    )
+
+    rewritten_path = initial_caret + os.path.join(
+        normalized_runtime_directory,
+        'zfs_snapshots',
+        '.',  # Borg 1.4+ "slashdot" hack.
+        # Included so that the source directory ends up in the Borg archive at its "original" path.
+        pattern.path.lstrip('^').lstrip(os.path.sep),
+    )
+
+    return borgmatic.borg.pattern.Pattern(
+        rewritten_path,
+        pattern.type,
+        pattern.style,
+        pattern.device,
+    )
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given a ZFS configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any ZFS dataset mount points listed in the given patterns and any
+    dataset with a borgmatic-specific user property. Also update those patterns, replacing dataset
+    mount points with corresponding snapshot directories so they get stored in the Borg archive
+    instead.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+
+    If this is a dry run, then don't actually snapshot anything.
+    '''
+    dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
+    logger.info(f'Snapshotting ZFS datasets{dry_run_label}')
+
+    # List ZFS datasets to get their mount points.
+    zfs_command = hook_config.get('zfs_command', 'zfs')
+    requested_datasets = get_datasets_to_backup(zfs_command, patterns)
+
+    # Snapshot each dataset, rewriting patterns to use the snapshot paths.
+    snapshot_name = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+    normalized_runtime_directory = os.path.normpath(borgmatic_runtime_directory)
+
+    if not requested_datasets:
+        logger.warning(f'No ZFS datasets found to snapshot{dry_run_label}')
+
+    for dataset in requested_datasets:
+        full_snapshot_name = f'{dataset.name}@{snapshot_name}'
+        logger.debug(
+            f'Creating ZFS snapshot {full_snapshot_name} of {dataset.mount_point}{dry_run_label}'
+        )
+
+        if not dry_run:
+            snapshot_dataset(zfs_command, full_snapshot_name)
+
+        # Mount the snapshot into a particular named temporary directory so that the snapshot ends
+        # up in the Borg archive at the "original" dataset mount point path.
+        snapshot_mount_path = os.path.join(
+            normalized_runtime_directory,
+            'zfs_snapshots',
+            dataset.mount_point.lstrip(os.path.sep),
+        )
+
+        logger.debug(
+            f'Mounting ZFS snapshot {full_snapshot_name} at {snapshot_mount_path}{dry_run_label}'
+        )
+
+        if dry_run:
+            continue
+
+        mount_snapshot(
+            hook_config.get('mount_command', 'mount'), full_snapshot_name, snapshot_mount_path
+        )
+
+        for pattern in dataset.contained_patterns:
+            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+
+            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            try:
+                patterns[patterns.index(pattern)] = snapshot_pattern
+            except ValueError:
+                patterns.append(snapshot_pattern)
+
+    return []
+
+
+def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
+    '''
+    Given a umount command to run and the mount path of a snapshot, unmount it.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(umount_command.split(' ')) + (snapshot_mount_path,),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def destroy_snapshot(zfs_command, full_snapshot_name):  # pragma: no cover
+    '''
+    Given a ZFS command to run and the name of a snapshot in the form "dataset@snapshot", destroy
+    it.
+    '''
+    borgmatic.execute.execute_command(
+        tuple(zfs_command.split(' '))
+        + (
+            'destroy',
+            full_snapshot_name,
+        ),
+        output_log_level=logging.DEBUG,
+    )
+
+
+def get_all_snapshots(zfs_command):
+    '''
+    Given a ZFS command to run, return all ZFS snapshots as a sequence of full snapshot names of the
+    form "dataset@snapshot".
+    '''
+    list_output = borgmatic.execute.execute_command_and_capture_output(
+        tuple(zfs_command.split(' '))
+        + (
+            'list',
+            '-H',
+            '-t',
+            'snapshot',
+            '-o',
+            'name',
+        )
+    )
+
+    return tuple(line.rstrip() for line in list_output.splitlines())
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given a ZFS configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, unmount and destroy any ZFS snapshots created by borgmatic. If this
+    is a dry run or ZFS isn't configured in borgmatic's configuration, then don't actually remove
+    anything.
+    '''
+    if hook_config is None:
+        return
+
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    # Unmount snapshots.
+    zfs_command = hook_config.get('zfs_command', 'zfs')
+
+    try:
+        dataset_mount_points = get_all_dataset_mount_points(zfs_command)
+    except FileNotFoundError:
+        logger.debug(f'Could not find "{zfs_command}" command')
+        return
+    except subprocess.CalledProcessError as error:
+        logger.debug(error)
+        return
+
+    snapshots_glob = os.path.join(
+        borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(borgmatic_runtime_directory),
+        ),
+        'zfs_snapshots',
+    )
+    logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
+    umount_command = hook_config.get('umount_command', 'umount')
+
+    for snapshots_directory in glob.glob(snapshots_glob):
+        if not os.path.isdir(snapshots_directory):
+            continue
+
+        # Reversing the sorted datasets ensures that we unmount the longer mount point paths of
+        # child datasets before the shorter mount point paths of parent datasets.
+        for mount_point in reversed(dataset_mount_points):
+            snapshot_mount_path = os.path.join(snapshots_directory, mount_point.lstrip(os.path.sep))
+            if not os.path.isdir(snapshot_mount_path):
+                continue
+
+            # This might fail if the path is already mounted, but we swallow errors here since we'll
+            # do another recursive delete below. The point of doing it here is that we don't want to
+            # try to unmount a non-mounted directory (which *will* fail), and probing for whether a
+            # directory is mounted is tough to do in a cross-platform way.
+            if not dry_run:
+                shutil.rmtree(snapshot_mount_path, ignore_errors=True)
+
+                # If the delete was successful, that means there's nothing to unmount.
+                if not os.path.isdir(snapshot_mount_path):
+                    continue
+
+            logger.debug(f'Unmounting ZFS snapshot at {snapshot_mount_path}{dry_run_label}')
+
+            if not dry_run:
+                try:
+                    unmount_snapshot(umount_command, snapshot_mount_path)
+                except FileNotFoundError:
+                    logger.debug(f'Could not find "{umount_command}" command')
+                    return
+                except subprocess.CalledProcessError as error:
+                    logger.debug(error)
+                    return
+
+        if not dry_run:
+            shutil.rmtree(snapshots_directory)
+
+    # Destroy snapshots.
+    full_snapshot_names = get_all_snapshots(zfs_command)
+
+    for full_snapshot_name in full_snapshot_names:
+        # Only destroy snapshots that borgmatic actually created!
+        if not full_snapshot_name.split('@')[-1].startswith(BORGMATIC_SNAPSHOT_PREFIX):
+            continue
+
+        logger.debug(f'Destroying ZFS snapshot {full_snapshot_name}{dry_run_label}')
+
+        if not dry_run:
+            destroy_snapshot(zfs_command, full_snapshot_name)
+
+
+def make_data_source_dump_patterns(
+    hook_config, config, borgmatic_runtime_directory, name=None
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/dispatch.py

@@ -1,71 +1,79 @@
+import enum
+import importlib
 import logging
+import pkgutil
 
-from borgmatic.hooks import (
-    apprise,
-    cronhub,
-    cronitor,
-    healthchecks,
-    loki,
-    mariadb,
-    mongodb,
-    mysql,
-    ntfy,
-    pagerduty,
-    postgresql,
-    pushover,
-    sqlite,
-    uptimekuma,
-    zabbix,
-)
+import borgmatic.hooks.credential
+import borgmatic.hooks.data_source
+import borgmatic.hooks.monitoring
 
 logger = logging.getLogger(__name__)
 
-HOOK_NAME_TO_MODULE = {
-    'apprise': apprise,
-    'cronhub': cronhub,
-    'cronitor': cronitor,
-    'healthchecks': healthchecks,
-    'loki': loki,
-    'mariadb_databases': mariadb,
-    'mongodb_databases': mongodb,
-    'mysql_databases': mysql,
-    'ntfy': ntfy,
-    'pagerduty': pagerduty,
-    'postgresql_databases': postgresql,
-    'pushover': pushover,
-    'sqlite_databases': sqlite,
-    'uptime_kuma': uptimekuma,
-    'zabbix': zabbix,
-}
+
+class Hook_type(enum.Enum):
+    CREDENTIAL = 'credential'
+    DATA_SOURCE = 'data_source'
+    MONITORING = 'monitoring'
 
 
-def call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs):
+def get_submodule_names(parent_module):  # pragma: no cover
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to the given hook name. Supply that call with the configuration
-    for this hook (if any), the log prefix, and any given args and kwargs. Return any return value.
+    Given a parent module, return the names of its direct submodules as a tuple of strings.
+    '''
+    return tuple(module_info.name for module_info in pkgutil.iter_modules(parent_module.__path__))
+
+
+def call_hook(function_name, config, hook_name, *args, **kwargs):
+    '''
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    the given hook name. Supply that call with the configuration for this hook (if any) and any
+    given args and kwargs. Return the return value of that call or None if the module in question is
+    not a hook.
 
     Raise ValueError if the hook name is unknown.
     Raise AttributeError if the function name is not found in the module.
     Raise anything else that the called function raises.
     '''
-    hook_config = config.get(hook_name, {})
+    if hook_name in config or f'{hook_name}_databases' in config:
+        hook_config = config.get(hook_name) or config.get(f'{hook_name}_databases') or {}
+    else:
+        hook_config = None
 
-    try:
-        module = HOOK_NAME_TO_MODULE[hook_name]
-    except KeyError:
+    module_name = hook_name.split('_databases')[0]
+
+    # Probe for a data source or monitoring hook module corresponding to the hook name.
+    for parent_module in (
+        borgmatic.hooks.credential,
+        borgmatic.hooks.data_source,
+        borgmatic.hooks.monitoring,
+    ):
+        if module_name not in get_submodule_names(parent_module):
+            continue
+
+        module = importlib.import_module(f'{parent_module.__name__}.{module_name}')
+
+        # If this module is explicitly flagged as not a hook, bail.
+        if not getattr(module, 'IS_A_HOOK', True):
+            return None
+
+        break
+    else:
         raise ValueError(f'Unknown hook name: {hook_name}')
 
-    logger.debug(f'{log_prefix}: Calling {hook_name} hook function {function_name}')
-    return getattr(module, function_name)(hook_config, config, log_prefix, *args, **kwargs)
+    logger.debug(f'Calling {hook_name} hook function {function_name}')
+
+    return getattr(module, function_name)(hook_config, config, *args, **kwargs)
 
 
-def call_hooks(function_name, config, log_prefix, hook_names, *args, **kwargs):
+def call_hooks(function_name, config, hook_type, *args, **kwargs):
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to each given hook name. Supply each call with the configuration
-    for that hook, the log prefix, and any given args and kwargs. Collect any return values into a
-    dict from hook name to return value.
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    each hook of the given hook type ("credential", "data_source", or "monitoring"). Supply each
+    call with the configuration for that hook, and any given args and kwargs.
+
+    Collect any return values into a dict from module name to return value. Note that the module
+    name is the name of the hook module itself, which might be different from the hook configuration
+    option (e.g. "postgresql" for the former vs. "postgresql_databases" for the latter).
 
     If the hook name is not present in the hooks configuration, then don't call the function for it
     and omit it from the return values.
@@ -75,23 +83,27 @@ def call_hooks(function_name, config, log_prefix, hook_names, *args, **kwargs):
     Raise anything else that a called function raises. An error stops calls to subsequent functions.
     '''
     return {
-        hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
-        for hook_name in hook_names
-        if config.get(hook_name)
+        hook_name: call_hook(function_name, config, hook_name, *args, **kwargs)
+        for hook_name in get_submodule_names(
+            importlib.import_module(f'borgmatic.hooks.{hook_type.value}')
+        )
+        if hook_name in config or f'{hook_name}_databases' in config
     }
 
 
-def call_hooks_even_if_unconfigured(function_name, config, log_prefix, hook_names, *args, **kwargs):
+def call_hooks_even_if_unconfigured(function_name, config, hook_type, *args, **kwargs):
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to each given hook name. Supply each call with the configuration
-    for that hook, the log prefix, and any given args and kwargs. Collect any return values into a
-    dict from hook name to return value.
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    each hook of the given hook type ("credential", "data_source", or "monitoring"). Supply each
+    call with the configuration for that hook and any given args and kwargs. Collect any return
+    values into a dict from hook name to return value.
 
     Raise AttributeError if the function name is not found in the module.
     Raise anything else that a called function raises. An error stops calls to subsequent functions.
     '''
     return {
-        hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
-        for hook_name in hook_names
+        hook_name: call_hook(function_name, config, hook_name, *args, **kwargs)
+        for hook_name in get_submodule_names(
+            importlib.import_module(f'borgmatic.hooks.{hook_type.value}')
+        )
     }

borgmatic/hooks/monitor.py

@@ -1,21 +0,0 @@
-from enum import Enum
-
-MONITOR_HOOK_NAMES = (
-    'apprise',
-    'cronhub',
-    'cronitor',
-    'healthchecks',
-    'loki',
-    'ntfy',
-    'pagerduty',
-    'pushover',
-    'uptime_kuma',
-    'zabbix',
-)
-
-
-class State(Enum):
-    START = 1
-    FINISH = 2
-    FAIL = 3
-    LOG = 4

borgmatic/hooks/monitoring/apprise.py

@@ -1,8 +1,8 @@
 import logging
 import operator
 
-import borgmatic.hooks.logs
-import borgmatic.hooks.monitor
+import borgmatic.hooks.monitoring.logs
+import borgmatic.hooks.monitoring.monitor
 
 logger = logging.getLogger(__name__)
 
@@ -22,12 +22,12 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
 
     logs_size_limit = max(
         hook_config.get('logs_size_limit', DEFAULT_LOGS_SIZE_LIMIT_BYTES)
-        - len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
+        - len(borgmatic.hooks.monitoring.logs.PAYLOAD_TRUNCATION_INDICATOR),
         0,
     )
 
-    borgmatic.hooks.logs.add_handler(
-        borgmatic.hooks.logs.Forgetful_buffering_handler(
+    borgmatic.hooks.monitoring.logs.add_handler(
+        borgmatic.hooks.monitoring.logs.Forgetful_buffering_handler(
             HANDLER_IDENTIFIER, logs_size_limit, monitoring_log_level
         )
     )
@@ -66,12 +66,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     )
 
     if not hook_config.get('services'):
-        logger.info(f'{config_filename}: No Apprise services to ping')
+        logger.info('No Apprise services to ping')
         return
 
     dry_run_string = ' (dry run; not actually pinging)' if dry_run else ''
     labels_string = ', '.join(map(operator.itemgetter('label'), hook_config.get('services')))
-    logger.info(f'{config_filename}: Pinging Apprise services: {labels_string}{dry_run_string}')
+    logger.info(f'Pinging Apprise services: {labels_string}{dry_run_string}')
 
     apprise_object = apprise.Apprise()
     apprise_object.add(list(map(operator.itemgetter('url'), hook_config.get('services'))))
@@ -82,11 +82,13 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     body = state_config.get('body')
 
     if state in (
-        borgmatic.hooks.monitor.State.FINISH,
-        borgmatic.hooks.monitor.State.FAIL,
-        borgmatic.hooks.monitor.State.LOG,
+        borgmatic.hooks.monitoring.monitor.State.FINISH,
+        borgmatic.hooks.monitoring.monitor.State.FAIL,
+        borgmatic.hooks.monitoring.monitor.State.LOG,
     ):
-        formatted_logs = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
+        formatted_logs = borgmatic.hooks.monitoring.logs.format_buffered_logs_for_payload(
+            HANDLER_IDENTIFIER
+        )
         if formatted_logs:
             body += f'\n\n{formatted_logs}'
 
@@ -98,12 +100,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     )
 
     if result is False:
-        logger.warning(f'{config_filename}: Error sending some Apprise notifications')
+        logger.warning('Error sending some Apprise notifications')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger. This prevents the handler from
     getting reused by other instances of this monitor.
     '''
-    borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)
+    borgmatic.hooks.monitoring.logs.remove_handler(HANDLER_IDENTIFIER)

borgmatic/hooks/monitoring/cronhub.py

@@ -2,7 +2,7 @@ import logging
 
 import requests
 
-from borgmatic.hooks import monitor
+from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
@@ -28,9 +28,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONHUB:
-        logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook'
-        )
+        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
@@ -41,8 +39,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         .replace('/ping/', formatted_state)
     )
 
-    logger.info(f'{config_filename}: Pinging Cronhub {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Cronhub ping URL {ping_url}')
+    logger.info(f'Pinging Cronhub {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Cronhub ping URL {ping_url}')
 
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -51,12 +49,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Cronhub error: {error}')
+            logger.warning(f'Cronhub error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/cronitor.py

@@ -2,7 +2,7 @@ import logging
 
 import requests
 
-from borgmatic.hooks import monitor
+from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
@@ -28,16 +28,14 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONITOR:
-        logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook'
-        )
+        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
     ping_url = f"{hook_config['ping_url']}/{MONITOR_STATE_TO_CRONITOR[state]}"
 
-    logger.info(f'{config_filename}: Pinging Cronitor {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Cronitor ping URL {ping_url}')
+    logger.info(f'Pinging Cronitor {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Cronitor ping URL {ping_url}')
 
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -46,12 +44,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Cronitor error: {error}')
+            logger.warning(f'Cronitor error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/healthchecks.py

@@ -3,8 +3,8 @@ import re
 
 import requests
 
-import borgmatic.hooks.logs
-from borgmatic.hooks import monitor
+import borgmatic.hooks.monitoring.logs
+from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
@@ -30,12 +30,12 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
 
     ping_body_limit = max(
         hook_config.get('ping_body_limit', DEFAULT_PING_BODY_LIMIT_BYTES)
-        - len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
+        - len(borgmatic.hooks.monitoring.logs.PAYLOAD_TRUNCATION_INDICATOR),
         0,
     )
 
-    borgmatic.hooks.logs.add_handler(
-        borgmatic.hooks.logs.Forgetful_buffering_handler(
+    borgmatic.hooks.monitoring.logs.add_handler(
+        borgmatic.hooks.monitoring.logs.Forgetful_buffering_handler(
             HANDLER_IDENTIFIER, ping_body_limit, monitoring_log_level
         )
     )
@@ -55,9 +55,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
 
     if 'states' in hook_config and state.name.lower() not in hook_config['states']:
-        logger.info(
-            f'{config_filename}: Skipping Healthchecks {state.name.lower()} ping due to configured states'
-        )
+        logger.info(f'Skipping Healthchecks {state.name.lower()} ping due to configured states')
         return
 
     ping_url_is_uuid = re.search(r'\w{8}-\w{4}-\w{4}-\w{4}-\w{12}$', ping_url)
@@ -68,17 +66,17 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     if hook_config.get('create_slug'):
         if ping_url_is_uuid:
-            logger.warning(
-                f'{config_filename}: Healthchecks UUIDs do not support auto provisionning; ignoring'
-            )
+            logger.warning('Healthchecks UUIDs do not support auto provisionning; ignoring')
         else:
             ping_url = f'{ping_url}?create=1'
 
-    logger.info(f'{config_filename}: Pinging Healthchecks {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Healthchecks ping URL {ping_url}')
+    logger.info(f'Pinging Healthchecks {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Healthchecks ping URL {ping_url}')
 
     if state in (monitor.State.FINISH, monitor.State.FAIL, monitor.State.LOG):
-        payload = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
+        payload = borgmatic.hooks.monitoring.logs.format_buffered_logs_for_payload(
+            HANDLER_IDENTIFIER
+        )
     else:
         payload = ''
 
@@ -91,12 +89,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Healthchecks error: {error}')
+            logger.warning(f'Healthchecks error: {error}')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger. This prevents the handler from
     getting reused by other instances of this monitor.
     '''
-    borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)
+    borgmatic.hooks.monitoring.logs.remove_handler(HANDLER_IDENTIFIER)

borgmatic/hooks/monitoring/logs.py

@@ -1,5 +1,6 @@
 import logging
 
+IS_A_HOOK = False
 PAYLOAD_TRUNCATION_INDICATOR = '...\n'
 
 

borgmatic/hooks/monitoring/loki.py

@@ -6,7 +6,7 @@ import time
 
 import requests
 
-from borgmatic.hooks import monitor
+from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
@@ -139,10 +139,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     for handler in tuple(logging.getLogger().handlers):
         if isinstance(handler, Loki_log_handler):
             if state in MONITOR_STATE_TO_LOKI.keys():
-                handler.raw(f'{config_filename}: {MONITOR_STATE_TO_LOKI[state]} backup')
+                handler.raw(f'{MONITOR_STATE_TO_LOKI[state]} backup')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger.
     '''

borgmatic/hooks/monitoring/monitor.py

@@ -0,0 +1,10 @@
+import enum
+
+IS_A_HOOK = False
+
+
+class State(enum.Enum):
+    START = 1
+    FINISH = 2
+    FAIL = 3
+    LOG = 4

borgmatic/hooks/monitoring/ntfy.py

@@ -19,7 +19,6 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     Ping the configured Ntfy topic. Use the given configuration filename in any log entries.
     If this is a dry run, then don't actually ping anything.
     '''
-
     run_states = hook_config.get('states', ['fail'])
 
     if state.name.lower() in run_states:
@@ -38,8 +37,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         base_url = hook_config.get('server', 'https://ntfy.sh')
         topic = hook_config.get('topic')
 
-        logger.info(f'{config_filename}: Pinging ntfy topic {topic}{dry_run_label}')
-        logger.debug(f'{config_filename}: Using Ntfy ping URL {base_url}/{topic}')
+        logger.info(f'Pinging ntfy topic {topic}{dry_run_label}')
+        logger.debug(f'Using Ntfy ping URL {base_url}/{topic}')
 
         headers = {
             'X-Title': state_config.get('title'),
@@ -56,20 +55,16 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if access_token is not None:
             if username or password:
                 logger.warning(
-                    f'{config_filename}: ntfy access_token is set but so is username/password, only using access_token'
+                    'ntfy access_token is set but so is username/password, only using access_token'
                 )
             auth = requests.auth.HTTPBasicAuth('', access_token)
         elif (username and password) is not None:
             auth = requests.auth.HTTPBasicAuth(username, password)
-            logger.info(f'{config_filename}: Using basic auth with user {username} for ntfy')
+            logger.info(f'Using basic auth with user {username} for ntfy')
         elif username is not None:
-            logger.warning(
-                f'{config_filename}: Password missing for ntfy authentication, defaulting to no auth'
-            )
+            logger.warning('Password missing for ntfy authentication, defaulting to no auth')
         elif password is not None:
-            logger.warning(
-                f'{config_filename}: Username missing for ntfy authentication, defaulting to no auth'
-            )
+            logger.warning('Username missing for ntfy authentication, defaulting to no auth')
 
         if not dry_run:
             logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -78,12 +73,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                 if not response.ok:
                     response.raise_for_status()
             except requests.exceptions.RequestException as error:
-                logger.warning(f'{config_filename}: ntfy error: {error}')
+                logger.warning(f'ntfy error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/pagerduty.py

@@ -5,7 +5,7 @@ import platform
 
 import requests
 
-from borgmatic.hooks import monitor
+from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
 
@@ -29,12 +29,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     '''
     if state != monitor.State.FAIL:
         logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
+            f'Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
         )
         return
 
     dry_run_label = ' (dry run; not actually sending)' if dry_run else ''
-    logger.info(f'{config_filename}: Sending failure event to PagerDuty {dry_run_label}')
+    logger.info(f'Sending failure event to PagerDuty {dry_run_label}')
 
     if dry_run:
         return
@@ -61,7 +61,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             },
         }
     )
-    logger.debug(f'{config_filename}: Using PagerDuty payload: {payload}')
+    logger.debug(f'Using PagerDuty payload: {payload}')
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
@@ -69,12 +69,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:
-        logger.warning(f'{config_filename}: PagerDuty error: {error}')
+        logger.warning(f'PagerDuty error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/pushover.py

@@ -35,14 +35,14 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     token = hook_config.get('token')
     user = hook_config.get('user')
 
-    logger.info(f'{config_filename}: Updating Pushover{dry_run_label}')
+    logger.info(f'Updating Pushover{dry_run_label}')
 
     if state_config.get('priority') == EMERGENCY_PRIORITY:
         if 'expire' not in state_config:
-            logger.info(f'{config_filename}: Setting expire to default (10 min).')
+            logger.info('Setting expire to default (10 min)')
             state_config['expire'] = 600
         if 'retry' not in state_config:
-            logger.info(f'{config_filename}: Setting retry to default (30 sec).')
+            logger.info('Setting retry to default (30 sec)')
             state_config['retry'] = 30
     else:
         if 'expire' in state_config or 'retry' in state_config:
@@ -75,12 +75,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Pushover error: {error}')
+            logger.warning(f'Pushover error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/sentry.py

@@ -0,0 +1,75 @@
+import logging
+import re
+
+import requests
+
+logger = logging.getLogger(__name__)
+
+
+def initialize_monitor(
+    ping_url, config, config_filename, monitoring_log_level, dry_run
+):  # pragma: no cover
+    '''
+    No initialization is necessary for this monitor.
+    '''
+    pass
+
+
+DATA_SOURCE_NAME_URL_PATTERN = re.compile(
+    '^(?P<protocol>.+)://(?P<username>.+)@(?P<hostname>.+)/(?P<project_id>.+)$'
+)
+
+
+def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
+    '''
+    Construct and ping a Sentry cron URL, based on the configured DSN URL and monitor slug. Use the
+    given configuration filename in any log entries. If this is a dry run, then don't actually ping
+    anything.
+    '''
+    run_states = hook_config.get('states', ['start', 'finish', 'fail'])
+
+    if not state.name.lower() in run_states:
+        return
+
+    dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
+
+    data_source_name_url = hook_config.get('data_source_name_url')
+    monitor_slug = hook_config.get('monitor_slug')
+    match = DATA_SOURCE_NAME_URL_PATTERN.match(data_source_name_url)
+
+    if not match:
+        logger.warning(f'Invalid Sentry data source name URL: {data_source_name_url}')
+        return
+
+    cron_url = f'{match.group("protocol")}://{match.group("hostname")}/api/{match.group("project_id")}/cron/{monitor_slug}/{match.group("username")}/'
+
+    logger.info(f'Pinging Sentry {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Sentry cron URL {cron_url}')
+
+    status = {
+        'start': 'in_progress',
+        'finish': 'ok',
+        'fail': 'error',
+    }.get(state.name.lower())
+
+    if not status:
+        logger.warning('Invalid Sentry state')
+        return
+
+    if dry_run:
+        return
+
+    logging.getLogger('urllib3').setLevel(logging.ERROR)
+    try:
+        response = requests.post(f'{cron_url}?status={status}')
+        if not response.ok:
+            response.raise_for_status()
+    except requests.exceptions.RequestException as error:
+        logger.warning(f'Sentry error: {error}')
+
+
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
+    '''
+    No destruction is necessary for this monitor.
+    '''
+    pass

borgmatic/hooks/monitoring/uptime_kuma.py

@@ -28,10 +28,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     status = 'down' if state.name.lower() == 'fail' else 'up'
     push_url = hook_config.get('push_url', 'https://example.uptime.kuma/api/push/abcd1234')
     query = f'status={status}&msg={state.name.lower()}'
-    logger.info(
-        f'{config_filename}: Pushing Uptime Kuma push_url {push_url}?{query} {dry_run_label}'
-    )
-    logger.debug(f'{config_filename}: Full Uptime Kuma state URL {push_url}?{query}')
+    logger.info(f'Pushing Uptime Kuma push_url {push_url}?{query} {dry_run_label}')
+    logger.debug(f'Full Uptime Kuma state URL {push_url}?{query}')
 
     if dry_run:
         return
@@ -43,12 +41,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:
-        logger.warning(f'{config_filename}: Uptime Kuma error: {error}')
+        logger.warning(f'Uptime Kuma error: {error}')
 
 
-def destroy_monitor(
-    push_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(push_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/zabbix.py

@@ -44,16 +44,16 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     value = state_config.get('value')
     headers = {'Content-Type': 'application/json-rpc'}
 
-    logger.info(f'{config_filename}: Updating Zabbix{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Zabbix URL: {server}')
+    logger.info(f'Updating Zabbix{dry_run_label}')
+    logger.debug(f'Using Zabbix URL: {server}')
 
     if server is None:
-        logger.warning(f'{config_filename}: Server missing for Zabbix')
+        logger.warning('Server missing for Zabbix')
         return
 
     # Determine the Zabbix method used to store the value: itemid or host/key
     if itemid is not None:
-        logger.info(f'{config_filename}: Updating {itemid} on Zabbix')
+        logger.info(f'Updating {itemid} on Zabbix')
         data = {
             'jsonrpc': '2.0',
             'method': 'history.push',
@@ -62,7 +62,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         }
 
     elif (host and key) is not None:
-        logger.info(f'{config_filename}: Updating Host:{host} and Key:{key} on Zabbix')
+        logger.info(f'Updating Host:{host} and Key:{key} on Zabbix')
         data = {
             'jsonrpc': '2.0',
             'method': 'history.push',
@@ -71,23 +71,23 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         }
 
     elif host is not None:
-        logger.warning(f'{config_filename}: Key missing for Zabbix')
+        logger.warning('Key missing for Zabbix')
         return
 
     elif key is not None:
-        logger.warning(f'{config_filename}: Host missing for Zabbix.')
+        logger.warning('Host missing for Zabbix')
         return
     else:
-        logger.warning(f'{config_filename}: No zabbix itemid or host/key provided.')
+        logger.warning('No Zabbix itemid or host/key provided')
         return
 
     # Determine the authentication method: API key or username/password
     if api_key is not None:
-        logger.info(f'{config_filename}: Using API key auth for Zabbix')
+        logger.info('Using API key auth for Zabbix')
         headers['Authorization'] = 'Bearer ' + api_key
 
     elif (username and password) is not None:
-        logger.info(f'{config_filename}: Using user/pass auth with user {username} for Zabbix')
+        logger.info('Using user/pass auth with user {username} for Zabbix')
         auth_data = {
             'jsonrpc': '2.0',
             'method': 'user.login',
@@ -102,18 +102,18 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                 if not response.ok:
                     response.raise_for_status()
             except requests.exceptions.RequestException as error:
-                logger.warning(f'{config_filename}: Zabbix error: {error}')
+                logger.warning(f'Zabbix error: {error}')
                 return
 
     elif username is not None:
-        logger.warning(f'{config_filename}: Password missing for Zabbix authentication')
+        logger.warning('Password missing for Zabbix authentication')
         return
 
     elif password is not None:
-        logger.warning(f'{config_filename}: Username missing for Zabbix authentication')
+        logger.warning('Username missing for Zabbix authentication')
         return
     else:
-        logger.warning(f'{config_filename}: Authentication data missing for Zabbix')
+        logger.warning('Authentication data missing for Zabbix')
         return
 
     if not dry_run:
@@ -123,12 +123,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Zabbix error: {error}')
+            logger.warning(f'Zabbix error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/logger.py

@@ -1,10 +1,9 @@
+import enum
 import logging
 import logging.handlers
 import os
 import sys
 
-import colorama
-
 
 def to_bool(arg):
     '''
@@ -33,7 +32,7 @@ def interactive_console():
 def should_do_markup(no_color, configs):
     '''
     Given the value of the command-line no-color argument, and a dict of configuration filename to
-    corresponding parsed configuration, determine if we should enable colorama marking up.
+    corresponding parsed configuration, determine if we should enable color marking up.
     '''
     if no_color:
         return False
@@ -89,34 +88,61 @@ class Multi_stream_handler(logging.Handler):
 
 
 class Console_no_color_formatter(logging.Formatter):
-    def format(self, record):  # pragma: no cover
-        return record.msg
+    def __init__(self, *args, **kwargs):  # pragma: no cover
+        super(Console_no_color_formatter, self).__init__(
+            '{prefix}{message}', style='{', defaults={'prefix': ''}, *args, **kwargs
+        )
+
+
+class Color(enum.Enum):
+    RESET = 0
+    RED = 31
+    GREEN = 32
+    YELLOW = 33
+    MAGENTA = 35
+    CYAN = 36
 
 
 class Console_color_formatter(logging.Formatter):
+    def __init__(self, *args, **kwargs):
+        super(Console_color_formatter, self).__init__(
+            '{prefix}{message}', style='{', defaults={'prefix': ''}, *args, **kwargs
+        )
+
     def format(self, record):
         add_custom_log_levels()
 
-        color = {
-            logging.CRITICAL: colorama.Fore.RED,
-            logging.ERROR: colorama.Fore.RED,
-            logging.WARN: colorama.Fore.YELLOW,
-            logging.ANSWER: colorama.Fore.MAGENTA,
-            logging.INFO: colorama.Fore.GREEN,
-            logging.DEBUG: colorama.Fore.CYAN,
-        }.get(record.levelno)
+        color = (
+            {
+                logging.CRITICAL: Color.RED,
+                logging.ERROR: Color.RED,
+                logging.WARN: Color.YELLOW,
+                logging.ANSWER: Color.MAGENTA,
+                logging.INFO: Color.GREEN,
+                logging.DEBUG: Color.CYAN,
+            }
+            .get(record.levelno)
+            .value
+        )
 
-        return color_text(color, record.msg)
+        return color_text(color, super(Console_color_formatter, self).format(record))
+
+
+def ansi_escape_code(color):  # pragma: no cover
+    '''
+    Given a color value, produce the corresponding ANSI escape code.
+    '''
+    return f'\x1b[{color}m'
 
 
 def color_text(color, message):
     '''
-    Give colored text.
+    Given a color value and a message, return the message colored with ANSI escape codes.
     '''
     if not color:
         return message
 
-    return f'{color}{message}{colorama.Style.RESET_ALL}'
+    return f'{ansi_escape_code(color)}{message}{ansi_escape_code(Color.RESET.value)}'
 
 
 def add_logging_level(level_name, level_number):
@@ -160,6 +186,72 @@ def add_custom_log_levels():  # pragma: no cover
     add_logging_level('DISABLED', DISABLED)
 
 
+def get_log_prefix():
+    '''
+    Return the current log prefix from the defaults for the formatter on the first logging handler,
+    set by set_log_prefix(). Return None if no such prefix exists.
+    '''
+    try:
+        return next(
+            handler.formatter._style._defaults.get('prefix').rstrip().rstrip(':')
+            for handler in logging.getLogger().handlers
+        )
+    except (StopIteration, AttributeError):
+        return None
+
+
+def set_log_prefix(prefix):
+    '''
+    Given a log prefix as a string, set it into the defaults for the formatters on all logging
+    handlers. Note that this overwrites any existing defaults.
+    '''
+    for handler in logging.getLogger().handlers:
+        try:
+            handler.formatter._style._defaults = {'prefix': f'{prefix}: ' if prefix else ''}
+        except AttributeError:
+            pass
+
+
+class Log_prefix:
+    '''
+    A Python context manager for setting a log prefix so that it shows up in every subsequent
+    logging message for the duration of the context manager. For this to work, it relies on each
+    logging formatter to be initialized with "{prefix}" somewhere in its logging format.
+
+    Example use as a context manager:
+
+
+       with borgmatic.logger.Log_prefix('myprefix'):
+            do_something_that_logs()
+
+    For the scope of that "with" statement, any logs created are prefixed with "myprefix: ".
+    Afterwards, the prefix gets restored to whatever it was prior to the context manager.
+    '''
+
+    def __init__(self, prefix):
+        '''
+        Given the desired log prefix, save it for use below. Set prefix to None to disable any
+        prefix from getting logged.
+        '''
+        self.prefix = prefix
+        self.original_prefix = None
+
+    def __enter__(self):
+        '''
+        Set the prefix onto the formatter defaults for every logging handler so that the prefix ends
+        up in every log message. But first, save off any original prefix so that it can be restored
+        below.
+        '''
+        self.original_prefix = get_log_prefix()
+        set_log_prefix(self.prefix)
+
+    def __exit__(self, exception, value, traceback):
+        '''
+        Restore any original prefix.
+        '''
+        set_log_prefix(self.original_prefix)
+
+
 def configure_logging(
     console_log_level,
     syslog_log_level=None,
@@ -223,7 +315,11 @@ def configure_logging(
         if syslog_path:
             syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
             syslog_handler.setFormatter(
-                logging.Formatter('borgmatic: {levelname} {message}', style='{')  # noqa: FS003
+                logging.Formatter(
+                    'borgmatic: {levelname} {prefix}{message}',  # noqa: FS003
+                    style='{',
+                    defaults={'prefix': ''},
+                )
             )
             syslog_handler.setLevel(syslog_log_level)
             handlers.append(syslog_handler)
@@ -232,7 +328,9 @@ def configure_logging(
         file_handler = logging.handlers.WatchedFileHandler(log_file)
         file_handler.setFormatter(
             logging.Formatter(
-                log_file_format or '[{asctime}] {levelname}: {message}', style='{'  # noqa: FS003
+                log_file_format or '[{asctime}] {levelname}: {prefix}{message}',  # noqa: FS003
+                style='{',
+                defaults={'prefix': ''},
             )
         )
         file_handler.setLevel(log_file_log_level)

docs/Dockerfile

@@ -2,7 +2,7 @@ FROM docker.io/alpine:3.20.1 AS borgmatic
 
 COPY . /app
 RUN apk add --no-cache py3-pip py3-ruamel.yaml py3-ruamel.yaml.clib
-RUN pip install --break-system-packages --no-cache /app && generate-borgmatic-config && chmod +r /etc/borgmatic/config.yaml
+RUN pip install --break-system-packages --no-cache /app && borgmatic config generate && chmod +r /etc/borgmatic/config.yaml
 RUN borgmatic --help > /command-line.txt \
     && for action in repo-create transfer create prune compact check delete extract config "config bootstrap" "config generate" "config validate" export-tar mount umount repo-delete restore repo-list list repo-info info break-lock "key export" "key change-passphrase" borg; do \
            echo -e "\n--------------------------------------------------------------------------------\n" >> /command-line.txt \

docs/fetch-contributors

@@ -25,27 +25,43 @@ def list_merged_pulls(url):
     return tuple(pull for pull in response.json() if pull.get('merged_at'))
 
 
-API_ENDPOINT_URLS = (
+def list_contributing_issues(url):
+    # labels = bug, design finalized, etc.
+    response = requests.get(f'{url}?labels=19,20,22,23,32,52,53,54', headers={'Accept': 'application/json', 'Content-Type': 'application/json'})
+
+    if not response.ok:
+        response.raise_for_status()
+
+    return tuple(response.json())
+
+
+PULLS_API_ENDPOINT_URLS = (
     'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/pulls',
     'https://api.github.com/repos/borgmatic-collective/borgmatic/pulls',
 )
+ISSUES_API_ENDPOINT_URL = 'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/issues'
 RECENT_CONTRIBUTORS_CUTOFF_DAYS = 365
 
 
+def get_item_timestamp(item):
+    return item.get('merged_at') or item.get('created_at')
+
+
 def print_contributors():
     '''
     Display the recent contributors as a row of avatars in an HTML fragment.
     '''
-    pulls = tuple(itertools.chain.from_iterable(list_merged_pulls(url) for url in API_ENDPOINT_URLS))
+    pulls = tuple(itertools.chain.from_iterable(list_merged_pulls(url) for url in PULLS_API_ENDPOINT_URLS))
+    issues = list_contributing_issues(ISSUES_API_ENDPOINT_URL)
     seen_user_ids = set()
 
     print('<p>')
 
-    for pull in sorted(pulls, key=operator.itemgetter('merged_at'), reverse=True):
-        merged_at = pull.get('merged_at')
-        user = pull.get('user')
+    for item in sorted(pulls + issues, key=get_item_timestamp, reverse=True):
+        timestamp = get_item_timestamp(item)
+        user = item.get('user')
 
-        if not merged_at or not user:
+        if not timestamp or not user:
             continue
 
         user_id = user.get('id')
@@ -53,7 +69,7 @@ def print_contributors():
         if not user_id or user_id in seen_user_ids:
             continue
 
-        if datetime.datetime.fromisoformat(merged_at) < datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=RECENT_CONTRIBUTORS_CUTOFF_DAYS):
+        if datetime.datetime.fromisoformat(timestamp) < datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=RECENT_CONTRIBUTORS_CUTOFF_DAYS):
             continue
 
         seen_user_ids.add(user_id)

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -3,7 +3,7 @@ title: How to add preparation and cleanup steps to backups
 eleventyNavigation:
   key: 🧹 Add preparation and cleanup steps
   parent: How-to guides
-  order: 9
+  order: 10
 ---
 ## Preparation and cleanup hooks
 

docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server.md

@@ -3,7 +3,7 @@ title: How to backup to a removable drive or an intermittent server
 eleventyNavigation:
   key: 💾 Backup to a removable drive/server
   parent: How-to guides
-  order: 10
+  order: 11
 ---
 ## Occasional backups
 

docs/how-to/backup-your-databases.md

@@ -58,7 +58,7 @@ mariadb_databases:
 
 As part of each backup, borgmatic streams a database dump for each configured
 database directly to Borg, so it's included in the backup without consuming
-additional disk space. (The exceptions are the PostgreSQL/MongoDB "directory"
+additional disk space. (The exceptions are the PostgreSQL/MongoDB `directory`
 dump formats, which can't stream and therefore do consume temporary disk
 space. Additionally, prior to borgmatic 1.5.3, all database dumps consumed
 temporary disk space.)
@@ -135,6 +135,9 @@ temporary file storage, probing the following locations (in order) to find it:
     Hard-coded `/tmp`. <span class="minilink minilink-addedin">Prior to
     version 1.9.2</span>This was instead hard-coded to `/run/user/$UID`.
 
+You can see the runtime directory path that borgmatic selects by running with
+`--verbosity 2` and looking for "Using runtime directory" in the output.
+
 Regardless of the runtime directory selected, borgmatic stores its files
 within a `borgmatic` subdirectory of the runtime directory. Additionally, in
 the case of `TMPDIR`, `TEMP`, and the hard-coded `/tmp`, borgmatic creates a
@@ -258,19 +261,23 @@ hooks:
 
 ... where `my_pg_container` is the name of your database container. In this
 example, you'd also need to set the `pg_restore_command` and `psql_command`
-options. If you choose to use the `pg_dump` command within the container
-though, note that it will output the database dump to a file inside the
-container. So you'll have to mount the `.borgmatic` folder from your host's
-home folder into the container using the same directory structure.
+options.
 
-See the following Docker compose file an as example:
+If you choose to use the `pg_dump` command within the container, and you're
+using the `directory` format in particular, you'll also need to mount the
+[runtime directory](#runtime-directory) from your host into the container at the
+same path. Otherwise, the `directory` format dump will remain locked away inside
+the database container where Borg can't read it.
+
+For example, with Docker Compose and a runtime directory located at
+`/run/user/1000`:
 
 ```yaml
 services:
   db:
     image: postgres
     volumes:
-      - /home/USERNAME/.borgmatic:/home/USERNAME/.borgmatic
+      - /run/user/1000:/run/user/1000
 ```
 
 Similar command override options are available for (some of) the other
@@ -391,7 +398,7 @@ with the repository's path or its label as configured in your borgmatic
 configuration file.
 
 ```bash
-borgmatic restore --repository repo.borg --archive host-2023-...
+borgmatic restore --repository repo.borg --archive latest
 ```
 
 ### Restore particular databases
@@ -401,7 +408,7 @@ restore one of them, use the `--database` flag to select one or more
 databases. For instance:
 
 ```bash
-borgmatic restore --archive host-2023-... --database users --database orders
+borgmatic restore --archive latest --database users --database orders
 ```
 
 <span class="minilink minilink-addedin">New in version 1.7.6</span> You can
@@ -409,29 +416,99 @@ also restore individual databases even if you dumped them as "all"—as long as
 you dumped them into separate files via use of the "format" option. See above
 for more information.
 
+### Restore databases sharing a name
+
+<span class="minilink minilink-addedin">New in version 1.9.5</span> If you've
+backed up multiple databases that happen to share the same name but different
+hostnames, ports, or hooks, you can include additional flags to disambiguate
+which database you'd like to restore. For instance, let's say you've backed up
+the following configured databases:
+
+```yaml
+postgresql_databases:
+    - name: users
+      hostname: host1.example.org
+    - name: users
+      hostname: host2.example.org
+```
+
+... then you can run the following command to restore only one of them:
+
+```bash
+borgmatic restore --archive latest --database users --original-hostname host1.example.org
+```
+
+This selects a `users` database to restore, but only if it originally came
+from the host `host1.example.org`. This command won't restore `users`
+databases from any other hosts.
+
+Here's another example configuration:
+
+```yaml
+postgresql_databases:
+    - name: users
+      hostname: example.org
+      port: 5433
+    - name: users
+      hostname: example.org
+      port: 5434
+```
+
+And a command to restore just one of the databases:
+
+```bash
+borgmatic restore --archive latest --database users --original-port 5433
+```
+
+That restores a `users` database only if it originally came from port `5433`
+*and* if that port is in borgmatic's configuration, e.g. `port: 5433`.
+
+Finally, check out this configuration:
+
+```yaml
+postgresql_databases:
+    - name: users
+      hostname: example.org
+mariadb_databases:
+    - name: users
+      hostname: example.org
+```
+
+And to select just one of the databases to restore:
+
+```bash
+borgmatic restore --archive latest --database users --hook postgresql
+```
+
+That restores a `users` database only if it was dumped using the
+`postgresql_databases:` data source hook. This command won't restore `users`
+databases that were dumped using other hooks.
+
+Note that these flags don't change the hostname or port to which the database
+is actually restored. For that, see below about restoring to an alternate
+host.
+
 
 ### Restore all databases
 
 To restore all databases:
 
 ```bash
-borgmatic restore --archive host-2023-... --database all
+borgmatic restore --archive latest --database all
 ```
 
 Or omit the `--database` flag entirely:
 
 
 ```bash
-borgmatic restore --archive host-2023-...
+borgmatic restore --archive latest
 ```
 
-Prior to borgmatic version 1.7.6, this restores a combined "all" database
-dump from the archive.
-
 <span class="minilink minilink-addedin">New in version 1.7.6</span> Restoring
 "all" databases restores each database found in the selected archive. That
 includes any combined dump file named "all" and any other individual database
-dumps found in the archive.
+dumps found in the archive. Prior to borgmatic version 1.7.6, restoring "all"
+only restored a combined "all" database dump from the archive.
 
 
 ### Restore particular schemas
@@ -466,21 +543,35 @@ postgresql_databases:
         restore_password: trustsome1
 ```
 
+### Manual restoration
 
-### Limitations
+If you prefer to restore a database without the help of borgmatic, first
+[extract](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) an
+archive containing a database dump.
+
+borgmatic extracts the dump file into the `borgmatic/` directory within the
+extraction destination path. For example, if you're extracting to `/tmp`, then
+the dump will be in `/tmp/borgmatic/`.
+
+<span class="minilink minilink-addedin">Prior to version 1.9.0</span> borgmatic
+extracted the dump file into the *`username`*`/.borgmatic/` directory within the
+extraction destination path, where *`username`* is the user that created the
+backup. For example, if you created the backup with the `root` user and you're
+extracting to `/tmp`, then the dump will be in `/tmp/root/.borgmatic`.
+
+After extraction, you can manually restore the dump file using native database
+commands like `pg_restore`, `mysql`, `mongorestore`, `sqlite`, or similar.
+
+Also see the documentation on [listing database
+dumps](https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#listing-database-dumps).
+
+
+## Limitations
 
 There are a few important limitations with borgmatic's current database
-restoration feature that you should know about:
+hooks that you should know about:
 
-1. You must restore as the same Unix user that created the archive containing
-the database dump. That's because the user's home directory path is encoded
-into the path of the database dump within the archive.
-2. As mentioned above, borgmatic can only restore a database that's defined in
-borgmatic's own configuration file. So include your configuration file in
-backups to avoid getting caught without a way to restore a database.
-3. borgmatic does not currently support backing up or restoring multiple
-databases that share the exact same name on different hosts.
-4. When database hooks are enabled, borgmatic instructs Borg to consume
+1. When database hooks are enabled, borgmatic instructs Borg to consume
 special files (via `--read-special`) to support database dump
 streaming—regardless of the value of your `read_special` configuration option.
 And because this can cause Borg to hang, borgmatic also automatically excludes
@@ -495,7 +586,10 @@ exclude them. <span class="minilink minilink-addedin">Prior to version
 1.7.3</span>Special files were not auto-excluded, and you were responsible for
 excluding them yourself. Common directories to exclude are `/dev` and `/run`,
 but that may not be exhaustive.
-5. <span class="minilink minilink-addedin">Prior to version 1.9.0</span>
+2. <span class="minilink minilink-addedin">Prior to version 1.9.5</span>
+borgmatic did not support backing up or restoring multiple databases that
+shared the exact same name on different hosts or with different ports.
+3. <span class="minilink minilink-addedin">Prior to version 1.9.0</span>
 Database hooks also implicitly enabled the `one_file_system` option, which
 meant Borg wouldn't cross filesystem boundaries when looking for files to
 backup. When borgmatic was running in a container, this often required a
@@ -503,25 +597,16 @@ work-around to explicitly add each mounted backup volume to
 `source_directories` instead of relying on Borg to include them implicitly via
 a parent directory. But as of borgmatic 1.9.0, `one_file_system` is no longer
 auto-enabled and such work-arounds aren't necessary.
-
-
-### Manual restoration
-
-If you prefer to restore a database without the help of borgmatic, first
-[extract](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) an
-archive containing a database dump.
-
-borgmatic extracts the dump file into the *`username`*`/.borgmatic/` directory
-within the extraction destination path, where *`username`* is the user that
-created the backup. For example, if you created the backup with the `root`
-user and you're extracting to `/tmp`, then the dump will be in
-`/tmp/root/.borgmatic`.
-
-After extraction, you can manually restore the dump file using native database
-commands like `pg_restore`, `mysql`, `mongorestore`, `sqlite`, or similar.
-
-Also see the documentation on [listing database
-dumps](https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#listing-database-dumps).
+4. <span class="minilink minilink-addedin">Prior to version 1.9.0</span> You
+must restore as the same Unix user that created the archive containing the
+database dump. That's because the user's home directory path is encoded into
+the path of the database dump within the archive.
+5. <span class="minilink minilink-addedin">Prior to version 1.7.15</span> As
+mentioned above, borgmatic can only restore a database that's defined in
+borgmatic's own configuration file. So include your configuration files in
+backups to avoid getting caught without a way to restore a database. But
+starting from version 1.7.15, borgmatic includes your configuration files
+automatically.
 
 
 ## Preparation and cleanup hooks

docs/how-to/customize-warnings-and-errors.md

@@ -3,7 +3,7 @@ title: How to customize warnings and errors
 eleventyNavigation:
   key: 💥 Customize warnings/errors
   parent: How-to guides
-  order: 12
+  order: 13
 ---
 ## When things go wrong
 

docs/how-to/deal-with-very-large-backups.md

@@ -62,8 +62,8 @@ actions](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#skipping-acti
 Another option is to customize your consistency checks. By default, if you
 omit consistency checks from configuration, borgmatic runs full-repository
 checks (`repository`) and per-archive checks (`archives`) within each
-repository. (Although see below about check frequency.) This is equivalent to
-what `borg check` does if run without options.
+repository, running the checks on a monthly basis. (See below about setting
+your own check frequency.)
 
 But if you find that archive checks are too slow, for example, you can
 configure borgmatic to run repository checks only. Configure this in the
@@ -87,13 +87,16 @@ checks:
 ```
 
 
-Here are the available checks from fastest to slowest:
+Here are the available checks, roughly from fastest to slowest:
 
- * `repository`: Checks the consistency of the repository itself.
- * `archives`: Checks all of the archives in the repository.
+ * `archives`: Checks all of the archives' metadata in the repository.
+ * `repository`: Checks the consistency of the whole repository. The checks run
+   on the server and do not cause significant network traffic.
  * `extract`: Performs an extraction dry-run of the latest archive.
- * `data`: Verifies the data integrity of all archives contents, decrypting and decompressing all data.
- * `spot`: Compares file counts and contents between your source files and the latest archive.
+ * `data`: Verifies the data integrity of all archives contents, decrypting
+   and decompressing all data.
+ * `spot`: Compares file counts and contents between your source files and the
+   latest archive.
 
 Note that the `data` check is a more thorough version of the `archives` check,
 so enabling the `data` check implicitly enables the `archives` check as well.
@@ -113,11 +116,11 @@ file). Because if something goes wrong with your source files, most
 consistency checks will still pass with flying colors and you won't discover
 there's a problem until you go to restore.
 
-<span class="minilink minilink-addedin">New in version 1.8.10</span> <span
-class="minilink minilink-addedin">Beta feature</span> That's where the spot
-check comes in. This check actually compares your source file counts and data
-against those in the latest archive, potentially catching problems like
-incorrect excludes, inadvertent deletes, files changed by malware, etc.
+<span class="minilink minilink-addedin">New in version 1.8.10</span> That's
+where the spot check comes in. This check actually compares your source file
+counts and data against those in the latest archive, potentially catching
+problems like incorrect excludes, inadvertent deletes, files changed by
+malware, etc.
 
 But because an exhaustive comparison of all source files against the latest
 archive might be too slow, the spot check supports *sampling* a percentage of
@@ -180,11 +183,6 @@ to run it immediately after a `create` action (borgmatic's default behavior).
 Instead, it may make more sense to run the spot check on a separate schedule
 from `create`.
 
-As long as the spot check feature is in beta, it may be subject to breaking
-changes. But feel free to use it in production if you're okay with that
-caveat, and please [provide any
-feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
-
 
 ### Check frequency
 

docs/how-to/develop-on-borgmatic.md

@@ -3,7 +3,7 @@ title: How to develop on borgmatic
 eleventyNavigation:
   key: 🏗️ Develop on borgmatic
   parent: How-to guides
-  order: 14
+  order: 15
 ---
 ## Source code
 

docs/how-to/extract-a-backup.md

@@ -202,13 +202,25 @@ borgmatic config bootstrap --repository repo.borg --archive host-2023-01-02T04:0
 See the output of `config bootstrap --help` for additional flags you may need
 for bootstrapping.
 
-<span class="minilink minilink-addedin">New in version 1.8.1</span> Set the
-`store_config_files` option to `false` to disable the automatic backup of
-borgmatic configuration files, for instance if they contain sensitive
-information you don't want to store even inside your encrypted backups. If you
-do this though, the `config bootstrap` action will no longer work.
+<span class="minilink minilink-addedin">New in version 1.9.3</span>
+If your borgmatic configuration files contain sensitive information you don't
+want to store even inside your encrypted backups, you can disable the
+automatic backup of the configuration files. To do this, set the
+`store_config_files` option under the `bootstrap` hook to `false`. For
+instance:
 
-<span class="minilink minilink-addedin">New in version 1.8.7</span> Included
-configuration files are stored in each backup archive. This means that the
-`config bootstrap` action not only extracts the top-level configuration files
-but also the includes they depend upon.
+```yaml
+bootstrap:
+    store_config_files: false
+```
+
+If you do this though, the `config bootstrap` action will no longer work.
+
+<span class="minilink minilink-addedin">In version 1.8.1 through 1.9.2</span>
+The `store_config_files` option was at the global scope instead of under the
+`bootstrap` hook.
+
+<span class="minilink minilink-addedin">New in version 1.8.7</span>
+Configuration file includes are stored in each backup archive. This means that
+the `config bootstrap` action not only extracts the top-level configuration
+files but also the includes they depend upon.

docs/how-to/inspect-your-backups.md

@@ -119,10 +119,10 @@ archive, regardless of the user who performs the backup. (Note that Borg
 doesn't store the leading `/`.)
 
 <span class="minilink minilink-addedin">With Borg version 1.2 and
-earlier</span>Database dump files are stored at a path dependent on the
-[runtime
+earlier</span>Database dump files are stored at a path dependent on the [runtime
 directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)
-in use at the time the archive was created.
+in use at the time the archive was created, as Borg 1.2 and earlier do not
+support path rewriting.
 
 <span class="minilink minilink-addedin">Prior to borgmatic version
 1.9.0</span>Database dump files were instead stored at `~/.borgmatic` within

docs/how-to/monitor-your-backups.md

@@ -47,6 +47,7 @@ them as backups happen:
  * [ntfy](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#ntfy-hook)
  * [PagerDuty](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook)
  * [Pushover](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pushover-hook)
+ * [Sentry](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook)
  * [Uptime Kuma](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#uptime-kuma-hook)
  * [Zabbix](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#zabbix-hook)
 
@@ -361,10 +362,49 @@ pushover:
 ```
 
 
+## Sentry hook
+
+<span class="minilink minilink-addedin">New in version 1.9.7</span>
+[Sentry](https://sentry.io/) is an application monitoring service that
+includes cron-style monitoring (either cloud-hosted or
+[self-hosted](https://develop.sentry.dev/self-hosted/)).
+
+To get started, create a [Sentry cron
+monitor](https://docs.sentry.io/product/crons/) in the Sentry UI. Under
+"Instrument your monitor," select "Sentry CLI" and copy the URL value for the
+displayed
+[`SENTRY_DSN`](https://docs.sentry.io/concepts/key-terms/dsn-explainer/)
+environment variable into borgmatic's Sentry `data_source_name_url`
+configuration option. For example:
+
+```
+sentry:
+    data_source_name_url: https://5f80ec@o294220.ingest.us.sentry.io/203069
+    monitor_slug: mymonitor
+```
+
+The `monitor_slug` value comes from the "Monitor Slug" under "Cron Details" on
+the same Sentry monitor page.
+
+With this configuration, borgmatic pings Sentry whenever borgmatic starts,
+finishes, or fails, but only when any of the `create`, `prune`, `compact`, or
+`check` actions are run. You can optionally override the start/finish/fail
+behavior with the `states` configuration option. For instance, to only ping
+Sentry on failure:
+
+```
+sentry:
+    data_source_name_url: https://5f80ec@o294220.ingest.us.sentry.io/203069
+    monitor_slug: mymonitor
+    states:
+      - fail
+```
+
+
 ## ntfy hook
 
 <span class="minilink minilink-addedin">New in version 1.6.3</span>
-[ntfy](https://ntfy.sh) is a free, simple, service (either hosted or
+[ntfy](https://ntfy.sh) is a free, simple, service (either cloud-hosted or
 self-hosted) which offers simple pub/sub push notifications to multiple
 platforms including [web](https://ntfy.sh/stats),
 [Android](https://play.google.com/store/apps/details?id=io.heckel.ntfy) and
@@ -438,22 +478,18 @@ your Loki API push URL. Here's an example:
 ```yaml
 loki:
     url: http://localhost:3100/loki/api/v1/push
+
+    labels:
+        app: borgmatic
+        hostname: example.org
 ```
 
 With this configuration, borgmatic sends its logs to your Loki instance as any
 of the `create`, `prune`, `compact`, or `check` actions are run. Then, after
 the actions complete, borgmatic notifies Loki of success or failure.
 
-This hook supports sending arbitrary labels to Loki. For instance:
-
-```yaml
-loki:
-    url: http://localhost:3100/loki/api/v1/push
-
-    labels:
-        app: borgmatic
-        hostname: example.org
-```
+This hook supports sending arbitrary labels to Loki. At least one label is
+required.
 
 There are also a few placeholders you can optionally use as label values:
 

docs/how-to/provide-your-passwords.md

@@ -24,11 +24,30 @@ or security reasons, read on.
 borgmatic supports calling another application such as a password manager to 
 obtain the Borg passphrase to a repository.
 
-For example, to ask the *Pass* password manager to provide the passphrase:
+For example, to ask the [Pass](https://www.passwordstore.org/) password manager
+to provide the passphrase:
+
 ```yaml
-encryption_passcommand: pass path/to/borg-repokey
+encryption_passcommand: pass path/to/borg-passphrase
 ```
 
+Another example for [KeePassXC](https://keepassxc.org/):
+
+```yaml
+encryption_passcommand: keepassxc-cli show --show-protected --attributes Password credentials.kdbx borg_passphrase
+```
+
+... where `borg_passphrase` is the title of the KeePassXC entry containing your
+Borg encryption passphrase in its `Password` field.
+
+<span class="minilink minilink-addedin">New in version 1.9.9</span> Instead of
+letting Borg run the passcommand—potentially multiple times since borgmatic runs
+Borg multiple times—borgmatic now runs the passcommand itself and passes the
+resulting passprhase securely to Borg via an anonymous pipe. This means you
+should only ever get prompted for your password manager's passphrase at most
+once per borgmatic run.
+
+
 ### Using systemd service credentials
 
 Borgmatic supports using encrypted [credentials](https://systemd.io/CREDENTIALS/).

docs/how-to/run-arbitrary-borg-commands.md

@@ -3,7 +3,7 @@ title: How to run arbitrary Borg commands
 eleventyNavigation:
   key: 🔧 Run arbitrary Borg commands
   parent: How-to guides
-  order: 11
+  order: 12
 ---
 ## Running Borg with borgmatic
 

docs/how-to/snapshot-your-filesystems.md

@@ -0,0 +1,297 @@
+---
+title: How to snapshot your filesystems
+eleventyNavigation:
+  key: 📸 Snapshot your filesystems
+  parent: How-to guides
+  order: 9
+---
+## Filesystem hooks
+
+Many filesystems support taking snapshots—point-in-time, read-only "copies" of
+your data, ideal for backing up files that may change during the backup. These
+snapshots initially don't use any additional storage space and can be made
+almost instantly.
+
+To help automate backup of these filesystems, borgmatic can use them to take
+snapshots.
+
+
+### ZFS
+
+<span class="minilink minilink-addedin">New in version 1.9.3</span> <span
+class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+taking snapshots with the [ZFS filesystem](https://openzfs.org/) and sending
+those snapshots to Borg for backup.
+
+To use this feature, first you need one or more mounted ZFS datasets. Then,
+enable ZFS within borgmatic by adding the following line to your configuration
+file:
+
+```yaml
+zfs:
+```
+
+No other options are necessary to enable ZFS support, but if desired you can
+override some of the commands used by the ZFS hook. For instance:
+
+```yaml
+zfs:
+    zfs_command: /usr/local/bin/zfs
+    mount_command: /usr/local/bin/mount
+    umount_command: /usr/local/bin/umount
+```
+
+As long as the ZFS hook is in beta, it may be subject to breaking changes
+and/or may not work well for your use cases. But feel free to use it in
+production if you're okay with these caveats, and please [provide any
+feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
+
+
+#### Dataset discovery
+
+You have a couple of options for borgmatic to find and backup your ZFS datasets:
+
+ * For any dataset you'd like backed up, add its mount point to borgmatic's
+   `source_directories` option.
+ * <span class="minilink minilink-addedin">New in version 1.9.6</span> Or
+   include the mount point with borgmatic's `patterns` or `patterns_from`
+   options.
+ * Or set the borgmatic-specific user property
+   `org.torsion.borgmatic:backup=auto` onto your dataset, e.g. by running `zfs
+   set org.torsion.borgmatic:backup=auto datasetname`. Then borgmatic can find
+   and backup these datasets.
+
+If you have multiple borgmatic configuration files with ZFS enabled, and you'd
+like particular datasets to be backed up only for particular configuration
+files, use the `source_directories` option instead of the user property.
+
+During a backup, borgmatic automatically snapshots these discovered datasets
+(non-recursively), temporarily mounts the snapshots within its [runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory),
+and includes the snapshotted files in the paths sent to Borg. borgmatic is also
+responsible for cleaning up (destroying) these snapshots after a backup
+completes.
+
+Additionally, borgmatic rewrites the snapshot file paths so that they appear
+at their original dataset locations in a Borg archive. For instance, if your
+dataset is mounted at `/var/dataset`, then the snapshotted files will appear
+in an archive at `/var/dataset` as well—even if borgmatic has to mount the
+snapshot somewhere in `/run/user/1000/borgmatic/zfs_snapshots/` to perform the
+backup.
+
+<span class="minilink minilink-addedin">New in version 1.9.4</span> borgmatic
+is smart enough to look at the parent (and grandparent, etc.) directories of
+each of your `source_directories` to discover any datasets. For instance,
+let's say you add `/var/log` and `/var/lib` to your source directories, but
+`/var` is a dataset. borgmatic will discover that and snapshot `/var`
+accordingly. This also works even with nested datasets; borgmatic selects
+the dataset that's the "closest" parent to your source directories.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> When using
+[patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns),
+the initial portion of a pattern's path that you intend borgmatic to match
+against a dataset can't have globs or other non-literal characters in it—or it
+won't actually match. For instance, a mount point of `/var` would match a
+pattern of `+ fm:/var/*/data`, but borgmatic isn't currently smart enough to
+match `/var` to a pattern like `+ fm:/v*/lib/data`.
+
+<span class="minilink minilink-addedin">With Borg version 1.2 and
+earlier</span>Snapshotted files are instead stored at a path dependent on the
+[runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)
+in use at the time the archive was created, as Borg 1.2 and earlier do not
+support path rewriting.
+
+
+#### Extract a dataset
+
+Filesystem snapshots are stored in a Borg archive as normal files, so
+you can use the standard
+[extract action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) to
+extract them.
+
+
+### Btrfs
+
+<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+class="minilink minilink-addedin">Beta feature</span> borgmatic supports taking
+snapshots with the [Btrfs filesystem](https://btrfs.readthedocs.io/) and sending
+those snapshots to Borg for backup.
+
+To use this feature, first you need one or more Btrfs subvolumes on mounted
+filesystems. Then, enable Btrfs within borgmatic by adding the following line to
+your configuration file:
+
+```yaml
+btrfs:
+```
+
+No other options are necessary to enable Btrfs support, but if desired you can
+override some of the commands used by the Btrfs hook. For instance:
+
+```yaml
+btrfs:
+    btrfs_command: /usr/local/bin/btrfs
+    findmnt_command: /usr/local/bin/findmnt
+```
+
+As long as the Btrfs hook is in beta, it may be subject to breaking changes
+and/or may not work well for your use cases. But feel free to use it in
+production if you're okay with these caveats, and please [provide any
+feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
+
+
+#### Subvolume discovery
+
+For any subvolume you'd like backed up, add its path to borgmatic's
+`source_directories` option.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
+the mount point with borgmatic's `patterns` or `patterns_from` options.
+
+During a backup, borgmatic snapshots these subvolumes (non-recursively) and
+includes the snapshotted files in the paths sent to Borg. borgmatic is also
+responsible for cleaning up (deleting) these snapshots after a backup completes.
+
+borgmatic is smart enough to look at the parent (and grandparent, etc.)
+directories of each of your `source_directories` to discover any subvolumes.
+For instance, let's say you add `/var/log` and `/var/lib` to your source
+directories, but `/var` is a subvolume. borgmatic will discover that and
+snapshot `/var` accordingly. This also works even with nested subvolumes;
+borgmatic selects the subvolume that's the "closest" parent to your source
+directories.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> When using
+[patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns),
+the initial portion of a pattern's path that you intend borgmatic to match
+against a subvolume can't have globs or other non-literal characters in it—or it
+won't actually match. For instance, a subvolume of `/var` would match a pattern
+of `+ fm:/var/*/data`, but borgmatic isn't currently smart enough to match
+`/var` to a pattern like `+ fm:/v*/lib/data`.
+
+Additionally, borgmatic rewrites the snapshot file paths so that they appear
+at their original subvolume locations in a Borg archive. For instance, if your
+subvolume exists at `/var/subvolume`, then the snapshotted files will appear
+in an archive at `/var/subvolume` as well—even if borgmatic has to mount the
+snapshot somewhere in `/var/subvolume/.borgmatic-snapshot-1234/` to perform
+the backup.
+
+<span class="minilink minilink-addedin">With Borg version 1.2 and
+earlier</span>Snapshotted files are instead stored at a path dependent on the
+temporary snapshot directory in use at the time the archive was created, as Borg
+1.2 and earlier do not support path rewriting.
+
+
+#### Extract a subvolume
+
+Subvolume snapshots are stored in a Borg archive as normal files, so you can use
+the standard [extract
+action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) to extract
+them.
+
+
+### LVM
+
+<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+taking snapshots with [LVM](https://sourceware.org/lvm2/) (Linux Logical
+Volume Manager) and sending those snapshots to Borg for backup. LVM isn't
+itself a filesystem, but it can take snapshots at the layer right below your
+filesystem.
+
+To use this feature, first you need one or more mounted LVM logical volumes.
+Then, enable LVM within borgmatic by adding the following line to your
+configuration file:
+
+```yaml
+lvm:
+```
+
+No other options are necessary to enable LVM support, but if desired you can
+override some of the options used by the LVM hook. For instance:
+
+```yaml
+lvm:
+    snapshot_size: 5GB  # See below for details.
+    lvcreate_command: /usr/local/bin/lvcreate
+    lvremove_command: /usr/local/bin/lvremove
+    lvs_command: /usr/local/bin/lvs
+    lsbrk_command: /usr/local/bin/lsbrk
+    mount_command: /usr/local/bin/mount
+    umount_command: /usr/local/bin/umount
+```
+
+As long as the LVM hook is in beta, it may be subject to breaking changes
+and/or may not work well for your use cases. But feel free to use it in
+production if you're okay with these caveats, and please [provide any
+feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
+
+
+#### Snapshot size
+
+The `snapshot_size` option is the size to allocate for each snapshot taken,
+including the units to use for that size. While borgmatic's snapshots
+themselves are read-only and don't change during backups, the logical volume
+being snapshotted *can* change—therefore requiring additional snapshot storage
+since LVM snapshots are copy-on-write. And if the configured snapshot size is
+too small (and LVM isn't configured to grow snapshots automatically), then the
+snapshots will fail to allocate enough space, resulting in a broken backup.
+
+If not specified, the `snapshot_size` option defaults to `10%ORIGIN`, which
+means 10% of the size of the logical volume being snapshotted. See the
+[`lvcreate --size` and `--extents`
+documentation](https://www.man7.org/linux/man-pages/man8/lvcreate.8.html) for
+more information about possible values here. (Under the hood, borgmatic uses
+`lvcreate --extents` if the `snapshot_size` is a percentage value, and `lvcreate
+--size` otherwise.)
+
+
+#### Logical volume discovery
+
+For any logical volume you'd like backed up, add its mount point to
+borgmatic's `source_directories` option.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
+the mount point with borgmatic's `patterns` or `patterns_from` options.
+
+During a backup, borgmatic automatically snapshots these discovered logical volumes
+(non-recursively), temporarily mounts the snapshots within its [runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory), and
+includes the snapshotted files in the paths sent to Borg. borgmatic is also responsible for cleaning
+up (deleting) these snapshots after a backup completes.
+
+borgmatic is smart enough to look at the parent (and grandparent, etc.)
+directories of each of your `source_directories` to discover any logical
+volumes. For instance, let's say you add `/var/log` and `/var/lib` to your
+source directories, but `/var` is a logical volume. borgmatic will discover
+that and snapshot `/var` accordingly.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> When using
+[patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns),
+the initial portion of a pattern's path that you intend borgmatic to match
+against a logical volume can't have globs or other non-literal characters in
+it—or it won't actually match. For instance, a logical volume of `/var` would
+match a pattern of `+ fm:/var/*/data`, but borgmatic isn't currently smart
+enough to match `/var` to a pattern like `+ fm:/v*/lib/data`.
+
+Additionally, borgmatic rewrites the snapshot file paths so that they appear
+at their original logical volume locations in a Borg archive. For instance, if
+your logical volume is mounted at `/var/lvolume`, then the snapshotted files
+will appear in an archive at `/var/lvolume` as well—even if borgmatic has to
+mount the snapshot somewhere in `/run/user/1000/borgmatic/lvm_snapshots/` to
+perform the backup.
+
+<span class="minilink minilink-addedin">With Borg version 1.2 and
+earlier</span>Snapshotted files are instead stored at a path dependent on the
+[runtime
+directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)
+in use at the time the archive was created, as Borg 1.2 and earlier do not
+support path rewriting.
+
+
+#### Extract a logical volume
+
+Logical volume snapshots are stored in a Borg archive as normal files, so
+you can use the standard
+[extract action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) to
+extract them.

docs/how-to/upgrade.md

@@ -3,7 +3,7 @@ title: How to upgrade borgmatic and Borg
 eleventyNavigation:
   key: 📦 Upgrade borgmatic/Borg
   parent: How-to guides
-  order: 13
+  order: 14
 ---
 ## Upgrading borgmatic
 

docs/reference/source-code.md

@@ -16,8 +16,11 @@ you get started. Starting at the top level, we have:
    * [actions](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/actions): borgmatic-specific logic for running each action (create, list, check, etc.).
    * [borg](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/borg): Lower-level code that's responsible for interacting with Borg to run each action.
    * [commands](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/commands): Looking to add a new flag or action? Start here. This contains borgmatic's entry point, argument parsing, and shell completion. 
-   * [config](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/config): Code responsible for loading, normalizing, and validating borgmatic's configuration.
-   * [hooks](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks): Looking to add a new database or monitoring integration? Start here.
+   * [config](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/config): Code responsible for loading, normalizing, and validating borgmatic's configuration. Interested in adding a new configuration option? Check out `schema.yaml` here.
+   * [hooks](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks): Looking to add a new database, filesystem, or monitoring integration? Start here.
+     * [credential](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/credential): Credential hooks—for loading passphrases and secrets from external providers.
+     * [data_source](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/data_source): Database and filesystem hooks—anything that produces data or files to go into a backup archive.
+     * [monitoring](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/monitoring): Monitoring hooks—integrations with third-party or self-hosted monitoring services.
  * [docs](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/docs): How-to and reference documentation, including the document you're reading now.
  * [sample](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/sample): Example configurations for cron and systemd.
  * [scripts](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/scripts): Dev-facing scripts for things like building documentation and running end-to-end tests.

pyproject.toml

@@ -1,12 +1,12 @@
 [project]
 name = "borgmatic"
-version = "1.9.2"
+version = "1.9.9"
 authors = [
   { name="Dan Helfman", email="witten@torsion.org" },
 ]
 description = "Simple, configuration-driven backup software for servers and workstations"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 classifiers=[
     "Development Status :: 5 - Production/Stable",
     "Environment :: Console",
@@ -17,11 +17,10 @@ classifiers=[
     "Topic :: System :: Archiving :: Backup",
 ]
 dependencies = [
-  "colorama>=0.4.1,<0.5",
-  "jsonschema",
-  "packaging",
-  "requests",
-  "ruamel.yaml>0.15.0",
+    "jsonschema",
+    "packaging",
+    "requests",
+    "ruamel.yaml>0.15.0",
 ]
 
 [project.scripts]
@@ -49,7 +48,7 @@ skip-string-normalization = true
 
 [tool.pytest.ini_options]
 testpaths = "tests"
-addopts = "--cov-report term-missing:skip-covered --cov=borgmatic --ignore=tests/end-to-end"
+addopts = "--cov-report term-missing:skip-covered --cov=borgmatic --no-cov-on-fail --cov-fail-under=100 --ignore=tests/end-to-end"
 
 [tool.isort]
 profile = "black"

scripts/run-full-tests

@@ -17,12 +17,9 @@ if [ -z "$TEST_CONTAINER" ]; then
     exit 1
 fi
 
-apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client mongodb-tools \
-    py3-ruamel.yaml py3-ruamel.yaml.clib py3-yaml bash sqlite fish
-# If certain dependencies of black are available in this version of Alpine, install them.
-apk add --no-cache py3-typed-ast py3-regex || true
-python3 -m pip install --no-cache --upgrade pip==24.2 setuptools==72.1.0
-pip3 install --ignore-installed tox==4.11.3
+apk add --no-cache python3 py3-pip borgbackup postgresql17-client mariadb-client mongodb-tools \
+    py3-mongo py3-regex py3-ruamel.yaml py3-ruamel.yaml.clib py3-tox py3-typed-ast py3-yaml bash \
+    sqlite fish
 export COVERAGE_FILE=/tmp/.coverage
 
 tox --workdir /tmp/.tox --sitepackages

test_requirements.txt

@@ -6,7 +6,6 @@ certifi==2024.7.4
 chardet==5.2.0
 click==8.1.7
 codespell==2.2.6
-colorama==0.4.6
 coverage==7.5.1
 flake8==7.0.0
 flake8-quotes==3.4.0

tests/end-to-end/commands/fake_btrfs.py

@@ -0,0 +1,90 @@
+import argparse
+import json
+import os
+import shutil
+import sys
+
+
+def parse_arguments(*unparsed_arguments):
+    global_parser = argparse.ArgumentParser(add_help=False)
+    action_parsers = global_parser.add_subparsers(dest='action')
+
+    subvolume_parser = action_parsers.add_parser('subvolume')
+    subvolume_subparser = subvolume_parser.add_subparsers(dest='subaction')
+
+    list_parser = subvolume_subparser.add_parser('list')
+    list_parser.add_argument('-s', dest='snapshots_only', action='store_true')
+    list_parser.add_argument('subvolume_path')
+
+    snapshot_parser = subvolume_subparser.add_parser('snapshot')
+    snapshot_parser.add_argument('-r', dest='read_only', action='store_true')
+    snapshot_parser.add_argument('subvolume_path')
+    snapshot_parser.add_argument('snapshot_path')
+
+    delete_parser = subvolume_subparser.add_parser('delete')
+    delete_parser.add_argument('snapshot_path')
+
+    return global_parser.parse_args(unparsed_arguments)
+
+
+BUILTIN_SUBVOLUME_LIST_LINES = (
+    '261 gen 29 top level 5 path sub',
+    '262 gen 29 top level 5 path other',
+)
+SUBVOLUME_LIST_LINE_PREFIX = '263 gen 29 top level 5 path '
+
+
+def load_snapshots():
+    try:
+        return json.load(open('/tmp/fake_btrfs.json'))
+    except FileNotFoundError:
+        return []
+
+
+def save_snapshots(snapshot_paths):
+    json.dump(snapshot_paths, open('/tmp/fake_btrfs.json', 'w'))
+
+
+def print_subvolume_list(arguments, snapshot_paths):
+    assert arguments.subvolume_path == '/mnt/subvolume'
+
+    if not arguments.snapshots_only:
+        for line in BUILTIN_SUBVOLUME_LIST_LINES:
+            print(line)
+
+    for snapshot_path in snapshot_paths:
+        print(
+            SUBVOLUME_LIST_LINE_PREFIX
+            + snapshot_path[snapshot_path.index('.borgmatic-snapshot-') :]
+        )
+
+
+def main():
+    arguments = parse_arguments(*sys.argv[1:])
+    snapshot_paths = load_snapshots()
+
+    if arguments.subaction == 'list':
+        print_subvolume_list(arguments, snapshot_paths)
+    elif arguments.subaction == 'snapshot':
+        snapshot_paths.append(arguments.snapshot_path)
+        save_snapshots(snapshot_paths)
+
+        subdirectory = os.path.join(arguments.snapshot_path, 'subdir')
+        os.makedirs(subdirectory, mode=0o700, exist_ok=True)
+        test_file = open(os.path.join(subdirectory, 'file.txt'), 'w')
+        test_file.write('contents')
+        test_file.close()
+    elif arguments.subaction == 'delete':
+        subdirectory = os.path.join(arguments.snapshot_path, 'subdir')
+        shutil.rmtree(subdirectory)
+
+        snapshot_paths = [
+            snapshot_path
+            for snapshot_path in snapshot_paths
+            if snapshot_path.endswith('/' + arguments.snapshot_path)
+        ]
+        save_snapshots(snapshot_paths)
+
+
+if __name__ == '__main__':
+    main()

tests/end-to-end/commands/fake_findmnt.py

@@ -0,0 +1,42 @@
+import argparse
+import sys
+
+
+def parse_arguments(*unparsed_arguments):
+    parser = argparse.ArgumentParser(add_help=False)
+    parser.add_argument('-t', dest='type')
+    parser.add_argument('--json', action='store_true')
+    parser.add_argument('--list', action='store_true')
+
+    return parser.parse_args(unparsed_arguments)
+
+
+BUILTIN_FILESYSTEM_MOUNT_OUTPUT = '''{
+       "filesystems": [
+          {
+             "target": "/mnt/subvolume",
+             "source": "/dev/loop0",
+             "fstype": "btrfs",
+             "options": "rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/"
+          }
+       ]
+    }
+    '''
+
+
+def print_filesystem_mounts():
+    print(BUILTIN_FILESYSTEM_MOUNT_OUTPUT)
+
+
+def main():
+    arguments = parse_arguments(*sys.argv[1:])
+
+    assert arguments.type == 'btrfs'
+    assert arguments.json
+    assert arguments.list
+
+    print_filesystem_mounts()
+
+
+if __name__ == '__main__':
+    main()

tests/end-to-end/commands/fake_lsblk.py

@@ -0,0 +1,70 @@
+import argparse
+import json
+import sys
+
+
+def parse_arguments(*unparsed_arguments):
+    parser = argparse.ArgumentParser(add_help=False)
+
+    parser.add_argument('--output', required=True)
+    parser.add_argument('--json', action='store_true', required=True)
+    parser.add_argument('--list', action='store_true', required=True)
+
+    return parser.parse_args(unparsed_arguments)
+
+
+BUILTIN_BLOCK_DEVICES = {
+    'blockdevices': [
+        {'name': 'loop0', 'path': '/dev/loop0', 'mountpoint': None, 'type': 'loop'},
+        {'name': 'cryptroot', 'path': '/dev/mapper/cryptroot', 'mountpoint': '/', 'type': 'crypt'},
+        {
+            'name': 'vgroup-lvolume',
+            'path': '/dev/mapper/vgroup-lvolume',
+            'mountpoint': '/mnt/lvolume',
+            'type': 'lvm',
+        },
+        {
+            'name': 'vgroup-lvolume-real',
+            'path': '/dev/mapper/vgroup-lvolume-real',
+            'mountpoint': None,
+            'type': 'lvm',
+        },
+    ]
+}
+
+
+def load_snapshots():
+    try:
+        return json.load(open('/tmp/fake_lvm.json'))
+    except FileNotFoundError:
+        return []
+
+
+def print_logical_volumes_json(arguments, snapshots):
+    data = dict(BUILTIN_BLOCK_DEVICES)
+
+    for snapshot in snapshots:
+        data['blockdevices'].extend(
+            {
+                'name': snapshot['lv_name'],
+                'path': snapshot['lv_path'],
+                'mountpoint': None,
+                'type': 'lvm',
+            }
+            for snapshot in snapshots
+        )
+
+    print(json.dumps(data))
+
+
+def main():
+    arguments = parse_arguments(*sys.argv[1:])
+    snapshots = load_snapshots()
+
+    assert arguments.output == 'name,path,mountpoint,type'
+
+    print_logical_volumes_json(arguments, snapshots)
+
+
+if __name__ == '__main__':
+    main()