Bump version for release.

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1014

NEWS

@@ -1,3 +1,75 @@
+1.9.13
+ * #975: Add a "compression" option to the PostgreSQL database hook.
+ * #1001: Fix a ZFS error during snapshot cleanup.
+ * #1003: In the Zabbix monitoring hook, support Zabbix 7.2's authentication changes.
+ * #1009: Send database passwords to MariaDB and MySQL via anonymous pipe, which is more secure than
+   using an environment variable.
+ * #1013: Send database passwords to MongoDB via anonymous pipe, which is more secure than using
+   "--password" on the command-line!
+ * #1015: When ctrl-C is pressed, more strongly encourage Borg to actually exit.
+ * Add a "verify_tls" option to the Uptime Kuma monitoring hook for disabling TLS verification.
+ * Add "tls" options to the MariaDB and MySQL database hooks to enable or disable TLS encryption
+   between client and server.
+
+1.9.12
+ * #1005: Fix the credential hooks to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
+1.9.11
+ * #795: Add credential loading from file, KeePassXC, and Docker/Podman secrets. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #996: Fix the "create" action to omit the repository label prefix from Borg's output when
+   databases are enabled.
+ * #998: Send the "encryption_passphrase" option to Borg via an anonymous pipe, which is more secure
+   than using an environment variable.
+ * #999: Fix a runtime directory error from a conflict between "extra_borg_options" and special file
+   detection.
+ * #1001: For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from
+   a borgmatic configuration option (e.g. "source_directories")—not from other hooks within
+   borgmatic.
+ * #1001: Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical
+   volumes.
+ * #1001: Don't try to snapshot ZFS datasets that have the "canmount=off" property.
+ * Fix another error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+
+1.9.10
+ * #966: Add a "{credential ...}" syntax for loading systemd credentials into borgmatic
+   configuration files. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #987: Fix a "list" action error when the "encryption_passcommand" option is set.
+ * #987: When both "encryption_passcommand" and "encryption_passphrase" are configured, prefer
+   "encryption_passphrase" even if it's an empty value.
+ * #988: With the "max_duration" option or the "--max-duration" flag, run the archives and
+   repository checks separately so they don't interfere with one another. Previously, borgmatic
+   refused to run checks in this situation.
+ * #989: Fix the log message code to avoid using Python 3.10+ logging features. Now borgmatic will
+   work with Python 3.9 again.
+ * Capture and delay any log records produced before logging is fully configured, so early log
+   records don't get lost.
+ * Add support for Python 3.13.
+
+1.9.9
+ * #635: Log the repository path or label on every relevant log message, not just some logs.
+ * #961: When the "encryption_passcommand" option is set, call the command once from borgmatic to
+   collect the encryption passphrase and then pass it to Borg multiple times. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #981: Fix a "spot" check file count delta error.
+ * #982: Fix for borgmatic "exclude_patterns" and "exclude_from" recursing into excluded
+   subdirectories.
+ * #983: Fix the Btrfs hook to support subvolumes with names like "@home" different from their
+   mount points.
+ * #985: Change the default value for the "--original-hostname" flag from "localhost" to no host
+   specified. This way, the "restore" action works without a hostname if there's a single matching
+   database dump.
+
+1.9.8
+ * #979: Fix root patterns so they don't have an invalid "sh:" prefix before getting passed to Borg.
+ * Expand the recent contributors documentation section to include ticket submitters—not just code
+   contributors—because there are multiple ways to contribute to the project! See:
+   https://torsion.org/borgmatic/#recent-contributors
+
 1.9.7
  * #855: Add a Sentry monitoring hook. See the documentation for more information:
    https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook

README.md

@@ -56,6 +56,8 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 
 ## Integrations
 
+### Data
+
 <a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
@@ -65,6 +67,11 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 <a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Monitoring
+
 <a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
@@ -76,7 +83,14 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 <a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Credentials
+
+<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ## Getting started
@@ -165,4 +179,8 @@ info on cloning source code, running tests, etc.
 
 ### Recent contributors
 
+Thanks to all borgmatic contributors! There are multiple ways to contribute to
+this project, so the following includes those who have fixed bugs, contributed
+features, *or* filed tickets.
+
 {% include borgmatic/contributors.html %}

borgmatic/actions/borg.py

@@ -22,9 +22,7 @@ def run_borg(
     if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, borg_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Running arbitrary Borg command'
-        )
+        logger.info('Running arbitrary Borg command')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             borg_arguments.archive,

borgmatic/actions/break_lock.py

@@ -21,9 +21,7 @@ def run_break_lock(
     if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, break_lock_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Breaking repository and cache locks'
-        )
+        logger.info('Breaking repository and cache locks')
         borgmatic.borg.break_lock.break_lock(
             repository['path'],
             config,

borgmatic/actions/change_passphrase.py

@@ -16,7 +16,7 @@ def run_change_passphrase(
     remote_path,
 ):
     '''
-    Run the "key change-passprhase" action for the given repository.
+    Run the "key change-passphrase" action for the given repository.
     '''
     if (
         change_passphrase_arguments.repository is None
@@ -24,9 +24,7 @@ def run_change_passphrase(
             repository, change_passphrase_arguments.repository
         )
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Changing repository passphrase'
-        )
+        logger.info('Changing repository passphrase')
         borgmatic.borg.change_passphrase.change_passphrase(
             repository['path'],
             config,

borgmatic/actions/check.py

@@ -363,7 +363,6 @@ def collect_spot_check_source_paths(
         borgmatic.hooks.dispatch.call_hooks(
             'use_streaming',
             config,
-            repository['path'],
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         ).values()
     )
@@ -387,13 +386,12 @@ def collect_spot_check_source_paths(
             stream_processes=stream_processes,
         )
     )
-    borg_environment = borgmatic.borg.environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     paths_output = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
-        extra_environment=borg_environment,
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -443,7 +441,7 @@ def collect_spot_check_archive_paths(
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} {path}{NL}',  # noqa: FS003
+            path_format='{type} {path}{NUL}',  # noqa: FS003
             local_path=local_path,
             remote_path=remote_path,
         )
@@ -468,15 +466,14 @@ def compare_spot_check_hashes(
     global_arguments,
     local_path,
     remote_path,
-    log_prefix,
     source_paths,
 ):
     '''
     Given a repository configuration dict, the name of the latest archive, a configuration dict, the
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
-    remote Borg path, a log label, and spot check source paths, compare the hashes for a sampling of
-    the source paths with hashes from corresponding paths in the given archive. Return a sequence of
-    the paths that fail that hash comparison.
+    remote Borg path, and spot check source paths, compare the hashes for a sampling of the source
+    paths with hashes from corresponding paths in the given archive. Return a sequence of the paths
+    that fail that hash comparison.
     '''
     # Based on the configured sample percentage, come up with a list of random sample files from the
     # source directories.
@@ -492,7 +489,7 @@ def compare_spot_check_hashes(
         if os.path.exists(os.path.join(working_directory or '', source_path))
     }
     logger.debug(
-        f'{log_prefix}: Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
+        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
     )
 
     source_sample_paths_iterator = iter(source_sample_paths)
@@ -540,7 +537,7 @@ def compare_spot_check_hashes(
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} {path}{NL}',  # noqa: FS003
+                    path_format='{xxh64} {path}{NUL}',  # noqa: FS003
                     local_path=local_path,
                     remote_path=remote_path,
                 )
@@ -580,8 +577,7 @@ def spot_check(
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    log_prefix = f'{repository.get("label", repository["path"])}'
-    logger.debug(f'{log_prefix}: Running spot check')
+    logger.debug('Running spot check')
 
     try:
         spot_check_config = next(
@@ -604,7 +600,7 @@ def spot_check(
         remote_path,
         borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(source_paths)} total source paths for spot check')
+    logger.debug(f'{len(source_paths)} total source paths for spot check')
 
     archive = borgmatic.borg.repo_list.resolve_archive_name(
         repository['path'],
@@ -615,7 +611,7 @@ def spot_check(
         local_path,
         remote_path,
     )
-    logger.debug(f'{log_prefix}: Using archive {archive} for spot check')
+    logger.debug(f'Using archive {archive} for spot check')
 
     archive_paths = collect_spot_check_archive_paths(
         repository,
@@ -627,11 +623,11 @@ def spot_check(
         remote_path,
         borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_prefix}: {len(archive_paths)} total archive paths for spot check')
+    logger.debug(f'{len(archive_paths)} total archive paths for spot check')
 
     if len(source_paths) == 0:
         logger.debug(
-            f'{log_prefix}: Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
         )
         raise ValueError(
             'Spot check failed: There are no source paths to compare against the archive'
@@ -644,10 +640,10 @@ def spot_check(
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
         rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
         logger.debug(
-            f'{log_prefix}: Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
+            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
         )
         logger.debug(
-            f'{log_prefix}: Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
+            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
         )
         raise ValueError(
             f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
@@ -661,25 +657,24 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
-        log_prefix,
         source_paths,
     )
 
     # Error if the percentage of failing hashes exceeds the configured tolerance percentage.
-    logger.debug(f'{log_prefix}: {len(failing_paths)} non-matching spot check hashes')
+    logger.debug(f'{len(failing_paths)} non-matching spot check hashes')
     data_tolerance_percentage = spot_check_config['data_tolerance_percentage']
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
         logger.debug(
-            f'{log_prefix}: Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
         )
         raise ValueError(
             f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'
         )
 
     logger.info(
-        f'{log_prefix}: Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
+        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
     )
 
 
@@ -713,8 +708,7 @@ def run_check(
         **hook_context,
     )
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Running consistency checks')
+    logger.info('Running consistency checks')
 
     repository_id = borgmatic.borg.check.get_repository_id(
         repository['path'],
@@ -767,9 +761,7 @@ def run_check(
         write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
-        with borgmatic.config.paths.Runtime_directory(
-            config, log_prefix
-        ) as borgmatic_runtime_directory:
+        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
             spot_check(
                 repository,
                 config,

borgmatic/actions/compact.py

@@ -37,9 +37,7 @@ def run_compact(
         **hook_context,
     )
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Compacting segments{dry_run_label}'
-        )
+        logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(
             global_arguments.dry_run,
             repository['path'],
@@ -53,9 +51,8 @@ def run_compact(
             threshold=compact_arguments.threshold,
         )
     else:  # pragma: nocover
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Skipping compact (only available/needed in Borg 1.2+)'
-        )
+        logger.info('Skipping compact (only available/needed in Borg 1.2+)')
+
     borgmatic.hooks.command.execute_hook(
         config.get('after_compact'),
         config.get('umask'),

borgmatic/actions/config/bootstrap.py

@@ -45,7 +45,6 @@ def get_config_paths(archive_name, bootstrap_arguments, global_arguments, local_
     # still want to support reading the manifest from previously created archives as well.
     with borgmatic.config.paths.Runtime_directory(
         {'user_runtime_directory': bootstrap_arguments.user_runtime_directory},
-        bootstrap_arguments.repository,
     ) as borgmatic_runtime_directory:
         for base_directory in (
             'borgmatic',

borgmatic/actions/create.py

@@ -36,6 +36,7 @@ def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_sty
         path,
         borgmatic.borg.pattern.Pattern_type(pattern_type),
         borgmatic.borg.pattern.Pattern_style(pattern_style),
+        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
     )
 
 
@@ -51,24 +52,26 @@ def collect_patterns(config):
     try:
         return (
             tuple(
-                borgmatic.borg.pattern.Pattern(source_directory)
+                borgmatic.borg.pattern.Pattern(
+                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
+                )
                 for source_directory in config.get('source_directories', ())
             )
             + tuple(
-                parse_pattern(pattern_line.strip(), borgmatic.borg.pattern.Pattern_style.SHELL)
+                parse_pattern(pattern_line.strip())
                 for pattern_line in config.get('patterns', ())
                 if not pattern_line.lstrip().startswith('#')
                 if pattern_line.strip()
             )
             + tuple(
                 parse_pattern(
-                    f'{borgmatic.borg.pattern.Pattern_type.EXCLUDE.value} {exclude_line.strip()}',
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
                     borgmatic.borg.pattern.Pattern_style.FNMATCH,
                 )
                 for exclude_line in config.get('exclude_patterns', ())
             )
             + tuple(
-                parse_pattern(pattern_line.strip(), borgmatic.borg.pattern.Pattern_style.SHELL)
+                parse_pattern(pattern_line.strip())
                 for filename in config.get('patterns_from', ())
                 for pattern_line in open(filename).readlines()
                 if not pattern_line.lstrip().startswith('#')
@@ -76,7 +79,7 @@ def collect_patterns(config):
             )
             + tuple(
                 parse_pattern(
-                    f'{borgmatic.borg.pattern.Pattern_type.EXCLUDE.value} {exclude_line.strip()}',
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
                     borgmatic.borg.pattern.Pattern_style.FNMATCH,
                 )
                 for filename in config.get('exclude_from', ())
@@ -144,6 +147,7 @@ def expand_patterns(patterns, working_directory=None, skip_paths=None):
                         pattern.type,
                         pattern.style,
                         pattern.device,
+                        pattern.source,
                     )
                     for expanded_path in expand_directory(pattern.path, working_directory)
                 )
@@ -178,6 +182,7 @@ def device_map_patterns(patterns, working_directory=None):
                 and os.path.exists(full_path)
                 else None
             ),
+            source=pattern.source,
         )
         for pattern in patterns
         for full_path in (os.path.join(working_directory or '', pattern.path),)
@@ -283,17 +288,13 @@ def run_create(
         **hook_context,
     )
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Creating archive{dry_run_label}')
+    logger.info(f'Creating archive{dry_run_label}')
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
@@ -302,7 +303,6 @@ def run_create(
         active_dumps = borgmatic.hooks.dispatch.call_hooks(
             'dump_data_sources',
             config,
-            repository['path'],
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             config_paths,
             borgmatic_runtime_directory,
@@ -339,7 +339,6 @@ def run_create(
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            config_filename,
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,

borgmatic/actions/delete.py

@@ -23,7 +23,7 @@ def run_delete(
     if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, delete_arguments.repository
     ):
-        logger.answer(f'{repository.get("label", repository["path"])}: Deleting archives')
+        logger.answer('Deleting archives')
 
         archive_name = (
             borgmatic.borg.repo_list.resolve_archive_name(

borgmatic/actions/export_key.py

@@ -21,7 +21,7 @@ def run_export_key(
     if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_arguments.repository
     ):
-        logger.info(f'{repository.get("label", repository["path"])}: Exporting repository key')
+        logger.info('Exporting repository key')
         borgmatic.borg.export_key.export_key(
             repository['path'],
             config,

borgmatic/actions/export_tar.py

@@ -22,9 +22,7 @@ def run_export_tar(
     if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, export_tar_arguments.repository
     ):
-        logger.info(
-            f'{repository["path"]}: Exporting archive {export_tar_arguments.archive} as tar file'
-        )
+        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
         borgmatic.borg.export_tar.export_tar_archive(
             global_arguments.dry_run,
             repository['path'],

borgmatic/actions/extract.py

@@ -33,9 +33,7 @@ def run_extract(
     if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
         repository, extract_arguments.repository
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Extracting archive {extract_arguments.archive}'
-        )
+        logger.info(f'Extracting archive {extract_arguments.archive}')
         borgmatic.borg.extract.extract_archive(
             global_arguments.dry_run,
             repository['path'],

borgmatic/actions/info.py

@@ -27,9 +27,7 @@ def run_info(
         repository, info_arguments.repository
     ):
         if not info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying archive summary information'
-            )
+            logger.answer('Displaying archive summary information')
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             info_arguments.archive,

borgmatic/actions/list.py

@@ -27,9 +27,9 @@ def run_list(
     ):
         if not list_arguments.json:
             if list_arguments.find_paths:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
+                logger.answer('Searching archives')
             elif not list_arguments.archive:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
+                logger.answer('Listing archives')
 
         archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],

borgmatic/actions/mount.py

@@ -23,11 +23,9 @@ def run_mount(
         repository, mount_arguments.repository
     ):
         if mount_arguments.archive:
-            logger.info(
-                f'{repository.get("label", repository["path"])}: Mounting archive {mount_arguments.archive}'
-            )
+            logger.info(f'Mounting archive {mount_arguments.archive}')
         else:  # pragma: nocover
-            logger.info(f'{repository.get("label", repository["path"])}: Mounting repository')
+            logger.info('Mounting repository')
 
         borgmatic.borg.mount.mount_archive(
             repository['path'],

borgmatic/actions/prune.py

@@ -35,7 +35,7 @@ def run_prune(
         global_arguments.dry_run,
         **hook_context,
     )
-    logger.info(f'{repository.get("label", repository["path"])}: Pruning archives{dry_run_label}')
+    logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/actions/repo_create.py

@@ -23,7 +23,7 @@ def run_repo_create(
     ):
         return
 
-    logger.info(f'{repository.get("label", repository["path"])}: Creating repository')
+    logger.info('Creating repository')
     borgmatic.borg.repo_create.create_repository(
         global_arguments.dry_run,
         repository['path'],

borgmatic/actions/repo_delete.py

@@ -21,8 +21,7 @@ def run_repo_delete(
         repository, repo_delete_arguments.repository
     ):
         logger.answer(
-            f'{repository.get("label", repository["path"])}: Deleting repository'
-            + (' cache' if repo_delete_arguments.cache_only else '')
+            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else '')
         )
 
         borgmatic.borg.repo_delete.delete_repository(

borgmatic/actions/repo_info.py

@@ -25,9 +25,7 @@ def run_repo_info(
         repository, repo_info_arguments.repository
     ):
         if not repo_info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying repository summary information'
-            )
+            logger.answer('Displaying repository summary information')
 
         json_output = borgmatic.borg.repo_info.display_repository_info(
             repository['path'],

borgmatic/actions/repo_list.py

@@ -25,7 +25,7 @@ def run_repo_list(
         repository, repo_list_arguments.repository
     ):
         if not repo_list_arguments.json:
-            logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
+            logger.answer('Listing repository')
 
         json_output = borgmatic.borg.repo_list.list_repository(
             repository['path'],

borgmatic/actions/restore.py

@@ -57,7 +57,7 @@ def render_dump_metadata(dump):
     Given a Dump instance, make a display string describing it for use in log messages.
     '''
     name = 'unspecified' if dump.data_source_name is UNSPECIFIED else dump.data_source_name
-    hostname = dump.hostname or 'localhost'
+    hostname = dump.hostname or UNSPECIFIED
     port = None if dump.port is UNSPECIFIED else dump.port
 
     if port:
@@ -71,10 +71,10 @@ def render_dump_metadata(dump):
     return metadata
 
 
-def get_configured_data_source(config, restore_dump, log_prefix):
+def get_configured_data_source(config, restore_dump):
     '''
     Search in the given configuration dict for dumps corresponding to the given dump to restore. If
-    there are multiple matches, error. Log using the given log prefix.
+    there are multiple matches, error.
 
     Return the found data source as a data source configuration dict or None if not found.
     '''
@@ -91,7 +91,6 @@ def get_configured_data_source(config, restore_dump, log_prefix):
             borgmatic.hooks.dispatch.call_hook(
                 function_name='get_default_port',
                 config=config,
-                log_prefix=log_prefix,
                 hook_name=hook_name,
             ),
         )
@@ -173,14 +172,11 @@ def restore_single_dump(
         Dump(hook_name, data_source['name'], data_source.get('hostname'), data_source.get('port'))
     )
 
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Restoring data source {dump_metadata}'
-    )
+    logger.info(f'Restoring data source {dump_metadata}')
 
     dump_patterns = borgmatic.hooks.dispatch.call_hooks(
         'make_data_source_dump_patterns',
         config,
-        repository['path'],
         borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         borgmatic_runtime_directory,
         data_source['name'],
@@ -227,7 +223,6 @@ def restore_single_dump(
     borgmatic.hooks.dispatch.call_hook(
         function_name='restore_data_source_dump',
         config=config,
-        log_prefix=repository['path'],
         hook_name=hook_name,
         data_source=data_source,
         dry_run=global_arguments.dry_run,
@@ -319,7 +314,7 @@ def collect_dumps_from_archive(
             break
         else:
             logger.warning(
-                f'{repository}: Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
+                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
             )
 
     return dumps_from_archive
@@ -348,12 +343,15 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                     else UNSPECIFIED
                 ),
                 data_source_name=name,
-                hostname=restore_arguments.original_hostname or 'localhost',
+                hostname=restore_arguments.original_hostname or UNSPECIFIED,
                 port=restore_arguments.original_port,
             )
-            for name in restore_arguments.data_sources
+            for name in restore_arguments.data_sources or (UNSPECIFIED,)
         }
-        if restore_arguments.data_sources
+        if restore_arguments.hook
+        or restore_arguments.data_sources
+        or restore_arguments.original_hostname
+        or restore_arguments.original_port
         else {
             Dump(
                 hook_name=UNSPECIFIED,
@@ -444,16 +442,12 @@ def run_restore(
     ):
         return
 
-    log_prefix = repository.get('label', repository['path'])
-    logger.info(f'{log_prefix}: Restoring data sources from archive {restore_arguments.archive}')
+    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
 
-    with borgmatic.config.paths.Runtime_directory(
-        config, log_prefix
-    ) as borgmatic_runtime_directory:
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,
@@ -494,7 +488,6 @@ def run_restore(
             found_data_source = get_configured_data_source(
                 config,
                 restore_dump,
-                log_prefix=repository['path'],
             )
 
             # For a dump that wasn't found via an exact match in the configuration, try to fallback
@@ -503,7 +496,6 @@ def run_restore(
                 found_data_source = get_configured_data_source(
                     config,
                     Dump(restore_dump.hook_name, 'all', restore_dump.hostname, restore_dump.port),
-                    log_prefix=repository['path'],
                 )
 
                 if not found_data_source:
@@ -531,7 +523,6 @@ def run_restore(
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
             config,
-            repository['path'],
             borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             borgmatic_runtime_directory,
             global_arguments.dry_run,

borgmatic/actions/transfer.py

@@ -17,9 +17,7 @@ def run_transfer(
     '''
     Run the "transfer" action for the given repository.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Transferring archives to repository'
-    )
+    logger.info('Transferring archives to repository')
     borgmatic.borg.transfer.transfer_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/borg/borg.py

@@ -61,7 +61,7 @@ def run_arbitrary_borg(
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
         shell=True,
-        extra_environment=dict(
+        environment=dict(
             (environment.make_environment(config) or {}),
             **{
                 'BORG_REPO': repository_path,

borgmatic/borg/break_lock.py

@@ -34,10 +34,9 @@ def break_lock(
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    borg_environment = environment.make_environment(config)
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/change_passphrase.py

@@ -41,7 +41,7 @@ def change_passphrase(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping change password (dry run)')
+        logger.info('Skipping change password (dry run)')
         return
 
     # If the original passphrase is set programmatically, then Borg won't prompt for a new one! So
@@ -56,7 +56,7 @@ def change_passphrase(
         full_command,
         output_file=borgmatic.execute.DO_NOT_CAPTURE,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config_without_passphrase),
+        environment=environment.make_environment(config_without_passphrase),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/check.py

@@ -64,15 +64,11 @@ def make_check_name_flags(checks, archive_filter_flags):
 
         ('--repository-only',)
 
-    However, if both "repository" and "archives" are in checks, then omit them from the returned
-    flags because Borg does both checks by default. If "data" is in checks, that implies "archives".
+    However, if both "repository" and "archives" are in checks, then omit the "only" flags from the
+    returned flags because Borg does both checks by default. Note that a "data" check only works
+    along with an "archives" check.
     '''
-    if 'data' in checks:
-        data_flags = ('--verify-data',)
-        checks.update({'archives'})
-    else:
-        data_flags = ()
-
+    data_flags = ('--verify-data',) if 'data' in checks else ()
     common_flags = (archive_filter_flags if 'archives' in checks else ()) + data_flags
 
     if {'repository', 'archives'}.issubset(checks):
@@ -142,51 +138,51 @@ def check_archives(
     except StopIteration:
         repository_check_config = {}
 
-    if check_arguments.max_duration and 'archives' in checks:
-        raise ValueError('The archives check cannot run when the --max-duration flag is used')
-    if repository_check_config.get('max_duration') and 'archives' in checks:
-        raise ValueError(
-            'The archives check cannot run when the repository check has the max_duration option set'
-        )
-
     max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')
+
     umask = config.get('umask')
-
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
-
-    full_command = (
-        (local_path, 'check')
-        + (('--repair',) if check_arguments.repair else ())
-        + (('--max-duration', str(max_duration)) if max_duration else ())
-        + make_check_name_flags(checks, archive_filter_flags)
-        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
-        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + verbosity_flags
-        + (('--progress',) if check_arguments.progress else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
-        + flags.make_repository_flags(repository_path, local_borg_version)
-    )
-
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    # The Borg repair option triggers an interactive prompt, which won't work when output is
-    # captured. And progress messes with the terminal directly.
-    if check_arguments.repair or check_arguments.progress:
+    if 'data' in checks:
+        checks.add('archives')
+
+    grouped_checks = (checks,)
+
+    # If max_duration is set, then archives and repository checks need to be run separately, as Borg
+    # doesn't support --max-duration along with an archives checks.
+    if max_duration and 'archives' in checks and 'repository' in checks:
+        checks.remove('repository')
+        grouped_checks = (checks, {'repository'})
+
+    for checks_subset in grouped_checks:
+        full_command = (
+            (local_path, 'check')
+            + (('--repair',) if check_arguments.repair else ())
+            + (
+                ('--max-duration', str(max_duration))
+                if max_duration and 'repository' in checks_subset
+                else ()
+            )
+            + make_check_name_flags(checks_subset, archive_filter_flags)
+            + (('--remote-path', remote_path) if remote_path else ())
+            + (('--umask', str(umask)) if umask else ())
+            + (('--log-json',) if global_arguments.log_json else ())
+            + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+            + verbosity_flags
+            + (('--progress',) if check_arguments.progress else ())
+            + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+            + flags.make_repository_flags(repository_path, local_borg_version)
+        )
+
         execute_command(
             full_command,
-            output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
-            working_directory=working_directory,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-    else:
-        execute_command(
-            full_command,
-            extra_environment=borg_environment,
+            # The Borg repair option triggers an interactive prompt, which won't work when output is
+            # captured. And progress messes with the terminal directly.
+            output_file=(
+                DO_NOT_CAPTURE if check_arguments.repair or check_arguments.progress else None
+            ),
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/compact.py

@@ -43,13 +43,13 @@ def compact_segments(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping compact (dry run)')
+        logging.info('Skipping compact (dry run)')
         return
 
     execute_command(
         full_command,
         output_log_level=logging.INFO,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/create.py

@@ -20,14 +20,12 @@ from borgmatic.execute import (
 logger = logging.getLogger(__name__)
 
 
-def write_patterns_file(patterns, borgmatic_runtime_directory, log_prefix, patterns_file=None):
+def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
     '''
     Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
     temporary file in the given borgmatic runtime directory and return the file object so it can
     continue to exist on disk as long as the caller needs it.
 
-    Use the given log prefix in any logging.
-
     If an optional open pattern file is given, append to it instead of making a new temporary file.
     Return None if no patterns are provided.
     '''
@@ -36,14 +34,16 @@ def write_patterns_file(patterns, borgmatic_runtime_directory, log_prefix, patte
 
     if patterns_file is None:
         patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
+        operation_name = 'Writing'
     else:
         patterns_file.write('\n')
+        operation_name = 'Appending'
 
     patterns_output = '\n'.join(
         f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
         for pattern in patterns
     )
-    logger.debug(f'{log_prefix}: Writing patterns to {patterns_file.name}:\n{patterns_output}')
+    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
 
     patterns_file.write(patterns_output)
     patterns_file.flush()
@@ -122,52 +122,63 @@ def collect_special_file_paths(
     config,
     local_path,
     working_directory,
-    borg_environment,
     borgmatic_runtime_directory,
 ):
     '''
     Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
-    a working directory, a dict of environment variables to pass to Borg, and the borgmatic runtime
-    directory, collect the paths for any special files (character devices, block devices, and named
-    pipes / FIFOs) that Borg would encounter during a create. These are all paths that could cause
-    Borg to hang if its --read-special flag is used.
+    a working directory, and the borgmatic runtime directory, collect the paths for any special
+    files (character devices, block devices, and named pipes / FIFOs) that Borg would encounter
+    during a create. These are all paths that could cause Borg to hang if its --read-special flag is
+    used.
 
     Skip looking for special files in the given borgmatic runtime directory, as borgmatic creates
-    its own special files there for database dumps. And if the borgmatic runtime directory is
-    configured to be excluded from the files Borg backs up, error, because this means Borg won't be
-    able to consume any database dumps and therefore borgmatic will hang.
+    its own special files there for database dumps and we don't want those omitted.
+
+    Additionally, if the borgmatic runtime directory is not contained somewhere in the files Borg
+    plans to backup, that means the user must have excluded the runtime directory (e.g. via
+    "exclude_patterns" or similar). Therefore, raise, because this means Borg won't be able to
+    consume any database dumps and therefore borgmatic will hang when it tries to do so.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
-    # files including any named pipe we've created.
+    # files including any named pipe we've created. And omit "--filter" because that can break the
+    # paths output parsing below such that path lines no longer start with th expected "- ".
     paths_output = execute_command_and_capture_output(
-        tuple(argument for argument in create_command if argument != '--exclude-nodump')
+        flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
         + ('--dry-run', '--list'),
         capture_stderr=True,
         working_directory=working_directory,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
+    # These are all the individual files that Borg is planning to backup as determined by the Borg
+    # create dry run above.
     paths = tuple(
         path_line.split(' ', 1)[1]
         for path_line in paths_output.split('\n')
         if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
     )
-    skip_paths = {}
+
+    # These are the subset of those files that contain the borgmatic runtime directory.
+    paths_containing_runtime_directory = {}
 
     if os.path.exists(borgmatic_runtime_directory):
-        skip_paths = {
+        paths_containing_runtime_directory = {
             path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
         }
 
-        if not skip_paths and not dry_run:
+        # If no paths to backup contain the runtime directory, it must've been excluded.
+        if not paths_containing_runtime_directory and not dry_run:
             raise ValueError(
                 f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.'
             )
 
     return tuple(
-        path for path in paths if special_file(path, working_directory) if path not in skip_paths
+        path
+        for path in paths
+        if special_file(path, working_directory)
+        if path not in paths_containing_runtime_directory
     )
 
 
@@ -217,9 +228,7 @@ def make_base_create_command(
     if config.get('source_directories_must_exist', False):
         check_all_root_patterns_exist(patterns)
 
-    patterns_file = write_patterns_file(
-        patterns, borgmatic_runtime_directory, log_prefix=repository_path
-    )
+    patterns_file = write_patterns_file(patterns, borgmatic_runtime_directory)
     checkpoint_interval = config.get('checkpoint_interval', None)
     checkpoint_volume = config.get('checkpoint_volume', None)
     chunker_params = config.get('chunker_params', None)
@@ -299,19 +308,17 @@ def make_base_create_command(
     # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
     if stream_processes and not config.get('read_special'):
         logger.warning(
-            f'{repository_path}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
+            'Ignoring configured "read_special" value of false, as true is needed for database hooks.'
         )
-        borg_environment = environment.make_environment(config)
         working_directory = borgmatic.config.paths.get_working_directory(config)
 
-        logger.debug(f'{repository_path}: Collecting special file paths')
+        logger.debug('Collecting special file paths')
         special_file_paths = collect_special_file_paths(
             dry_run,
             create_flags + create_positional_arguments,
             config,
             local_path,
             working_directory,
-            borg_environment,
             borgmatic_runtime_directory=borgmatic_runtime_directory,
         )
 
@@ -322,19 +329,19 @@ def make_base_create_command(
                 placeholder=' ...',
             )
             logger.warning(
-                f'{repository_path}: Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
+                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
             )
             patterns_file = write_patterns_file(
                 tuple(
                     borgmatic.borg.pattern.Pattern(
                         special_file_path,
-                        borgmatic.borg.pattern.Pattern_type.EXCLUDE,
+                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
                         borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
                     )
                     for special_file_path in special_file_paths
                 ),
                 borgmatic_runtime_directory,
-                log_prefix=repository_path,
                 patterns_file=patterns_file,
             )
 
@@ -399,8 +406,6 @@ def create_archive(
     # the terminal directly.
     output_file = DO_NOT_CAPTURE if progress else None
 
-    borg_environment = environment.make_environment(config)
-
     create_flags += (
         (('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
         + (('--stats',) if stats and not json and not dry_run else ())
@@ -417,7 +422,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -425,7 +430,7 @@ def create_archive(
         return execute_command_and_capture_output(
             create_flags + create_positional_arguments,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -435,7 +440,7 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )

borgmatic/borg/delete.py

@@ -128,7 +128,7 @@ def delete_archives(
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/environment.py

@@ -1,5 +1,8 @@
 import os
 
+import borgmatic.borg.passcommand
+import borgmatic.hooks.credential.parse
+
 OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_base_directory': 'BORG_BASE_DIR',
     'borg_config_directory': 'BORG_CONFIG_DIR',
@@ -7,8 +10,6 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
-    'encryption_passcommand': 'BORG_PASSCOMMAND',
-    'encryption_passphrase': 'BORG_PASSPHRASE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
 }
@@ -25,17 +26,59 @@ DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
 
 def make_environment(config):
     '''
-    Given a borgmatic configuration dict, return its options converted to a Borg environment
-    variable dict.
+    Given a borgmatic configuration dict, convert it to a Borg environment variable dict, merge it
+    with a copy of the current environment variables, and return the result.
+
+    Do not reuse this environment across multiple Borg invocations, because it can include
+    references to resources like anonymous pipes for passphrases—which can only be consumed once.
+
+    Here's how native Borg precedence works for a few of the environment variables:
+
+      1. BORG_PASSPHRASE, if set, is used first.
+      2. BORG_PASSCOMMAND is used only if BORG_PASSPHRASE isn't set.
+      3. BORG_PASSPHRASE_FD is used only if neither of the above are set.
+
+    In borgmatic, we want to simulate this precedence order, but there are some additional
+    complications. First, values can come from either configuration or from environment variables
+    set outside borgmatic; configured options should take precedence. Second, when borgmatic gets a
+    passphrase—directly from configuration or indirectly via a credential hook or a passcommand—we
+    want to pass that passphrase to Borg via an anonymous pipe (+ BORG_PASSPHRASE_FD), since that's
+    more secure than using an environment variable (BORG_PASSPHRASE).
     '''
-    environment = {}
+    environment = dict(os.environ)
 
     for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
 
-        if value:
+        if value is not None:
             environment[environment_variable_name] = str(value)
 
+    if 'encryption_passphrase' in config:
+        environment.pop('BORG_PASSPHRASE', None)
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    if 'encryption_passcommand' in config:
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    passphrase = borgmatic.hooks.credential.parse.resolve_credential(
+        config.get('encryption_passphrase'), config
+    )
+
+    if passphrase is None:
+        passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+
+    # If there's a passphrase (from configuration, from a configured credential, or from a
+    # configured passcommand), send it to Borg via an anonymous pipe.
+    if passphrase is not None:
+        read_file_descriptor, write_file_descriptor = os.pipe()
+        os.write(write_file_descriptor, passphrase.encode('utf-8'))
+        os.close(write_file_descriptor)
+
+        # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the Borg
+        # child process to inherit the file descriptor.
+        os.set_inheritable(read_file_descriptor, True)
+        environment['BORG_PASSPHRASE_FD'] = str(read_file_descriptor)
+
     for (
         option_name,
         environment_variable_name,

borgmatic/borg/export_key.py

@@ -60,14 +60,14 @@ def export_key(
     )
 
     if global_arguments.dry_run:
-        logger.info(f'{repository_path}: Skipping key export (dry run)')
+        logger.info('Skipping key export (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=output_file,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/export_tar.py

@@ -63,14 +63,14 @@ def export_tar_archive(
         output_log_level = logging.INFO
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping export to tar file (dry run)')
+        logging.info('Skipping export to tar file (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/extract.py

@@ -44,7 +44,6 @@ def extract_last_archive_dry_run(
         return
 
     list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
-    borg_environment = environment.make_environment(config)
     full_extract_command = (
         (local_path, 'extract', '--dry-run')
         + (('--remote-path', remote_path) if remote_path else ())
@@ -59,7 +58,7 @@ def extract_last_archive_dry_run(
 
     execute_command(
         full_extract_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -144,7 +143,6 @@ def extract_archive(
         + (tuple(paths) if paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
     full_destination_path = (
         os.path.join(working_directory or '', destination_path) if destination_path else None
@@ -156,7 +154,7 @@ def extract_archive(
         return execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -168,7 +166,7 @@ def extract_archive(
             full_command,
             output_file=subprocess.PIPE,
             run_to_completion=False,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -178,7 +176,7 @@ def extract_archive(
     # if the restore paths don't exist in the archive.
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=full_destination_path,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/flags.py

@@ -156,3 +156,44 @@ def warn_for_aggressive_archive_flags(json_command, json_output):
         logger.debug(f'Cannot parse JSON output from archive command: {error}')
     except (TypeError, KeyError):
         logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
+
+
+def omit_flag(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given (valueless) flag
+    omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    return tuple(argument for argument in arguments if argument != flag)
+
+
+def omit_flag_and_value(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given flag and its
+    corresponding value omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', 'value', '--other-flag')
+
+    ... or:
+
+        ('borg', 'create', '--flag=value', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    # This works by zipping together a list of overlapping pairwise arguments. E.g., ('one', 'two',
+    # 'three', 'four') becomes ((None, 'one'), ('one, 'two'), ('two', 'three'), ('three', 'four')).
+    # This makes it easy to "look back" at the previous arguments so we can exclude both a flag and
+    # its value.
+    return tuple(
+        argument
+        for (previous_argument, argument) in zip((None,) + arguments, arguments)
+        if flag not in (previous_argument, argument)
+        if not argument.startswith(f'{flag}=')
+    )

borgmatic/borg/info.py

@@ -102,7 +102,7 @@ def display_archives_info(
 
     json_info = execute_command_and_capture_output(
         json_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -116,7 +116,7 @@ def display_archives_info(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/list.py

@@ -106,8 +106,6 @@ def capture_archive_listing(
     format to use for the output, and local and remote Borg paths, capture the
     output of listing that archive and return it as a list of file paths.
     '''
-    borg_environment = environment.make_environment(config)
-
     return tuple(
         execute_command_and_capture_output(
             make_list_command(
@@ -126,12 +124,12 @@ def capture_archive_listing(
                 local_path,
                 remote_path,
             ),
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\n')
+        .strip('\0')
         .split('\0')
     )
 
@@ -194,7 +192,6 @@ def list_archive(
             'The --json flag on the list action is not supported when using the --archive/--find flags.'
         )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     # If there are any paths to find (and there's not a single archive already selected), start by
@@ -224,7 +221,7 @@ def list_archive(
                     local_path,
                     remote_path,
                 ),
-                extra_environment=borg_environment,
+                environment=environment.make_environment(config),
                 working_directory=borgmatic.config.paths.get_working_directory(config),
                 borg_local_path=local_path,
                 borg_exit_codes=borg_exit_codes,
@@ -237,7 +234,7 @@ def list_archive(
 
     # For each archive listed by Borg, run list on the contents of that archive.
     for archive in archive_lines:
-        logger.answer(f'{repository_path}: Listing archive {archive}')
+        logger.answer(f'Listing archive {archive}')
 
         archive_arguments = copy.copy(list_arguments)
         archive_arguments.archive = archive
@@ -260,7 +257,7 @@ def list_archive(
         execute_command(
             main_command,
             output_log_level=logging.ANSWER,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/mount.py

@@ -59,7 +59,6 @@ def mount_archive(
         + (tuple(mount_arguments.paths) if mount_arguments.paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     # Don't capture the output when foreground mode is used so that ctrl-C can work properly.
@@ -67,7 +66,7 @@ def mount_archive(
         execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
@@ -76,7 +75,7 @@ def mount_archive(
 
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/passcommand.py

@@ -0,0 +1,40 @@
+import functools
+import logging
+import shlex
+
+import borgmatic.config.paths
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+@functools.cache
+def run_passcommand(passcommand, working_directory):
+    '''
+    Run the given passcommand using the given working directory and return the passphrase produced
+    by the command.
+
+    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
+    per borgmatic invocation.
+    '''
+    return borgmatic.execute.execute_command_and_capture_output(
+        shlex.split(passcommand),
+        working_directory=working_directory,
+    )
+
+
+def get_passphrase_from_passcommand(config):
+    '''
+    Given the configuration dict, call the configured passcommand to produce and return an
+    encryption passphrase. In effect, we're doing an end-run around Borg by invoking its passcommand
+    ourselves. This allows us to pass the resulting passphrase to multiple different Borg
+    invocations without the user having to be prompted multiple times.
+
+    If no passcommand is configured, then return None.
+    '''
+    passcommand = config.get('encryption_passcommand')
+
+    if not passcommand:
+        return None
+
+    return run_passcommand(passcommand, borgmatic.config.paths.get_working_directory(config))

borgmatic/borg/pattern.py

@@ -20,12 +20,31 @@ class Pattern_style(enum.Enum):
     PATH_FULL_MATCH = 'pf'
 
 
+class Pattern_source(enum.Enum):
+    '''
+    Where the pattern came from within borgmatic. This is important because certain use cases (like
+    filesystem snapshotting) only want to consider patterns that the user actually put in a
+    configuration file and not patterns from other sources.
+    '''
+
+    # The pattern is from a borgmatic configuration option, e.g. listed in "source_directories".
+    CONFIG = 'config'
+
+    # The pattern is generated internally within borgmatic, e.g. for special file excludes.
+    INTERNAL = 'internal'
+
+    # The pattern originates from within a borgmatic hook, e.g. a database hook that adds its dump
+    # directory.
+    HOOK = 'hook'
+
+
 Pattern = collections.namedtuple(
     'Pattern',
-    ('path', 'type', 'style', 'device'),
+    ('path', 'type', 'style', 'device', 'source'),
     defaults=(
         Pattern_type.ROOT,
         Pattern_style.NONE,
         None,
+        Pattern_source.HOOK,
     ),
 )

borgmatic/borg/prune.py

@@ -96,7 +96,7 @@ def prune_archives(
     execute_command(
         full_command,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_create.py

@@ -57,7 +57,7 @@ def create_repository(
                 f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
             )
 
-        logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
+        logger.info('Repository already exists. Skipping creation.')
         return
     except subprocess.CalledProcessError as error:
         if error.returncode not in REPO_INFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
@@ -91,14 +91,14 @@ def create_repository(
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping repository creation (dry run)')
+        logging.info('Skipping repository creation (dry run)')
         return
 
     # Do not capture output here, so as to support interactive prompts.
     execute_command(
         repo_create_command,
         output_file=DO_NOT_CAPTURE,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_delete.py

@@ -88,7 +88,7 @@ def delete_repository(
             if repo_delete_arguments.force or repo_delete_arguments.cache_only
             else borgmatic.execute.DO_NOT_CAPTURE
         ),
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/repo_info.py

@@ -50,14 +50,13 @@ def display_repository_info(
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    extra_environment = environment.make_environment(config)
     working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     if repo_info_arguments.json:
         return execute_command_and_capture_output(
             full_command,
-            extra_environment=extra_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
@@ -66,7 +65,7 @@ def display_repository_info(
         execute_command(
             full_command,
             output_log_level=logging.ANSWER,
-            extra_environment=extra_environment,
+            environment=environment.make_environment(config),
             working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,

borgmatic/borg/repo_list.py

@@ -49,7 +49,7 @@ def resolve_archive_name(
 
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
@@ -59,7 +59,7 @@ def resolve_archive_name(
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'{repository_path}: Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive}')
 
     return latest_archive
 
@@ -140,7 +140,6 @@ def list_repository(
     return JSON output).
     '''
     borgmatic.logger.add_custom_log_levels()
-    borg_environment = environment.make_environment(config)
 
     main_command = make_repo_list_command(
         repository_path,
@@ -165,7 +164,7 @@ def list_repository(
 
     json_listing = execute_command_and_capture_output(
         json_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
@@ -179,7 +178,7 @@ def list_repository(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,

borgmatic/borg/transfer.py

@@ -57,7 +57,7 @@ def transfer_archives(
         full_command,
         output_log_level=logging.ANSWER,
         output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/borg/version.py

@@ -21,7 +21,7 @@ def local_borg_version(config, local_path='borg'):
     )
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),

borgmatic/commands/arguments.py

@@ -349,12 +349,12 @@ def make_parsers():
     global_group.add_argument(
         '--log-file-format',
         type=str,
-        help='Log format string used for log messages written to the log file',
+        help='Python format string used for log messages written to the log file',
     )
     global_group.add_argument(
         '--log-json',
         action='store_true',
-        help='Write log messages and console output as one JSON object per log line instead of formatted text',
+        help='Write Borg log messages and console output as one JSON object per log line instead of formatted text',
     )
     global_group.add_argument(
         '--override',

borgmatic/commands/borgmatic.py

@@ -39,7 +39,14 @@ from borgmatic.commands.arguments import parse_arguments
 from borgmatic.config import checks, collect, validate
 from borgmatic.hooks import command, dispatch
 from borgmatic.hooks.monitoring import monitor
-from borgmatic.logger import DISABLED, add_custom_log_levels, configure_logging, should_do_markup
+from borgmatic.logger import (
+    DISABLED,
+    Log_prefix,
+    add_custom_log_levels,
+    configure_delayed_logging,
+    configure_logging,
+    should_do_markup,
+)
 from borgmatic.signals import configure_signals
 from borgmatic.verbosity import verbosity_to_log_level
 
@@ -86,12 +93,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
 
     if skip_actions:
         logger.debug(
-            f"{config_filename}: Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
+            f"Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
         )
 
     try:
         local_borg_version = borg_version.local_borg_version(config, local_path)
-        logger.debug(f'{config_filename}: Borg {local_borg_version}')
+        logger.debug(f'Borg {local_borg_version}')
     except (OSError, CalledProcessError, ValueError) as error:
         yield from log_error_records(f'{config_filename}: Error getting local Borg version', error)
         return
@@ -101,8 +108,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'initialize_monitor',
                 config,
-                config_filename,
                 dispatch.Hook_type.MONITORING,
+                config_filename,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
@@ -110,14 +117,14 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
-                config_filename,
                 dispatch.Hook_type.MONITORING,
+                config_filename,
                 monitor.State.START,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
     except (OSError, CalledProcessError) as error:
-        if command.considered_soft_failure(config_filename, error):
+        if command.considered_soft_failure(error):
             return
 
         encountered_error = error
@@ -132,53 +139,49 @@ def run_configuration(config_filename, config, config_paths, arguments):
 
         while not repo_queue.empty():
             repository, retry_num = repo_queue.get()
-            logger.debug(
-                f'{repository.get("label", repository["path"])}: Running actions for repository'
-            )
-            timeout = retry_num * retry_wait
-            if timeout:
-                logger.warning(
-                    f'{repository.get("label", repository["path"])}: Sleeping {timeout}s before next retry'
-                )
-                time.sleep(timeout)
-            try:
-                yield from run_actions(
-                    arguments=arguments,
-                    config_filename=config_filename,
-                    config=config,
-                    config_paths=config_paths,
-                    local_path=local_path,
-                    remote_path=remote_path,
-                    local_borg_version=local_borg_version,
-                    repository=repository,
-                )
-            except (OSError, CalledProcessError, ValueError) as error:
-                if retry_num < retries:
-                    repo_queue.put(
-                        (repository, retry_num + 1),
+
+            with Log_prefix(repository.get('label', repository['path'])):
+                logger.debug('Running actions for repository')
+                timeout = retry_num * retry_wait
+                if timeout:
+                    logger.warning(f'Sleeping {timeout}s before next retry')
+                    time.sleep(timeout)
+                try:
+                    yield from run_actions(
+                        arguments=arguments,
+                        config_filename=config_filename,
+                        config=config,
+                        config_paths=config_paths,
+                        local_path=local_path,
+                        remote_path=remote_path,
+                        local_borg_version=local_borg_version,
+                        repository=repository,
                     )
-                    tuple(  # Consume the generator so as to trigger logging.
-                        log_error_records(
-                            f'{repository.get("label", repository["path"])}: Error running actions for repository',
-                            error,
-                            levelno=logging.WARNING,
-                            log_command_error_output=True,
+                except (OSError, CalledProcessError, ValueError) as error:
+                    if retry_num < retries:
+                        repo_queue.put(
+                            (repository, retry_num + 1),
                         )
-                    )
-                    logger.warning(
-                        f'{repository.get("label", repository["path"])}: Retrying... attempt {retry_num + 1}/{retries}'
-                    )
-                    continue
+                        tuple(  # Consume the generator so as to trigger logging.
+                            log_error_records(
+                                'Error running actions for repository',
+                                error,
+                                levelno=logging.WARNING,
+                                log_command_error_output=True,
+                            )
+                        )
+                        logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
+                        continue
 
-                if command.considered_soft_failure(config_filename, error):
-                    continue
+                    if command.considered_soft_failure(error):
+                        continue
 
-                yield from log_error_records(
-                    f'{repository.get("label", repository["path"])}: Error running actions for repository',
-                    error,
-                )
-                encountered_error = error
-                error_repository = repository['path']
+                    yield from log_error_records(
+                        'Error running actions for repository',
+                        error,
+                    )
+                    encountered_error = error
+                    error_repository = repository['path']
 
     try:
         if monitoring_hooks_are_activated:
@@ -186,16 +189,16 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
-                config_filename,
                 dispatch.Hook_type.MONITORING,
+                config_filename,
                 monitor.State.LOG,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
     except (OSError, CalledProcessError) as error:
-        if not command.considered_soft_failure(config_filename, error):
+        if not command.considered_soft_failure(error):
             encountered_error = error
-            yield from log_error_records(f'{repository["path"]}: Error pinging monitor', error)
+            yield from log_error_records('Error pinging monitor', error)
 
     if not encountered_error:
         try:
@@ -203,8 +206,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
                 dispatch.call_hooks(
                     'ping_monitor',
                     config,
-                    config_filename,
                     dispatch.Hook_type.MONITORING,
+                    config_filename,
                     monitor.State.FINISH,
                     monitoring_log_level,
                     global_arguments.dry_run,
@@ -212,13 +215,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
                 dispatch.call_hooks(
                     'destroy_monitor',
                     config,
-                    config_filename,
                     dispatch.Hook_type.MONITORING,
                     monitoring_log_level,
                     global_arguments.dry_run,
                 )
         except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(config_filename, error):
+            if command.considered_soft_failure(error):
                 return
 
             encountered_error = error
@@ -239,8 +241,8 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'ping_monitor',
                 config,
-                config_filename,
                 dispatch.Hook_type.MONITORING,
+                config_filename,
                 monitor.State.FAIL,
                 monitoring_log_level,
                 global_arguments.dry_run,
@@ -248,13 +250,12 @@ def run_configuration(config_filename, config, config_paths, arguments):
             dispatch.call_hooks(
                 'destroy_monitor',
                 config,
-                config_filename,
                 dispatch.Hook_type.MONITORING,
                 monitoring_log_level,
                 global_arguments.dry_run,
             )
         except (OSError, CalledProcessError) as error:
-            if command.considered_soft_failure(config_filename, error):
+            if command.considered_soft_failure(error):
                 return
 
             yield from log_error_records(f'{config_filename}: Error running on-error hook', error)
@@ -534,13 +535,16 @@ def run_actions(
 
 def load_configurations(config_filenames, overrides=None, resolve_env=True):
     '''
-    Given a sequence of configuration filenames, load and validate each configuration file. Return
-    the results as a tuple of: dict of configuration filename to corresponding parsed configuration,
-    a sequence of paths for all loaded configuration files (including includes), and a sequence of
-    logging.LogRecord instances containing any parse errors.
+    Given a sequence of configuration filenames, a sequence of configuration file override strings
+    in the form of "option.suboption=value", and whether to resolve environment variables, load and
+    validate each configuration file. Return the results as a tuple of: dict of configuration
+    filename to corresponding parsed configuration, a sequence of paths for all loaded configuration
+    files (including includes), and a sequence of logging.LogRecord instances containing any parse
+    errors.
 
     Log records are returned here instead of being logged directly because logging isn't yet
-    initialized at this point!
+    initialized at this point! (Although with the Delayed_logging_handler now in place, maybe this
+    approach could change.)
     '''
     # Dict mapping from config filename to corresponding parsed config dict.
     configs = collections.OrderedDict()
@@ -562,7 +566,10 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
         )
         try:
             configs[config_filename], paths, parse_logs = validate.parse_configuration(
-                config_filename, validate.schema_filename(), overrides, resolve_env
+                config_filename,
+                validate.schema_filename(),
+                overrides,
+                resolve_env,
             )
             config_paths.update(paths)
             logs.extend(parse_logs)
@@ -819,23 +826,27 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments):
 
     # Execute the actions corresponding to each configuration file.
     json_results = []
-    for config_filename, config in configs.items():
-        results = list(run_configuration(config_filename, config, config_paths, arguments))
-        error_logs = tuple(result for result in results if isinstance(result, logging.LogRecord))
 
-        if error_logs:
-            yield from log_error_records(f'{config_filename}: An error occurred')
-            yield from error_logs
-        else:
-            yield logging.makeLogRecord(
-                dict(
-                    levelno=logging.INFO,
-                    levelname='INFO',
-                    msg=f'{config_filename}: Successfully ran configuration file',
-                )
+    for config_filename, config in configs.items():
+        with Log_prefix(config_filename):
+            results = list(run_configuration(config_filename, config, config_paths, arguments))
+            error_logs = tuple(
+                result for result in results if isinstance(result, logging.LogRecord)
             )
-            if results:
-                json_results.extend(results)
+
+            if error_logs:
+                yield from log_error_records('An error occurred')
+                yield from error_logs
+            else:
+                yield logging.makeLogRecord(
+                    dict(
+                        levelno=logging.INFO,
+                        levelname='INFO',
+                        msg='Successfully ran configuration file',
+                    )
+                )
+                if results:
+                    json_results.extend(results)
 
     if 'umount' in arguments:
         logger.info(f"Unmounting mount point {arguments['umount'].mount_point}")
@@ -876,6 +887,7 @@ def exit_with_help_link():  # pragma: no cover
 
 def main(extra_summary_logs=[]):  # pragma: no cover
     configure_signals()
+    configure_delayed_logging()
 
     try:
         arguments = parse_arguments(*sys.argv[1:])
@@ -901,9 +913,12 @@ def main(extra_summary_logs=[]):  # pragma: no cover
         print(borgmatic.commands.completion.fish.fish_completion())
         sys.exit(0)
 
+    validate = bool('validate' in arguments)
     config_filenames = tuple(collect.collect_config_filenames(global_arguments.config_paths))
     configs, config_paths, parse_logs = load_configurations(
-        config_filenames, global_arguments.overrides, global_arguments.resolve_env
+        config_filenames,
+        global_arguments.overrides,
+        resolve_env=global_arguments.resolve_env and not validate,
     )
     configuration_parse_errors = (
         (max(log.levelno for log in parse_logs) >= logging.CRITICAL) if parse_logs else False

borgmatic/config/generate.py

@@ -41,6 +41,7 @@ def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
     '''
     schema_type = schema.get('type')
     example = schema.get('example')
+
     if example is not None:
         return example
 

borgmatic/config/load.py

@@ -69,7 +69,7 @@ def include_configuration(loader, filename_node, include_directory, config_paths
         ]
 
     raise ValueError(
-        '!include value is not supported; use a single filename or a list of filenames'
+        'The value given for the !include tag is invalid; use a single filename or a list of filenames instead'
     )
 
 

borgmatic/config/paths.py

@@ -76,14 +76,13 @@ class Runtime_directory:
     automatically gets cleaned up as necessary.
     '''
 
-    def __init__(self, config, log_prefix):
+    def __init__(self, config):
         '''
-        Given a configuration dict and a log prefix, determine the borgmatic runtime directory,
-        creating a secure, temporary directory within it if necessary. Defaults to
-        $XDG_RUNTIME_DIR/./borgmatic or $RUNTIME_DIRECTORY/./borgmatic or
-        $TMPDIR/borgmatic-[random]/./borgmatic or $TEMP/borgmatic-[random]/./borgmatic or
-        /tmp/borgmatic-[random]/./borgmatic where "[random]" is a randomly generated string intended
-        to avoid path collisions.
+        Given a configuration dict determine the borgmatic runtime directory, creating a secure,
+        temporary directory within it if necessary. Defaults to $XDG_RUNTIME_DIR/./borgmatic or
+        $RUNTIME_DIRECTORY/./borgmatic or $TMPDIR/borgmatic-[random]/./borgmatic or
+        $TEMP/borgmatic-[random]/./borgmatic or /tmp/borgmatic-[random]/./borgmatic where "[random]"
+        is a randomly generated string intended to avoid path collisions.
 
         If XDG_RUNTIME_DIR or RUNTIME_DIRECTORY is set and already ends in "/borgmatic", then don't
         tack on a second "/borgmatic" path component.
@@ -127,7 +126,7 @@ class Runtime_directory:
         )
         os.makedirs(self.runtime_path, mode=0o700, exist_ok=True)
 
-        logger.debug(f'{log_prefix}: Using runtime directory {os.path.normpath(self.runtime_path)}')
+        logger.debug(f'Using runtime directory {os.path.normpath(self.runtime_path)}')
 
     def __enter__(self):
         '''

borgmatic/config/schema.yaml

@@ -205,8 +205,8 @@ properties:
         description: |
             Deprecated. Only used for locating database dumps and bootstrap
             metadata within backup archives created prior to deprecation.
-            Replaced by borgmatic_runtime_directory and
-            borgmatic_state_directory. Defaults to ~/.borgmatic
+            Replaced by user_runtime_directory and user_state_directory.
+            Defaults to ~/.borgmatic
         example: /tmp/borgmatic
     user_runtime_directory:
         type: string
@@ -250,7 +250,7 @@ properties:
             repositories that were initialized with passphrase/repokey/keyfile
             encryption. Quote the value if it contains punctuation, so it parses
             correctly. And backslash any quote or backslash literals as well.
-            Defaults to not set.
+            Defaults to not set. Supports the "{credential ...}" syntax.
         example: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
     checkpoint_interval:
         type: integer
@@ -632,8 +632,8 @@ properties:
                               long-running repository check into multiple
                               partial checks. Defaults to no interruption. Only
                               applies to the "repository" check, does not check
-                              the repository index, and is not compatible with a
-                              simultaneous "archives" check or "--repair" flag.
+                              the repository index and is not compatible with
+                              the "--repair" flag.
                           example: 3600
                 - required:
                     - name
@@ -989,13 +989,15 @@ properties:
                         Username with which to connect to the database. Defaults
                         to the username of the current user. You probably want
                         to specify the "postgres" superuser here when the
-                        database name is "all".
+                        database name is "all". Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1003,13 +1005,15 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if PostgreSQL is configured to
                         trust the configured username without a password or you
-                        create a ~/.pgpass file.
+                        create a ~/.pgpass file. Supports the "{credential ...}"
+                        syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 no_owner:
                     type: boolean
@@ -1036,6 +1040,18 @@ properties:
                         individual databases. See the pg_dump documentation for
                         more about formats.
                     example: directory
+                compression:
+                    type: ["string", "integer"]
+                    description: |
+                        Database dump compression level (integer) or method
+                        ("gzip", "lz4", "zstd", or "none") and optional
+                        colon-separated detail. Defaults to moderate "gzip" for
+                        "custom" and "directory" formats and no compression for
+                        the "plain" format. Compression is not supported for the
+                        "tar" format. Be aware that Borg does its own
+                        compression as well, so you may not need it in both
+                        places.
+                    example: none
                 ssl_mode:
                     type: string
                     enum: ['disable', 'allow', 'prefer',
@@ -1169,13 +1185,15 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Defaults
-                        to the username of the current user.
+                        to the username of the current user. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1183,7 +1201,29 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if MariaDB is configured to
                         trust the configured username without a password.
+                        Supports the "{credential ...}" syntax.
                     example: trustsome1
+                restore_password:
+                    type: string
+                    description: |
+                        Password with which to connect to the restore database.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
+                    example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MariaDB version.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MariaDB version.
+                    example: false
                 mariadb_dump_command:
                     type: string
                     description: |
@@ -1201,12 +1241,6 @@ properties:
                         run a specific mariadb version (e.g., one inside a
                         running container). Defaults to "mariadb".
                     example: docker exec mariadb_container mariadb           
-                restore_password:
-                    type: string
-                    description: |
-                        Password with which to connect to the restore database.
-                        Defaults to the "password" option.
-                    example: trustsome1
                 format:
                     type: string
                     enum: ['sql']
@@ -1295,13 +1329,15 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Defaults
-                        to the username of the current user.
+                        to the username of the current user. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
@@ -1309,13 +1345,29 @@ properties:
                         Password with which to connect to the database. Omitting
                         a password will only work if MySQL is configured to
                         trust the configured username without a password.
+                        Supports the "{credential ...}" syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
+                tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and server. The default varies based on the
+                        MySQL installation.
+                    example: false
+                restore_tls:
+                    type: boolean
+                    description: |
+                        Whether to TLS-encrypt data transmitted between the
+                        client and restore server. The default varies based on
+                        the MySQL installation.
+                    example: false
                 mysql_dump_command:
                     type: string
                     description: |
@@ -1451,25 +1503,29 @@ properties:
                     type: string
                     description: |
                         Username with which to connect to the database. Skip it
-                        if no authentication is needed.
+                        if no authentication is needed. Supports the
+                        "{credential ...}" syntax.
                     example: dbuser
                 restore_username:
                     type: string
                     description: |
                         Username with which to restore the database. Defaults to
-                        the "username" option.
+                        the "username" option. Supports the "{credential ...}"
+                        syntax.
                     example: dbuser
                 password:
                     type: string
                     description: |
                         Password with which to connect to the database. Skip it
-                        if no authentication is needed.
+                        if no authentication is needed. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 restore_password:
                     type: string
                     description: |
                         Password with which to connect to the restore database.
-                        Defaults to the "password" option.
+                        Defaults to the "password" option. Supports the
+                        "{credential ...}" syntax.
                     example: trustsome1
                 authentication_database:
                     type: string
@@ -1528,18 +1584,20 @@ properties:
             username:
                 type: string
                 description: |
-                    The username used for authentication.
+                    The username used for authentication. Supports the
+                    "{credential ...}" syntax.
                 example: testuser
             password:
                 type: string
                 description: |
-                    The password used for authentication.
+                    The password used for authentication. Supports the
+                    "{credential ...}" syntax.
                 example: fakepassword
             access_token:
                 type: string
                 description: |
                     An ntfy access token to authenticate with instead of
-                    username/password.
+                    username/password. Supports the "{credential ...}" syntax.
                 example: tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2
             start:
                 type: object
@@ -1634,14 +1692,16 @@ properties:
             token:
                 type: string
                 description: |
-                    Your application's API token.
+                    Your application's API token. Supports the "{credential
+                    ...}" syntax.
                 example: 7ms6TXHpTokTou2P6x4SodDeentHRa
             user:
                 type: string
                 description: |
-                    Your user/group key (or that of your target user), viewable 
-                    when logged into your dashboard: often referred to as 
+                    Your user/group key (or that of your target user), viewable
+                    when logged into your dashboard: often referred to as
                     USER_KEY in Pushover documentation and code examples.
+                    Supports the "{credential ...}" syntax.
                 example: hwRwoWsXMBWwgrSecfa9EfPey55WSN
             start:
                 type: object
@@ -1887,6 +1947,8 @@ properties:
     zabbix:
         type: object
         additionalProperties: false
+        required:
+            - server
         properties:
             itemid:
                 type: integer
@@ -1909,25 +1971,26 @@ properties:
             server:
                 type: string
                 description: |
-                    The address of your Zabbix instance.
+                    The API endpoint URL of your Zabbix instance, usually ending
+                    with "/api_jsonrpc.php". Required.
                 example: https://zabbix.your-domain.com
             username:
                 type: string
                 description: |
                     The username used for authentication. Not needed if using
-                    an API key.
+                    an API key. Supports the "{credential ...}" syntax.
                 example: testuser
             password:
                 type: string
                 description: |
                     The password used for authentication. Not needed if using
-                    an API key.
+                    an API key. Supports the "{credential ...}" syntax.
                 example: fakepassword
             api_key:
                 type: string
                 description: |
-                    The API key used for authentication. Not needed if using
-                    an username/password.
+                    The API key used for authentication. Not needed if using an
+                    username/password. Supports the "{credential ...}" syntax.
                 example: fakekey
             start:
                 type: object
@@ -2180,6 +2243,12 @@ properties:
                     - start
                     - finish
                     - fail
+            verify_tls:
+                type: boolean
+                description: |
+                    Verify the TLS certificate of the push URL host. Defaults to
+                    true.
+                example: false
         description: |
             Configuration for a monitoring integration with Uptime Kuma using
             the Push monitor type.
@@ -2207,8 +2276,8 @@ properties:
             integration_key:
                 type: string
                 description: |
-                    PagerDuty integration key used to notify PagerDuty
-                    when a backup errors.
+                    PagerDuty integration key used to notify PagerDuty when a
+                    backup errors. Supports the "{credential ...}" syntax.
                 example: a177cad45bd374409f78906a810a3074
         description: |
             Configuration for a monitoring integration with PagerDuty. Create an
@@ -2382,3 +2451,25 @@ properties:
         description: |
             Configuration for integration with Linux LVM (Logical Volume
             Manager).
+    container:
+        type: object
+        additionalProperties: false
+        properties:
+            secrets_directory:
+                type: string
+                description: |
+                    Secrets directory to use instead of "/run/secrets".
+                example: /path/to/secrets
+        description: |
+            Configuration for integration with Docker or Podman secrets.
+    keepassxc:
+        type: object
+        additionalProperties: false
+        properties:
+            keepassxc_cli_command:
+                type: string
+                description: |
+                    Command to use instead of "keepassxc-cli".
+                example: /usr/local/bin/keepassxc-cli
+        description: |
+            Configuration for integration with the KeePassXC password manager.

borgmatic/config/validate.py

@@ -88,8 +88,9 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     '''
     Given the path to a config filename in YAML format, the path to a schema filename in a YAML
     rendition of JSON Schema format, a sequence of configuration file override strings in the form
-    of "option.suboption=value", return the parsed configuration as a data structure of nested dicts
-    and lists corresponding to the schema. Example return value:
+    of "option.suboption=value", and whether to resolve environment variables, return the parsed
+    configuration as a data structure of nested dicts and lists corresponding to the schema. Example
+    return value:
 
         {
             'source_directories': ['/home', '/etc'],
@@ -124,6 +125,7 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
         validator = jsonschema.Draft7Validator(schema)
     except AttributeError:  # pragma: no cover
         validator = jsonschema.Draft4Validator(schema)
+
     validation_errors = tuple(validator.iter_errors(config))
 
     if validation_errors:

borgmatic/execute.py

@@ -1,11 +1,12 @@
 import collections
 import enum
 import logging
-import os
 import select
 import subprocess
 import textwrap
 
+import borgmatic.logger
+
 logger = logging.getLogger(__name__)
 
 
@@ -241,6 +242,9 @@ def mask_command_secrets(full_command):
 MAX_LOGGED_COMMAND_LENGTH = 1000
 
 
+PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG = ('BORG_', 'PG', 'MARIADB_', 'MYSQL_')
+
+
 def log_command(full_command, input_file=None, output_file=None, environment=None):
     '''
     Log the given command (a sequence of command/argument strings), along with its input/output file
@@ -249,14 +253,21 @@ def log_command(full_command, input_file=None, output_file=None, environment=Non
     logger.debug(
         textwrap.shorten(
             ' '.join(
-                tuple(f'{key}=***' for key in (environment or {}).keys())
+                tuple(
+                    f'{key}=***'
+                    for key in (environment or {}).keys()
+                    if any(
+                        key.startswith(prefix)
+                        for prefix in PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG
+                    )
+                )
                 + mask_command_secrets(full_command)
             ),
             width=MAX_LOGGED_COMMAND_LENGTH,
             placeholder=' ...',
         )
-        + (f" < {getattr(input_file, 'name', '')}" if input_file else '')
-        + (f" > {getattr(output_file, 'name', '')}" if output_file else '')
+        + (f" < {getattr(input_file, 'name', input_file)}" if input_file else '')
+        + (f" > {getattr(output_file, 'name', output_file)}" if output_file else '')
     )
 
 
@@ -272,7 +283,7 @@ def execute_command(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
@@ -282,18 +293,17 @@ def execute_command(
     Execute the given command (a sequence of command/argument strings) and log its output at the
     given log level. If an open output file object is given, then write stdout to the file and only
     log stderr. If an open input file object is given, then read stdin from the file. If shell is
-    True, execute the command within a shell. If an extra environment dict is given, then use it to
-    augment the current environment, and pass the result into the command. If a working directory is
-    given, use that as the present working directory when running the command. If a Borg local path
-    is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
-    instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
-    dicts, then use that configuration to decide what's an error and what's a warning. If run to
-    completion is False, then return the process for the command without executing it to completion.
+    True, execute the command within a shell. If an environment variables dict is given, then pass
+    it into the command. If a working directory is given, use that as the present working directory
+    when running the command. If a Borg local path is given, and the command matches it (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning. If run to completion is False, then return the process for
+    the command without executing it to completion.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
@@ -305,52 +315,58 @@ def execute_command(
         shell=shell,
         env=environment,
         cwd=working_directory,
+        # Necessary for passing credentials via anonymous pipe.
+        close_fds=False,
     )
     if not run_to_completion:
         return process
 
-    log_outputs(
-        (process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        log_outputs(
+            (process,),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
 
 
 def execute_command_and_capture_output(
     full_command,
+    input_file=None,
     capture_stderr=False,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If capture stderr is True, then capture and return stderr in addition to
-    stdout. If shell is True, execute the command within a shell. If an extra environment dict is
-    given, then use it to augment the current environment, and pass the result into the command. If
-    a working directory is given, use that as the present working directory when running the
-    command. If a Borg local path is given, and the command matches it (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    execute the command within a shell. If an environment variables dict is given, then pass it into
+    the command. If a working directory is given, use that as the present working directory when
+    running the command. If a Borg local path is given, and the command matches it (regardless of
+    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, environment=extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, environment=environment)
     command = ' '.join(full_command) if shell else full_command
 
     try:
         output = subprocess.check_output(
             command,
+            stdin=input_file,
             stderr=subprocess.STDOUT if capture_stderr else None,
             shell=shell,
             env=environment,
             cwd=working_directory,
+            # Necessary for passing credentials via anonymous pipe.
+            close_fds=False,
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -370,7 +386,7 @@ def execute_command_with_processes(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
@@ -384,19 +400,17 @@ def execute_command_with_processes(
     If an open output file object is given, then write stdout to the file and only log stderr. But
     if output log level is None, instead suppress logging and return the captured output for (only)
     the given command. If an open input file object is given, then read stdin from the file. If
-    shell is True, execute the command within a shell. If an extra environment dict is given, then
-    use it to augment the current environment, and pass the result into the command. If a working
-    directory is given, use that as the present working directory when running the command. If a
-    Borg local path is given, then for any matching command or process (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    shell is True, execute the command within a shell. If an environment variables dict is given,
+    then pass it into the command. If a working directory is given, use that as the present working
+    directory when running the command. If a Borg local path is given, then for any matching command
+    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    configuration to decide what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
     upstream process.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
@@ -411,6 +425,8 @@ def execute_command_with_processes(
             shell=shell,
             env=environment,
             cwd=working_directory,
+            # Necessary for passing credentials via anonymous pipe.
+            close_fds=False,
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.
@@ -421,13 +437,14 @@ def execute_command_with_processes(
                 process.kill()
         raise
 
-    captured_outputs = log_outputs(
-        tuple(processes) + (command_process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        captured_outputs = log_outputs(
+            tuple(processes) + (command_process,),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
 
     if output_log_level is None:
         return captured_outputs.get(command_process)

borgmatic/hooks/command.py

@@ -12,7 +12,7 @@ logger = logging.getLogger(__name__)
 SOFT_FAIL_EXIT_CODE = 75
 
 
-def interpolate_context(config_filename, hook_description, command, context):
+def interpolate_context(hook_description, command, context):
     '''
     Given a config filename, a hook description, a single hook command, and a dict of context
     names/values, interpolate the values by "{name}" into the command and return the result.
@@ -22,7 +22,7 @@ def interpolate_context(config_filename, hook_description, command, context):
 
     for unsupported_variable in re.findall(r'{\w+}', command):
         logger.warning(
-            f"{config_filename}: Variable '{unsupported_variable}' is not supported in {hook_description} hook"
+            f"Variable '{unsupported_variable}' is not supported in {hook_description} hook"
         )
 
     return command
@@ -30,16 +30,18 @@ def interpolate_context(config_filename, hook_description, command, context):
 
 def make_environment(current_environment, sys_module=sys):
     '''
-    Given the existing system environment as a map from environment variable name to value, return
-    (in the same form) any extra environment variables that should be used when running command
-    hooks.
+    Given the existing system environment as a map from environment variable name to value, return a
+    copy of it, augmented with any extra environment variables that should be used when running
+    command hooks.
     '''
+    environment = dict(current_environment)
+
     # Detect whether we're running within a PyInstaller bundle. If so, set or clear LD_LIBRARY_PATH
     # based on the value of LD_LIBRARY_PATH_ORIG. This prevents library version information errors.
     if getattr(sys_module, 'frozen', False) and hasattr(sys_module, '_MEIPASS'):
-        return {'LD_LIBRARY_PATH': current_environment.get('LD_LIBRARY_PATH_ORIG', '')}
+        environment['LD_LIBRARY_PATH'] = environment.get('LD_LIBRARY_PATH_ORIG', '')
 
-    return {}
+    return environment
 
 
 def execute_hook(commands, umask, config_filename, description, dry_run, **context):
@@ -54,26 +56,24 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
     Raise subprocesses.CalledProcessError if an error occurs in a hook.
     '''
     if not commands:
-        logger.debug(f'{config_filename}: No commands to run for {description} hook')
+        logger.debug(f'No commands to run for {description} hook')
         return
 
     dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
 
     context['configuration_filename'] = config_filename
-    commands = [
-        interpolate_context(config_filename, description, command, context) for command in commands
-    ]
+    commands = [interpolate_context(description, command, context) for command in commands]
 
     if len(commands) == 1:
-        logger.info(f'{config_filename}: Running command for {description} hook{dry_run_label}')
+        logger.info(f'Running command for {description} hook{dry_run_label}')
     else:
         logger.info(
-            f'{config_filename}: Running {len(commands)} commands for {description} hook{dry_run_label}',
+            f'Running {len(commands)} commands for {description} hook{dry_run_label}',
         )
 
     if umask:
         parsed_umask = int(str(umask), 8)
-        logger.debug(f'{config_filename}: Set hook umask to {oct(parsed_umask)}')
+        logger.debug(f'Set hook umask to {oct(parsed_umask)}')
         original_umask = os.umask(parsed_umask)
     else:
         original_umask = None
@@ -87,14 +87,14 @@ def execute_hook(commands, umask, config_filename, description, dry_run, **conte
                 [command],
                 output_log_level=(logging.ERROR if description == 'on-error' else logging.WARNING),
                 shell=True,
-                extra_environment=make_environment(os.environ),
+                environment=make_environment(os.environ),
             )
     finally:
         if original_umask:
             os.umask(original_umask)
 
 
-def considered_soft_failure(config_filename, error):
+def considered_soft_failure(error):
     '''
     Given a configuration filename and an exception object, return whether the exception object
     represents a subprocess.CalledProcessError with a return code of SOFT_FAIL_EXIT_CODE. If so,
@@ -106,7 +106,7 @@ def considered_soft_failure(config_filename, error):
 
     if exit_code == SOFT_FAIL_EXIT_CODE:
         logger.info(
-            f'{config_filename}: Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining repository actions',
+            f'Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining repository actions',
         )
         return True
 

borgmatic/hooks/credential/container.py

@@ -0,0 +1,43 @@
+import logging
+import os
+import re
+
+logger = logging.getLogger(__name__)
+
+
+SECRET_NAME_PATTERN = re.compile(r'^\w+$')
+DEFAULT_SECRETS_DIRECTORY = '/run/secrets'
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a secret name to load, read the secret from the corresponding container secrets file
+    and return it.
+
+    Raise ValueError if the credential parameters is not one element, the secret name is invalid, or
+    the secret file cannot be read.
+    '''
+    try:
+        (secret_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid secret name: "{name}"')
+
+    if not SECRET_NAME_PATTERN.match(secret_name):
+        raise ValueError(f'Cannot load invalid secret name: "{secret_name}"')
+
+    try:
+        with open(
+            os.path.join(
+                config.get('working_directory', ''),
+                (hook_config or {}).get('secrets_directory', DEFAULT_SECRETS_DIRECTORY),
+                secret_name,
+            )
+        ) as secret_file:
+            return secret_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load secret "{secret_name}" from file: {error.filename}')

borgmatic/hooks/credential/file.py

@@ -0,0 +1,30 @@
+import logging
+import os
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential path to load, load the credential from file and return it.
+
+    Raise ValueError if the credential parameters is not one element or the secret file cannot be
+    read.
+    '''
+    try:
+        (credential_path,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential: "{name}"')
+
+    try:
+        with open(
+            os.path.join(config.get('working_directory', ''), credential_path)
+        ) as credential_file:
+            return credential_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load credential file: {error.filename}')

borgmatic/hooks/credential/keepassxc.py

@@ -0,0 +1,42 @@
+import logging
+import os
+import shlex
+
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a KeePassXC database path and an attribute name to load, run keepassxc-cli to fetch
+    the corresponidng KeePassXC credential and return it.
+
+    Raise ValueError if keepassxc-cli can't retrieve the credential.
+    '''
+    try:
+        (database_path, attribute_name) = credential_parameters
+    except ValueError:
+        path_and_name = ' '.join(credential_parameters)
+
+        raise ValueError(
+            f'Cannot load credential with invalid KeePassXC database path and attribute name: "{path_and_name}"'
+        )
+
+    if not os.path.exists(database_path):
+        raise ValueError(
+            f'Cannot load credential because KeePassXC database path does not exist: {database_path}'
+        )
+
+    return borgmatic.execute.execute_command_and_capture_output(
+        tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
+        + (
+            'show',
+            '--show-protected',
+            '--attributes',
+            'Password',
+            database_path,
+            attribute_name,
+        )
+    ).rstrip(os.linesep)

borgmatic/hooks/credential/parse.py

@@ -0,0 +1,124 @@
+import functools
+import re
+import shlex
+
+import borgmatic.hooks.dispatch
+
+IS_A_HOOK = False
+
+
+class Hash_adapter:
+    '''
+    A Hash_adapter instance wraps an unhashable object and pretends it's hashable. This is intended
+    for passing to a @functools.cache-decorated function to prevent it from complaining that an
+    argument is unhashable. It should only be used for arguments that you don't want to actually
+    impact the cache hashing, because Hash_adapter doesn't actually hash the object's contents.
+
+    Example usage:
+
+        @functools.cache
+        def func(a, b):
+            print(a, b.actual_value)
+            return a
+
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Calls func(), prints, and returns.
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Hits the cache and just returns the value.
+        func(5, Hash_adapter({5: 6, 7: 8}))  # Also uses cache, since the Hash_adapter is ignored.
+
+    In the above function, the "b" value is one that has been wrapped with Hash_adappter, and
+    therefore "b.actual_value" is necessary to access the original value.
+    '''
+
+    def __init__(self, actual_value):
+        self.actual_value = actual_value
+
+    def __eq__(self, other):
+        return True
+
+    def __hash__(self):
+        return 0
+
+
+UNHASHABLE_TYPES = (dict, list, set)
+
+
+def cache_ignoring_unhashable_arguments(function):
+    '''
+    A function decorator that caches calls to the decorated function but ignores any unhashable
+    arguments when performing cache lookups. This is intended to be a drop-in replacement for
+    functools.cache.
+
+    Example usage:
+
+        @cache_ignoring_unhashable_arguments
+        def func(a, b):
+            print(a, b)
+            return a
+
+        func(5, {1: 2, 3: 4})  # Calls func(), prints, and returns.
+        func(5, {1: 2, 3: 4})  # Hits the cache and just returns the value.
+        func(5, {5: 6, 7: 8})  # Also uses cache, since the unhashable value (the dict) is ignored.
+    '''
+
+    @functools.cache
+    def cached_function(*args, **kwargs):
+        return function(
+            *(arg.actual_value if isinstance(arg, Hash_adapter) else arg for arg in args),
+            **{
+                key: value.actual_value if isinstance(value, Hash_adapter) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    @functools.wraps(function)
+    def wrapper_function(*args, **kwargs):
+        return cached_function(
+            *(Hash_adapter(arg) if isinstance(arg, UNHASHABLE_TYPES) else arg for arg in args),
+            **{
+                key: Hash_adapter(value) if isinstance(value, UNHASHABLE_TYPES) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    wrapper_function.cache_clear = cached_function.cache_clear
+
+    return wrapper_function
+
+
+CREDENTIAL_PATTERN = re.compile(r'\{credential( +(?P<hook_and_parameters>.*))?\}')
+
+
+@cache_ignoring_unhashable_arguments
+def resolve_credential(value, config):
+    '''
+    Given a configuration value containing a string like "{credential hookname credentialname}" and
+    a configuration dict, resolve the credential by calling the relevant hook to get the actual
+    credential value. If the given value does not actually contain a credential tag, then return it
+    unchanged.
+
+    Cache the value (ignoring the config for purposes of caching), so repeated calls to this
+    function don't need to load the credential repeatedly.
+
+    Raise ValueError if the config could not be parsed or the credential could not be loaded.
+    '''
+    if value is None:
+        return value
+
+    matcher = CREDENTIAL_PATTERN.match(value)
+
+    if not matcher:
+        return value
+
+    hook_and_parameters = matcher.group('hook_and_parameters')
+
+    if not hook_and_parameters:
+        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
+
+    (hook_name, *credential_parameters) = shlex.split(hook_and_parameters)
+
+    if not credential_parameters:
+        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
+
+    return borgmatic.hooks.dispatch.call_hook(
+        'load_credential', config, hook_name, tuple(credential_parameters)
+    )

borgmatic/hooks/credential/systemd.py

@@ -0,0 +1,43 @@
+import logging
+import os
+import re
+
+logger = logging.getLogger(__name__)
+
+
+CREDENTIAL_NAME_PATTERN = re.compile(r'^\w+$')
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential name to load, read the credential from the corresponding systemd
+    credential file and return it.
+
+    Raise ValueError if the systemd CREDENTIALS_DIRECTORY environment variable is not set, the
+    credential name is invalid, or the credential file cannot be read.
+    '''
+    try:
+        (credential_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential name: "{name}"')
+
+    credentials_directory = os.environ.get('CREDENTIALS_DIRECTORY')
+
+    if not credentials_directory:
+        raise ValueError(
+            f'Cannot load credential "{credential_name}" because the systemd CREDENTIALS_DIRECTORY environment variable is not set'
+        )
+
+    if not CREDENTIAL_NAME_PATTERN.match(credential_name):
+        raise ValueError(f'Cannot load invalid credential name "{credential_name}"')
+
+    try:
+        with open(os.path.join(credentials_directory, credential_name)) as credential_file:
+            return credential_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load credential "{credential_name}" from file: {error.filename}')

borgmatic/hooks/data_source/bootstrap.py

@@ -10,7 +10,7 @@ import borgmatic.config.paths
 logger = logging.getLogger(__name__)
 
 
-def use_streaming(hook_config, config, log_prefix):  # pragma: no cover
+def use_streaming(hook_config, config):  # pragma: no cover
     '''
     Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
     '''
@@ -20,18 +20,17 @@ def use_streaming(hook_config, config, log_prefix):  # pragma: no cover
 def dump_data_sources(
     hook_config,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
     dry_run,
 ):
     '''
-    Given a bootstrap configuration dict, a configuration dict, a log prefix, the borgmatic
-    configuration file paths, the borgmatic runtime directory, the configured patterns, and whether
-    this is a dry run, create a borgmatic manifest file to store the paths of the configuration
-    files used to create the archive. But skip this if the bootstrap store_config_files option is
-    False or if this is a dry run.
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic configuration file
+    paths, the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    create a borgmatic manifest file to store the paths of the configuration files used to create
+    the archive. But skip this if the bootstrap store_config_files option is False or if this is a
+    dry run.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
     '''
@@ -56,19 +55,27 @@ def dump_data_sources(
             manifest_file,
         )
 
-    patterns.extend(borgmatic.borg.pattern.Pattern(config_path) for config_path in config_paths)
+    patterns.extend(
+        borgmatic.borg.pattern.Pattern(
+            config_path, source=borgmatic.borg.pattern.Pattern_source.HOOK
+        )
+        for config_path in config_paths
+    )
     patterns.append(
-        borgmatic.borg.pattern.Pattern(os.path.join(borgmatic_runtime_directory, 'bootstrap'))
+        borgmatic.borg.pattern.Pattern(
+            os.path.join(borgmatic_runtime_directory, 'bootstrap'),
+            source=borgmatic.borg.pattern.Pattern_source.HOOK,
+        )
     )
 
     return []
 
 
-def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
     '''
-    Given a bootstrap configuration dict, a configuration dict, a log prefix, the borgmatic runtime
-    directory, and whether this is a dry run, then remove the manifest file created above. If this
-    is a dry run, then don't actually remove anything.
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, then remove the manifest file created above. If this is a dry run,
+    then don't actually remove anything.
     '''
     dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
 
@@ -79,14 +86,12 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
         'bootstrap',
     )
     logger.debug(
-        f'{log_prefix}: Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}'
+        f'Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}'
     )
 
     for manifest_directory in glob.glob(manifest_glob):
         manifest_file_path = os.path.join(manifest_directory, 'manifest.json')
-        logger.debug(
-            f'{log_prefix}: Removing bootstrap manifest at {manifest_file_path}{dry_run_label}'
-        )
+        logger.debug(f'Removing bootstrap manifest at {manifest_file_path}{dry_run_label}')
 
         if dry_run:
             continue
@@ -103,7 +108,7 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
 
 
 def make_data_source_dump_patterns(
-    hook_config, config, log_prefix, borgmatic_runtime_directory, name=None
+    hook_config, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
     Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
@@ -115,7 +120,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,

borgmatic/hooks/data_source/btrfs.py

@@ -14,16 +14,16 @@ import borgmatic.hooks.data_source.snapshot
 logger = logging.getLogger(__name__)
 
 
-def use_streaming(hook_config, config, log_prefix):  # pragma: no cover
+def use_streaming(hook_config, config):  # pragma: no cover
     '''
     Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
     '''
     return False
 
 
-def get_filesystem_mount_points(findmnt_command):
+def get_subvolume_mount_points(findmnt_command):
     '''
-    Given a findmnt command to run, get all top-level Btrfs filesystem mount points.
+    Given a findmnt command to run, get all sorted Btrfs subvolume mount points.
     '''
     findmnt_output = borgmatic.execute.execute_command_and_capture_output(
         tuple(findmnt_command.split(' '))
@@ -37,7 +37,7 @@ def get_filesystem_mount_points(findmnt_command):
 
     try:
         return tuple(
-            filesystem['target'] for filesystem in json.loads(findmnt_output)['filesystems']
+            sorted(filesystem['target'] for filesystem in json.loads(findmnt_output)['filesystems'])
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
@@ -45,34 +45,6 @@ def get_filesystem_mount_points(findmnt_command):
         raise ValueError(f'Invalid {findmnt_command} output: Missing key "{error}"')
 
 
-def get_subvolumes_for_filesystem(btrfs_command, filesystem_mount_point):
-    '''
-    Given a Btrfs command to run and a Btrfs filesystem mount point, get the sorted subvolumes for
-    that filesystem. Include the filesystem itself.
-    '''
-    btrfs_output = borgmatic.execute.execute_command_and_capture_output(
-        tuple(btrfs_command.split(' '))
-        + (
-            'subvolume',
-            'list',
-            filesystem_mount_point,
-        )
-    )
-
-    if not filesystem_mount_point.strip():
-        return ()
-
-    return (filesystem_mount_point,) + tuple(
-        sorted(
-            subvolume_path
-            for line in btrfs_output.splitlines()
-            for subvolume_subpath in (line.rstrip().split(' ')[-1],)
-            for subvolume_path in (os.path.join(filesystem_mount_point, subvolume_subpath),)
-            if subvolume_subpath.strip()
-        )
-    )
-
-
 Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
 
 
@@ -82,30 +54,33 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     between the current Btrfs filesystem and subvolume mount points and the paths of any patterns.
     The idea is that these pattern paths represent the requested subvolumes to snapshot.
 
-    If patterns is None, then return all subvolumes, sorted by path.
+    Only include subvolumes that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
+    return all subvolumes instead, sorted by path.
 
     Return the result as a sequence of matching subvolume mount points.
     '''
     candidate_patterns = set(patterns or ())
     subvolumes = []
 
-    # For each filesystem mount point, find its subvolumes and match them against the given patterns
-    # to find the subvolumes to backup. And within this loop, sort the subvolumes from longest to
-    # shortest mount points, so longer mount points get a whack at the candidate pattern piñata
-    # before their parents do. (Patterns are consumed during this process, so no two subvolumes end
-    # up with the same contained patterns.)
-    for mount_point in get_filesystem_mount_points(findmnt_command):
+    # For each subvolume mount point, match it against the given patterns to find the subvolumes to
+    # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
+    # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
+    # this process, so no two subvolumes end up with the same contained patterns.)
+    for mount_point in reversed(get_subvolume_mount_points(findmnt_command)):
         subvolumes.extend(
-            Subvolume(subvolume_path, contained_patterns)
-            for subvolume_path in reversed(
-                get_subvolumes_for_filesystem(btrfs_command, mount_point)
-            )
+            Subvolume(mount_point, contained_patterns)
             for contained_patterns in (
                 borgmatic.hooks.data_source.snapshot.get_contained_patterns(
-                    subvolume_path, candidate_patterns
+                    mount_point, candidate_patterns
                 ),
             )
-            if patterns is None or contained_patterns
+            if patterns is None
+            or any(
+                pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for pattern in contained_patterns
+            )
         )
 
     return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))
@@ -151,8 +126,9 @@ def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
             subvolume_path.lstrip(os.path.sep),
             snapshot_directory,
         ),
-        borgmatic.borg.pattern.Pattern_type.EXCLUDE,
+        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
         borgmatic.borg.pattern.Pattern_style.FNMATCH,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -185,6 +161,7 @@ def make_borg_snapshot_pattern(subvolume_path, pattern):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -211,38 +188,37 @@ def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma:
 def dump_data_sources(
     hook_config,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
     dry_run,
 ):
     '''
-    Given a Btrfs configuration dict, a configuration dict, a log prefix, the borgmatic
-    configuration file paths, the borgmatic runtime directory, the configured patterns, and whether
-    this is a dry run, auto-detect and snapshot any Btrfs subvolume mount points listed in the given
-    patterns. Also update those patterns, replacing subvolume mount points with corresponding
-    snapshot directories so they get stored in the Borg archive instead. Use the log prefix in any
-    log entries.
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any Btrfs subvolume mount points listed in the given patterns. Also
+    update those patterns, replacing subvolume mount points with corresponding snapshot directories
+    so they get stored in the Borg archive instead.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
 
     If this is a dry run, then don't actually snapshot anything.
     '''
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
-    logger.info(f'{log_prefix}: Snapshotting Btrfs subvolumes{dry_run_label}')
+    logger.info(f'Snapshotting Btrfs subvolumes{dry_run_label}')
 
-    # Based on the configured patterns, determine Btrfs subvolumes to backup.
+    # Based on the configured patterns, determine Btrfs subvolumes to backup. Only consider those
+    # patterns that came from actual user configuration (as opposed to, say, other hooks).
     btrfs_command = hook_config.get('btrfs_command', 'btrfs')
     findmnt_command = hook_config.get('findmnt_command', 'findmnt')
     subvolumes = get_subvolumes(btrfs_command, findmnt_command, patterns)
 
     if not subvolumes:
-        logger.warning(f'{log_prefix}: No Btrfs subvolumes found to snapshot{dry_run_label}')
+        logger.warning(f'No Btrfs subvolumes found to snapshot{dry_run_label}')
 
     # Snapshot each subvolume, rewriting patterns to use their snapshot paths.
     for subvolume in subvolumes:
-        logger.debug(f'{log_prefix}: Creating Btrfs snapshot for {subvolume.path} subvolume')
+        logger.debug(f'Creating Btrfs snapshot for {subvolume.path} subvolume')
 
         snapshot_path = make_snapshot_path(subvolume.path)
 
@@ -280,12 +256,11 @@ def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
     )
 
 
-def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
     '''
-    Given a Btrfs configuration dict, a configuration dict, a log prefix, the borgmatic runtime
-    directory, and whether this is a dry run, delete any Btrfs snapshots created by borgmatic. Use
-    the log prefix in any log entries. If this is a dry run or Btrfs isn't configured in borgmatic's
-    configuration, then don't actually remove anything.
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, delete any Btrfs snapshots created by borgmatic. If this is a dry run
+    or Btrfs isn't configured in borgmatic's configuration, then don't actually remove anything.
     '''
     if hook_config is None:
         return
@@ -298,10 +273,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
     try:
         all_subvolumes = get_subvolumes(btrfs_command, findmnt_command)
     except FileNotFoundError as error:
-        logger.debug(f'{log_prefix}: Could not find "{error.filename}" command')
+        logger.debug(f'Could not find "{error.filename}" command')
         return
     except subprocess.CalledProcessError as error:
-        logger.debug(f'{log_prefix}: {error}')
+        logger.debug(error)
         return
 
     # Reversing the sorted subvolumes ensures that we remove longer mount point paths of child
@@ -313,14 +288,14 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
         )
 
         logger.debug(
-            f'{log_prefix}: Looking for snapshots to remove in {subvolume_snapshots_glob}{dry_run_label}'
+            f'Looking for snapshots to remove in {subvolume_snapshots_glob}{dry_run_label}'
         )
 
         for snapshot_path in glob.glob(subvolume_snapshots_glob):
             if not os.path.isdir(snapshot_path):
                 continue
 
-            logger.debug(f'{log_prefix}: Deleting Btrfs snapshot {snapshot_path}{dry_run_label}')
+            logger.debug(f'Deleting Btrfs snapshot {snapshot_path}{dry_run_label}')
 
             if dry_run:
                 continue
@@ -328,19 +303,22 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
             try:
                 delete_snapshot(btrfs_command, snapshot_path)
             except FileNotFoundError:
-                logger.debug(f'{log_prefix}: Could not find "{btrfs_command}" command')
+                logger.debug(f'Could not find "{btrfs_command}" command')
                 return
             except subprocess.CalledProcessError as error:
-                logger.debug(f'{log_prefix}: {error}')
+                logger.debug(error)
                 return
 
-            # Strip off the subvolume path from the end of the snapshot path and then delete the
-            # resulting directory.
-            shutil.rmtree(snapshot_path.rsplit(subvolume.path, 1)[0])
+            # Remove the snapshot parent directory if it still exists. (It might not exist if the
+            # snapshot was for "/".)
+            snapshot_parent_dir = snapshot_path.rsplit(subvolume.path, 1)[0]
+
+            if os.path.isdir(snapshot_parent_dir):
+                shutil.rmtree(snapshot_parent_dir)
 
 
 def make_data_source_dump_patterns(
-    hook_config, config, log_prefix, borgmatic_runtime_directory, name=None
+    hook_config, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".
@@ -351,7 +329,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,

borgmatic/hooks/data_source/dump.py

@@ -46,14 +46,14 @@ def create_named_pipe_for_dump(dump_path):
     os.mkfifo(dump_path, mode=0o600)
 
 
-def remove_data_source_dumps(dump_path, data_source_type_name, log_prefix, dry_run):
+def remove_data_source_dumps(dump_path, data_source_type_name, dry_run):
     '''
     Remove all data source dumps in the given dump directory path (including the directory itself).
     If this is a dry run, then don't actually remove anything.
     '''
     dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
 
-    logger.debug(f'{log_prefix}: Removing {data_source_type_name} data source dumps{dry_run_label}')
+    logger.debug(f'Removing {data_source_type_name} data source dumps{dry_run_label}')
 
     if dry_run:
         return

borgmatic/hooks/data_source/lvm.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import hashlib
 import json
 import logging
 import os
@@ -14,7 +15,7 @@ import borgmatic.hooks.data_source.snapshot
 logger = logging.getLogger(__name__)
 
 
-def use_streaming(hook_config, config, log_prefix):  # pragma: no cover
+def use_streaming(hook_config, config):  # pragma: no cover
     '''
     Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
     '''
@@ -33,7 +34,9 @@ def get_logical_volumes(lsblk_command, patterns=None):
     between the current LVM logical volume mount points and the paths of any patterns. The idea is
     that these pattern paths represent the requested logical volumes to snapshot.
 
-    If patterns is None, include all logical volume mounts points, not just those in patterns.
+    Only include logical volumes that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, include
+    all logical volume mounts points instead, not just those in patterns.
 
     Return the result as a sequence of Logical_volume instances.
     '''
@@ -72,7 +75,12 @@ def get_logical_volumes(lsblk_command, patterns=None):
                     device['mountpoint'], candidate_patterns
                 ),
             )
-            if not patterns or contained_patterns
+            if not patterns
+            or any(
+                pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for pattern in contained_patterns
+            )
         )
     except KeyError as error:
         raise ValueError(f'Invalid {lsblk_command} output: Missing key "{error}"')
@@ -124,10 +132,14 @@ def mount_snapshot(mount_command, snapshot_device, snapshot_mount_path):  # prag
     )
 
 
-def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+MOUNT_POINT_HASH_LENGTH = 10
+
+
+def make_borg_snapshot_pattern(pattern, logical_volume, normalized_runtime_directory):
     '''
-    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
-    path rewritten to be in a snapshot directory based on the given runtime directory.
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance and a Logical_volume
+    containing it, return a new Pattern with its path rewritten to be in a snapshot directory based
+    on both the given runtime directory and the given Logical_volume's mount point.
 
     Move any initial caret in a regular expression pattern path to the beginning, so as not to break
     the regular expression.
@@ -142,6 +154,13 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
     rewritten_path = initial_caret + os.path.join(
         normalized_runtime_directory,
         'lvm_snapshots',
+        # Including this hash prevents conflicts between snapshot patterns for different logical
+        # volumes. For instance, without this, snapshotting a logical volume at /var and another at
+        # /var/spool would result in overlapping snapshot patterns and therefore colliding mount
+        # attempts.
+        hashlib.shake_256(logical_volume.mount_point.encode('utf-8')).hexdigest(
+            MOUNT_POINT_HASH_LENGTH
+        ),
         '.',  # Borg 1.4+ "slashdot" hack.
         # Included so that the source directory ends up in the Borg archive at its "original" path.
         pattern.path.lstrip('^').lstrip(os.path.sep),
@@ -152,6 +171,7 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
@@ -161,28 +181,27 @@ DEFAULT_SNAPSHOT_SIZE = '10%ORIGIN'
 def dump_data_sources(
     hook_config,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
     dry_run,
 ):
     '''
-    Given an LVM configuration dict, a configuration dict, a log prefix, the borgmatic configuration
-    file paths, the borgmatic runtime directory, the configured patterns, and whether this is a dry
-    run, auto-detect and snapshot any LVM logical volume mount points listed in the given patterns.
-    Also update those patterns, replacing logical volume mount points with corresponding snapshot
-    directories so they get stored in the Borg archive instead. Use the log prefix in any log
-    entries.
+    Given an LVM configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any LVM logical volume mount points listed in the given patterns. Also
+    update those patterns, replacing logical volume mount points with corresponding snapshot
+    directories so they get stored in the Borg archive instead.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
 
     If this is a dry run, then don't actually snapshot anything.
     '''
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
-    logger.info(f'{log_prefix}: Snapshotting LVM logical volumes{dry_run_label}')
+    logger.info(f'Snapshotting LVM logical volumes{dry_run_label}')
 
-    # List logical volumes to get their mount points.
+    # List logical volumes to get their mount points, but only consider those patterns that came
+    # from actual user configuration (as opposed to, say, other hooks).
     lsblk_command = hook_config.get('lsblk_command', 'lsblk')
     requested_logical_volumes = get_logical_volumes(lsblk_command, patterns)
 
@@ -191,12 +210,12 @@ def dump_data_sources(
     normalized_runtime_directory = os.path.normpath(borgmatic_runtime_directory)
 
     if not requested_logical_volumes:
-        logger.warning(f'{log_prefix}: No LVM logical volumes found to snapshot{dry_run_label}')
+        logger.warning(f'No LVM logical volumes found to snapshot{dry_run_label}')
 
     for logical_volume in requested_logical_volumes:
         snapshot_name = f'{logical_volume.name}_{snapshot_suffix}'
         logger.debug(
-            f'{log_prefix}: Creating LVM snapshot {snapshot_name} of {logical_volume.mount_point}{dry_run_label}'
+            f'Creating LVM snapshot {snapshot_name} of {logical_volume.mount_point}{dry_run_label}'
         )
 
         if not dry_run:
@@ -220,11 +239,14 @@ def dump_data_sources(
         snapshot_mount_path = os.path.join(
             normalized_runtime_directory,
             'lvm_snapshots',
+            hashlib.shake_256(logical_volume.mount_point.encode('utf-8')).hexdigest(
+                MOUNT_POINT_HASH_LENGTH
+            ),
             logical_volume.mount_point.lstrip(os.path.sep),
         )
 
         logger.debug(
-            f'{log_prefix}: Mounting LVM snapshot {snapshot_name} at {snapshot_mount_path}{dry_run_label}'
+            f'Mounting LVM snapshot {snapshot_name} at {snapshot_mount_path}{dry_run_label}'
         )
 
         if dry_run:
@@ -235,7 +257,9 @@ def dump_data_sources(
         )
 
         for pattern in logical_volume.contained_patterns:
-            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+            snapshot_pattern = make_borg_snapshot_pattern(
+                pattern, logical_volume, normalized_runtime_directory
+            )
 
             # Attempt to update the pattern in place, since pattern order matters to Borg.
             try:
@@ -312,12 +336,12 @@ def get_snapshots(lvs_command, snapshot_name=None):
         raise ValueError(f'Invalid {lvs_command} output: Missing key "{error}"')
 
 
-def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
     '''
-    Given an LVM configuration dict, a configuration dict, a log prefix, the borgmatic runtime
-    directory, and whether this is a dry run, unmount and delete any LVM snapshots created by
-    borgmatic. Use the log prefix in any log entries. If this is a dry run or LVM isn't configured
-    in borgmatic's configuration, then don't actually remove anything.
+    Given an LVM configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, unmount and delete any LVM snapshots created by borgmatic. If this is
+    a dry run or LVM isn't configured in borgmatic's configuration, then don't actually remove
+    anything.
     '''
     if hook_config is None:
         return
@@ -328,10 +352,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
     try:
         logical_volumes = get_logical_volumes(hook_config.get('lsblk_command', 'lsblk'))
     except FileNotFoundError as error:
-        logger.debug(f'{log_prefix}: Could not find "{error.filename}" command')
+        logger.debug(f'Could not find "{error.filename}" command')
         return
     except subprocess.CalledProcessError as error:
-        logger.debug(f'{log_prefix}: {error}')
+        logger.debug(error)
         return
 
     snapshots_glob = os.path.join(
@@ -339,10 +363,9 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
             os.path.normpath(borgmatic_runtime_directory),
         ),
         'lvm_snapshots',
+        '*',
     )
-    logger.debug(
-        f'{log_prefix}: Looking for snapshots to remove in {snapshots_glob}{dry_run_label}'
-    )
+    logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
     umount_command = hook_config.get('umount_command', 'umount')
 
     for snapshots_directory in glob.glob(snapshots_glob):
@@ -353,7 +376,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
             snapshot_mount_path = os.path.join(
                 snapshots_directory, logical_volume.mount_point.lstrip(os.path.sep)
             )
-            if not os.path.isdir(snapshot_mount_path):
+
+            # If the snapshot mount path is empty, this is probably just a "shadow" of a nested
+            # logical volume and therefore there's nothing to unmount.
+            if not os.path.isdir(snapshot_mount_path) or not os.listdir(snapshot_mount_path):
                 continue
 
             # This might fail if the directory is already mounted, but we swallow errors here since
@@ -366,9 +392,7 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
                 if not os.path.isdir(snapshot_mount_path):
                     continue
 
-            logger.debug(
-                f'{log_prefix}: Unmounting LVM snapshot at {snapshot_mount_path}{dry_run_label}'
-            )
+            logger.debug(f'Unmounting LVM snapshot at {snapshot_mount_path}{dry_run_label}')
 
             if dry_run:
                 continue
@@ -376,11 +400,11 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
             try:
                 unmount_snapshot(umount_command, snapshot_mount_path)
             except FileNotFoundError:
-                logger.debug(f'{log_prefix}: Could not find "{umount_command}" command')
+                logger.debug(f'Could not find "{umount_command}" command')
                 return
             except subprocess.CalledProcessError as error:
-                logger.debug(f'{log_prefix}: {error}')
-                return
+                logger.debug(error)
+                continue
 
         if not dry_run:
             shutil.rmtree(snapshots_directory)
@@ -391,10 +415,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
     try:
         snapshots = get_snapshots(hook_config.get('lvs_command', 'lvs'))
     except FileNotFoundError as error:
-        logger.debug(f'{log_prefix}: Could not find "{error.filename}" command')
+        logger.debug(f'Could not find "{error.filename}" command')
         return
     except subprocess.CalledProcessError as error:
-        logger.debug(f'{log_prefix}: {error}')
+        logger.debug(error)
         return
 
     for snapshot in snapshots:
@@ -402,14 +426,14 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
         if not snapshot.name.split('_')[-1].startswith(BORGMATIC_SNAPSHOT_PREFIX):
             continue
 
-        logger.debug(f'{log_prefix}: Deleting LVM snapshot {snapshot.name}{dry_run_label}')
+        logger.debug(f'Deleting LVM snapshot {snapshot.name}{dry_run_label}')
 
         if not dry_run:
             remove_snapshot(lvremove_command, snapshot.device_path)
 
 
 def make_data_source_dump_patterns(
-    hook_config, config, log_prefix, borgmatic_runtime_directory, name=None
+    hook_config, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".
@@ -420,7 +444,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,

borgmatic/hooks/data_source/mariadb.py

@@ -1,10 +1,12 @@
 import copy
 import logging
 import os
+import re
 import shlex
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.credential.parse
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -22,14 +24,90 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'mariadb_databases')
 
 
-SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
+DEFAULTS_EXTRA_FILE_FLAG_PATTERN = re.compile('^--defaults-extra-file=(?P<filename>.*)$')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def parse_extra_options(extra_options):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all", query for the names of databases on the configured host and return them,
-    excluding any system databases that will cause problems during restore.
+    Given an extra options string, split the options into a tuple and return it. Additionally, if
+    the first option is "--defaults-extra-file=...", then remove it from the options and return the
+    filename.
+
+    So the return value is a tuple of: (parsed options, defaults extra filename).
+
+    The intent is to support downstream merging of multiple "--defaults-extra-file"s, as
+    MariaDB/MySQL only allows one at a time.
+    '''
+    split_extra_options = tuple(shlex.split(extra_options)) if extra_options else ()
+
+    if not split_extra_options:
+        return ((), None)
+
+    match = DEFAULTS_EXTRA_FILE_FLAG_PATTERN.match(split_extra_options[0])
+
+    if not match:
+        return (split_extra_options, None)
+
+    return (split_extra_options[1:], match.group('filename'))
+
+
+def make_defaults_file_options(username=None, password=None, defaults_extra_filename=None):
+    '''
+    Given a database username and/or password, write it to an anonymous pipe and return the flags
+    for passing that file descriptor to an executed command. The idea is that this is a more secure
+    way to transmit credentials to a database client than using an environment variable.
+
+    If no username or password are given, then return the options for the given defaults extra
+    filename (if any). But if there is a username and/or password and a defaults extra filename is
+    given, then "!include" it from the generated file, effectively allowing multiple defaults extra
+    files.
+
+    Do not use the returned value for multiple different command invocations. That will not work
+    because each pipe is "used up" once read.
+    '''
+    values = '\n'.join(
+        (
+            (f'user={username}' if username is not None else ''),
+            (f'password={password}' if password is not None else ''),
+        )
+    ).strip()
+
+    if not values:
+        if defaults_extra_filename:
+            return (f'--defaults-extra-file={defaults_extra_filename}',)
+
+        return ()
+
+    fields_message = ' and '.join(
+        field_name
+        for field_name in (
+            (f'username ({username})' if username is not None else None),
+            ('password' if password is not None else None),
+        )
+        if field_name is not None
+    )
+    include_message = f' (including {defaults_extra_filename})' if defaults_extra_filename else ''
+    logger.debug(f'Writing database {fields_message} to defaults extra file pipe{include_message}')
+
+    include = f'!include {defaults_extra_filename}\n' if defaults_extra_filename else ''
+
+    read_file_descriptor, write_file_descriptor = os.pipe()
+    os.write(write_file_descriptor, f'{include}[client]\n{values}'.encode('utf-8'))
+    os.close(write_file_descriptor)
+
+    # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the database
+    # client child process to inherit the file descriptor.
+    os.set_inheritable(read_file_descriptor, True)
+
+    return (f'--defaults-extra-file=/dev/fd/{read_file_descriptor}',)
+
+
+def database_names_to_dump(database, config, username, password, environment, dry_run):
+    '''
+    Given a requested database config, a configuration dict, a database username and password, an
+    environment dict, and whether this is a dry run, return the corresponding sequence of database
+    names to dump. In the case of "all", query for the names of databases on the configured host and
+    return them, excluding any system databases that will cause problems during restore.
     '''
     if database['name'] != 'all':
         return (database['name'],)
@@ -39,20 +117,23 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
     mariadb_show_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
     show_command = (
         mariadb_show_command
-        + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (('--user', database['username']) if 'username' in database else ())
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
-    logger.debug(f'{log_prefix}: Querying for "all" MariaDB databases to dump')
-    show_output = execute_command_and_capture_output(
-        show_command, extra_environment=extra_environment
-    )
+
+    logger.debug('Querying for "all" MariaDB databases to dump')
+
+    show_output = execute_command_and_capture_output(show_command, environment=environment)
 
     return tuple(
         show_name
@@ -61,13 +142,23 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
     )
 
 
+SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
+
+
 def execute_dump_command(
-    database, log_prefix, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database,
+    config,
+    username,
+    password,
+    dump_path,
+    database_names,
+    environment,
+    dry_run,
+    dry_run_label,
 ):
     '''
     Kick off a dump for the given MariaDB database (provided as a configuration dict) to a named
-    pipe constructed from the given dump path and database name. Use the given log prefix in any
-    log entries.
+    pipe constructed from the given dump path and database name.
 
     Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
     this is a dry run, then don't actually dump anything and return None.
@@ -82,7 +173,7 @@ def execute_dump_command(
 
     if os.path.exists(dump_filename):
         logger.warning(
-            f'{log_prefix}: Skipping duplicate dump of MariaDB database "{database_name}" to {dump_filename}'
+            f'Skipping duplicate dump of MariaDB database "{database_name}" to {dump_filename}'
         )
         return None
 
@@ -90,22 +181,23 @@ def execute_dump_command(
         shlex.quote(part)
         for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
     dump_command = (
         mariadb_dump_command
-        + (tuple(database['options'].split(' ')) if 'options' in database else ())
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (('--user', database['username']) if 'username' in database else ())
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
         + database_names
         + ('--result-file', dump_filename)
     )
 
-    logger.debug(
-        f'{log_prefix}: Dumping MariaDB database "{database_name}" to {dump_filename}{dry_run_label}'
-    )
+    logger.debug(f'Dumping MariaDB database "{database_name}" to {dump_filename}{dry_run_label}')
     if dry_run:
         return None
 
@@ -113,19 +205,19 @@ def execute_dump_command(
 
     return execute_command(
         dump_command,
-        extra_environment=extra_environment,
+        environment=environment,
         run_to_completion=False,
     )
 
 
-def get_default_port(databases, config, log_prefix):  # pragma: no cover
+def get_default_port(databases, config):  # pragma: no cover
     return 3306
 
 
-def use_streaming(databases, config, log_prefix):
+def use_streaming(databases, config):
     '''
-    Given a sequence of MariaDB database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MariaDB database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
@@ -133,7 +225,6 @@ def use_streaming(databases, config, log_prefix):
 def dump_data_sources(
     databases,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
@@ -142,8 +233,7 @@ def dump_data_sources(
     '''
     Dump the given MariaDB databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@@ -153,13 +243,19 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping MariaDB databases{dry_run_label}')
+    logger.info(f'Dumping MariaDB databases{dry_run_label}')
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('password'), config
+        )
+        environment = dict(os.environ)
         dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
+            database, config, username, password, environment, dry_run
         )
 
         if not dump_database_names:
@@ -175,10 +271,12 @@ def dump_data_sources(
                 processes.append(
                     execute_dump_command(
                         renamed_database,
-                        log_prefix,
+                        config,
+                        username,
+                        password,
                         dump_path,
                         (dump_name,),
-                        extra_environment,
+                        environment,
                         dry_run,
                         dry_run_label,
                     )
@@ -187,10 +285,12 @@ def dump_data_sources(
             processes.append(
                 execute_dump_command(
                     database,
-                    log_prefix,
+                    config,
+                    username,
+                    password,
                     dump_path,
                     dump_database_names,
-                    extra_environment,
+                    environment,
                     dry_run,
                     dry_run_label,
                 )
@@ -199,7 +299,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mariadb_databases')
+                os.path.join(borgmatic_runtime_directory, 'mariadb_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -207,25 +308,23 @@ def dump_data_sources(
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic_runtime_directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic_runtime_directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MariaDB', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MariaDB', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -243,7 +342,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -252,9 +350,9 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     hostname = connection_params['hostname'] or data_source.get(
@@ -263,32 +361,40 @@ def restore_data_source_dump(
     port = str(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
-    username = connection_params['username'] or data_source.get(
-        'restore_username', data_source.get('username')
+    tls = data_source.get('restore_tls', data_source.get('tls'))
+    username = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
-    password = connection_params['password'] or data_source.get(
-        'restore_password', data_source.get('password')
+    password = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['password']
+            or data_source.get('restore_password', data_source.get('password'))
+        ),
+        config,
     )
 
     mariadb_restore_command = tuple(
         shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
     )
+    extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
     restore_command = (
         mariadb_restore_command
+        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + extra_options
         + ('--batch',)
-        + (
-            tuple(data_source['restore_options'].split(' '))
-            if 'restore_options' in data_source
-            else ()
-        )
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
-        + (('--user', username) if username else ())
+        + (('--ssl',) if tls is True else ())
+        + (('--skip-ssl',) if tls is False else ())
     )
-    extra_environment = {'MYSQL_PWD': password} if password else None
+    environment = dict(os.environ)
 
-    logger.debug(f"{log_prefix}: Restoring MariaDB database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 
@@ -299,5 +405,5 @@ def restore_data_source_dump(
         [extract_process],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout,
-        extra_environment=extra_environment,
+        environment=environment,
     )

borgmatic/hooks/data_source/mongodb.py

@@ -4,6 +4,7 @@ import shlex
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.credential.parse
 from borgmatic.execute import execute_command, execute_command_with_processes
 from borgmatic.hooks.data_source import dump
 
@@ -17,14 +18,14 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'mongodb_databases')
 
 
-def get_default_port(databases, config, log_prefix):  # pragma: no cover
+def get_default_port(databases, config):  # pragma: no cover
     return 27017
 
 
-def use_streaming(databases, config, log_prefix):
+def use_streaming(databases, config):
     '''
-    Given a sequence of MongoDB database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MongoDB database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(database.get('format') != 'directory' for database in databases)
 
@@ -32,7 +33,6 @@ def use_streaming(databases, config, log_prefix):
 def dump_data_sources(
     databases,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
@@ -41,8 +41,7 @@ def dump_data_sources(
     '''
     Dump the given MongoDB databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the borgmatic
-    runtime directory to construct the destination path (used for the directory format and the given
-    log prefix in any log entries.
+    runtime directory to construct the destination path (used for the directory format.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@@ -51,7 +50,7 @@ def dump_data_sources(
     '''
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
 
-    logger.info(f'{log_prefix}: Dumping MongoDB databases{dry_run_label}')
+    logger.info(f'Dumping MongoDB databases{dry_run_label}')
 
     processes = []
     for database in databases:
@@ -65,12 +64,12 @@ def dump_data_sources(
         dump_format = database.get('format', 'archive')
 
         logger.debug(
-            f'{log_prefix}: Dumping MongoDB database {name} to {dump_filename}{dry_run_label}',
+            f'Dumping MongoDB database {name} to {dump_filename}{dry_run_label}',
         )
         if dry_run:
             continue
 
-        command = build_dump_command(database, dump_filename, dump_format)
+        command = build_dump_command(database, config, dump_filename, dump_format)
 
         if dump_format == 'directory':
             dump.create_parent_directory_for_dump(dump_filename)
@@ -82,26 +81,62 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mongodb_databases')
+                os.path.join(borgmatic_runtime_directory, 'mongodb_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
     return processes
 
 
-def build_dump_command(database, dump_filename, dump_format):
+def make_password_config_file(password):
+    '''
+    Given a database password, write it as a MongoDB configuration file to an anonymous pipe and
+    return its filename. The idea is that this is a more secure way to transmit a password to
+    MongoDB than providing it directly on the command-line.
+
+    Do not use the returned value for multiple different command invocations. That will not work
+    because each pipe is "used up" once read.
+    '''
+    logger.debug('Writing MongoDB password to configuration file pipe')
+
+    read_file_descriptor, write_file_descriptor = os.pipe()
+    os.write(write_file_descriptor, f'password: {password}'.encode('utf-8'))
+    os.close(write_file_descriptor)
+
+    # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the database
+    # client child process to inherit the file descriptor.
+    os.set_inheritable(read_file_descriptor, True)
+
+    return f'/dev/fd/{read_file_descriptor}'
+
+
+def build_dump_command(database, config, dump_filename, dump_format):
     '''
     Return the mongodump command from a single database configuration.
     '''
     all_databases = database['name'] == 'all'
 
+    password = borgmatic.hooks.credential.parse.resolve_credential(database.get('password'), config)
+
     return (
         ('mongodump',)
         + (('--out', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
         + (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
         + (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
-        + (('--username', shlex.quote(database['username'])) if 'username' in database else ())
-        + (('--password', shlex.quote(database['password'])) if 'password' in database else ())
+        + (
+            (
+                '--username',
+                shlex.quote(
+                    borgmatic.hooks.credential.parse.resolve_credential(
+                        database['username'], config
+                    )
+                ),
+            )
+            if 'username' in database
+            else ()
+        )
+        + (('--config', make_password_config_file(password)) if password else ())
         + (
             ('--authenticationDatabase', shlex.quote(database['authentication_database']))
             if 'authentication_database' in database
@@ -118,25 +153,23 @@ def build_dump_command(database, dump_filename, dump_format):
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic_runtime_directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic_runtime_directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MongoDB', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MongoDB', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -154,7 +187,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -164,9 +196,9 @@ def restore_data_source_dump(
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
     configuration dict, but the given hook configuration is ignored. The given configuration dict is
-    used to construct the destination path, and the given log prefix is used for any log entries. If
-    this is a dry run, then don't actually restore anything. Trigger the given active extract
-    process (an instance of subprocess.Popen) to produce output to consume.
+    used to construct the destination path. If this is a dry run, then don't actually restore
+    anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
+    output to consume.
 
     If the extract process is None, then restore the dump from the filesystem rather than from an
     extract stream.
@@ -178,10 +210,10 @@ def restore_data_source_dump(
         data_source.get('hostname'),
     )
     restore_command = build_restore_command(
-        extract_process, data_source, dump_filename, connection_params
+        extract_process, data_source, config, dump_filename, connection_params
     )
 
-    logger.debug(f"{log_prefix}: Restoring MongoDB database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MongoDB database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 
@@ -195,7 +227,7 @@ def restore_data_source_dump(
     )
 
 
-def build_restore_command(extract_process, database, dump_filename, connection_params):
+def build_restore_command(extract_process, database, config, dump_filename, connection_params):
     '''
     Return the mongorestore command from a single database configuration.
     '''
@@ -203,11 +235,19 @@ def build_restore_command(extract_process, database, dump_filename, connection_p
         'restore_hostname', database.get('hostname')
     )
     port = str(connection_params['port'] or database.get('restore_port', database.get('port', '')))
-    username = connection_params['username'] or database.get(
-        'restore_username', database.get('username')
+    username = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['username']
+            or database.get('restore_username', database.get('username'))
+        ),
+        config,
     )
-    password = connection_params['password'] or database.get(
-        'restore_password', database.get('password')
+    password = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['password']
+            or database.get('restore_password', database.get('password'))
+        ),
+        config,
     )
 
     command = ['mongorestore']
@@ -224,7 +264,7 @@ def build_restore_command(extract_process, database, dump_filename, connection_p
     if username:
         command.extend(('--username', username))
     if password:
-        command.extend(('--password', password))
+        command.extend(('--config', make_password_config_file(password)))
     if 'authentication_database' in database:
         command.extend(('--authenticationDatabase', database['authentication_database']))
     if 'restore_options' in database:

borgmatic/hooks/data_source/mysql.py

@@ -5,6 +5,8 @@ import shlex
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.mariadb
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -25,11 +27,12 @@ def make_dump_path(base_directory):  # pragma: no cover
 SYSTEM_DATABASE_NAMES = ('information_schema', 'mysql', 'performance_schema', 'sys')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def database_names_to_dump(database, config, username, password, environment, dry_run):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all", query for the names of databases on the configured host and return them,
-    excluding any system databases that will cause problems during restore.
+    Given a requested database config, a configuration dict, a database username and password, an
+    environment dict, and whether this is a dry run, return the corresponding sequence of database
+    names to dump. In the case of "all", query for the names of databases on the configured host and
+    return them, excluding any system databases that will cause problems during restore.
     '''
     if database['name'] != 'all':
         return (database['name'],)
@@ -39,20 +42,27 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
     mysql_show_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mysql_command') or 'mysql')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('list_options'))
+    )
     show_command = (
         mysql_show_command
-        + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
+        )
+        + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (('--user', database['username']) if 'username' in database else ())
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
         + ('--execute', 'show schemas')
     )
-    logger.debug(f'{log_prefix}: Querying for "all" MySQL databases to dump')
-    show_output = execute_command_and_capture_output(
-        show_command, extra_environment=extra_environment
-    )
+
+    logger.debug('Querying for "all" MySQL databases to dump')
+
+    show_output = execute_command_and_capture_output(show_command, environment=environment)
 
     return tuple(
         show_name
@@ -62,12 +72,19 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
 
 
 def execute_dump_command(
-    database, log_prefix, dump_path, database_names, extra_environment, dry_run, dry_run_label
+    database,
+    config,
+    username,
+    password,
+    dump_path,
+    database_names,
+    environment,
+    dry_run,
+    dry_run_label,
 ):
     '''
     Kick off a dump for the given MySQL/MariaDB database (provided as a configuration dict) to a
-    named pipe constructed from the given dump path and database name. Use the given log prefix in
-    any log entries.
+    named pipe constructed from the given dump path and database name.
 
     Return a subprocess.Popen instance for the dump process ready to spew to a named pipe. But if
     this is a dry run, then don't actually dump anything and return None.
@@ -82,29 +99,34 @@ def execute_dump_command(
 
     if os.path.exists(dump_filename):
         logger.warning(
-            f'{log_prefix}: Skipping duplicate dump of MySQL database "{database_name}" to {dump_filename}'
+            f'Skipping duplicate dump of MySQL database "{database_name}" to {dump_filename}'
         )
         return None
 
     mysql_dump_command = tuple(
         shlex.quote(part) for part in shlex.split(database.get('mysql_dump_command') or 'mysqldump')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('options'))
+    )
     dump_command = (
         mysql_dump_command
-        + (tuple(database['options'].split(' ')) if 'options' in database else ())
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
+        )
+        + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
-        + (('--user', database['username']) if 'username' in database else ())
+        + (('--ssl',) if database.get('tls') is True else ())
+        + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
         + database_names
         + ('--result-file', dump_filename)
     )
 
-    logger.debug(
-        f'{log_prefix}: Dumping MySQL database "{database_name}" to {dump_filename}{dry_run_label}'
-    )
+    logger.debug(f'Dumping MySQL database "{database_name}" to {dump_filename}{dry_run_label}')
     if dry_run:
         return None
 
@@ -112,19 +134,19 @@ def execute_dump_command(
 
     return execute_command(
         dump_command,
-        extra_environment=extra_environment,
+        environment=environment,
         run_to_completion=False,
     )
 
 
-def get_default_port(databases, config, log_prefix):  # pragma: no cover
+def get_default_port(databases, config):  # pragma: no cover
     return 3306
 
 
-def use_streaming(databases, config, log_prefix):
+def use_streaming(databases, config):
     '''
-    Given a sequence of MySQL database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of MySQL database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
@@ -132,7 +154,6 @@ def use_streaming(databases, config, log_prefix):
 def dump_data_sources(
     databases,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
@@ -141,8 +162,7 @@ def dump_data_sources(
     '''
     Dump the given MySQL/MariaDB databases to a named pipe. The databases are supplied as a sequence
     of dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@@ -152,13 +172,19 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping MySQL databases{dry_run_label}')
+    logger.info(f'Dumping MySQL databases{dry_run_label}')
 
     for database in databases:
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        extra_environment = {'MYSQL_PWD': database['password']} if 'password' in database else None
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            database.get('password'), config
+        )
+        environment = dict(os.environ)
         dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
+            database, config, username, password, environment, dry_run
         )
 
         if not dump_database_names:
@@ -174,10 +200,12 @@ def dump_data_sources(
                 processes.append(
                     execute_dump_command(
                         renamed_database,
-                        log_prefix,
+                        config,
+                        username,
+                        password,
                         dump_path,
                         (dump_name,),
-                        extra_environment,
+                        environment,
                         dry_run,
                         dry_run_label,
                     )
@@ -186,10 +214,12 @@ def dump_data_sources(
             processes.append(
                 execute_dump_command(
                     database,
-                    log_prefix,
+                    config,
+                    username,
+                    password,
                     dump_path,
                     dump_database_names,
-                    extra_environment,
+                    environment,
                     dry_run,
                     dry_run_label,
                 )
@@ -198,7 +228,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'mysql_databases')
+                os.path.join(borgmatic_runtime_directory, 'mysql_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -206,25 +237,23 @@ def dump_data_sources(
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'MySQL', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'MySQL', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -242,7 +271,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -251,9 +279,9 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     hostname = connection_params['hostname'] or data_source.get(
@@ -262,32 +290,44 @@ def restore_data_source_dump(
     port = str(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
-    username = connection_params['username'] or data_source.get(
-        'restore_username', data_source.get('username')
+    tls = data_source.get('restore_tls', data_source.get('tls'))
+    username = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
-    password = connection_params['password'] or data_source.get(
-        'restore_password', data_source.get('password')
+    password = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['password']
+            or data_source.get('restore_password', data_source.get('password'))
+        ),
+        config,
     )
 
     mysql_restore_command = tuple(
         shlex.quote(part) for part in shlex.split(data_source.get('mysql_command') or 'mysql')
     )
+    extra_options, defaults_extra_filename = (
+        borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
+    )
     restore_command = (
         mysql_restore_command
-        + ('--batch',)
-        + (
-            tuple(data_source['restore_options'].split(' '))
-            if 'restore_options' in data_source
-            else ()
+        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+            username, password, defaults_extra_filename
         )
+        + extra_options
+        + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
-        + (('--user', username) if username else ())
+        + (('--ssl',) if tls is True else ())
+        + (('--skip-ssl',) if tls is False else ())
     )
-    extra_environment = {'MYSQL_PWD': password} if password else None
+    environment = dict(os.environ)
 
-    logger.debug(f"{log_prefix}: Restoring MySQL database {data_source['name']}{dry_run_label}")
+    logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 
@@ -298,5 +338,5 @@ def restore_data_source_dump(
         [extract_process],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout,
-        extra_environment=extra_environment,
+        environment=environment,
     )

borgmatic/hooks/data_source/postgresql.py

@@ -7,6 +7,7 @@ import shlex
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.credential.parse
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -24,46 +25,52 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'postgresql_databases')
 
 
-def make_extra_environment(database, restore_connection_params=None):
+def make_environment(database, config, restore_connection_params=None):
     '''
-    Make the extra_environment dict from the given database configuration. If restore connection
-    params are given, this is for a restore operation.
+    Make an environment dict from the current environment variables and the given database
+    configuration. If restore connection params are given, this is for a restore operation.
     '''
-    extra = dict()
+    environment = dict(os.environ)
 
     try:
         if restore_connection_params:
-            extra['PGPASSWORD'] = restore_connection_params.get('password') or database.get(
-                'restore_password', database['password']
+            environment['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
+                (
+                    restore_connection_params.get('password')
+                    or database.get('restore_password', database['password'])
+                ),
+                config,
             )
         else:
-            extra['PGPASSWORD'] = database['password']
+            environment['PGPASSWORD'] = borgmatic.hooks.credential.parse.resolve_credential(
+                database['password'], config
+            )
     except (AttributeError, KeyError):
         pass
 
     if 'ssl_mode' in database:
-        extra['PGSSLMODE'] = database['ssl_mode']
+        environment['PGSSLMODE'] = database['ssl_mode']
     if 'ssl_cert' in database:
-        extra['PGSSLCERT'] = database['ssl_cert']
+        environment['PGSSLCERT'] = database['ssl_cert']
     if 'ssl_key' in database:
-        extra['PGSSLKEY'] = database['ssl_key']
+        environment['PGSSLKEY'] = database['ssl_key']
     if 'ssl_root_cert' in database:
-        extra['PGSSLROOTCERT'] = database['ssl_root_cert']
+        environment['PGSSLROOTCERT'] = database['ssl_root_cert']
     if 'ssl_crl' in database:
-        extra['PGSSLCRL'] = database['ssl_crl']
+        environment['PGSSLCRL'] = database['ssl_crl']
 
-    return extra
+    return environment
 
 
 EXCLUDED_DATABASE_NAMES = ('template0', 'template1')
 
 
-def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
+def database_names_to_dump(database, config, environment, dry_run):
     '''
-    Given a requested database config, return the corresponding sequence of database names to dump.
-    In the case of "all" when a database format is given, query for the names of databases on the
-    configured host and return them. For "all" without a database format, just return a sequence
-    containing "all".
+    Given a requested database config and a configuration dict, return the corresponding sequence of
+    database names to dump. In the case of "all" when a database format is given, query for the
+    names of databases on the configured host and return them. For "all" without a database format,
+    just return a sequence containing "all".
     '''
     requested_name = database['name']
 
@@ -82,13 +89,18 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
         + ('--list', '--no-password', '--no-psqlrc', '--csv', '--tuples-only')
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
-        + (('--username', database['username']) if 'username' in database else ())
+        + (
+            (
+                '--username',
+                borgmatic.hooks.credential.parse.resolve_credential(database['username'], config),
+            )
+            if 'username' in database
+            else ()
+        )
         + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
     )
-    logger.debug(f'{log_prefix}: Querying for "all" PostgreSQL databases to dump')
-    list_output = execute_command_and_capture_output(
-        list_command, extra_environment=extra_environment
-    )
+    logger.debug('Querying for "all" PostgreSQL databases to dump')
+    list_output = execute_command_and_capture_output(list_command, environment=environment)
 
     return tuple(
         row[0]
@@ -97,14 +109,14 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
     )
 
 
-def get_default_port(databases, config, log_prefix):  # pragma: no cover
+def get_default_port(databases, config):  # pragma: no cover
     return 5432
 
 
-def use_streaming(databases, config, log_prefix):
+def use_streaming(databases, config):
     '''
-    Given a sequence of PostgreSQL database configuration dicts, a configuration dict (ignored), and
-    a log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of PostgreSQL database configuration dicts, a configuration dict (ignored),
+    return whether streaming will be using during dumps.
     '''
     return any(database.get('format') != 'directory' for database in databases)
 
@@ -112,7 +124,6 @@ def use_streaming(databases, config, log_prefix):
 def dump_data_sources(
     databases,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
@@ -121,8 +132,7 @@ def dump_data_sources(
     '''
     Dump the given PostgreSQL databases to a named pipe. The databases are supplied as a sequence of
     dicts, one dict describing each database as per the configuration schema. Use the given
-    borgmatic runtime directory to construct the destination path and the given log prefix in any
-    log entries.
+    borgmatic runtime directory to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@@ -134,14 +144,12 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping PostgreSQL databases{dry_run_label}')
+    logger.info(f'Dumping PostgreSQL databases{dry_run_label}')
 
     for database in databases:
-        extra_environment = make_extra_environment(database)
+        environment = make_environment(database, config)
         dump_path = make_dump_path(borgmatic_runtime_directory)
-        dump_database_names = database_names_to_dump(
-            database, extra_environment, log_prefix, dry_run
-        )
+        dump_database_names = database_names_to_dump(database, config, environment, dry_run)
 
         if not dump_database_names:
             if dry_run:
@@ -151,6 +159,7 @@ def dump_data_sources(
 
         for database_name in dump_database_names:
             dump_format = database.get('format', None if database_name == 'all' else 'custom')
+            compression = database.get('compression')
             default_dump_command = 'pg_dumpall' if database_name == 'all' else 'pg_dump'
             dump_command = tuple(
                 shlex.quote(part)
@@ -164,7 +173,7 @@ def dump_data_sources(
             )
             if os.path.exists(dump_filename):
                 logger.warning(
-                    f'{log_prefix}: Skipping duplicate dump of PostgreSQL database "{database_name}" to {dump_filename}'
+                    f'Skipping duplicate dump of PostgreSQL database "{database_name}" to {dump_filename}'
                 )
                 continue
 
@@ -178,12 +187,20 @@ def dump_data_sources(
                 + (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
                 + (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
                 + (
-                    ('--username', shlex.quote(database['username']))
+                    (
+                        '--username',
+                        shlex.quote(
+                            borgmatic.hooks.credential.parse.resolve_credential(
+                                database['username'], config
+                            )
+                        ),
+                    )
                     if 'username' in database
                     else ()
                 )
                 + (('--no-owner',) if database.get('no_owner', False) else ())
                 + (('--format', shlex.quote(dump_format)) if dump_format else ())
+                + (('--compress', shlex.quote(str(compression))) if compression is not None else ())
                 + (('--file', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
                 + (
                     tuple(shlex.quote(option) for option in database['options'].split(' '))
@@ -198,7 +215,7 @@ def dump_data_sources(
             )
 
             logger.debug(
-                f'{log_prefix}: Dumping PostgreSQL database "{database_name}" to {dump_filename}{dry_run_label}'
+                f'Dumping PostgreSQL database "{database_name}" to {dump_filename}{dry_run_label}'
             )
             if dry_run:
                 continue
@@ -208,7 +225,7 @@ def dump_data_sources(
                 execute_command(
                     command,
                     shell=True,
-                    extra_environment=extra_environment,
+                    environment=environment,
                 )
             else:
                 dump.create_named_pipe_for_dump(dump_filename)
@@ -216,7 +233,7 @@ def dump_data_sources(
                     execute_command(
                         command,
                         shell=True,
-                        extra_environment=extra_environment,
+                        environment=environment,
                         run_to_completion=False,
                     )
                 )
@@ -224,7 +241,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'postgresql_databases')
+                os.path.join(borgmatic_runtime_directory, 'postgresql_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -232,25 +250,25 @@ def dump_data_sources(
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
     dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'PostgreSQL', log_prefix, dry_run
+        make_dump_path(borgmatic_runtime_directory), 'PostgreSQL', dry_run
     )
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -268,7 +286,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -278,10 +295,9 @@ def restore_data_source_dump(
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
     configuration dict, but the given hook configuration is ignored. The given borgmatic runtime
-    directory is used to construct the destination path (used for the directory format), and the
-    given log prefix is used for any log entries. If this is a dry run, then don't actually restore
-    anything. Trigger the given active extract process (an instance of subprocess.Popen) to produce
-    output to consume.
+    directory is used to construct the destination path (used for the directory format). If this is
+    a dry run, then don't actually restore anything. Trigger the given active extract process (an
+    instance of subprocess.Popen) to produce output to consume.
 
     If the extract process is None, then restore the dump from the filesystem rather than from an
     extract stream.
@@ -296,8 +312,12 @@ def restore_data_source_dump(
     port = str(
         connection_params['port'] or data_source.get('restore_port', data_source.get('port', ''))
     )
-    username = connection_params['username'] or data_source.get(
-        'restore_username', data_source.get('username')
+    username = borgmatic.hooks.credential.parse.resolve_credential(
+        (
+            connection_params['username']
+            or data_source.get('restore_username', data_source.get('username'))
+        ),
+        config,
     )
 
     all_databases = bool(data_source['name'] == 'all')
@@ -350,13 +370,9 @@ def restore_data_source_dump(
         )
     )
 
-    extra_environment = make_extra_environment(
-        data_source, restore_connection_params=connection_params
-    )
+    environment = make_environment(data_source, config, restore_connection_params=connection_params)
 
-    logger.debug(
-        f"{log_prefix}: Restoring PostgreSQL database {data_source['name']}{dry_run_label}"
-    )
+    logger.debug(f"Restoring PostgreSQL database {data_source['name']}{dry_run_label}")
     if dry_run:
         return
 
@@ -367,6 +383,6 @@ def restore_data_source_dump(
         [extract_process] if extract_process else [],
         output_log_level=logging.DEBUG,
         input_file=extract_process.stdout if extract_process else None,
-        extra_environment=extra_environment,
+        environment=environment,
     )
-    execute_command(analyze_command, extra_environment=extra_environment)
+    execute_command(analyze_command, environment=environment)

borgmatic/hooks/data_source/snapshot.py

@@ -11,10 +11,10 @@ def get_contained_patterns(parent_directory, candidate_patterns):
     paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at /var, then
     /var is what we want to snapshot.
 
-    For this to work, a candidate pattern path can't have any globs or other non-literal characters
-    in the initial portion of the path that matches the parent directory. For instance, a parent
-    directory of /var would match a candidate pattern path of /var/log/*/data, but not a pattern
-    path like /v*/log/*/data.
+    For this function to work, a candidate pattern path can't have any globs or other non-literal
+    characters in the initial portion of the path that matches the parent directory. For instance, a
+    parent directory of /var would match a candidate pattern path of /var/log/*/data, but not a
+    pattern path like /v*/log/*/data.
 
     The one exception is that if a regular expression pattern path starts with "^", that will get
     stripped off for purposes of matching against a parent directory.
@@ -31,8 +31,10 @@ def get_contained_patterns(parent_directory, candidate_patterns):
         candidate
         for candidate in candidate_patterns
         for candidate_path in (pathlib.PurePath(candidate.path.lstrip('^')),)
-        if pathlib.PurePath(parent_directory) == candidate_path
-        or pathlib.PurePath(parent_directory) in candidate_path.parents
+        if (
+            pathlib.PurePath(parent_directory) == candidate_path
+            or pathlib.PurePath(parent_directory) in candidate_path.parents
+        )
     )
     candidate_patterns -= set(contained_patterns)
 

borgmatic/hooks/data_source/sqlite.py

@@ -17,14 +17,14 @@ def make_dump_path(base_directory):  # pragma: no cover
     return dump.make_data_source_dump_path(base_directory, 'sqlite_databases')
 
 
-def get_default_port(databases, config, log_prefix):  # pragma: no cover
+def get_default_port(databases, config):  # pragma: no cover
     return None  # SQLite doesn't use a port.
 
 
-def use_streaming(databases, config, log_prefix):
+def use_streaming(databases, config):
     '''
-    Given a sequence of SQLite database configuration dicts, a configuration dict (ignored), and a
-    log prefix (ignored), return whether streaming will be using during dumps.
+    Given a sequence of SQLite database configuration dicts, a configuration dict (ignored), return
+    whether streaming will be using during dumps.
     '''
     return any(databases)
 
@@ -32,7 +32,6 @@ def use_streaming(databases, config, log_prefix):
 def dump_data_sources(
     databases,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
@@ -41,7 +40,7 @@ def dump_data_sources(
     '''
     Dump the given SQLite databases to a named pipe. The databases are supplied as a sequence of
     configuration dicts, as per the configuration schema. Use the given borgmatic runtime directory
-    to construct the destination path and the given log prefix in any log entries.
+    to construct the destination path.
 
     Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
     pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
@@ -51,7 +50,7 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
     processes = []
 
-    logger.info(f'{log_prefix}: Dumping SQLite databases{dry_run_label}')
+    logger.info(f'Dumping SQLite databases{dry_run_label}')
 
     for database in databases:
         database_path = database['path']
@@ -60,7 +59,7 @@ def dump_data_sources(
             logger.warning('The "all" database name has no meaning for SQLite databases')
         if not os.path.exists(database_path):
             logger.warning(
-                f'{log_prefix}: No SQLite database at {database_path}; an empty database will be created and dumped'
+                f'No SQLite database at {database_path}; an empty database will be created and dumped'
             )
 
         dump_path = make_dump_path(borgmatic_runtime_directory)
@@ -68,7 +67,7 @@ def dump_data_sources(
 
         if os.path.exists(dump_filename):
             logger.warning(
-                f'{log_prefix}: Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
+                f'Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
             )
             continue
 
@@ -80,7 +79,7 @@ def dump_data_sources(
             shlex.quote(dump_filename),
         )
         logger.debug(
-            f'{log_prefix}: Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
+            f'Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
         )
         if dry_run:
             continue
@@ -91,7 +90,8 @@ def dump_data_sources(
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(
-                os.path.join(borgmatic_runtime_directory, 'sqlite_databases')
+                os.path.join(borgmatic_runtime_directory, 'sqlite_databases'),
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
             )
         )
 
@@ -99,25 +99,23 @@ def dump_data_sources(
 
 
 def remove_data_source_dumps(
-    databases, config, log_prefix, borgmatic_runtime_directory, dry_run
+    databases, config, borgmatic_runtime_directory, dry_run
 ):  # pragma: no cover
     '''
     Remove all database dump files for this hook regardless of the given databases. Use the
-    borgmatic runtime directory to construct the destination path and the log prefix in any log
-    entries. If this is a dry run, then don't actually remove anything.
+    borgmatic runtime directory to construct the destination path. If this is a dry run, then don't
+    actually remove anything.
     '''
-    dump.remove_data_source_dumps(
-        make_dump_path(borgmatic_runtime_directory), 'SQLite', log_prefix, dry_run
-    )
+    dump.remove_data_source_dumps(make_dump_path(borgmatic_runtime_directory), 'SQLite', dry_run)
 
 
 def make_data_source_dump_patterns(
-    databases, config, log_prefix, borgmatic_runtime_directory, name=None
+    databases, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
-    Given a sequence of configurations dicts, a configuration dict, a prefix to log with, the
-    borgmatic runtime directory, and a database name to match, return the corresponding glob
-    patterns to match the database dump in an archive.
+    Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
+    and a database name to match, return the corresponding glob patterns to match the database dump
+    in an archive.
     '''
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
@@ -135,7 +133,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,
@@ -144,22 +141,22 @@ def restore_data_source_dump(
 ):
     '''
     Restore a database from the given extract stream. The database is supplied as a data source
-    configuration dict, but the given hook configuration is ignored. The given log prefix is used
-    for any log entries. If this is a dry run, then don't actually restore anything. Trigger the
-    given active extract process (an instance of subprocess.Popen) to produce output to consume.
+    configuration dict, but the given hook configuration is ignored. If this is a dry run, then
+    don't actually restore anything. Trigger the given active extract process (an instance of
+    subprocess.Popen) to produce output to consume.
     '''
     dry_run_label = ' (dry run; not actually restoring anything)' if dry_run else ''
     database_path = connection_params['restore_path'] or data_source.get(
         'restore_path', data_source.get('path')
     )
 
-    logger.debug(f'{log_prefix}: Restoring SQLite database at {database_path}{dry_run_label}')
+    logger.debug(f'Restoring SQLite database at {database_path}{dry_run_label}')
     if dry_run:
         return
 
     try:
         os.remove(database_path)
-        logger.warning(f'{log_prefix}: Removed existing SQLite database at {database_path}')
+        logger.warning(f'Removed existing SQLite database at {database_path}')
     except FileNotFoundError:  # pragma: no cover
         pass
 

borgmatic/hooks/data_source/zfs.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import hashlib
 import logging
 import os
 import shutil
@@ -13,7 +14,7 @@ import borgmatic.hooks.data_source.snapshot
 logger = logging.getLogger(__name__)
 
 
-def use_streaming(hook_config, config, log_prefix):  # pragma: no cover
+def use_streaming(hook_config, config):  # pragma: no cover
     '''
     Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
     '''
@@ -38,6 +39,9 @@ def get_datasets_to_backup(zfs_command, patterns):
     pattern paths represent the requested datasets to snapshot. But also include any datasets tagged
     with a borgmatic-specific user property, whether or not they appear in the patterns.
 
+    Only include datasets that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic).
+
     Return the result as a sequence of Dataset instances, sorted by mount point.
     '''
     list_output = borgmatic.execute.execute_command_and_capture_output(
@@ -48,7 +52,7 @@ def get_datasets_to_backup(zfs_command, patterns):
             '-t',
             'filesystem',
             '-o',
-            f'name,mountpoint,{BORGMATIC_USER_PROPERTY}',
+            f'name,mountpoint,canmount,{BORGMATIC_USER_PROPERTY}',
         )
     )
 
@@ -60,7 +64,12 @@ def get_datasets_to_backup(zfs_command, patterns):
             (
                 Dataset(dataset_name, mount_point, (user_property_value == 'auto'), ())
                 for line in list_output.splitlines()
-                for (dataset_name, mount_point, user_property_value) in (line.rstrip().split('\t'),)
+                for (dataset_name, mount_point, can_mount, user_property_value) in (
+                    line.rstrip().split('\t'),
+                )
+                # Skip datasets that are marked "canmount=off", because mounting their snapshots will
+                # result in completely empty mount points—thereby preventing us from backing them up.
+                if can_mount == 'on'
             ),
             key=lambda dataset: dataset.mount_point,
             reverse=True,
@@ -83,7 +92,12 @@ def get_datasets_to_backup(zfs_command, patterns):
                 for contained_patterns in (
                     (
                         (
-                            (borgmatic.borg.pattern.Pattern(dataset.mount_point),)
+                            (
+                                borgmatic.borg.pattern.Pattern(
+                                    dataset.mount_point,
+                                    source=borgmatic.borg.pattern.Pattern_source.HOOK,
+                                ),
+                            )
                             if dataset.auto_backup
                             else ()
                         )
@@ -92,7 +106,12 @@ def get_datasets_to_backup(zfs_command, patterns):
                         )
                     ),
                 )
-                if contained_patterns
+                if dataset.auto_backup
+                or any(
+                    pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                    and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                    for pattern in contained_patterns
+                )
             ),
             key=lambda dataset: dataset.mount_point,
         )
@@ -115,7 +134,16 @@ def get_all_dataset_mount_points(zfs_command):
         )
     )
 
-    return tuple(sorted(line.rstrip() for line in list_output.splitlines()))
+    return tuple(
+        sorted(
+            {
+                mount_point
+                for line in list_output.splitlines()
+                for mount_point in (line.rstrip(),)
+                if mount_point != 'none'
+            }
+        )
+    )
 
 
 def snapshot_dataset(zfs_command, full_snapshot_name):  # pragma: no cover
@@ -155,10 +183,14 @@ def mount_snapshot(mount_command, full_snapshot_name, snapshot_mount_path):  # p
     )
 
 
-def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
+MOUNT_POINT_HASH_LENGTH = 10
+
+
+def make_borg_snapshot_pattern(pattern, dataset, normalized_runtime_directory):
     '''
-    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance, return a new Pattern with its
-    path rewritten to be in a snapshot directory based on the given runtime directory.
+    Given a Borg pattern as a borgmatic.borg.pattern.Pattern instance and the Dataset containing it,
+    return a new Pattern with its path rewritten to be in a snapshot directory based on both the
+    given runtime directory and the given Dataset's mount point.
 
     Move any initial caret in a regular expression pattern path to the beginning, so as not to break
     the regular expression.
@@ -173,6 +205,10 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
     rewritten_path = initial_caret + os.path.join(
         normalized_runtime_directory,
         'zfs_snapshots',
+        # Including this hash prevents conflicts between snapshot patterns for different datasets.
+        # For instance, without this, snapshotting a dataset at /var and another at /var/spool would
+        # result in overlapping snapshot patterns and therefore colliding mount attempts.
+        hashlib.shake_256(dataset.mount_point.encode('utf-8')).hexdigest(MOUNT_POINT_HASH_LENGTH),
         '.',  # Borg 1.4+ "slashdot" hack.
         # Included so that the source directory ends up in the Borg archive at its "original" path.
         pattern.path.lstrip('^').lstrip(os.path.sep),
@@ -183,34 +219,35 @@ def make_borg_snapshot_pattern(pattern, normalized_runtime_directory):
         pattern.type,
         pattern.style,
         pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
     )
 
 
 def dump_data_sources(
     hook_config,
     config,
-    log_prefix,
     config_paths,
     borgmatic_runtime_directory,
     patterns,
     dry_run,
 ):
     '''
-    Given a ZFS configuration dict, a configuration dict, a log prefix, the borgmatic configuration
-    file paths, the borgmatic runtime directory, the configured patterns, and whether this is a dry
-    run, auto-detect and snapshot any ZFS dataset mount points listed in the given patterns and any
+    Given a ZFS configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any ZFS dataset mount points listed in the given patterns and any
     dataset with a borgmatic-specific user property. Also update those patterns, replacing dataset
     mount points with corresponding snapshot directories so they get stored in the Borg archive
-    instead. Use the log prefix in any log entries.
+    instead.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
 
     If this is a dry run, then don't actually snapshot anything.
     '''
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
-    logger.info(f'{log_prefix}: Snapshotting ZFS datasets{dry_run_label}')
+    logger.info(f'Snapshotting ZFS datasets{dry_run_label}')
 
-    # List ZFS datasets to get their mount points.
+    # List ZFS datasets to get their mount points, but only consider those patterns that came from
+    # actual user configuration (as opposed to, say, other hooks).
     zfs_command = hook_config.get('zfs_command', 'zfs')
     requested_datasets = get_datasets_to_backup(zfs_command, patterns)
 
@@ -219,12 +256,12 @@ def dump_data_sources(
     normalized_runtime_directory = os.path.normpath(borgmatic_runtime_directory)
 
     if not requested_datasets:
-        logger.warning(f'{log_prefix}: No ZFS datasets found to snapshot{dry_run_label}')
+        logger.warning(f'No ZFS datasets found to snapshot{dry_run_label}')
 
     for dataset in requested_datasets:
         full_snapshot_name = f'{dataset.name}@{snapshot_name}'
         logger.debug(
-            f'{log_prefix}: Creating ZFS snapshot {full_snapshot_name} of {dataset.mount_point}{dry_run_label}'
+            f'Creating ZFS snapshot {full_snapshot_name} of {dataset.mount_point}{dry_run_label}'
         )
 
         if not dry_run:
@@ -235,11 +272,14 @@ def dump_data_sources(
         snapshot_mount_path = os.path.join(
             normalized_runtime_directory,
             'zfs_snapshots',
+            hashlib.shake_256(dataset.mount_point.encode('utf-8')).hexdigest(
+                MOUNT_POINT_HASH_LENGTH
+            ),
             dataset.mount_point.lstrip(os.path.sep),
         )
 
         logger.debug(
-            f'{log_prefix}: Mounting ZFS snapshot {full_snapshot_name} at {snapshot_mount_path}{dry_run_label}'
+            f'Mounting ZFS snapshot {full_snapshot_name} at {snapshot_mount_path}{dry_run_label}'
         )
 
         if dry_run:
@@ -250,7 +290,9 @@ def dump_data_sources(
         )
 
         for pattern in dataset.contained_patterns:
-            snapshot_pattern = make_borg_snapshot_pattern(pattern, normalized_runtime_directory)
+            snapshot_pattern = make_borg_snapshot_pattern(
+                pattern, dataset, normalized_runtime_directory
+            )
 
             # Attempt to update the pattern in place, since pattern order matters to Borg.
             try:
@@ -306,12 +348,12 @@ def get_all_snapshots(zfs_command):
     return tuple(line.rstrip() for line in list_output.splitlines())
 
 
-def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
     '''
-    Given a ZFS configuration dict, a configuration dict, a log prefix, the borgmatic runtime
-    directory, and whether this is a dry run, unmount and destroy any ZFS snapshots created by
-    borgmatic. Use the log prefix in any log entries. If this is a dry run or ZFS isn't configured
-    in borgmatic's configuration, then don't actually remove anything.
+    Given a ZFS configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, unmount and destroy any ZFS snapshots created by borgmatic. If this
+    is a dry run or ZFS isn't configured in borgmatic's configuration, then don't actually remove
+    anything.
     '''
     if hook_config is None:
         return
@@ -324,10 +366,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
     try:
         dataset_mount_points = get_all_dataset_mount_points(zfs_command)
     except FileNotFoundError:
-        logger.debug(f'{log_prefix}: Could not find "{zfs_command}" command')
+        logger.debug(f'Could not find "{zfs_command}" command')
         return
     except subprocess.CalledProcessError as error:
-        logger.debug(f'{log_prefix}: {error}')
+        logger.debug(error)
         return
 
     snapshots_glob = os.path.join(
@@ -335,10 +377,9 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
             os.path.normpath(borgmatic_runtime_directory),
         ),
         'zfs_snapshots',
+        '*',
     )
-    logger.debug(
-        f'{log_prefix}: Looking for snapshots to remove in {snapshots_glob}{dry_run_label}'
-    )
+    logger.debug(f'Looking for snapshots to remove in {snapshots_glob}{dry_run_label}')
     umount_command = hook_config.get('umount_command', 'umount')
 
     for snapshots_directory in glob.glob(snapshots_glob):
@@ -349,7 +390,10 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
         # child datasets before the shorter mount point paths of parent datasets.
         for mount_point in reversed(dataset_mount_points):
             snapshot_mount_path = os.path.join(snapshots_directory, mount_point.lstrip(os.path.sep))
-            if not os.path.isdir(snapshot_mount_path):
+
+            # If the snapshot mount path is empty, this is probably just a "shadow" of a nested
+            # dataset and therefore there's nothing to unmount.
+            if not os.path.isdir(snapshot_mount_path) or not os.listdir(snapshot_mount_path):
                 continue
 
             # This might fail if the path is already mounted, but we swallow errors here since we'll
@@ -363,22 +407,20 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
                 if not os.path.isdir(snapshot_mount_path):
                     continue
 
-            logger.debug(
-                f'{log_prefix}: Unmounting ZFS snapshot at {snapshot_mount_path}{dry_run_label}'
-            )
+            logger.debug(f'Unmounting ZFS snapshot at {snapshot_mount_path}{dry_run_label}')
 
             if not dry_run:
                 try:
                     unmount_snapshot(umount_command, snapshot_mount_path)
                 except FileNotFoundError:
-                    logger.debug(f'{log_prefix}: Could not find "{umount_command}" command')
+                    logger.debug(f'Could not find "{umount_command}" command')
                     return
                 except subprocess.CalledProcessError as error:
-                    logger.debug(f'{log_prefix}: {error}')
-                    return
+                    logger.debug(error)
+                    continue
 
         if not dry_run:
-            shutil.rmtree(snapshots_directory)
+            shutil.rmtree(snapshot_mount_path, ignore_errors=True)
 
     # Destroy snapshots.
     full_snapshot_names = get_all_snapshots(zfs_command)
@@ -388,14 +430,14 @@ def remove_data_source_dumps(hook_config, config, log_prefix, borgmatic_runtime_
         if not full_snapshot_name.split('@')[-1].startswith(BORGMATIC_SNAPSHOT_PREFIX):
             continue
 
-        logger.debug(f'{log_prefix}: Destroying ZFS snapshot {full_snapshot_name}{dry_run_label}')
+        logger.debug(f'Destroying ZFS snapshot {full_snapshot_name}{dry_run_label}')
 
         if not dry_run:
             destroy_snapshot(zfs_command, full_snapshot_name)
 
 
 def make_data_source_dump_patterns(
-    hook_config, config, log_prefix, borgmatic_runtime_directory, name=None
+    hook_config, config, borgmatic_runtime_directory, name=None
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".
@@ -406,7 +448,6 @@ def make_data_source_dump_patterns(
 def restore_data_source_dump(
     hook_config,
     config,
-    log_prefix,
     data_source,
     dry_run,
     extract_process,

borgmatic/hooks/dispatch.py

@@ -3,6 +3,7 @@ import importlib
 import logging
 import pkgutil
 
+import borgmatic.hooks.credential
 import borgmatic.hooks.data_source
 import borgmatic.hooks.monitoring
 
@@ -10,6 +11,7 @@ logger = logging.getLogger(__name__)
 
 
 class Hook_type(enum.Enum):
+    CREDENTIAL = 'credential'
     DATA_SOURCE = 'data_source'
     MONITORING = 'monitoring'
 
@@ -21,12 +23,12 @@ def get_submodule_names(parent_module):  # pragma: no cover
     return tuple(module_info.name for module_info in pkgutil.iter_modules(parent_module.__path__))
 
 
-def call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs):
+def call_hook(function_name, config, hook_name, *args, **kwargs):
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to the given hook name. Supply that call with the configuration
-    for this hook (if any), the log prefix, and any given args and kwargs. Return the return value
-    of that call or None if the module in question is not a hook.
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    the given hook name. Supply that call with the configuration for this hook (if any) and any
+    given args and kwargs. Return the return value of that call or None if the module in question is
+    not a hook.
 
     Raise ValueError if the hook name is unknown.
     Raise AttributeError if the function name is not found in the module.
@@ -40,7 +42,11 @@ def call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs):
     module_name = hook_name.split('_databases')[0]
 
     # Probe for a data source or monitoring hook module corresponding to the hook name.
-    for parent_module in (borgmatic.hooks.data_source, borgmatic.hooks.monitoring):
+    for parent_module in (
+        borgmatic.hooks.credential,
+        borgmatic.hooks.data_source,
+        borgmatic.hooks.monitoring,
+    ):
         if module_name not in get_submodule_names(parent_module):
             continue
 
@@ -54,17 +60,16 @@ def call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs):
     else:
         raise ValueError(f'Unknown hook name: {hook_name}')
 
-    logger.debug(f'{log_prefix}: Calling {hook_name} hook function {function_name}')
+    logger.debug(f'Calling {hook_name} hook function {function_name}')
 
-    return getattr(module, function_name)(hook_config, config, log_prefix, *args, **kwargs)
+    return getattr(module, function_name)(hook_config, config, *args, **kwargs)
 
 
-def call_hooks(function_name, config, log_prefix, hook_type, *args, **kwargs):
+def call_hooks(function_name, config, hook_type, *args, **kwargs):
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to each hook of the given hook type (either "data_source" or
-    "monitoring"). Supply each call with the configuration for that hook, the log prefix, and any
-    given args and kwargs.
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    each hook of the given hook type ("credential", "data_source", or "monitoring"). Supply each
+    call with the configuration for that hook, and any given args and kwargs.
 
     Collect any return values into a dict from module name to return value. Note that the module
     name is the name of the hook module itself, which might be different from the hook configuration
@@ -78,7 +83,7 @@ def call_hooks(function_name, config, log_prefix, hook_type, *args, **kwargs):
     Raise anything else that a called function raises. An error stops calls to subsequent functions.
     '''
     return {
-        hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
+        hook_name: call_hook(function_name, config, hook_name, *args, **kwargs)
         for hook_name in get_submodule_names(
             importlib.import_module(f'borgmatic.hooks.{hook_type.value}')
         )
@@ -86,18 +91,18 @@ def call_hooks(function_name, config, log_prefix, hook_type, *args, **kwargs):
     }
 
 
-def call_hooks_even_if_unconfigured(function_name, config, log_prefix, hook_type, *args, **kwargs):
+def call_hooks_even_if_unconfigured(function_name, config, hook_type, *args, **kwargs):
     '''
-    Given a configuration dict and a prefix to use in log entries, call the requested function of
-    the Python module corresponding to each hook of the given hook type (either "data_source" or
-    "monitoring"). Supply each call with the configuration for that hook, the log prefix, and any
-    given args and kwargs. Collect any return values into a dict from hook name to return value.
+    Given a configuration dict, call the requested function of the Python module corresponding to
+    each hook of the given hook type ("credential", "data_source", or "monitoring"). Supply each
+    call with the configuration for that hook and any given args and kwargs. Collect any return
+    values into a dict from hook name to return value.
 
     Raise AttributeError if the function name is not found in the module.
     Raise anything else that a called function raises. An error stops calls to subsequent functions.
     '''
     return {
-        hook_name: call_hook(function_name, config, log_prefix, hook_name, *args, **kwargs)
+        hook_name: call_hook(function_name, config, hook_name, *args, **kwargs)
         for hook_name in get_submodule_names(
             importlib.import_module(f'borgmatic.hooks.{hook_type.value}')
         )

borgmatic/hooks/monitoring/apprise.py

@@ -66,12 +66,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     )
 
     if not hook_config.get('services'):
-        logger.info(f'{config_filename}: No Apprise services to ping')
+        logger.info('No Apprise services to ping')
         return
 
     dry_run_string = ' (dry run; not actually pinging)' if dry_run else ''
     labels_string = ', '.join(map(operator.itemgetter('label'), hook_config.get('services')))
-    logger.info(f'{config_filename}: Pinging Apprise services: {labels_string}{dry_run_string}')
+    logger.info(f'Pinging Apprise services: {labels_string}{dry_run_string}')
 
     apprise_object = apprise.Apprise()
     apprise_object.add(list(map(operator.itemgetter('url'), hook_config.get('services'))))
@@ -100,10 +100,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     )
 
     if result is False:
-        logger.warning(f'{config_filename}: Error sending some Apprise notifications')
+        logger.warning('Error sending some Apprise notifications')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger. This prevents the handler from
     getting reused by other instances of this monitor.

borgmatic/hooks/monitoring/cronhub.py

@@ -28,9 +28,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONHUB:
-        logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook'
-        )
+        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronhub hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
@@ -41,8 +39,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         .replace('/ping/', formatted_state)
     )
 
-    logger.info(f'{config_filename}: Pinging Cronhub {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Cronhub ping URL {ping_url}')
+    logger.info(f'Pinging Cronhub {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Cronhub ping URL {ping_url}')
 
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -51,12 +49,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Cronhub error: {error}')
+            logger.warning(f'Cronhub error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/cronitor.py

@@ -28,16 +28,14 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     filename in any log entries. If this is a dry run, then don't actually ping anything.
     '''
     if state not in MONITOR_STATE_TO_CRONITOR:
-        logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook'
-        )
+        logger.debug(f'Ignoring unsupported monitoring {state.name.lower()} in Cronitor hook')
         return
 
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
     ping_url = f"{hook_config['ping_url']}/{MONITOR_STATE_TO_CRONITOR[state]}"
 
-    logger.info(f'{config_filename}: Pinging Cronitor {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Cronitor ping URL {ping_url}')
+    logger.info(f'Pinging Cronitor {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Cronitor ping URL {ping_url}')
 
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -46,12 +44,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Cronitor error: {error}')
+            logger.warning(f'Cronitor error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/healthchecks.py

@@ -55,9 +55,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
 
     if 'states' in hook_config and state.name.lower() not in hook_config['states']:
-        logger.info(
-            f'{config_filename}: Skipping Healthchecks {state.name.lower()} ping due to configured states'
-        )
+        logger.info(f'Skipping Healthchecks {state.name.lower()} ping due to configured states')
         return
 
     ping_url_is_uuid = re.search(r'\w{8}-\w{4}-\w{4}-\w{4}-\w{12}$', ping_url)
@@ -68,14 +66,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     if hook_config.get('create_slug'):
         if ping_url_is_uuid:
-            logger.warning(
-                f'{config_filename}: Healthchecks UUIDs do not support auto provisionning; ignoring'
-            )
+            logger.warning('Healthchecks UUIDs do not support auto provisionning; ignoring')
         else:
             ping_url = f'{ping_url}?create=1'
 
-    logger.info(f'{config_filename}: Pinging Healthchecks {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Healthchecks ping URL {ping_url}')
+    logger.info(f'Pinging Healthchecks {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Healthchecks ping URL {ping_url}')
 
     if state in (monitor.State.FINISH, monitor.State.FAIL, monitor.State.LOG):
         payload = borgmatic.hooks.monitoring.logs.format_buffered_logs_for_payload(
@@ -93,10 +89,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Healthchecks error: {error}')
+            logger.warning(f'Healthchecks error: {error}')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger. This prevents the handler from
     getting reused by other instances of this monitor.

borgmatic/hooks/monitoring/loki.py

@@ -139,10 +139,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     for handler in tuple(logging.getLogger().handlers):
         if isinstance(handler, Loki_log_handler):
             if state in MONITOR_STATE_TO_LOKI.keys():
-                handler.raw(f'{config_filename}: {MONITOR_STATE_TO_LOKI[state]} backup')
+                handler.raw(f'{MONITOR_STATE_TO_LOKI[state]} backup')
 
 
-def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
+def destroy_monitor(hook_config, config, monitoring_log_level, dry_run):
     '''
     Remove the monitor handler that was added to the root logger.
     '''

borgmatic/hooks/monitoring/ntfy.py

@@ -2,6 +2,8 @@ import logging
 
 import requests
 
+import borgmatic.hooks.credential.parse
+
 logger = logging.getLogger(__name__)
 
 
@@ -37,8 +39,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         base_url = hook_config.get('server', 'https://ntfy.sh')
         topic = hook_config.get('topic')
 
-        logger.info(f'{config_filename}: Pinging ntfy topic {topic}{dry_run_label}')
-        logger.debug(f'{config_filename}: Using Ntfy ping URL {base_url}/{topic}')
+        logger.info(f'Pinging ntfy topic {topic}{dry_run_label}')
+        logger.debug(f'Using Ntfy ping URL {base_url}/{topic}')
 
         headers = {
             'X-Title': state_config.get('title'),
@@ -47,28 +49,35 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             'X-Tags': state_config.get('tags'),
         }
 
-        username = hook_config.get('username')
-        password = hook_config.get('password')
-        access_token = hook_config.get('access_token')
+        try:
+            username = borgmatic.hooks.credential.parse.resolve_credential(
+                hook_config.get('username'), config
+            )
+            password = borgmatic.hooks.credential.parse.resolve_credential(
+                hook_config.get('password'), config
+            )
+            access_token = borgmatic.hooks.credential.parse.resolve_credential(
+                hook_config.get('access_token'), config
+            )
+        except ValueError as error:
+            logger.warning(f'Ntfy credential error: {error}')
+            return
+
         auth = None
 
         if access_token is not None:
             if username or password:
                 logger.warning(
-                    f'{config_filename}: ntfy access_token is set but so is username/password, only using access_token'
+                    'ntfy access_token is set but so is username/password, only using access_token'
                 )
             auth = requests.auth.HTTPBasicAuth('', access_token)
         elif (username and password) is not None:
             auth = requests.auth.HTTPBasicAuth(username, password)
-            logger.info(f'{config_filename}: Using basic auth with user {username} for ntfy')
+            logger.info(f'Using basic auth with user {username} for ntfy')
         elif username is not None:
-            logger.warning(
-                f'{config_filename}: Password missing for ntfy authentication, defaulting to no auth'
-            )
+            logger.warning('Password missing for ntfy authentication, defaulting to no auth')
         elif password is not None:
-            logger.warning(
-                f'{config_filename}: Username missing for ntfy authentication, defaulting to no auth'
-            )
+            logger.warning('Username missing for ntfy authentication, defaulting to no auth')
 
         if not dry_run:
             logging.getLogger('urllib3').setLevel(logging.ERROR)
@@ -77,12 +86,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                 if not response.ok:
                     response.raise_for_status()
             except requests.exceptions.RequestException as error:
-                logger.warning(f'{config_filename}: ntfy error: {error}')
+                logger.warning(f'ntfy error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/pagerduty.py

@@ -5,6 +5,7 @@ import platform
 
 import requests
 
+import borgmatic.hooks.credential.parse
 from borgmatic.hooks.monitoring import monitor
 
 logger = logging.getLogger(__name__)
@@ -29,21 +30,29 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     '''
     if state != monitor.State.FAIL:
         logger.debug(
-            f'{config_filename}: Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
+            f'Ignoring unsupported monitoring {state.name.lower()} in PagerDuty hook',
         )
         return
 
     dry_run_label = ' (dry run; not actually sending)' if dry_run else ''
-    logger.info(f'{config_filename}: Sending failure event to PagerDuty {dry_run_label}')
+    logger.info(f'Sending failure event to PagerDuty {dry_run_label}')
 
     if dry_run:
         return
 
+    try:
+        integration_key = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('integration_key'), config
+        )
+    except ValueError as error:
+        logger.warning(f'PagerDuty credential error: {error}')
+        return
+
     hostname = platform.node()
     local_timestamp = datetime.datetime.now(datetime.timezone.utc).astimezone().isoformat()
     payload = json.dumps(
         {
-            'routing_key': hook_config['integration_key'],
+            'routing_key': integration_key,
             'event_action': 'trigger',
             'payload': {
                 'summary': f'backup failed on {hostname}',
@@ -61,7 +70,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             },
         }
     )
-    logger.debug(f'{config_filename}: Using PagerDuty payload: {payload}')
+    logger.debug(f'Using PagerDuty payload: {payload}')
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
@@ -69,12 +78,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:
-        logger.warning(f'{config_filename}: PagerDuty error: {error}')
+        logger.warning(f'PagerDuty error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/pushover.py

@@ -2,6 +2,8 @@ import logging
 
 import requests
 
+import borgmatic.hooks.credential.parse
+
 logger = logging.getLogger(__name__)
 
 
@@ -32,17 +34,23 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     state_config = hook_config.get(state.name.lower(), {})
 
-    token = hook_config.get('token')
-    user = hook_config.get('user')
+    try:
+        token = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('token'), config
+        )
+        user = borgmatic.hooks.credential.parse.resolve_credential(hook_config.get('user'), config)
+    except ValueError as error:
+        logger.warning(f'Pushover credential error: {error}')
+        return
 
-    logger.info(f'{config_filename}: Updating Pushover{dry_run_label}')
+    logger.info(f'Updating Pushover{dry_run_label}')
 
     if state_config.get('priority') == EMERGENCY_PRIORITY:
         if 'expire' not in state_config:
-            logger.info(f'{config_filename}: Setting expire to default (10 min).')
+            logger.info('Setting expire to default (10 min)')
             state_config['expire'] = 600
         if 'retry' not in state_config:
-            logger.info(f'{config_filename}: Setting retry to default (30 sec).')
+            logger.info('Setting retry to default (30 sec)')
             state_config['retry'] = 30
     else:
         if 'expire' in state_config or 'retry' in state_config:
@@ -75,12 +83,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Pushover error: {error}')
+            logger.warning(f'Pushover error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/sentry.py

@@ -16,7 +16,7 @@ def initialize_monitor(
 
 
 DATA_SOURCE_NAME_URL_PATTERN = re.compile(
-    '^(?P<protocol>.*)://(?P<username>.*)@(?P<hostname>.*)/(?P<project_id>.*)$'
+    '^(?P<protocol>.+)://(?P<username>.+)@(?P<hostname>.+)/(?P<project_id>.+)$'
 )
 
 
@@ -38,15 +38,13 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     match = DATA_SOURCE_NAME_URL_PATTERN.match(data_source_name_url)
 
     if not match:
-        logger.warning(
-            'f{config_filename}: Invalid Sentry data source name URL: {data_source_name_url}'
-        )
+        logger.warning(f'Invalid Sentry data source name URL: {data_source_name_url}')
         return
 
     cron_url = f'{match.group("protocol")}://{match.group("hostname")}/api/{match.group("project_id")}/cron/{monitor_slug}/{match.group("username")}/'
 
-    logger.info(f'{config_filename}: Pinging Sentry {state.name.lower()}{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Sentry cron URL {cron_url}')
+    logger.info(f'Pinging Sentry {state.name.lower()}{dry_run_label}')
+    logger.debug(f'Using Sentry cron URL {cron_url}')
 
     status = {
         'start': 'in_progress',
@@ -55,7 +53,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     }.get(state.name.lower())
 
     if not status:
-        logger.warning('f{config_filename}: Invalid Sentry state')
+        logger.warning('Invalid Sentry state')
         return
 
     if dry_run:
@@ -67,12 +65,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:
-        logger.warning(f'{config_filename}: Sentry error: {error}')
+        logger.warning(f'Sentry error: {error}')
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/uptime_kuma.py

@@ -28,10 +28,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     status = 'down' if state.name.lower() == 'fail' else 'up'
     push_url = hook_config.get('push_url', 'https://example.uptime.kuma/api/push/abcd1234')
     query = f'status={status}&msg={state.name.lower()}'
-    logger.info(
-        f'{config_filename}: Pushing Uptime Kuma push_url {push_url}?{query} {dry_run_label}'
-    )
-    logger.debug(f'{config_filename}: Full Uptime Kuma state URL {push_url}?{query}')
+    logger.info(f'Pushing Uptime Kuma push_url {push_url}?{query} {dry_run_label}')
+    logger.debug(f'Full Uptime Kuma state URL {push_url}?{query}')
 
     if dry_run:
         return
@@ -39,16 +37,14 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     logging.getLogger('urllib3').setLevel(logging.ERROR)
 
     try:
-        response = requests.get(f'{push_url}?{query}')
+        response = requests.get(f'{push_url}?{query}', verify=hook_config.get('verify_tls', True))
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:
-        logger.warning(f'{config_filename}: Uptime Kuma error: {error}')
+        logger.warning(f'Uptime Kuma error: {error}')
 
 
-def destroy_monitor(
-    push_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(push_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/hooks/monitoring/zabbix.py

@@ -2,6 +2,8 @@ import logging
 
 import requests
 
+import borgmatic.hooks.credential.parse
+
 logger = logging.getLogger(__name__)
 
 
@@ -14,6 +16,42 @@ def initialize_monitor(
     pass
 
 
+def send_zabbix_request(server, headers, data):
+    '''
+    Given a Zabbix server URL, HTTP headers as a dict, and valid Zabbix JSON payload data as a dict,
+    send a request to the Zabbix server via API.
+
+    Return the response "result" value or None.
+    '''
+    logging.getLogger('urllib3').setLevel(logging.ERROR)
+
+    logger.debug(f'Sending a "{data["method"]}" request to the Zabbix server')
+
+    try:
+        response = requests.post(server, headers=headers, json=data)
+
+        if not response.ok:
+            response.raise_for_status()
+    except requests.exceptions.RequestException as error:
+        logger.warning(f'Zabbix error: {error}')
+
+        return None
+
+    try:
+        result = response.json().get('result')
+        error_message = result['data'][0]['error']
+    except requests.exceptions.JSONDecodeError:
+        logger.warning('Zabbix error: Cannot parse API response')
+
+        return None
+    except (TypeError, KeyError, IndexError):
+        return result
+    else:
+        logger.warning(f'Zabbix error: {error_message}')
+
+        return None
+
+
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
     '''
     Update the configured Zabbix item using either the itemid, or a host and key.
@@ -34,26 +72,34 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         },
     )
 
+    try:
+        username = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('username'), config
+        )
+        password = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('password'), config
+        )
+        api_key = borgmatic.hooks.credential.parse.resolve_credential(
+            hook_config.get('api_key'), config
+        )
+    except ValueError as error:
+        logger.warning(f'Zabbix credential error: {error}')
+
+        return
+
     server = hook_config.get('server')
-    username = hook_config.get('username')
-    password = hook_config.get('password')
-    api_key = hook_config.get('api_key')
     itemid = hook_config.get('itemid')
     host = hook_config.get('host')
     key = hook_config.get('key')
     value = state_config.get('value')
     headers = {'Content-Type': 'application/json-rpc'}
 
-    logger.info(f'{config_filename}: Updating Zabbix{dry_run_label}')
-    logger.debug(f'{config_filename}: Using Zabbix URL: {server}')
-
-    if server is None:
-        logger.warning(f'{config_filename}: Server missing for Zabbix')
-        return
+    logger.info(f'Pinging Zabbix{dry_run_label}')
+    logger.debug(f'Using Zabbix URL: {server}')
 
     # Determine the Zabbix method used to store the value: itemid or host/key
     if itemid is not None:
-        logger.info(f'{config_filename}: Updating {itemid} on Zabbix')
+        logger.info(f'Updating {itemid} on Zabbix')
         data = {
             'jsonrpc': '2.0',
             'method': 'history.push',
@@ -61,8 +107,8 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             'id': 1,
         }
 
-    elif (host and key) is not None:
-        logger.info(f'{config_filename}: Updating Host:{host} and Key:{key} on Zabbix')
+    elif host is not None and key is not None:
+        logger.info(f'Updating Host: "{host}" and Key: "{key}" on Zabbix')
         data = {
             'jsonrpc': '2.0',
             'method': 'history.push',
@@ -71,64 +117,67 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         }
 
     elif host is not None:
-        logger.warning(f'{config_filename}: Key missing for Zabbix')
-        return
+        logger.warning('Key missing for Zabbix')
 
+        return
     elif key is not None:
-        logger.warning(f'{config_filename}: Host missing for Zabbix.')
+        logger.warning('Host missing for Zabbix')
+
         return
     else:
-        logger.warning(f'{config_filename}: No zabbix itemid or host/key provided.')
+        logger.warning('No Zabbix itemid or host/key provided')
+
         return
 
     # Determine the authentication method: API key or username/password
     if api_key is not None:
-        logger.info(f'{config_filename}: Using API key auth for Zabbix')
-        headers['Authorization'] = 'Bearer ' + api_key
-
-    elif (username and password) is not None:
-        logger.info(f'{config_filename}: Using user/pass auth with user {username} for Zabbix')
-        auth_data = {
+        logger.info('Using API key auth for Zabbix')
+        headers['Authorization'] = f'Bearer {api_key}'
+    elif username is not None and password is not None:
+        logger.info(f'Using user/pass auth with user {username} for Zabbix')
+        login_data = {
             'jsonrpc': '2.0',
             'method': 'user.login',
             'params': {'username': username, 'password': password},
             'id': 1,
         }
+
         if not dry_run:
-            logging.getLogger('urllib3').setLevel(logging.ERROR)
-            try:
-                response = requests.post(server, headers=headers, json=auth_data)
-                data['auth'] = response.json().get('result')
-                if not response.ok:
-                    response.raise_for_status()
-            except requests.exceptions.RequestException as error:
-                logger.warning(f'{config_filename}: Zabbix error: {error}')
+            result = send_zabbix_request(server, headers, login_data)
+
+            if not result:
                 return
 
+            headers['Authorization'] = f'Bearer {result}'
     elif username is not None:
-        logger.warning(f'{config_filename}: Password missing for Zabbix authentication')
-        return
+        logger.warning('Password missing for Zabbix authentication')
 
+        return
     elif password is not None:
-        logger.warning(f'{config_filename}: Username missing for Zabbix authentication')
+        logger.warning('Username missing for Zabbix authentication')
+
         return
     else:
-        logger.warning(f'{config_filename}: Authentication data missing for Zabbix')
+        logger.warning('Authentication data missing for Zabbix')
+
         return
 
     if not dry_run:
-        logging.getLogger('urllib3').setLevel(logging.ERROR)
-        try:
-            response = requests.post(server, headers=headers, json=data)
-            if not response.ok:
-                response.raise_for_status()
-        except requests.exceptions.RequestException as error:
-            logger.warning(f'{config_filename}: Zabbix error: {error}')
+        send_zabbix_request(server, headers, data)
+
+    if username is not None and password is not None:
+        logout_data = {
+            'jsonrpc': '2.0',
+            'method': 'user.logout',
+            'params': [],
+            'id': 1,
+        }
+
+        if not dry_run:
+            send_zabbix_request(server, headers, logout_data)
 
 
-def destroy_monitor(
-    ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
-):  # pragma: no cover
+def destroy_monitor(ping_url_or_uuid, config, monitoring_log_level, dry_run):  # pragma: no cover
     '''
     No destruction is necessary for this monitor.
     '''

borgmatic/logger.py

@@ -87,9 +87,16 @@ class Multi_stream_handler(logging.Handler):
             handler.setLevel(level)
 
 
-class Console_no_color_formatter(logging.Formatter):
+class Log_prefix_formatter(logging.Formatter):
+    def __init__(self, fmt='{prefix}{message}', style='{', *args, **kwargs):  # pragma: no cover
+        self.prefix = None
+
+        super(Log_prefix_formatter, self).__init__(fmt=fmt, style=style, *args, **kwargs)
+
     def format(self, record):  # pragma: no cover
-        return record.msg
+        record.prefix = f'{self.prefix}: ' if self.prefix else ''
+
+        return super(Log_prefix_formatter, self).format(record)
 
 
 class Color(enum.Enum):
@@ -102,6 +109,12 @@ class Color(enum.Enum):
 
 
 class Console_color_formatter(logging.Formatter):
+    def __init__(self, *args, **kwargs):
+        self.prefix = None
+        super(Console_color_formatter, self).__init__(
+            '{prefix}{message}', style='{', *args, **kwargs
+        )
+
     def format(self, record):
         add_custom_log_levels()
 
@@ -117,8 +130,9 @@ class Console_color_formatter(logging.Formatter):
             .get(record.levelno)
             .value
         )
+        record.prefix = f'{self.prefix}: ' if self.prefix else ''
 
-        return color_text(color, record.msg)
+        return color_text(color, super(Console_color_formatter, self).format(record))
 
 
 def ansi_escape_code(color):  # pragma: no cover
@@ -179,6 +193,137 @@ def add_custom_log_levels():  # pragma: no cover
     add_logging_level('DISABLED', DISABLED)
 
 
+def get_log_prefix():
+    '''
+    Return the current log prefix set by set_log_prefix(). Return None if no such prefix exists.
+
+    It would be a whole lot easier to use logger.Formatter(defaults=...) instead, but that argument
+    doesn't exist until Python 3.10+.
+    '''
+    try:
+        formatter = next(
+            handler.formatter
+            for handler in logging.getLogger().handlers
+            if handler.formatter
+            if hasattr(handler.formatter, 'prefix')
+        )
+    except StopIteration:
+        return None
+
+    return formatter.prefix
+
+
+def set_log_prefix(prefix):
+    '''
+    Given a log prefix as a string, set it into the each handler's formatter so that it can inject
+    the prefix into each logged record.
+    '''
+    for handler in logging.getLogger().handlers:
+        if handler.formatter and hasattr(handler.formatter, 'prefix'):
+            handler.formatter.prefix = prefix
+
+
+class Log_prefix:
+    '''
+    A Python context manager for setting a log prefix so that it shows up in every subsequent
+    logging message for the duration of the context manager. For this to work, it relies on each
+    logging formatter to be initialized with "{prefix}" somewhere in its logging format.
+
+    Example use as a context manager:
+
+
+       with borgmatic.logger.Log_prefix('myprefix'):
+            do_something_that_logs()
+
+    For the scope of that "with" statement, any logs created are prefixed with "myprefix: ".
+    Afterwards, the prefix gets restored to whatever it was prior to the context manager.
+    '''
+
+    def __init__(self, prefix):
+        '''
+        Given the desired log prefix, save it for use below. Set prefix to None to disable any
+        prefix from getting logged.
+        '''
+        self.prefix = prefix
+        self.original_prefix = None
+
+    def __enter__(self):
+        '''
+        Set the prefix onto the formatter defaults for every logging handler so that the prefix ends
+        up in every log message. But first, save off any original prefix so that it can be restored
+        below.
+        '''
+        self.original_prefix = get_log_prefix()
+        set_log_prefix(self.prefix)
+
+    def __exit__(self, exception, value, traceback):
+        '''
+        Restore any original prefix.
+        '''
+        set_log_prefix(self.original_prefix)
+
+
+class Delayed_logging_handler(logging.handlers.BufferingHandler):
+    '''
+    A logging handler that buffers logs and doesn't flush them until explicitly flushed (after
+    target handlers are actually set). It's useful for holding onto messages logged before logging
+    is configured, ensuring those records eventually make their way to the relevant logging
+    handlers.
+
+    When flushing, don't forward log records to a target handler if the record's log level is below
+    that of the handler. This recreates the standard logging behavior of, say, logging.DEBUG records
+    getting suppressed if a handler's level is only set to logging.INFO.
+    '''
+
+    def __init__(self):
+        super(Delayed_logging_handler, self).__init__(capacity=0)
+
+        self.targets = None
+
+    def shouldFlush(self, record):
+        return self.targets is not None
+
+    def flush(self):
+        self.acquire()
+
+        try:
+            if not self.targets:
+                return
+
+            for record in self.buffer:
+                for target in self.targets:
+                    if record.levelno >= target.level:
+                        target.handle(record)
+
+            self.buffer.clear()
+        finally:
+            self.release()
+
+
+def configure_delayed_logging():  # pragma: no cover
+    '''
+    Configure a delayed logging handler to buffer anything that gets logged until we're ready to
+    deal with it.
+    '''
+    logging.basicConfig(
+        level=logging.DEBUG,
+        handlers=[Delayed_logging_handler()],
+    )
+
+
+def flush_delayed_logging(target_handlers):
+    '''
+    Flush any previously buffered logs to our "real" logging handlers.
+    '''
+    root_logger = logging.getLogger()
+
+    if root_logger.handlers and isinstance(root_logger.handlers[0], Delayed_logging_handler):
+        delayed_handler = root_logger.handlers[0]
+        delayed_handler.targets = target_handlers
+        delayed_handler.flush()
+        root_logger.removeHandler(delayed_handler)
+
+
 def configure_logging(
     console_log_level,
     syslog_log_level=None,
@@ -223,10 +368,9 @@ def configure_logging(
     if color_enabled:
         console_handler.setFormatter(Console_color_formatter())
     else:
-        console_handler.setFormatter(Console_no_color_formatter())
+        console_handler.setFormatter(Log_prefix_formatter())
 
     console_handler.setLevel(console_log_level)
-
     handlers = [console_handler]
 
     if syslog_log_level != logging.DISABLED:
@@ -242,7 +386,9 @@ def configure_logging(
         if syslog_path:
             syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
             syslog_handler.setFormatter(
-                logging.Formatter('borgmatic: {levelname} {message}', style='{')  # noqa: FS003
+                Log_prefix_formatter(
+                    'borgmatic: {levelname} {prefix}{message}',  # noqa: FS003
+                )
             )
             syslog_handler.setLevel(syslog_log_level)
             handlers.append(syslog_handler)
@@ -250,13 +396,15 @@ def configure_logging(
     if log_file and log_file_log_level != logging.DISABLED:
         file_handler = logging.handlers.WatchedFileHandler(log_file)
         file_handler.setFormatter(
-            logging.Formatter(
-                log_file_format or '[{asctime}] {levelname}: {message}', style='{'  # noqa: FS003
+            Log_prefix_formatter(
+                log_file_format or '[{asctime}] {levelname}: {prefix}{message}',  # noqa: FS003
             )
         )
         file_handler.setLevel(log_file_log_level)
         handlers.append(file_handler)
 
+    flush_delayed_logging(handlers)
+
     logging.basicConfig(
         level=min(handler.level for handler in handlers),
         handlers=handlers,

borgmatic/signals.py

@@ -24,6 +24,9 @@ def handle_signal(signal_number, frame):
         logger.critical('Exiting due to TERM signal')
         sys.exit(EXIT_CODE_FROM_SIGNAL + signal.SIGTERM)
     elif signal_number == signal.SIGINT:
+        # Borg doesn't always exit on a SIGINT, so give it a little encouragement.
+        os.killpg(os.getpgrp(), signal.SIGTERM)
+
         raise KeyboardInterrupt()
 
 

docs/_includes/layouts/base.njk

@@ -4,7 +4,7 @@
 		<meta charset="utf-8">
 		<meta name="viewport" content="width=device-width, initial-scale=1.0">
                 <meta name="generator" content="{{ eleventy.generator }}">
-		<link rel="icon" href="docs/static/borgmatic.png" type="image/x-icon">
+		<link rel="icon" href="https://torsion.org/borgmatic/docs/static/borgmatic.png" type="image/x-icon">
 		<title>{{ subtitle + ' - ' if subtitle}}{{ title }}</title>
 {%- set css %}
 {% include 'index.css' %}

docs/fetch-contributors

@@ -25,27 +25,42 @@ def list_merged_pulls(url):
     return tuple(pull for pull in response.json() if pull.get('merged_at'))
 
 
-API_ENDPOINT_URLS = (
+def list_contributing_issues(url):
+    response = requests.get(url, headers={'Accept': 'application/json', 'Content-Type': 'application/json'})
+
+    if not response.ok:
+        response.raise_for_status()
+
+    return tuple(response.json())
+
+
+PULLS_API_ENDPOINT_URLS = (
     'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/pulls',
     'https://api.github.com/repos/borgmatic-collective/borgmatic/pulls',
 )
+ISSUES_API_ENDPOINT_URL = 'https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/issues?state=all'
 RECENT_CONTRIBUTORS_CUTOFF_DAYS = 365
 
 
+def get_item_timestamp(item):
+    return item.get('merged_at') or item.get('created_at')
+
+
 def print_contributors():
     '''
     Display the recent contributors as a row of avatars in an HTML fragment.
     '''
-    pulls = tuple(itertools.chain.from_iterable(list_merged_pulls(url) for url in API_ENDPOINT_URLS))
+    pulls = tuple(itertools.chain.from_iterable(list_merged_pulls(url) for url in PULLS_API_ENDPOINT_URLS))
+    issues = list_contributing_issues(ISSUES_API_ENDPOINT_URL)
     seen_user_ids = set()
 
     print('<p>')
 
-    for pull in sorted(pulls, key=operator.itemgetter('merged_at'), reverse=True):
-        merged_at = pull.get('merged_at')
-        user = pull.get('user')
+    for item in sorted(pulls + issues, key=get_item_timestamp, reverse=True):
+        timestamp = get_item_timestamp(item)
+        user = item.get('user')
 
-        if not merged_at or not user:
+        if not timestamp or not user:
             continue
 
         user_id = user.get('id')
@@ -53,7 +68,7 @@ def print_contributors():
         if not user_id or user_id in seen_user_ids:
             continue
 
-        if datetime.datetime.fromisoformat(merged_at) < datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=RECENT_CONTRIBUTORS_CUTOFF_DAYS):
+        if datetime.datetime.fromisoformat(timestamp) < datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=RECENT_CONTRIBUTORS_CUTOFF_DAYS):
             continue
 
         seen_user_ids.add(user_id)

docs/how-to/backup-your-databases.md

@@ -58,7 +58,7 @@ mariadb_databases:
 
 As part of each backup, borgmatic streams a database dump for each configured
 database directly to Borg, so it's included in the backup without consuming
-additional disk space. (The exceptions are the PostgreSQL/MongoDB "directory"
+additional disk space. (The exceptions are the PostgreSQL/MongoDB `directory`
 dump formats, which can't stream and therefore do consume temporary disk
 space. Additionally, prior to borgmatic 1.5.3, all database dumps consumed
 temporary disk space.)
@@ -135,6 +135,9 @@ temporary file storage, probing the following locations (in order) to find it:
     Hard-coded `/tmp`. <span class="minilink minilink-addedin">Prior to
     version 1.9.2</span>This was instead hard-coded to `/run/user/$UID`.
 
+You can see the runtime directory path that borgmatic selects by running with
+`--verbosity 2` and looking for "Using runtime directory" in the output.
+
 Regardless of the runtime directory selected, borgmatic stores its files
 within a `borgmatic` subdirectory of the runtime directory. Additionally, in
 the case of `TMPDIR`, `TEMP`, and the hard-coded `/tmp`, borgmatic creates a
@@ -258,19 +261,23 @@ hooks:
 
 ... where `my_pg_container` is the name of your database container. In this
 example, you'd also need to set the `pg_restore_command` and `psql_command`
-options. If you choose to use the `pg_dump` command within the container
-though, note that it will output the database dump to a file inside the
-container. So you'll have to mount the `.borgmatic` folder from your host's
-home folder into the container using the same directory structure.
+options.
 
-See the following Docker compose file an as example:
+If you choose to use the `pg_dump` command within the container, and you're
+using the `directory` format in particular, you'll also need to mount the
+[runtime directory](#runtime-directory) from your host into the container at the
+same path. Otherwise, the `directory` format dump will remain locked away inside
+the database container where Borg can't read it.
+
+For example, with Docker Compose and a runtime directory located at
+`/run/user/1000`:
 
 ```yaml
 services:
   db:
     image: postgres
     volumes:
-      - /home/USERNAME/.borgmatic:/home/USERNAME/.borgmatic
+      - /run/user/1000:/run/user/1000
 ```
 
 Similar command override options are available for (some of) the other
@@ -302,10 +309,8 @@ hooks:
 ### External passwords
 
 If you don't want to keep your database passwords in your borgmatic
-configuration file, you can instead pass them in via [environment
-variables](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/)
-or command-line [configuration
-overrides](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides).
+configuration file, you can instead pass them in [from external credential
+sources](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
 
 
 ### Configuration backups
@@ -547,7 +552,7 @@ extraction destination path. For example, if you're extracting to `/tmp`, then
 the dump will be in `/tmp/borgmatic/`.
 
 <span class="minilink minilink-addedin">Prior to version 1.9.0</span> borgmatic
-extracts the dump file into the *`username`*`/.borgmatic/` directory within the
+extracted the dump file into the *`username`*`/.borgmatic/` directory within the
 extraction destination path, where *`username`* is the user that created the
 backup. For example, if you created the backup with the `root` user and you're
 extracting to `/tmp`, then the dump will be in `/tmp/root/.borgmatic`.

docs/how-to/monitor-your-backups.md

@@ -478,22 +478,18 @@ your Loki API push URL. Here's an example:
 ```yaml
 loki:
     url: http://localhost:3100/loki/api/v1/push
+
+    labels:
+        app: borgmatic
+        hostname: example.org
 ```
 
 With this configuration, borgmatic sends its logs to your Loki instance as any
 of the `create`, `prune`, `compact`, or `check` actions are run. Then, after
 the actions complete, borgmatic notifies Loki of success or failure.
 
-This hook supports sending arbitrary labels to Loki. For instance:
-
-```yaml
-loki:
-    url: http://localhost:3100/loki/api/v1/push
-
-    labels:
-        app: borgmatic
-        hostname: example.org
-```
+This hook supports sending arbitrary labels to Loki. At least one label is
+required.
 
 There are also a few placeholders you can optionally use as label values:
 
@@ -714,7 +710,10 @@ zabbix:
         - fail
 ```
 
-This hook requires the Zabbix server be running version 7.0+
+This hook requires the Zabbix server be running version 7.0.
+
+<span class="minilink minilink-addedin">New in version 1.9.3</span> Zabbix 7.2+
+is supported as well.
 
 
 ### Authentication methods

docs/how-to/provide-your-passwords.md

@@ -19,64 +19,275 @@ encryption_passphrase: yourpassphrase
 But if you'd rather store them outside of borgmatic, whether for convenience
 or security reasons, read on.
 
+
 ### Delegating to another application
 
 borgmatic supports calling another application such as a password manager to 
 obtain the Borg passphrase to a repository.
 
-For example, to ask the *Pass* password manager to provide the passphrase:
-```yaml
-encryption_passcommand: pass path/to/borg-repokey
-```
-
-### Using systemd service credentials
-
-Borgmatic supports using encrypted [credentials](https://systemd.io/CREDENTIALS/).
-
-Save your password as an encrypted credential to `/etc/credstore.encrypted/borgmatic.pw`, e.g.,
-
-```
-# systemd-ask-password -n | systemd-creds encrypt - /etc/credstore.encrypted/borgmatic.pw
-```
-
-Then uncomment or use the following in your configuration file:
+For example, to ask the [Pass](https://www.passwordstore.org/) password manager
+to provide the passphrase:
 
 ```yaml
-encryption_passcommand: "cat ${CREDENTIALS_DIRECTORY}/borgmatic.pw"
+encryption_passcommand: pass path/to/borg-passphrase
+```
+
+<span class="minilink minilink-addedin">New in version 1.9.9</span> Instead of
+letting Borg run the passcommand—potentially multiple times since borgmatic runs
+Borg multiple times—borgmatic now runs the passcommand itself and passes the
+resulting passphrase securely to Borg via an anonymous pipe. This means you
+should only ever get prompted for your password manager's passphrase at most
+once per borgmatic run.
+
+
+### systemd service credentials
+
+borgmatic supports reading encrypted [systemd
+credentials](https://systemd.io/CREDENTIALS/). To use this feature, start by
+saving your password as an encrypted credential to
+`/etc/credstore.encrypted/borgmatic.pw`, e.g.,
+
+```bash
+systemd-ask-password -n | systemd-creds encrypt - /etc/credstore.encrypted/borgmatic.pw
+```
+
+Then use the following in your configuration file:
+
+```yaml
+encryption_passphrase: "{credential systemd borgmatic.pw}"
+```
+
+<span class="minilink minilink-addedin">Prior to version 1.9.10</span> You can
+accomplish the same thing with this configuration:
+
+```yaml
+encryption_passcommand: cat ${CREDENTIALS_DIRECTORY}/borgmatic.pw
 ```
 
 Note that the name `borgmatic.pw` is hardcoded in the systemd service file.
 
-To use multiple different passwords, save them as encrypted credentials to `/etc/credstore.encrypted/borgmatic/`, e.g.,
+The `{credential ...}` syntax works for several different options in a borgmatic
+configuration file besides just `encryption_passphrase`. For instance, the
+username, password, and API token options within database and monitoring hooks
+support `{credential ...}`:
 
+```yaml
+postgresql_databases:
+    - name: invoices
+      username: postgres
+      password: "{credential systemd borgmatic_db1}"
 ```
-# mkdir /etc/credstore.encrypted/borgmatic
-# systemd-ask-password -n | systemd-creds encrypt --name=borgmatic_backupserver1 - /etc/credstore.encrypted/borgmatic/backupserver1
-# systemd-ask-password -n | systemd-creds encrypt --name=borgmatic_pw2 - /etc/credstore.encrypted/borgmatic/pw2
+
+For specifics about which options are supported, see the
+[configuration
+reference](https://torsion.org/borgmatic/docs/reference/configuration/).
+
+To use these credentials, you'll need to modify the borgmatic systemd service
+file to support loading multiple credentials (assuming you need to load more
+than one or anything not named `borgmatic.pw`).
+
+Start by saving each encrypted credentials to
+`/etc/credstore.encrypted/borgmatic/`. E.g.,
+
+```bash
+mkdir /etc/credstore.encrypted/borgmatic
+systemd-ask-password -n | systemd-creds encrypt --name=borgmatic_backupserver1 - /etc/credstore.encrypted/borgmatic/backupserver1
+systemd-ask-password -n | systemd-creds encrypt --name=borgmatic_pw2 - /etc/credstore.encrypted/borgmatic/pw2
 ...
 ```
 
-Ensure that the file names, (e.g. `backupserver1`) match the corresponding part of
-the `--name` option *after* the underscore (_), and that the part *before* 
+Ensure that the file names, (e.g. `backupserver1`) match the corresponding part
+of the `--name` option *after* the underscore (_), and that the part *before*
 the underscore matches the directory name (e.g. `borgmatic`).
 
 Then, uncomment the appropriate line in the systemd service file:
 
 ```
-# systemctl edit borgmatic.service
+systemctl edit borgmatic.service
 ...
 # Load multiple encrypted credentials.
 LoadCredentialEncrypted=borgmatic:/etc/credstore.encrypted/borgmatic/
 ```
 
-Finally, use the following in your configuration file:
+Finally, use something like the following in your borgmatic configuration file
+for each option value you'd like to load from systemd:
 
-```
-encryption_passcommand: "cat ${CREDENTIALS_DIRECTORY}/borgmatic_backupserver1"
+```yaml
+encryption_passphrase: "{credential systemd borgmatic_backupserver1}"
 ```
 
-Adjust `borgmatic_backupserver1` according to the name given to the credential 
-and the directory set in the service file.
+<span class="minilink minilink-addedin">Prior to version 1.9.10</span> Use the
+following instead, but only for the `encryption_passcommand` option and
+not other options:
+
+```yaml
+encryption_passcommand: cat ${CREDENTIALS_DIRECTORY}/borgmatic_backupserver1
+```
+
+Adjust `borgmatic_backupserver1` according to the name of the credential and the
+directory set in the service file.
+
+Be aware that when using this systemd `{credential ...}` feature, you may no
+longer be able to run certain borgmatic actions outside of the systemd service,
+as the credentials are only available from within the context of that service.
+So for instance, `borgmatic list` necessarily relies on the
+`encryption_passphrase` in order to access the Borg repository, but `list`
+shouldn't need to load any credentials for your database or monitoring hooks.
+
+The one exception is `borgmatic config validate`, which doesn't actually load
+any credentials and should continue working anywhere.
+
+
+### Container secrets
+
+<span class="minilink minilink-addedin">New in version 1.9.11</span> When
+running inside a container, borgmatic can read [Docker
+secrets](https://docs.docker.com/compose/how-tos/use-secrets/) and [Podman
+secrets](https://www.redhat.com/en/blog/new-podman-secrets-command). Creating
+those secrets and passing them into your borgmatic container is outside the
+scope of this documentation, but here's a simple example of that with [Docker
+Compose](https://docs.docker.com/compose/):
+
+```yaml
+services:
+  borgmatic:
+    # Use the actual image name of your borgmatic container here.
+    image: borgmatic:latest
+    secrets:
+      - borgmatic_passphrase
+secrets:
+  borgmatic_passphrase:
+    file: /etc/borgmatic/passphrase.txt
+```
+
+This assumes there's a file on the host at `/etc/borgmatic/passphrase.txt`
+containing your passphrase. Docker or Podman mounts the contents of that file
+into a secret named `borgmatic_passphrase` in the borgmatic container at
+`/run/secrets/`.
+
+Once your container secret is in place, you can consume it within your borgmatic
+configuration file:
+
+```yaml
+encryption_passphrase: "{credential container borgmatic_passphrase}"
+```
+
+This reads the secret securely from a file mounted at
+`/run/secrets/borgmatic_passphrase` within the borgmatic container.
+
+The `{credential ...}` syntax works for several different options in a borgmatic
+configuration file besides just `encryption_passphrase`. For instance, the
+username, password, and API token options within database and monitoring hooks
+support `{credential ...}`:
+
+```yaml
+postgresql_databases:
+    - name: invoices
+      username: postgres
+      password: "{credential container borgmatic_db1}"
+```
+
+For specifics about which options are supported, see the
+[configuration
+reference](https://torsion.org/borgmatic/docs/reference/configuration/).
+
+You can also optionally override the `/run/secrets` directory that borgmatic reads secrets from
+inside a container:
+
+```yaml
+container:
+    secrets_directory: /path/to/secrets
+```
+
+But you should only need to do this for development or testing purposes.
+
+
+### KeePassXC passwords
+
+<span class="minilink minilink-addedin">New in version 1.9.11</span> borgmatic
+supports reading passwords from the [KeePassXC](https://keepassxc.org/) password
+manager. To use this feature, start by creating an entry in your KeePassXC
+database, putting your password into the "Password" field of that entry and
+making sure it's saved.
+
+Then, you can consume that password in your borgmatic configuration file. For
+instance, if the entry's title is "borgmatic" and your KeePassXC database is
+located at `/etc/keys.kdbx`, do this:
+
+```yaml
+encryption_passphrase: "{credential keepassxc /etc/keys.kdbx borgmatic}"
+```
+
+But if the entry's title is multiple words like `borg pw`, you'll
+need to quote it:
+
+```yaml
+encryption_passphrase: "{credential keepassxc /etc/keys.kdbx 'borg pw'}"
+```
+
+With this in place, borgmatic runs the `keepassxc-cli` command to retrieve the
+passphrase on demand. But note that `keepassxc-cli` will prompt for its own
+passphrase in order to unlock its database, so be prepared to enter it when
+running borgmatic.
+
+The `{credential ...}` syntax works for several different options in a borgmatic
+configuration file besides just `encryption_passphrase`. For instance, the
+username, password, and API token options within database and monitoring hooks
+support `{credential ...}`:
+
+```yaml
+postgresql_databases:
+    - name: invoices
+      username: postgres
+      password: "{credential keepassxc /etc/keys.kdbx database}"
+```
+
+For specifics about which options are supported, see the
+[configuration
+reference](https://torsion.org/borgmatic/docs/reference/configuration/).
+
+You can also optionally override the `keepassxc-cli` command that borgmatic calls to load
+passwords:
+
+```yaml
+keepassxc:
+    keepassxc_cli_command: /usr/local/bin/keepassxc-cli
+```
+
+
+### File-based credentials
+
+<span class="minilink minilink-addedin">New in version 1.9.11</span> borgmatic
+supports reading credentials from arbitrary file paths. To use this feature,
+start by writing your credential into a file that borgmatic has permission to
+read. Take care not to include anything in the file other than your credential.
+(borgmatic is smart enough to strip off a trailing newline though.)
+
+You can consume that credential file in your borgmatic configuration. For
+instance, if your credential file is at `/credentials/borgmatic.txt`, do this:
+
+```yaml
+encryption_passphrase: "{credential file /credentials/borgmatic.txt}"
+```
+
+With this in place, borgmatic reads the credential from the file path.
+
+The `{credential ...}` syntax works for several different options in a borgmatic
+configuration file besides just `encryption_passphrase`. For instance, the
+username, password, and API token options within database and monitoring hooks
+support `{credential ...}`:
+
+```yaml
+postgresql_databases:
+    - name: invoices
+      username: postgres
+      password: "{credential file /credentials/database.txt}"
+```
+
+For specifics about which options are supported, see the
+[configuration
+reference](https://torsion.org/borgmatic/docs/reference/configuration/).
+
 
 ### Environment variable interpolation
 
@@ -84,7 +295,15 @@ and the directory set in the service file.
 supports interpolating arbitrary environment variables directly into option
 values in your configuration file. That means you can instruct borgmatic to
 pull your repository passphrase, your database passwords, or any other option
-values from environment variables. For instance:
+values from environment variables.
+
+Be aware though that environment variables may be less secure than some of the
+other approaches above for getting credentials into borgmatic. That's because
+environment variables may be visible from within child processes and/or OS-level
+process metadata.
+
+Here's an example of using an environment variable from borgmatic's
+configuration file:
 
 ```yaml
 encryption_passphrase: ${YOUR_PASSPHRASE}
@@ -146,6 +365,7 @@ can escape it with a backslash. For instance, if your password is literally
 encryption_passphrase: \${A}@!
 ```
 
+
 ## Related features
 
 Another way to override particular options within a borgmatic configuration
@@ -158,9 +378,3 @@ Additionally, borgmatic action hooks support their own [variable
 interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation),
 although in that case it's for particular borgmatic runtime values rather than
 (only) environment variables.
-
-Lastly, if you do want to specify your passhprase directly within borgmatic
-configuration, but you'd like to keep it in a separate file from your main
-configuration, you can [use a configuration include or a merge
-include](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-includes)
-to pull in an external password.

docs/how-to/snapshot-your-filesystems.md

@@ -54,8 +54,8 @@ You have a couple of options for borgmatic to find and backup your ZFS datasets:
  * For any dataset you'd like backed up, add its mount point to borgmatic's
    `source_directories` option.
  * <span class="minilink minilink-addedin">New in version 1.9.6</span> Or
-   include the mount point with borgmatic's `patterns` or `patterns_from`
-   options.
+   include the mount point as a root pattern with borgmatic's `patterns` or
+   `patterns_from` options.
  * Or set the borgmatic-specific user property
    `org.torsion.borgmatic:backup=auto` onto your dataset, e.g. by running `zfs
    set org.torsion.borgmatic:backup=auto datasetname`. Then borgmatic can find
@@ -65,6 +65,11 @@ If you have multiple borgmatic configuration files with ZFS enabled, and you'd
 like particular datasets to be backed up only for particular configuration
 files, use the `source_directories` option instead of the user property.
 
+<span class="minilink minilink-addedin">New in version 1.9.11</span> borgmatic
+won't snapshot datasets with the `canmount=off` property, which is often set on
+datasets that only serve as a container for other datasets. Use `zfs get
+canmount datasetname` to see the `canmount` value for a dataset.
+
 During a backup, borgmatic automatically snapshots these discovered datasets
 (non-recursively), temporarily mounts the snapshots within its [runtime
 directory](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory),
@@ -147,7 +152,8 @@ For any subvolume you'd like backed up, add its path to borgmatic's
 `source_directories` option.
 
 <span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
-the mount point with borgmatic's `patterns` or `patterns_from` options.
+the mount point as a root pattern with borgmatic's `patterns` or `patterns_from`
+options.
 
 During a backup, borgmatic snapshots these subvolumes (non-recursively) and
 includes the snapshotted files in the paths sent to Borg. borgmatic is also
@@ -199,6 +205,14 @@ Volume Manager) and sending those snapshots to Borg for backup. LVM isn't
 itself a filesystem, but it can take snapshots at the layer right below your
 filesystem.
 
+Note that, due to Borg being a file-level backup, this feature is really only
+suitable for filesystems, not whole disk or raw images containing multiple
+filesystems (for example, if you're using a LVM volume to run a Windows
+KVM that contains an MBR, partitions, etc.).
+
+In those cases, you can omit the `lvm:` option and use Borg's own support for
+[image backup](https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html).
+
 To use this feature, first you need one or more mounted LVM logical volumes.
 Then, enable LVM within borgmatic by adding the following line to your
 configuration file:
@@ -252,7 +266,8 @@ For any logical volume you'd like backed up, add its mount point to
 borgmatic's `source_directories` option.
 
 <span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
-the mount point with borgmatic's `patterns` or `patterns_from` options.
+the mount point as a root pattern with borgmatic's `patterns` or `patterns_from`
+options.
 
 During a backup, borgmatic automatically snapshots these discovered logical volumes
 (non-recursively), temporarily mounts the snapshots within its [runtime

docs/reference/source-code.md

@@ -18,6 +18,7 @@ you get started. Starting at the top level, we have:
    * [commands](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/commands): Looking to add a new flag or action? Start here. This contains borgmatic's entry point, argument parsing, and shell completion. 
    * [config](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/config): Code responsible for loading, normalizing, and validating borgmatic's configuration. Interested in adding a new configuration option? Check out `schema.yaml` here.
    * [hooks](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks): Looking to add a new database, filesystem, or monitoring integration? Start here.
+     * [credential](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/credential): Credential hooks—for loading passphrases and secrets from external providers.
      * [data_source](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/data_source): Database and filesystem hooks—anything that produces data or files to go into a backup archive.
      * [monitoring](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/monitoring): Monitoring hooks—integrations with third-party or self-hosted monitoring services.
  * [docs](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/docs): How-to and reference documentation, including the document you're reading now.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "borgmatic"
-version = "1.9.7"
+version = "1.9.13"
 authors = [
   { name="Dan Helfman", email="witten@torsion.org" },
 ]