Bump version for release.

SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests.
SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to check.

.bandit

@@ -0,0 +1,3 @@
+[bandit]
+exclude=tests
+skips=S105,S404

.github/pull_request_template.md

@@ -0,0 +1,12 @@
+## Hold up, GitHub users
+
+Thanks for your contribution! 
+
+If you're submitting this PR on **projects.torsion.org**: Please ignore the following. You can safely delete this text before submitting your PR.
+
+If you're submitting this PR on **GitHub**: Unfortunately, we don't use GitHub pull requests to manage code contributions to this repository (and GitHub doesn't have any way to disable pull requests entirely). Instead, please see:
+
+https://torsion.org/borgmatic/#contributing
+
+... which provides full instructions on how to submit pull requests. You can even use your GitHub account to login.
+

NEWS

@@ -1,3 +1,53 @@
+2.0.7
+ * #1032: Fix a bug in which a Borg archive gets created even when a database hook fails.
+ * #1043: Support Btrfs subvolume paths in "source_directories" even when the subvolume is mounted
+   elsewhere.
+ * #1048: Ignore Btrfs subvolumes whose read-only status can't be determined.
+ * #1083: Add "debug_passphrase"/"display_passphrase" options and a "{unixtime}" placeholder in
+   support of Borg 2 features.
+ * #1099: Clarify documentation on command hooks order of execution.
+ * #1100: Fix a bug in which "borg --version" failing isn't considered a "fail" state in a command
+   hook.
+ * #1108: Fix a bug in which quoted "extra_borg_options" values containing spaces are passed to Borg
+   incorrectly.
+ * #1108: Add a "--comment" flag to the "create" action for creating an archive with a comment.
+ * Use the Bandit security analysis tool when running tests.
+ * SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests, e.g. due to
+   a compromised monitoring server holding requests open.
+ * SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to
+   check.
+
+2.0.6
+ * #1068: Fix a warning from LVM about leaked file descriptors.
+ * #1086: Fix for the "spot" check breaking when the "--progress" flag is used.
+ * #1089: Fix for the "spot" check erroring when a checksum command errors.
+ * #1091: Fix for the "config generate" action generating invalid configuration when upgrading
+   deprecated command hooks.
+ * #1093: Fix for the LVM hook erroring when the "--dry-run" flag is used.
+ * #1094: Fix incorrect documentation about customizing Borg exit codes:
+   https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
+ * #1095: Fix for the "spot" check's "xxh64sum_command" option erroring on commands containing
+   spaces.
+ * Add support for Borg 2's "s3:" and "b2:" repository URLs, so you can backup to S3 or B2 cloud
+   storage services even without using Rclone.
+ * During the "spot" check, truncate log messages containing many file paths.
+
+2.0.5
+ * #1033: Add a "password_transport" option to the MariaDB and MySQL database hooks for customizing
+   how borgmatic transmits passwords to the database client.
+ * #1078: Add "keep_3monthly" and "keep_13weekly" options for customizing "prune" action archive
+   retention.
+ * #1078: Add a "use_chunks_archive" option for controlling whether Borg uses its chunks cache
+   directory.
+ * #1078: For the "compact" action, pass "--dry-run" through to Borg.
+ * #1085: Fix a regression in which the default monitoring verbosity is 0 (warnings only) instead of
+   1 (info about steps borgmatic is taking). This prevented logs from showing up in monitoring
+   services like Healthchecks unless you had an explicit monitoring verbosity set.
+ * Move Mastodon social hosting from Fosstodon to FLOSS.social: https://floss.social/@borgmatic
+ * The borgmatic project no longer accepts pull requests on GitHub. But see
+   https://torsion.org/borgmatic/#contributing for how you can still submit pull requests. You can
+   even use your GitHub account to login.
+
 2.0.4
  * #1072: Fix path rewriting for non-root patterns in the ZFS, Btrfs, and LVM hooks.
  * #1073: Clarify the documentation about when an "after: error" command hook runs and how it
@@ -62,7 +112,7 @@
  * #837: Add custom command options for the MongoDB hook.
  * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
  * #1020: Document a database use case involving a temporary database client container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #1037: Fix an error with the "extract" action when both a remote repository and a
    "working_directory" are used.
  * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
@@ -312,7 +362,7 @@
    paths when a "working_directory" is set.
  * #906: Add documentation details for how to run custom database dump commands using binaries from
    running containers:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * Fix a regression in which the "color" option had no effect.
  * Add a recent contributors section to the documentation, because credit where credit's due! See:
    https://torsion.org/borgmatic/#recent-contributors
@@ -368,7 +418,7 @@
  * Fix handling of the NO_COLOR environment variable to ignore an empty value.
  * Add documentation about backing up containerized databases by configuring borgmatic to exec into
    a container to run a dump command:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 
 1.8.9
  * #311: Add custom dump/restore command options for MySQL and MariaDB.
@@ -638,7 +688,7 @@
    at the command-line. See the configuration reference for more information:
    https://torsion.org/borgmatic/docs/reference/configuration/
  * #649: Add documentation on backing up a database running in a container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #655: Fix error when databases are configured and a source directory doesn't exist.
  * Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
    To join in the pedantry, refresh your test environment with "tox --recreate".

README.md

@@ -138,7 +138,8 @@ issues.
 
 ### Social
 
-Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
+Follow <a rel="me" href="https://floss.social/@borgmatic">borgmatic on
+Mastodon</a>.
 
 
 ### Chat
@@ -159,21 +160,21 @@ Other questions or comments? Contact
 ### Contributing
 
 borgmatic [source code is
-available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
+available](https://projects.torsion.org/borgmatic-collective/borgmatic) and also
-on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
+has a read-only mirror on
+[GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
 
 borgmatic is licensed under the GNU General Public License version 3 or any
 later version.
 
-If you'd like to contribute to borgmatic development, please feel free to
+If you'd like to contribute to borgmatic development, please feel free to submit
-submit a [Pull
+a [pull
-Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
+request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
 open an
 [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) to
 discuss your idea. Note that you'll need to
 [register](https://projects.torsion.org/user/sign_up?invite_code=borgmatic)
-first. We also accept Pull Requests on GitHub, if that's more your thing. In
+first. In general, contributions are very welcome. We don't bite!
-general, contributions are very welcome. We don't bite!
 
 Also, please check out the [borgmatic development
 how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for

borgmatic/actions/check.py

@@ -6,7 +6,9 @@ import logging
 import os
 import pathlib
 import random
+import shlex
 import shutil
+import textwrap
 
 import borgmatic.actions.pattern
 import borgmatic.borg.check
@@ -372,7 +374,11 @@ def collect_spot_check_source_paths(
         borgmatic.borg.create.make_base_create_command(
             dry_run=True,
             repository_path=repository['path'],
-            config=dict(config, list_details=True),
+            # Omit "progress" because it interferes with "list_details".
+            config=dict(
+                {option: value for option, value in config.items() if option != 'progress'},
+                list_details=True,
+            ),
             patterns=borgmatic.actions.pattern.process_patterns(
                 borgmatic.actions.pattern.collect_patterns(config),
                 working_directory,
@@ -454,7 +460,7 @@ def collect_spot_check_archive_paths(
     )
 
 
-SAMPLE_PATHS_SUBSET_COUNT = 10000
+SAMPLE_PATHS_SUBSET_COUNT = 5000
 
 
 def compare_spot_check_hashes(
@@ -480,7 +486,7 @@ def compare_spot_check_hashes(
     sample_count = max(
         int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)), 1
     )
-    source_sample_paths = tuple(random.sample(source_paths, sample_count))
+    source_sample_paths = tuple(random.SystemRandom().sample(source_paths, sample_count))
     working_directory = borgmatic.config.paths.get_working_directory(config)
     hashable_source_sample_path = {
         source_path
@@ -508,7 +514,10 @@ def compare_spot_check_hashes(
             break
 
         hash_output = borgmatic.execute.execute_command_and_capture_output(
-            (spot_check_config.get('xxh64sum_command', 'xxh64sum'),)
+            tuple(
+                shlex.quote(part)
+                for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
+            )
             + tuple(
                 path for path in source_sample_paths_subset if path in hashable_source_sample_path
             ),
@@ -562,6 +571,9 @@ def compare_spot_check_hashes(
     return tuple(failing_paths)
 
 
+MAX_SPOT_CHECK_PATHS_LENGTH = 1000
+
+
 def spot_check(
     repository,
     config,
@@ -629,9 +641,12 @@ def spot_check(
     logger.debug(f'{len(archive_paths)} total archive paths for spot check')
 
     if len(source_paths) == 0:
-        logger.debug(
+        truncated_archive_paths = textwrap.shorten(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+            ', '.join(set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
         )
+        logger.debug(f'Paths in latest archive but not source paths: {truncated_archive_paths}')
         raise ValueError(
             'Spot check failed: There are no source paths to compare against the archive'
         )
@@ -642,11 +657,21 @@ def spot_check(
 
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
         rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
-        logger.debug(
+        truncated_exclusive_source_paths = textwrap.shorten(
-            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
+            ', '.join(rootless_source_paths - set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
         )
         logger.debug(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
+            f'Paths in source paths but not latest archive: {truncated_exclusive_source_paths}'
+        )
+        truncated_exclusive_archive_paths = textwrap.shorten(
+            ', '.join(set(archive_paths) - rootless_source_paths) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.debug(
+            f'Paths in latest archive but not source paths: {truncated_exclusive_archive_paths}'
         )
         raise ValueError(
             f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
@@ -669,8 +694,13 @@ def spot_check(
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
+        truncated_failing_paths = textwrap.shorten(
+            ', '.join(failing_paths),
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
         logger.debug(
-            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {truncated_failing_paths}'
         )
         raise ValueError(
             f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'

borgmatic/actions/create.py

@@ -2,6 +2,8 @@ import logging
 
 import borgmatic.actions.json
 import borgmatic.borg.create
+import borgmatic.borg.rename
+import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.dispatch
@@ -72,6 +74,12 @@ def run_create(
         )
         stream_processes = [process for processes in active_dumps.values() for process in processes]
 
+        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+        # to make sure we only create a real archive if all the streaming processes completed
+        # successfully (create_archive will fail if a streaming process fails, but the archive might
+        # have already been created at this point).
+        use_checkpoint = bool(stream_processes)
+
         json_output = borgmatic.borg.create.create_archive(
             global_arguments.dry_run,
             repository['path'],
@@ -80,14 +88,41 @@ def run_create(
             local_borg_version,
             global_arguments,
             borgmatic_runtime_directory,
+            archive_suffix='.checkpoint' if use_checkpoint else '',
             local_path=local_path,
             remote_path=remote_path,
             json=create_arguments.json,
+            comment=create_arguments.comment,
             stream_processes=stream_processes,
         )
 
+        if use_checkpoint:
+            rename_checkpoint_archive(
+                repository['path'],
+                global_arguments,
+                config,
+                local_borg_version,
+                local_path,
+                remote_path,
+            )
+
         if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+            if use_checkpoint:
+                # Patch archive name and ID
+                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                    repository['path'],
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                )
+
+                output['archive']['name'] = renamed_archive['name']
+                output['archive']['id'] = renamed_archive['id']
+
+            yield output
 
         borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
             'remove_data_source_dumps',
@@ -96,3 +131,54 @@ def run_create(
             borgmatic_runtime_directory,
             global_arguments.dry_run,
         )
+
+
+def rename_checkpoint_archive(
+    repository_path,
+    global_arguments,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    '''
+    Renames the latest archive to not have a '.checkpoint' suffix.
+
+    Raises ValueError if
+    - there is not latest archive
+    - the latest archive does not have a '.checkpoint' suffix
+
+    Implementation note: We cannot reliably get the just created archive name.
+    So we resort to listing the archives and picking the last one.
+
+    A similar comment applies to retrieving the ID of the renamed archive.
+    '''
+    archive = borgmatic.borg.repo_list.get_latest_archive(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+        consider_checkpoints=True,
+    )
+
+    archive_name = archive['name']
+
+    if not archive_name.endswith('.checkpoint'):
+        raise ValueError(f'Latest archive did not have a .checkpoint suffix. Got: {archive_name}')
+
+    new_archive_name = archive_name.removesuffix('.checkpoint')
+
+    logger.info(f'Renaming archive {archive_name} -> {new_archive_name}')
+
+    borgmatic.borg.rename.rename_archive(
+        repository_path,
+        archive_name,
+        new_archive_name,
+        global_arguments.dry_run,
+        config,
+        local_borg_version,
+        local_path,
+        remote_path,
+    )

borgmatic/borg/borg.py

@@ -60,7 +60,7 @@ def run_arbitrary_borg(
     return execute_command(
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
-        shell=True,
+        shell=True,  # noqa: S604
         environment=dict(
             (environment.make_environment(config) or {}),
             **{

borgmatic/borg/check.py

@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
+import shlex
 
 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags, repo_info
@@ -171,7 +172,7 @@ def check_archives(
             + (('--lock-wait', str(lock_wait)) if lock_wait else ())
             + verbosity_flags
             + (('--progress',) if config.get('progress') else ())
-            + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+            + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
             + flags.make_repository_flags(repository_path, local_borg_version)
         )
 

borgmatic/borg/compact.py

@@ -1,7 +1,8 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
-from borgmatic.borg import environment, flags
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
@@ -37,11 +38,16 @@ def compact_segments(
         + (('--threshold', str(threshold)) if threshold else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (
+            ('--dry-run',)
+            if dry_run and feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version)
+            else ()
+        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    if dry_run:
+    if dry_run and not feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version):
         logging.info('Skipping compact (dry run)')
         return
 

borgmatic/borg/create.py

@@ -1,6 +1,7 @@
 import logging
 import os
 import pathlib
+import shlex
 import stat
 import textwrap
 
@@ -121,17 +122,21 @@ def make_base_create_command(
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
     json=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
     Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version,
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version, global
-    global arguments as an argparse.Namespace instance, and a sequence of borgmatic source
+    arguments as an argparse.Namespace instance, the borgmatic runtime directory, a string suffix to
-    directories, return a tuple of (base Borg create command flags, Borg create command positional
+    add to the archive name, the local Borg path, the remote Borg path, whether to output JSON,
-    arguments, open pattern file handle).
+    comment text to add to the created archive, and a sequence of processes streaming data to Borg,
+    return a tuple of (base Borg create command flags, Borg create command positional arguments,
+    open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
         borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
@@ -149,8 +154,9 @@ def make_base_create_command(
     lock_wait = config.get('lock_wait', None)
     list_filter_flags = flags.make_list_filter_flags(local_borg_version, dry_run)
     files_cache = config.get('files_cache')
-    archive_name_format = config.get(
+    archive_name_format = (
-        'archive_name_format', flags.get_default_archive_name_format(local_borg_version)
+        config.get('archive_name_format', flags.get_default_archive_name_format(local_borg_version))
+        + archive_suffix
     )
     extra_borg_options = config.get('extra_borg_options', {}).get('create', '')
 
@@ -183,6 +189,7 @@ def make_base_create_command(
         + ('create',)
         + (('--patterns-from', patterns_file.name) if patterns_file else ())
         + flags.make_exclude_flags(config)
+        + (('--comment', comment) if comment else ())
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
         + (('--chunker-params', chunker_params) if chunker_params else ())
@@ -207,7 +214,7 @@ def make_base_create_command(
             else ()
         )
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
     )
 
     create_positional_arguments = flags.make_repository_archive_flags(
@@ -269,15 +276,20 @@ def create_archive(
     local_borg_version,
     global_arguments,
     borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
     json=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
     Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, and global arguments as an
+    sequence of loaded configuration paths, the local Borg version, global arguments as an
-    argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
+    argparse.Namespace instance, the borgmatic runtime directory, a string suffix to add to the
+    archive name, the local Borg path, the remote Borg path, whether to output JSON, and comment
+    text to add to the created archive, and a sequence of processes streaming data to Borg, create a
+    Borg archive and return Borg's JSON output (if any).
 
     If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
     create command while also triggering the given processes to produce output.
@@ -294,9 +306,11 @@ def create_archive(
         local_borg_version,
         global_arguments,
         borgmatic_runtime_directory,
+        archive_suffix,
         local_path,
         remote_path,
         json,
+        comment,
         stream_processes,
     )
 

borgmatic/borg/environment.py

@@ -14,13 +14,16 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'temporary_directory': 'TMPDIR',
 }
 
-DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
+DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE = {
-    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
+    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
-    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
 }
 
-DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
+DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
-    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
+    'debug_passphrase': 'BORG_DEBUG_PASSPHRASE',
+    'display_passphrase': 'BORG_DISPLAY_PASSPHRASE',
+    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
+    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
 }
 
 
@@ -82,15 +85,15 @@ def make_environment(config):
     for (
         option_name,
         environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
         if os.environ.get(environment_variable_name) is None:
             value = config.get(option_name)
-            environment[environment_variable_name] = 'yes' if value else 'no'
+            environment[environment_variable_name] = 'YES' if value else 'NO'
 
     for (
         option_name,
         environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
         if value is not None:
             environment[environment_variable_name] = 'YES' if value else 'NO'

borgmatic/borg/feature.py

@@ -18,6 +18,7 @@ class Feature(Enum):
     EXCLUDED_FILES_MINUS = 12
     ARCHIVE_SERIES = 13
     NO_PRUNE_STATS = 14
+    DRY_RUN_COMPACT = 15
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -35,6 +36,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
     Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
     Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
+    Feature.DRY_RUN_COMPACT: parse('1.4.1'),  # borg compact --dry-run support
 }
 
 

borgmatic/borg/prune.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -90,7 +91,7 @@ def prune_archives(
         + (('--list',) if config.get('list_details') else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/rename.py

@@ -0,0 +1,62 @@
+import logging
+
+import borgmatic.borg.flags
+
+logger = logging.getLogger(__name__)
+
+
+def make_rename_command(
+    dry_run,
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    return (
+        (local_path, 'rename')
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + borgmatic.borg.flags.make_flags('dry-run', dry_run)
+        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
+        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
+        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
+        + borgmatic.borg.flags.make_repository_archive_flags(
+            repository_name, old_archive_name, local_borg_version
+        )
+        + (new_archive_name,)
+    )
+
+
+def rename_archive(
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    dry_run,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    command = make_rename_command(
+        dry_run,
+        repository_name,
+        old_archive_name,
+        new_archive_name,
+        config,
+        local_borg_version,
+        local_path,
+        remote_path,
+    )
+
+    borgmatic.execute.execute_command(
+        command,
+        output_log_level=logging.INFO,
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/repo_create.py

@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
+import shlex
 import subprocess
 
 import borgmatic.config.paths
@@ -86,7 +87,7 @@ def create_repository(
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/repo_list.py

@@ -1,4 +1,5 @@
 import argparse
+import json
 import logging
 
 import borgmatic.config.paths
@@ -29,6 +30,33 @@ def resolve_archive_name(
     if archive != 'latest':
         return archive
 
+    latest_archive = get_latest_archive(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
+
+    return latest_archive['name']
+
+
+def get_latest_archive(
+    repository_path,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+    consider_checkpoints=False,
+):
+    '''
+    Returns a dict with information about the latest archive of a repository.
+
+    Raises ValueError if there are no archives in the repository.
+    '''
+
     full_command = (
         (
             local_path,
@@ -42,24 +70,28 @@ def resolve_archive_name(
         + flags.make_flags('umask', config.get('umask'))
         + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', config.get('lock_wait'))
+        + flags.make_flags('consider-checkpoints', consider_checkpoints)
         + flags.make_flags('last', 1)
-        + ('--short',)
+        + ('--json',)
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    output = execute_command_and_capture_output(
+    json_output = execute_command_and_capture_output(
         full_command,
         environment=environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
+
+    archives = json.loads(json_output)['archives']
+
     try:
-        latest_archive = output.strip().splitlines()[-1]
+        latest_archive = archives[-1]
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive["name"]}')
 
     return latest_archive
 

borgmatic/commands/arguments.py

@@ -667,7 +667,7 @@ def make_parsers(schema, unparsed_arguments):
         dest='make_parent_directories',
         default=None,
         action='store_true',
-        help='Create any missing parent directories of the repository directory',
+        help='Create any missing parent directories of the repository directory [Borg 1.x only]',
     )
     repo_create_group.add_argument(
         '-h', '--help', action='help', help='Show this help message and exit'
@@ -872,6 +872,11 @@ def make_parsers(schema, unparsed_arguments):
     create_group.add_argument(
         '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
     )
+    create_group.add_argument(
+        '--comment',
+        metavar='COMMENT',
+        help='Add a comment text to the archive',
+    )
     create_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
 
     check_parser = action_parsers.add_parser(

borgmatic/commands/borgmatic.py

@@ -95,7 +95,9 @@ class Monitoring_hooks:
         self.config_filename = config_filename
         self.config = config
         self.dry_run = global_arguments.dry_run
-        self.monitoring_log_level = verbosity_to_log_level(config.get('monitoring_verbosity'))
+        self.monitoring_log_level = verbosity_to_log_level(
+            get_verbosity({config_filename: config}, 'monitoring_verbosity')
+        )
         self.monitoring_hooks_are_activated = (
             using_primary_action and self.monitoring_log_level != DISABLED
         )
@@ -226,7 +228,7 @@ def run_configuration(config_filename, config, config_paths, arguments):
                     yield from log_error_records(
                         f'{config_filename}: Error getting local Borg version', error
                     )
-                    return
+                    raise
 
                 for repo in config['repositories']:
                     repo_queue.put(
@@ -287,11 +289,7 @@ def run_configuration(config_filename, config, config_paths, arguments):
                     raise encountered_error
 
     except (OSError, CalledProcessError, ValueError) as error:
-        # No need to repeat logging of the error if it was already logged above.
+        yield from log_error_records('Error running configuration')
-        if error_repository:
-            yield from log_error_records('Error running configuration')
-        else:
-            yield from log_error_records('Error running configuration', error)
 
         encountered_error = error
 
@@ -752,7 +750,7 @@ def log_error_records(
     except (ValueError, OSError) as error:
         yield log_record(levelno=levelno, levelname=level_name, msg=str(message))
         yield log_record(levelno=levelno, levelname=level_name, msg=str(error))
-    except:  # noqa: E722
+    except:  # noqa: E722, S110
         # Raising above only as a means of determining the error type. Swallow the exception here
         # because we don't want the exception to propagate out of this function.
         pass

borgmatic/config/generate.py

@@ -51,7 +51,7 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
     elif borgmatic.config.schema.compare_types(schema_type, {'object'}):
         if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
-            source_config = dict(collections.ChainMap(*source_config))
+            source_config = source_config[0]
 
         config = (
             ruamel.yaml.comments.CommentedMap(
@@ -71,7 +71,7 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
         )
         indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
         add_comments_to_configuration_object(
-            config, schema, source_config, indent=indent, skip_first=parent_is_sequence
+            config, schema, source_config, indent=indent, skip_first_field=parent_is_sequence
         )
     elif borgmatic.config.schema.compare_types(schema_type, SCALAR_SCHEMA_TYPES, match=all):
         return example
@@ -108,17 +108,30 @@ def comment_out_optional_configuration(rendered_config):
     '''
     lines = []
     optional = False
+    indent_characters = None
+    indent_characters_at_sentinel = None
 
     for line in rendered_config.split('\n'):
+        indent_characters = len(line) - len(line.lstrip())
+
         # Upon encountering an optional configuration option, comment out lines until the next blank
         # line.
         if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
             optional = True
+            indent_characters_at_sentinel = indent_characters
             continue
 
         # Hit a blank line, so reset commenting.
         if not line.strip():
             optional = False
+            indent_characters_at_sentinel = None
+        # Dedented, so reset commenting.
+        elif (
+            indent_characters_at_sentinel is not None
+            and indent_characters < indent_characters_at_sentinel
+        ):
+            optional = False
+            indent_characters_at_sentinel = None
 
         lines.append(comment_out_line(line) if optional else line)
 
@@ -198,25 +211,28 @@ COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 
 
 def add_comments_to_configuration_object(
-    config, schema, source_config=None, indent=0, skip_first=False
+    config, schema, source_config=None, indent=0, skip_first_field=False
 ):
     '''
     Using descriptions from a schema as a source, add those descriptions as comments to the given
     configuration dict, putting them before each field. Indent the comment the given number of
     characters.
 
+    If skip_first_field is True, omit the comment for the initial field. This is useful for
+    sequences, where the comment for the first field goes before the sequence itself.
+
     And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
     source configuration dict. The idea is that any options used in the source configuration should
     stay active in the generated configuration.
     '''
     for index, field_name in enumerate(config.keys()):
-        if skip_first and index == 0:
+        if skip_first_field and index == 0:
             continue
 
         field_schema = borgmatic.config.schema.get_properties(schema).get(field_name, {})
         description = field_schema.get('description', '').strip()
 
-        # If this isn't a default key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment, flagging it to be commented
         # out from the sample configuration. This sentinel is consumed by downstream processing that
         # does the actual commenting out.
         if field_name not in DEFAULT_KEYS and (
@@ -299,6 +315,12 @@ def generate_sample_configuration(
         source_config = load.load_configuration(source_filename)
         normalize.normalize(source_filename, source_config)
 
+        # The borgmatic.config.normalize.normalize() function tacks on an empty "bootstrap" if
+        # needed, so the hook gets used by default. But we don't want it to end up in the generated
+        # config unless the user has set it explicitly, as an empty "bootstrap:" won't validate.
+        if source_config and source_config.get('bootstrap') == {}:
+            del source_config['bootstrap']
+
     destination_config = merge_source_configuration_into_destination(
         schema_to_sample_configuration(schema, source_config), source_config
     )

borgmatic/config/normalize.py

@@ -159,7 +159,7 @@ def normalize(config_filename, config):
                 dict(
                     levelno=logging.WARNING,
                     levelname='WARNING',
-                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use borgmatic_runtime_directory and borgmatic_state_directory instead.',
+                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use user_runtime_directory and user_state_directory instead.',
                 )
             )
         )
@@ -356,6 +356,8 @@ def normalize(config_filename, config):
                     repository_path.startswith('ssh://')
                     or repository_path.startswith('sftp://')
                     or repository_path.startswith('rclone:')
+                    or repository_path.startswith('s3:')
+                    or repository_path.startswith('b2:')
                 ):
                     config['repositories'].append(repository_dict)
                 else:
@@ -365,7 +367,7 @@ def normalize(config_filename, config):
                             dict(
                                 levelno=logging.WARNING,
                                 levelname='WARNING',
-                                msg=f'{config_filename}: Remote repository paths without ssh:// or rclone: syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
+                                msg=f'{config_filename}: Remote repository paths without ssh://, sftp://, rclone:, s3:, or b2:, syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
                             )
                         )
                     )

borgmatic/config/paths.py

@@ -103,7 +103,9 @@ class Runtime_directory:
 
             self.temporary_directory = None
         else:
-            base_directory = os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'
+            base_directory = (
+                os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'  # noqa: S108
+            )
 
             if not base_directory.startswith(os.path.sep):
                 raise ValueError('The temporary directory must be an absolute path')

borgmatic/config/schema.yaml

@@ -72,7 +72,8 @@ properties:
                     description: |
                         Whether any missing parent directories of the repository
                         path should be created, only used for the repo-create
-                        action. Defaults to false.
+                        action. Defaults to false. (This option is supported 
+                        for Borg 1.x only)
                     example: true
         description: |
             A required list of local or remote repositories with paths and
@@ -389,6 +390,13 @@ properties:
             Path for Borg cache files. Defaults to
             $borg_base_directory/.cache/borg
         example: /path/to/base/cache
+    use_chunks_archive:
+        type: boolean
+        description: |
+            Enables or disables the use of chunks.archive.d for faster cache
+            resyncs in Borg. If true, value is set to "yes" (default) else
+            it's set to "no", reducing disk usage but slowing resyncs.
+        example: true
     borg_files_cache_ttl:
         type: integer
         description: |
@@ -483,6 +491,19 @@ properties:
             Bypass Borg error about a previously unknown unencrypted repository.
             Defaults to false.
         example: true
+    debug_passphrase:
+        type: boolean
+        description: |
+            When set true, display debugging information that includes 
+            passphrases used and passphrase related environment variables set. 
+            Defaults to false.
+        example: true
+    display_passphrase:
+        type: boolean
+        description: |
+            When set true, always shows passphrase and its hex UTF-8 byte
+            sequence. Defaults to false.
+        example: true
     check_i_know_what_i_am_doing:
         type: boolean
         description: |
@@ -558,6 +579,14 @@ properties:
         type: integer
         description: Number of yearly archives to keep.
         example: 1
+    keep_13weekly:
+        type: integer
+        description: Number of quarterly archives to keep (13 week strategy).
+        example: 13
+    keep_3monthly:
+        type: integer
+        description: Number of quarterly archives to keep (3 month strategy).
+        example: 3
     prefix:
         type: string
         description: |
@@ -1545,6 +1574,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MariaDB client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mariadb_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |
@@ -1693,6 +1739,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MySQL client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mysql_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |

borgmatic/execute.py

@@ -288,6 +288,7 @@ def execute_command(
     borg_local_path=None,
     borg_exit_codes=None,
     run_to_completion=True,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -312,11 +313,10 @@ def execute_command(
         stdin=input_file,
         stdout=None if do_not_capture else (output_file or subprocess.PIPE),
         stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
-        shell=shell,
+        shell=shell,  # noqa: S602
         env=environment,
         cwd=working_directory,
-        # Necessary for passing credentials via anonymous pipe.
+        close_fds=close_fds,
-        close_fds=False,
     )
     if not run_to_completion:
         return process
@@ -340,6 +340,7 @@ def execute_command_and_capture_output(
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
@@ -362,11 +363,10 @@ def execute_command_and_capture_output(
             command,
             stdin=input_file,
             stderr=subprocess.STDOUT if capture_stderr else None,
-            shell=shell,
+            shell=shell,  # noqa: S602
             env=environment,
             cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
+            close_fds=close_fds,
-            close_fds=False,
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -390,6 +390,7 @@ def execute_command_with_processes(
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -422,11 +423,10 @@ def execute_command_with_processes(
             stderr=(
                 None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
             ),
-            shell=shell,
+            shell=shell,  # noqa: S602
             env=environment,
             cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
+            close_fds=close_fds,
-            close_fds=False,
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.

borgmatic/hooks/command.py

@@ -19,6 +19,7 @@ BORG_PLACEHOLDERS = {
     '{reverse-fqdn}',
     '{now}',
     '{utcnow}',
+    '{unixtime}',
     '{user}',
     '{pid}',
     '{borgversion}',
@@ -143,7 +144,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                     output_log_level=(
                         logging.ERROR if hook_config.get('after') == 'error' else logging.ANSWER
                     ),
-                    shell=True,
+                    shell=True,  # noqa: S604
                     environment=make_environment(os.environ),
                     working_directory=working_directory,
                 )

borgmatic/hooks/data_source/btrfs.py

@@ -1,5 +1,6 @@
 import collections
 import glob
+import itertools
 import json
 import logging
 import os
@@ -21,9 +22,44 @@ def use_streaming(hook_config, config):  # pragma: no cover
     return False
 
 
-def get_subvolume_mount_points(findmnt_command):
+def get_contained_subvolume_paths(btrfs_command, subvolume_path):
     '''
-    Given a findmnt command to run, get all sorted Btrfs subvolume mount points.
+    Given the path of a Btrfs subvolume, return it in a sequence along with the paths of its
+    contained subvolumes.
+
+    If the btrfs command errors, log that error and return an empty sequence.
+    '''
+    try:
+        btrfs_output = borgmatic.execute.execute_command_and_capture_output(
+            tuple(btrfs_command.split(' '))
+            + (
+                'subvolume',
+                'list',
+                subvolume_path,
+            ),
+            close_fds=True,
+        )
+    except subprocess.CalledProcessError as error:
+        logger.debug(
+            f'Ignoring Btrfs subvolume {subvolume_path} because of error listing its subvolumes: {error}'
+        )
+
+        return ()
+
+    return (subvolume_path,) + tuple(
+        os.path.join(subvolume_path, line.split(' ')[-1])
+        for line in btrfs_output.splitlines()
+        if line.strip()
+    )
+
+
+FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION = 'subvolid=5'
+
+
+def get_all_subvolume_paths(btrfs_command, findmnt_command):
+    '''
+    Given btrfs and findmnt commands to run, get the sorted paths for all Btrfs subvolumes on the
+    system.
     '''
     findmnt_output = borgmatic.execute.execute_command_and_capture_output(
         tuple(findmnt_command.split(' '))
@@ -32,12 +68,28 @@ def get_subvolume_mount_points(findmnt_command):
             'btrfs',
             '--json',
             '--list',  # Request a flat list instead of a nested subvolume hierarchy.
-        )
+        ),
+        close_fds=True,
     )
 
     try:
         return tuple(
-            sorted(filesystem['target'] for filesystem in json.loads(findmnt_output)['filesystems'])
+            sorted(
+                itertools.chain.from_iterable(
+                    # If findmnt gave us a Btrfs root filesystem, list the subvolumes within it.
+                    # This is necessary because findmnt only returns a subvolume's mount point
+                    # rather than its original subvolume path (which can differ). For instance,
+                    # a subvolume might exist at /mnt/subvolume but be mounted at /home/myuser.
+                    # findmnt is still useful though because it's a global way to discover all
+                    # Btrfs subvolumes—even if we have to do some additional legwork ourselves.
+                    (
+                        get_contained_subvolume_paths(btrfs_command, filesystem['target'])
+                        if FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION in filesystem['options'].split(',')
+                        else (filesystem['target'],)
+                    )
+                    for filesystem in json.loads(findmnt_output)['filesystems']
+                )
+            )
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
@@ -49,6 +101,12 @@ Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'),
 
 
 def get_subvolume_property(btrfs_command, subvolume_path, property_name):
+    '''
+    Given a btrfs command, a subvolume path, and a property name to lookup, return the value of the
+    corresponding property.
+
+    Raise subprocess.CalledProcessError if the btrfs command errors.
+    '''
     output = borgmatic.execute.execute_command_and_capture_output(
         tuple(btrfs_command.split(' '))
         + (
@@ -59,6 +117,7 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
             subvolume_path,
             property_name,
         ),
+        close_fds=True,
     )
 
     try:
@@ -72,19 +131,24 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
     }.get(value, value)
 
 
-def omit_read_only_subvolume_mount_points(btrfs_command, subvolume_paths):
+def omit_read_only_subvolume_paths(btrfs_command, subvolume_paths):
     '''
-    Given a Btrfs command to run and a sequence of Btrfs subvolume mount points, filter them down to
+    Given a Btrfs command to run and a sequence of Btrfs subvolume paths, filter them down to just
-    just those that are read-write. The idea is that Btrfs can't actually snapshot a read-only
+    those that are read-write. The idea is that Btrfs can't actually snapshot a read-only subvolume,
-    subvolume, so we should just ignore them.
+    so we should just ignore them.
     '''
     retained_subvolume_paths = []
 
     for subvolume_path in subvolume_paths:
-        if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
+        try:
-            logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
+            if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
-        else:
+                logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
-            retained_subvolume_paths.append(subvolume_path)
+            else:
+                retained_subvolume_paths.append(subvolume_path)
+        except subprocess.CalledProcessError as error:
+            logger.debug(
+                f'Error determining read-only status of Btrfs subvolume {subvolume_path}: {error}'
+            )
 
     return tuple(retained_subvolume_paths)
 
@@ -92,32 +156,32 @@ def omit_read_only_subvolume_mount_points(btrfs_command, subvolume_paths):
 def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     '''
     Given a Btrfs command to run and a sequence of configured patterns, find the intersection
-    between the current Btrfs filesystem and subvolume mount points and the paths of any patterns.
+    between the current Btrfs filesystem and subvolume paths and the paths of any patterns.  The
-    The idea is that these pattern paths represent the requested subvolumes to snapshot.
+    idea is that these pattern paths represent the requested subvolumes to snapshot.
 
     Only include subvolumes that contain at least one root pattern sourced from borgmatic
     configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
     return all subvolumes instead, sorted by path.
 
-    Return the result as a sequence of matching subvolume mount points.
+    Return the result as a sequence of matching Subvolume instances.
     '''
     candidate_patterns = set(patterns or ())
     subvolumes = []
 
-    # For each subvolume mount point, match it against the given patterns to find the subvolumes to
+    # For each subvolume path, match it against the given patterns to find the subvolumes to
     # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
     # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
     # this process, so no two subvolumes end up with the same contained patterns.)
-    for mount_point in reversed(
+    for subvolume_path in reversed(
-        omit_read_only_subvolume_mount_points(
+        omit_read_only_subvolume_paths(
-            btrfs_command, get_subvolume_mount_points(findmnt_command)
+            btrfs_command, get_all_subvolume_paths(btrfs_command, findmnt_command)
         )
     ):
         subvolumes.extend(
-            Subvolume(mount_point, contained_patterns)
+            Subvolume(subvolume_path, contained_patterns)
             for contained_patterns in (
                 borgmatic.hooks.data_source.snapshot.get_contained_patterns(
-                    mount_point, candidate_patterns
+                    subvolume_path, candidate_patterns
                 ),
             )
             if patterns is None
@@ -227,6 +291,7 @@ def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma:
             snapshot_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -241,9 +306,9 @@ def dump_data_sources(
     '''
     Given a Btrfs configuration dict, a configuration dict, the borgmatic configuration file paths,
     the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
-    auto-detect and snapshot any Btrfs subvolume mount points listed in the given patterns. Also
+    auto-detect and snapshot any Btrfs subvolume paths listed in the given patterns. Also update
-    update those patterns, replacing subvolume mount points with corresponding snapshot directories
+    those patterns, replacing subvolume paths with corresponding snapshot directories so they get
-    so they get stored in the Borg archive instead.
+    stored in the Borg archive instead.
 
     Return an empty sequence, since there are no ongoing dump processes from this hook.
 
@@ -298,6 +363,7 @@ def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
             snapshot_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -324,8 +390,8 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
         logger.debug(error)
         return
 
-    # Reversing the sorted subvolumes ensures that we remove longer mount point paths of child
+    # Reversing the sorted subvolumes ensures that we remove longer paths of child subvolumes before
-    # subvolumes before the shorter mount point paths of parent subvolumes.
+    # the shorter paths of parent subvolumes.
     for subvolume in reversed(all_subvolumes):
         subvolume_snapshots_glob = borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
             os.path.normpath(make_snapshot_path(subvolume.path)),

borgmatic/hooks/data_source/lvm.py

@@ -50,7 +50,8 @@ def get_logical_volumes(lsblk_command, patterns=None):
                     'name,path,mountpoint,type',
                     '--json',
                     '--list',
-                )
+                ),
+                close_fds=True,
             )
         )
     except json.JSONDecodeError as error:
@@ -109,6 +110,7 @@ def snapshot_logical_volume(
             logical_volume_device,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -129,6 +131,7 @@ def mount_snapshot(mount_command, snapshot_device, snapshot_mount_path):  # prag
             snapshot_mount_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -227,12 +230,13 @@ def dump_data_sources(
             )
 
         # Get the device path for the snapshot we just created.
-        try:
+        if not dry_run:
-            snapshot = get_snapshots(
+            try:
-                hook_config.get('lvs_command', 'lvs'), snapshot_name=snapshot_name
+                snapshot = get_snapshots(
-            )[0]
+                    hook_config.get('lvs_command', 'lvs'), snapshot_name=snapshot_name
-        except IndexError:
+                )[0]
-            raise ValueError(f'Cannot find LVM snapshot {snapshot_name}')
+            except IndexError:
+                raise ValueError(f'Cannot find LVM snapshot {snapshot_name}')
 
         # Mount the snapshot into a particular named temporary directory so that the snapshot ends
         # up in the Borg archive at the "original" logical volume mount point path.
@@ -277,6 +281,7 @@ def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
     borgmatic.execute.execute_command(
         tuple(umount_command.split(' ')) + (snapshot_mount_path,),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -291,6 +296,7 @@ def remove_snapshot(lvremove_command, snapshot_device_path):  # pragma: no cover
             snapshot_device_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -318,7 +324,8 @@ def get_snapshots(lvs_command, snapshot_name=None):
                     'lv_name,lv_path',
                     '--select',
                     'lv_attr =~ ^s',  # Filter to just snapshots.
-                )
+                ),
+                close_fds=True,
             )
         )
     except json.JSONDecodeError as error:

borgmatic/hooks/data_source/mariadb.py

@@ -120,13 +120,19 @@ def database_names_to_dump(database, config, username, password, environment, dr
         shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mariadb_show_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -184,14 +190,20 @@ def execute_dump_command(
         for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mariadb_dump_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -255,7 +267,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -383,18 +402,27 @@ def restore_data_source_dump(
         shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mariadb_restore_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/mongodb.py

@@ -74,11 +74,12 @@ def dump_data_sources(
 
         if dump_format == 'directory':
             dump.create_parent_directory_for_dump(dump_filename)
-            execute_command(command, shell=True)
+            execute_command(command, shell=True)  # noqa: S604
         else:
             dump.create_named_pipe_for_dump(dump_filename)
-            processes.append(execute_command(command, shell=True, run_to_completion=False))
+            processes.append(
-
+                execute_command(command, shell=True, run_to_completion=False)  # noqa: S604
+            )
     if not dry_run:
         patterns.append(
             borgmatic.borg.pattern.Pattern(

borgmatic/hooks/data_source/mysql.py

@@ -45,15 +45,21 @@ def database_names_to_dump(database, config, username, password, environment, dr
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('list_options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mysql_show_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+        + (
-            username, password, defaults_extra_filename
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -109,16 +115,22 @@ def execute_dump_command(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mysql_dump_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+        + (
-            username, password, defaults_extra_filename
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -182,7 +194,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -312,20 +331,29 @@ def restore_data_source_dump(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
     )
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mysql_restore_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+        + (
-            username, password, defaults_extra_filename
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/postgresql.py

@@ -224,7 +224,7 @@ def dump_data_sources(
                 dump.create_parent_directory_for_dump(dump_filename)
                 execute_command(
                     command,
-                    shell=True,
+                    shell=True,  # noqa: S604
                     environment=environment,
                 )
             else:
@@ -232,7 +232,7 @@ def dump_data_sources(
                 processes.append(
                     execute_command(
                         command,
-                        shell=True,
+                        shell=True,  # noqa: S604
                         environment=environment,
                         run_to_completion=False,
                     )

borgmatic/hooks/data_source/sqlite.py

@@ -88,7 +88,9 @@ def dump_data_sources(
             continue
 
         dump.create_named_pipe_for_dump(dump_filename)
-        processes.append(execute_command(command, shell=True, run_to_completion=False))
+        processes.append(
+            execute_command(command, shell=True, run_to_completion=False)  # noqa: S604
+        )
 
     if not dry_run:
         patterns.append(

borgmatic/hooks/data_source/zfs.py

@@ -53,7 +53,8 @@ def get_datasets_to_backup(zfs_command, patterns):
             'filesystem',
             '-o',
             f'name,mountpoint,canmount,{BORGMATIC_USER_PROPERTY}',
-        )
+        ),
+        close_fds=True,
     )
 
     try:
@@ -131,7 +132,8 @@ def get_all_dataset_mount_points(zfs_command):
             'filesystem',
             '-o',
             'mountpoint',
-        )
+        ),
+        close_fds=True,
     )
 
     return tuple(
@@ -158,6 +160,7 @@ def snapshot_dataset(zfs_command, full_snapshot_name):  # pragma: no cover
             full_snapshot_name,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -180,6 +183,7 @@ def mount_snapshot(mount_command, full_snapshot_name, snapshot_mount_path):  # p
             snapshot_mount_path,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -310,6 +314,7 @@ def unmount_snapshot(umount_command, snapshot_mount_path):  # pragma: no cover
     borgmatic.execute.execute_command(
         tuple(umount_command.split(' ')) + (snapshot_mount_path,),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -325,6 +330,7 @@ def destroy_snapshot(zfs_command, full_snapshot_name):  # pragma: no cover
             full_snapshot_name,
         ),
         output_log_level=logging.DEBUG,
+        close_fds=True,
     )
 
 
@@ -342,7 +348,8 @@ def get_all_snapshots(zfs_command):
             'snapshot',
             '-o',
             'name',
-        )
+        ),
+        close_fds=True,
     )
 
     return tuple(line.rstrip() for line in list_output.splitlines())

borgmatic/hooks/monitoring/cronhub.py

@@ -11,6 +11,7 @@ MONITOR_STATE_TO_CRONHUB = {
     monitor.State.FINISH: 'finish',
     monitor.State.FAIL: 'fail',
 }
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -45,7 +46,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
-            response = requests.get(ping_url)
+            response = requests.get(ping_url, timeout=TIMEOUT_SECONDS)
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/cronitor.py

@@ -11,6 +11,7 @@ MONITOR_STATE_TO_CRONITOR = {
     monitor.State.FINISH: 'complete',
     monitor.State.FAIL: 'fail',
 }
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -40,7 +41,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     if not dry_run:
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
-            response = requests.get(ping_url)
+            response = requests.get(ping_url, timeout=TIMEOUT_SECONDS)
             if not response.ok:
                 response.raise_for_status()
         except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/healthchecks.py

@@ -17,6 +17,7 @@ MONITOR_STATE_TO_HEALTHCHECKS = {
 
 DEFAULT_PING_BODY_LIMIT_BYTES = 100000
 HANDLER_IDENTIFIER = 'healthchecks'
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
@@ -84,7 +85,10 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         logging.getLogger('urllib3').setLevel(logging.ERROR)
         try:
             response = requests.post(
-                ping_url, data=payload.encode('utf-8'), verify=hook_config.get('verify_tls', True)
+                ping_url,
+                data=payload.encode('utf-8'),
+                verify=hook_config.get('verify_tls', True),
+                timeout=TIMEOUT_SECONDS,
             )
             if not response.ok:
                 response.raise_for_status()

borgmatic/hooks/monitoring/loki.py

@@ -15,6 +15,7 @@ MONITOR_STATE_TO_LOKI = {
     monitor.State.FINISH: 'Finished',
     monitor.State.FAIL: 'Failed',
 }
+TIMEOUT_SECONDS = 10
 
 # Threshold at which logs get flushed to loki
 MAX_BUFFER_LINES = 100
@@ -69,7 +70,9 @@ class Loki_log_buffer:
         request_header = {'Content-Type': 'application/json'}
 
         try:
-            result = requests.post(self.url, headers=request_header, data=request_body, timeout=5)
+            result = requests.post(
+                self.url, headers=request_header, data=request_body, timeout=TIMEOUT_SECONDS
+            )
             result.raise_for_status()
         except requests.RequestException:
             logger.warning('Failed to upload logs to loki')

borgmatic/hooks/monitoring/ntfy.py

@@ -7,6 +7,9 @@ import borgmatic.hooks.credential.parse
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -82,7 +85,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         if not dry_run:
             logging.getLogger('urllib3').setLevel(logging.ERROR)
             try:
-                response = requests.post(f'{base_url}/{topic}', headers=headers, auth=auth)
+                response = requests.post(
+                    f'{base_url}/{topic}', headers=headers, auth=auth, timeout=TIMEOUT_SECONDS
+                )
                 if not response.ok:
                     response.raise_for_status()
             except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/pagerduty.py

@@ -14,6 +14,7 @@ logger = logging.getLogger(__name__)
 EVENTS_API_URL = 'https://events.pagerduty.com/v2/enqueue'
 DEFAULT_LOGS_PAYLOAD_LIMIT_BYTES = 10000
 HANDLER_IDENTIFIER = 'pagerduty'
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
@@ -94,7 +95,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
-        response = requests.post(EVENTS_API_URL, data=payload.encode('utf-8'))
+        response = requests.post(
+            EVENTS_API_URL, data=payload.encode('utf-8'), timeout=TIMEOUT_SECONDS
+        )
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/pushover.py

@@ -8,6 +8,7 @@ logger = logging.getLogger(__name__)
 
 
 EMERGENCY_PRIORITY = 2
+TIMEOUT_SECONDS = 10
 
 
 def initialize_monitor(
@@ -79,6 +80,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                 'https://api.pushover.net/1/messages.json',
                 headers={'Content-type': 'application/x-www-form-urlencoded'},
                 data=data,
+                timeout=TIMEOUT_SECONDS,
             )
             if not response.ok:
                 response.raise_for_status()

borgmatic/hooks/monitoring/sentry.py

@@ -6,6 +6,9 @@ import requests
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -61,7 +64,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
-        response = requests.post(f'{cron_url}?status={status}')
+        response = requests.post(f'{cron_url}?status={status}', timeout=TIMEOUT_SECONDS)
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/uptime_kuma.py

@@ -5,6 +5,9 @@ import requests
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     push_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -37,7 +40,11 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     logging.getLogger('urllib3').setLevel(logging.ERROR)
 
     try:
-        response = requests.get(f'{push_url}?{query}', verify=hook_config.get('verify_tls', True))
+        response = requests.get(
+            f'{push_url}?{query}',
+            verify=hook_config.get('verify_tls', True),
+            timeout=TIMEOUT_SECONDS,
+        )
         if not response.ok:
             response.raise_for_status()
     except requests.exceptions.RequestException as error:

borgmatic/hooks/monitoring/zabbix.py

@@ -7,6 +7,9 @@ import borgmatic.hooks.credential.parse
 logger = logging.getLogger(__name__)
 
 
+TIMEOUT_SECONDS = 10
+
+
 def initialize_monitor(
     ping_url, config, config_filename, monitoring_log_level, dry_run
 ):  # pragma: no cover
@@ -28,7 +31,7 @@ def send_zabbix_request(server, headers, data):
     logger.debug(f'Sending a "{data["method"]}" request to the Zabbix server')
 
     try:
-        response = requests.post(server, headers=headers, json=data)
+        response = requests.post(server, headers=headers, json=data, timeout=TIMEOUT_SECONDS)
 
         if not response.ok:
             response.raise_for_status()

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -95,6 +95,10 @@ configured, meaning that the hook commands are run in that directory.
 same `everything` command hook is present in multiple configuration files,
 borgmatic only runs it once.
 
+borgmatic's `--repository` flag does not impact which command hooks get run. But
+you can use the `--config` flag to limit the configuration files (and thus
+command hooks) used.
+
 
 ### Order of execution
 
@@ -130,6 +134,10 @@ have a chance to run. Whereas the `after: error` hook doesn't run until all
 actions for—and repositories in—a configuration file have had a chance to
 execute.
 
+And if there are multiple hooks defined for a particular step (e.g. `before:
+action` for `create`), then those hooks are run in the order they're defined in
+configuration.
+
 
 ### Deprecated command hooks
 

docs/how-to/backup-your-databases.md

@@ -191,7 +191,7 @@ mysql_databases:
       format: sql
 ```
 
-### Containers
+### Database containers
 
 If your database server is running within a container and borgmatic is too, no
 problem—configure borgmatic to connect to the container's name on its exposed
@@ -209,6 +209,9 @@ postgresql_databases:
 <span class="minilink minilink-addedin">Prior to version 1.8.0</span> Put
 these options in the `hooks:` section of your configuration.
 
+
+#### Database client on the host
+
 But what if borgmatic is running on the host? You can still connect to a
 database server container if its ports are properly exposed to the host. For
 instance, when running the database container, you can specify `--publish
@@ -228,17 +231,19 @@ And then you can configure borgmatic running on the host to connect to the
 database:
 
 ```yaml
-hooks:
+postgresql_databases:
-    postgresql_databases:
+    - name: users
-        - name: users
+      hostname: 127.0.0.1
-          hostname: 127.0.0.1
+      port: 5433
-          port: 5433
+      username: postgres
-          username: postgres
+      password: trustsome1
-          password: trustsome1
 ```
 
 Alter the ports in these examples to suit your particular database system.
 
+
+#### Database client in a running container
+
 Normally, borgmatic dumps a database by running a database dump command (e.g.
 `pg_dump`) on the host or wherever borgmatic is running, and this command
 connects to your containerized database via the given `hostname` and `port`. But
@@ -250,14 +255,13 @@ to run the dump command.
 For instance, if using Docker and PostgreSQL, something like this might work:
 
 ```yaml
-hooks:
+postgresql_databases:
-    postgresql_databases:
+    - name: users
-        - name: users
+      hostname: 127.0.0.1
-          hostname: 127.0.0.1
+      port: 5433
-          port: 5433
+      username: postgres
-          username: postgres
+      password: trustsome1
-          password: trustsome1
+      pg_dump_command: docker exec my_pg_container pg_dump
-          pg_dump_command: docker exec my_pg_container pg_dump
 ```
 
 ... where `my_pg_container` is the name of your running database container.
@@ -282,19 +286,43 @@ services:
       - /run/user/1000:/run/user/1000
 ```
 
+And here's an example of using a MariaDB database client within a running Docker
+container:
+
+```yaml
+mariadb_databases:
+    - name: users
+      hostname: 127.0.0.1
+      username: example
+      password: trustsome1
+      password_transport: environment
+      mariadb_dump_command: docker exec --env MYSQL_PWD my_mariadb_container mariadb-dump
+```
+
+The `password_transport: environment` option tells borgmatic to transmit the
+password via environment variable instead of the default behavior of using an
+anonymous pipe. The environment variable transport is potentially less secure
+than the pipe, but it's necessary when the database client is running in a
+container separate from borgmatic.
+
+A similar approach can work with MySQL, using `mysql_dump_command` instead of
+`mariadb_dump_command` to run `mysqldump` in a container.
+
+
+#### Database client in a temporary container
+
 Another variation: If you're running borgmatic on the host but want to spin up a
 temporary `pg_dump` container whenever borgmatic dumps a database, for
 instance to make use of a `pg_dump` version not present on the host, try
 something like this:
 
 ```yaml
-hooks:
+postgresql_databases:
-    postgresql_databases:
+    - name: users
-        - name: users
+      hostname: your-database-hostname
-          hostname: your-database-hostname
+      username: postgres
-          username: postgres
+      password: trustsome1
-          password: trustsome1
+      pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
-          pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
 ```
 
 The `--env PGPASSWORD` is necessary here for borgmatic to provide your database

docs/how-to/customize-warnings-and-errors.md

@@ -12,52 +12,18 @@ numeric ID indicating success, warning, or error. borgmatic consumes this exit
 code to decide how to respond. Normally, a Borg error results in a borgmatic
 error, while a Borg warning or success doesn't.
 
-But if that default behavior isn't sufficient for your needs, you can
+<span class="minilink minilink-addedin">With Borg version 1.4+</span> If the
-customize how borgmatic interprets [Borg's exit
+default behavior isn't sufficient for your needs, you can customize how
+borgmatic interprets [Borg's exit
 codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
-For instance, to elevate Borg warnings to errors, thereby causing borgmatic to
-error on them, use the following borgmatic configuration:
-
-```yaml
-borg_exit_codes:
-   - exit_code: 1
-     treat_as: error
-```
-
-Be aware though that Borg exits with a warning code for a variety of benign
-situations such as files changing while they're being read, so this example
-may not meet your needs. Keep reading though for more granular exit code
-configuration.
-
-Here's an example that squashes Borg errors to warnings:
-
-```yaml
-borg_exit_codes:
-   - exit_code: 2
-     treat_as: warning
-```
-
-Be careful with this example though, because it prevents borgmatic from
-erroring when Borg errors, which may not be desirable.
-
-
-### More granular configuration
-
-<span class="minilink minilink-addedin">New in Borg version 1.4</span> Borg
-support for [more granular exit
-codes](https://borgbackup.readthedocs.io/en/1.4-maint/usage/general.html#return-codes)
-means that you can configure borgmatic to respond to specific Borg conditions.
-See the full list of [Borg 1.4 error and warning exit
-codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
-The `rc:` numeric value there tells you the exit code for each.
 
 For instance, this borgmatic configuration elevates all Borg backup file
 permission warnings (exit code `105`)—and only those warnings—to errors:
 
 ```yaml
 borg_exit_codes:
-   - exit_code: 105
+    - code: 105
-     treat_as: error
+      treat_as: error
 ```
 
 The following configuration does that *and* elevates backup file not found
@@ -65,16 +31,20 @@ warnings (exit code `107`) to errors as well:
 
 ```yaml
 borg_exit_codes:
-   - exit_code: 105
+    - code: 105
-     treat_as: error
+      treat_as: error
-   - exit_code: 107
+    - code: 107
-     treat_as: error
+      treat_as: error
 ```
 
+See the full list of [Borg 1.4 error and warning exit
+codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
+The `rc:` numeric value there tells you the exit code for each.
+
 If you don't know the exit code for a particular Borg error or warning you're
-experiencing, you can usually find it in your borgmatic output when
+experiencing, you can usually find it in your borgmatic output when `--verbosity
-`--verbosity 2` is enabled. For instance, here's a snippet of that output when
+2` is enabled. For instance, here's a snippet of that output when a backup file
-a backup file is not found:
+is not found:
 
 ```
 /noexist: stat: [Errno 2] No such file or directory: '/noexist'
@@ -84,3 +54,30 @@ terminating with warning status, rc 107
 
 So if you want to configure borgmatic to treat this as an error instead of a
 warning, the exit status to use is `107`.
+
+<span class="minilink minilink-addedin">With Borg version 1.2 and earlier</span>
+Older versions of Borg didn't support granular exit codes, but still
+distinguished between Borg errors and warnings. For instance, to elevate Borg
+warnings to errors, thereby causing borgmatic to error on them, use the
+following borgmatic configuration with Borg 1.2 or below:
+
+```yaml
+borg_exit_codes:
+    - code: 1
+      treat_as: error
+```
+
+Be aware though that Borg exits with a warning code for a variety of benign
+situations such as files changing while they're being read, so this example
+may not meet your needs.
+
+Here's another Borg 1.2 example that squashes Borg errors to warnings:
+
+```yaml
+borg_exit_codes:
+    - code: 2
+      treat_as: warning
+```
+
+Be careful with this example though, because it prevents borgmatic from
+erroring when Borg errors, which may not be desirable.

docs/how-to/make-per-application-backups.md

@@ -141,7 +141,7 @@ for more information.
 
 <span class="minilink minilink-addedin">With Borg version 2.x</span>See the
 [match archives
-documentation](https://borgbackup.readthedocs.io/en/2.0.0b13/usage/help.html#borg-help-match-archives).
+documentation](https://borgbackup.readthedocs.io/en/2.0.0b16/usage/help.html#borg-help-match-archives).
 
 Some borgmatic command-line actions also have a `--match-archives` flag that
 overrides both the auto-matching behavior and the `match_archives`

docs/how-to/snapshot-your-filesystems.md

@@ -144,9 +144,8 @@ class="minilink minilink-addedin">Beta feature</span> borgmatic supports taking
 snapshots with the [Btrfs filesystem](https://btrfs.readthedocs.io/) and sending
 those snapshots to Borg for backup.
 
-To use this feature, first you need one or more Btrfs subvolumes on mounted
+To use this feature, first you need one or more Btrfs subvolumes. Then, enable
-filesystems. Then, enable Btrfs within borgmatic by adding the following line to
+Btrfs within borgmatic by adding the following line to your configuration file:
-your configuration file:
 
 ```yaml
 btrfs:
@@ -169,13 +168,18 @@ feedback](https://torsion.org/borgmatic/#issues) you have on this feature.
 
 #### Subvolume discovery
 
-For any read-write subvolume you'd like backed up, add its mount point path to
+For any read-write subvolume you'd like backed up, add its subvolume path to
 borgmatic's `source_directories` option. Btrfs does not support snapshotting
 read-only subvolumes.
 
-<span class="minilink minilink-addedin">New in version 1.9.6</span> Or include
+<span class="minilink minilink-addedin">New in version 2.0.7</span> The path can
-the mount point as a root pattern with borgmatic's `patterns` or `patterns_from`
+be either the path of the subvolume itself or the mount point where the
-options.
+subvolume is mounted. Prior to version 2.0.7, the subvolume path could not be
+used if the subvolume was mounted elsewhere; only the mount point could be used.
+
+<span class="minilink minilink-addedin">New in version 1.9.6</span> Instead of
+using `source_directories`, you can include the subvolume path as a root pattern
+with borgmatic's `patterns` or `patterns_from` options.
 
 During a backup, borgmatic snapshots these subvolumes (non-recursively) and
 includes the snapshotted files in the paths sent to Borg. borgmatic is also
@@ -184,25 +188,24 @@ responsible for cleaning up (deleting) these snapshots after a backup completes.
 borgmatic is smart enough to look at the parent (and grandparent, etc.)
 directories of each of your `source_directories` to discover any subvolumes. For
 instance, let's say you add `/var/log` and `/var/lib` to your source
-directories, but `/var` is a subvolume mount point. borgmatic will discover that
+directories, but `/var` is a subvolume path. borgmatic will discover that and
-and snapshot `/var` accordingly. This also works even with nested subvolumes;
+snapshot `/var` accordingly. This also works even with nested subvolumes;
 borgmatic selects the subvolume that's the "closest" parent to your source
 directories.
 
 <span class="minilink minilink-addedin">New in version 1.9.6</span> When using
 [patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns),
 the initial portion of a pattern's path that you intend borgmatic to match
-against a subvolume mount point can't have globs or other non-literal characters
+against a subvolume path can't have globs or other non-literal characters in
-in it—or it won't actually match. For instance, a subvolume mount point of
+it—or it won't actually match. For instance, a subvolume path of `/var` would
-`/var` would match a pattern of `+ fm:/var/*/data`, but borgmatic isn't
+match a pattern of `+ fm:/var/*/data`, but borgmatic isn't currently smart
-currently smart enough to match `/var` to a pattern like `+ fm:/v*/lib/data`.
+enough to match `/var` to a pattern like `+ fm:/v*/lib/data`.
 
 Additionally, borgmatic rewrites the snapshot file paths so that they appear at
 their original subvolume locations in a Borg archive. For instance, if your
-subvolume is mounted at `/var/subvolume`, then the snapshotted files will appear
+subvolume path is `/var/subvolume`, then the snapshotted files will appear in an
-in an archive at `/var/subvolume` as well—even if borgmatic has to mount the
+archive at `/var/subvolume` as well—even if borgmatic has to mount the snapshot
-snapshot somewhere in `/var/subvolume/.borgmatic-snapshot-1234/` to perform the
+somewhere in `/var/subvolume/.borgmatic-snapshot-1234/` to perform the backup.
-backup.
 
 <span class="minilink minilink-addedin">With Borg version 1.2 and
 earlier</span>Snapshotted files are instead stored at a path dependent on the

docs/how-to/upgrade.md

@@ -193,7 +193,7 @@ confirmation of success—or tells you if something hasn't been transferred yet.
 Note that by omitting the `--upgrader` flag, you can also do archive transfers
 between related Borg 2 repositories without upgrading, even down to individual
 archives. For more on that functionality, see the [Borg transfer
-documentation](https://borgbackup.readthedocs.io/en/2.0.0b13/usage/transfer.html).
+documentation](https://borgbackup.readthedocs.io/en/2.0.0b16/usage/transfer.html).
 
 That's it! Now you can use your new Borg 2 repository as normal with
 borgmatic. If you've got multiple repositories, repeat the above process for

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "borgmatic"
-version = "2.0.4"
+version = "2.0.7"
 authors = [
   { name="Dan Helfman", email="witten@torsion.org" },
 ]

test_requirements.txt

@@ -1,32 +1,35 @@
 appdirs==1.4.4
-apprise==1.8.0
+apprise==1.9.3
-attrs==23.2.0
+attrs==25.3.0
-black==24.4.2
+bandit==1.8.5
-certifi==2024.7.4
+black==25.1.0
+certifi==2025.6.15
 chardet==5.2.0
-click==8.1.7
+click==8.2.1
-codespell==2.2.6
+codespell==2.4.1
-coverage==7.5.1
+coverage==7.9.1
-flake8==7.0.0
+flake8==7.3.0
+flake8-bandit==4.1.1
 flake8-quotes==3.4.0
 flake8-use-fstring==1.4
 flake8-variables-names==0.0.6
 flexmock==0.12.1
-idna==3.7
+idna==3.10
-isort==5.13.2
+isort==6.0.1
-jsonschema==4.22.0
+jsonschema==4.24.0
-Markdown==3.6
+Markdown==3.8.2
 mccabe==0.7.0
-packaging==24.0
+packaging==25.0
 pathspec==0.12.1
-pluggy==1.5.0
+pbr==6.1.1
+pluggy==1.6.0
 py==1.11.0
-pycodestyle==2.11.1
+pycodestyle==2.14.0
-pyflakes==3.2.0
+pyflakes==3.4.0
-pytest==8.2.1
+pytest==8.4.1
-pytest-cov==5.0.0
+pytest-cov==6.2.1
 PyYAML>5.0.0
 regex
-requests==2.32.2
+requests==2.32.4
 ruamel.yaml>0.15.0
 toml==0.10.2

tests/integration/borg/test_rename.py

@@ -0,0 +1,120 @@
+import logging
+
+from borgmatic.borg import rename as module
+from tests.unit.test_verbosity import insert_logging_mock
+
+
+def test_make_rename_command_includes_log_info():
+    insert_logging_mock(logging.INFO)
+
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--info', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_log_debug():
+    insert_logging_mock(logging.DEBUG)
+
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--debug', '--show-rc', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_dry_run():
+    command = module.make_rename_command(
+        dry_run=True,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--dry-run', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_remote_path():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path='borg1',
+    )
+
+    assert command == (
+        'borg',
+        'rename',
+        '--remote-path',
+        'borg1',
+        'repo::old_archive',
+        'new_archive',
+    )
+
+
+def test_make_rename_command_includes_umask():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'umask': '077'},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--umask', '077', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_log_json():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'log_json': True},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--log-json', 'repo::old_archive', 'new_archive')
+
+
+def test_make_rename_command_includes_lock_wait():
+    command = module.make_rename_command(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={'lock_wait': 5},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )
+
+    assert command == ('borg', 'rename', '--lock-wait', '5', 'repo::old_archive', 'new_archive')

tests/integration/commands/test_borgmatic.py

@@ -2,6 +2,7 @@ import subprocess
 
 from flexmock import flexmock
 
+import borgmatic.hooks.command
 from borgmatic.commands import borgmatic as module
 
 
@@ -42,6 +43,28 @@ def test_run_configuration_without_error_pings_monitoring_hooks_start_and_finish
     list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
 
 
+def test_run_configuration_with_borg_version_error_pings_after_command_hook_with_fail_state():
+    config = {
+        'repositories': [{'path': 'foo'}],
+        'commands': ({'after': 'configuration', 'run': ['echo after'], 'states': ['fail']},),
+    }
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_raise(ValueError)
+    flexmock(module).should_receive('run_actions').and_return([])
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(borgmatic.hooks.command).should_receive('execute_hooks')
+    flexmock(borgmatic.hooks.command).should_receive('execute_hooks').with_args(
+        config['commands'],
+        umask=object,
+        working_directory=object,
+        dry_run=False,
+        log_file=object,
+        configuration_filename=object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
+
+
 def test_run_configuration_with_action_error_pings_monioring_hooks_start_and_fail():
     config = {'repositories': [{'path': 'foo'}]}
     arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}

tests/integration/config/test_generate.py

@@ -126,6 +126,45 @@ def test_schema_to_sample_configuration_comments_out_non_source_config_options_i
     assert 'field3' not in config[0].ca.items
 
 
+def test_schema_to_sample_configuration_comments_out_non_source_config_options_in_sequence_of_maps_with_different_subschemas():
+    schema = {
+        'type': 'array',
+        'items': {
+            'type': 'object',
+            'oneOf': [
+                {
+                    'properties': dict(
+                        [
+                            ('field1', {'type': 'string', 'example': 'Example 1'}),
+                            ('field2', {'type': 'string', 'example': 'Example 2'}),
+                        ]
+                    )
+                },
+                {
+                    'properties': dict(
+                        [
+                            ('field2', {'type': 'string', 'example': 'Example 2'}),
+                            ('field3', {'type': 'string', 'example': 'Example 3'}),
+                        ]
+                    )
+                },
+            ],
+        },
+    }
+    source_config = [{'field1': 'value'}, {'field3': 'value'}]
+
+    config = module.schema_to_sample_configuration(schema, source_config)
+
+    assert config == [
+        dict([('field1', 'Example 1'), ('field2', 'Example 2'), ('field3', 'Example 3')])
+    ]
+
+    # The first field in a sequence does not get commented.
+    assert 'field1' not in config[0].ca.items
+    assert 'COMMENT_OUT' in config[0].ca.items['field2'][1][-1]._value
+    assert 'COMMENT_OUT' in config[0].ca.items['field3'][1][-1]._value
+
+
 def test_comment_out_line_skips_blank_line():
     line = '    \n'
 
@@ -169,8 +208,11 @@ foo:
         - quux
 
 repositories:
-    - one
+    - path: foo
-    - two
+      # COMMENT_OUT
+      label: bar
+    - path: baz
+      label: quux
 
 # This comment should be kept.
 # COMMENT_OUT
@@ -185,8 +227,10 @@ other: thing
 #         - quux
 
 repositories:
-    - one
+    - path: foo
-    - two
+#       label: bar
+    - path: baz
+      label: quux
 
 # This comment should be kept.
 # other: thing
@@ -297,14 +341,28 @@ def test_add_comments_to_configuration_comments_out_non_source_config_options():
     assert 'baz' not in config.ca.items
 
 
-def test_add_comments_to_configuration_object_with_skip_first_does_not_comment_out_first_option():
+def test_add_comments_to_configuration_object_with_skip_first_field_does_not_comment_out_first_option():
     config = module.ruamel.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
     schema = {
         'type': 'object',
         'properties': {'foo': {'description': 'Foo'}, 'bar': {'description': 'Bar'}},
     }
 
-    module.add_comments_to_configuration_object(config, schema, skip_first=True)
+    module.add_comments_to_configuration_object(config, schema, skip_first_field=True)
+
+    assert 'foo' not in config.ca.items
+    assert 'COMMENT_OUT' in config.ca.items['bar'][1][-1]._value
+    assert 'baz' not in config.ca.items
+
+
+def test_add_comments_to_configuration_object_with_skip_first_field_does_not_comment_out_first_option():
+    config = module.ruamel.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
+    schema = {
+        'type': 'object',
+        'properties': {'foo': {'description': 'Foo'}, 'bar': {'description': 'Bar'}},
+    }
+
+    module.add_comments_to_configuration_object(config, schema, skip_first_field=True)
 
     assert 'foo' not in config.ca.items
     assert 'COMMENT_OUT' in config.ca.items['bar'][1][-1]._value
@@ -326,15 +384,39 @@ def test_generate_sample_configuration_does_not_raise():
     module.generate_sample_configuration(False, None, 'dest.yaml', 'schema.yaml')
 
 
-def test_generate_sample_configuration_with_source_filename_does_not_raise():
+def test_generate_sample_configuration_with_source_filename_omits_empty_bootstrap_field():
     builtins = flexmock(sys.modules['builtins'])
     builtins.should_receive('open').with_args('schema.yaml').and_return('')
     flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
         flexmock(load=lambda filename: {})
     )
-    flexmock(module.load).should_receive('load_configuration')
+    flexmock(module.load).should_receive('load_configuration').and_return(
+        {'bootstrap': {}, 'foo': 'bar'}
+    )
     flexmock(module.normalize).should_receive('normalize')
-    flexmock(module).should_receive('schema_to_sample_configuration')
+    flexmock(module).should_receive('schema_to_sample_configuration').with_args(
+        object, {'foo': 'bar'}
+    ).once()
+    flexmock(module).should_receive('merge_source_configuration_into_destination')
+    flexmock(module).should_receive('render_configuration')
+    flexmock(module).should_receive('comment_out_optional_configuration')
+    flexmock(module).should_receive('write_configuration')
+
+    module.generate_sample_configuration(False, 'source.yaml', 'dest.yaml', 'schema.yaml')
+
+
+def test_generate_sample_configuration_with_source_filename_keeps_non_empty_bootstrap_field():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_return('')
+    flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
+        flexmock(load=lambda filename: {})
+    )
+    source_config = {'bootstrap': {'stuff': 'here'}, 'foo': 'bar'}
+    flexmock(module.load).should_receive('load_configuration').and_return(source_config)
+    flexmock(module.normalize).should_receive('normalize')
+    flexmock(module).should_receive('schema_to_sample_configuration').with_args(
+        object, source_config
+    ).once()
     flexmock(module).should_receive('merge_source_configuration_into_destination')
     flexmock(module).should_receive('render_configuration')
     flexmock(module).should_receive('comment_out_optional_configuration')

tests/unit/actions/test_check.py

@@ -601,6 +601,53 @@ def test_collect_spot_check_source_paths_parses_borg_output():
     ) == ('/etc/path', '/etc/other')
 
 
+def test_collect_spot_check_source_paths_omits_progress_from_create_dry_run_command():
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'hook1': False, 'hook2': False}
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return(
+        [Pattern('foo'), Pattern('bar')]
+    )
+    flexmock(module.borgmatic.borg.create).should_receive('make_base_create_command').with_args(
+        dry_run=True,
+        repository_path='repo',
+        config={'working_directory': '/', 'list_details': True},
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version=object,
+        global_arguments=object,
+        borgmatic_runtime_directory='/run/borgmatic',
+        local_path=object,
+        remote_path=object,
+        stream_processes=False,
+    ).and_return((('borg', 'create'), ('repo::archive',), flexmock()))
+    flexmock(module.borgmatic.borg.environment).should_receive('make_environment').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).and_return(
+        'warning: stuff\n- /etc/path\n+ /etc/other\n? /nope',
+    )
+    flexmock(module.os.path).should_receive('isfile').and_return(True)
+
+    assert module.collect_spot_check_source_paths(
+        repository={'path': 'repo'},
+        config={'working_directory': '/', 'progress': True},
+        local_borg_version=flexmock(),
+        global_arguments=flexmock(),
+        local_path=flexmock(),
+        remote_path=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+    ) == ('/etc/path', '/etc/other')
+
+
 def test_collect_spot_check_source_paths_passes_through_stream_processes_false():
     flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
         {'hook1': False, 'hook2': False}
@@ -891,8 +938,8 @@ def test_collect_spot_check_source_paths_uses_working_directory():
 
 
 def test_compare_spot_check_hashes_returns_paths_having_failing_hashes():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -932,8 +979,8 @@ def test_compare_spot_check_hashes_returns_paths_having_failing_hashes():
 
 
 def test_compare_spot_check_hashes_returns_relative_paths_having_failing_hashes():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -973,8 +1020,8 @@ def test_compare_spot_check_hashes_returns_relative_paths_having_failing_hashes(
 
 
 def test_compare_spot_check_hashes_handles_data_sample_percentage_above_100():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1014,8 +1061,8 @@ def test_compare_spot_check_hashes_handles_data_sample_percentage_above_100():
 
 
 def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1024,7 +1071,9 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
     flexmock(module.os.path).should_receive('islink').and_return(False)
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
-    ).with_args(('/usr/local/bin/xxh64sum', '/foo', '/bar'), working_directory=None).and_return(
+    ).with_args(
+        ('/usr/local/bin/xxhsum', '-H64', '/foo', '/bar'), working_directory=None
+    ).and_return(
         'hash1  /foo\nhash2  /bar'
     )
     flexmock(module.borgmatic.borg.list).should_receive('capture_archive_listing').and_return(
@@ -1039,7 +1088,7 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
                 {
                     'name': 'spot',
                     'data_sample_percentage': 50,
-                    'xxh64sum_command': '/usr/local/bin/xxh64sum',
+                    'xxh64sum_command': '/usr/local/bin/xxhsum -H64',
                 },
             ]
         },
@@ -1052,8 +1101,8 @@ def test_compare_spot_check_hashes_uses_xxh64sum_command_option():
 
 
 def test_compare_spot_check_hashes_considers_path_missing_from_archive_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1089,8 +1138,8 @@ def test_compare_spot_check_hashes_considers_path_missing_from_archive_as_not_ma
 
 
 def test_compare_spot_check_hashes_considers_symlink_path_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1125,8 +1174,8 @@ def test_compare_spot_check_hashes_considers_symlink_path_as_not_matching():
 
 
 def test_compare_spot_check_hashes_considers_non_existent_path_as_not_matching():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1162,8 +1211,8 @@ def test_compare_spot_check_hashes_considers_non_existent_path_as_not_matching()
 
 def test_compare_spot_check_hashes_with_too_many_paths_feeds_them_to_commands_in_chunks():
     flexmock(module).SAMPLE_PATHS_SUBSET_COUNT = 2
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         None,
@@ -1208,8 +1257,8 @@ def test_compare_spot_check_hashes_with_too_many_paths_feeds_them_to_commands_in
 
 
 def test_compare_spot_check_hashes_uses_working_directory_to_access_source_paths():
-    flexmock(module.random).should_receive('sample').replace_with(
+    flexmock(module.random).should_receive('SystemRandom').and_return(
-        lambda population, count: population[:count]
+        flexmock(sample=lambda population, count: population[:count])
     )
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         '/working/dir',

tests/unit/actions/test_create.py

@@ -1,3 +1,4 @@
+import json
 import os
 
 import pytest
@@ -25,6 +26,7 @@ def test_run_create_executes_and_calls_hooks_for_configured_repository():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -66,6 +68,7 @@ def test_run_create_runs_with_selected_repository():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -98,6 +101,7 @@ def test_run_create_bails_if_repository_does_not_match():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -130,6 +134,7 @@ def test_run_create_with_both_list_and_json_errors():
         progress=flexmock(),
         statistics=flexmock(),
         json=True,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -163,6 +168,7 @@ def test_run_create_with_both_list_and_progress_errors():
         progress=flexmock(),
         statistics=flexmock(),
         json=False,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -209,6 +215,7 @@ def test_run_create_produces_json():
         progress=flexmock(),
         statistics=flexmock(),
         json=True,
+        comment=None,
         list_details=flexmock(),
     )
     global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
@@ -227,3 +234,193 @@ def test_run_create_produces_json():
             remote_path=None,
         )
     ) == [parsed_json]
+
+
+def test_run_create_with_active_dumps_roundtrips_via_checkpoint_archive():
+    mock_dump_process = flexmock()
+    mock_dump_process.should_receive('poll').and_return(None).and_return(0)
+
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').never()
+    flexmock(module.borgmatic.config.paths).should_receive('Runtime_directory').and_return(
+        flexmock()
+    )
+    flexmock(module.borgmatic.borg.create).should_receive('create_archive').once()
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'dump': mock_dump_process}
+    )
+    flexmock(module.borgmatic.hooks.dispatch).should_receive(
+        'call_hooks_even_if_unconfigured'
+    ).and_return({})
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'archive.checkpoint'}
+    )
+
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+
+    flexmock(module).should_receive('rename_checkpoint_archive').with_args(
+        repository_path='repo',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    ).once()
+    create_arguments = flexmock(
+        repository=None,
+        progress=flexmock(),
+        statistics=flexmock(),
+        json=False,
+        comment=None,
+        list_details=flexmock(),
+    )
+
+    list(
+        module.run_create(
+            config_filename='test.yaml',
+            repository={'path': 'repo'},
+            config={},
+            config_paths=['/tmp/test.yaml'],
+            local_borg_version=None,
+            create_arguments=create_arguments,
+            global_arguments=global_arguments,
+            dry_run_label='',
+            local_path=None,
+            remote_path=None,
+        )
+    )
+
+
+def test_run_create_with_active_dumps_json_updates_archive_info():
+    mock_dump_process = flexmock()
+    mock_dump_process.should_receive('poll').and_return(None).and_return(0)
+
+    borg_create_result = {
+        'archive': {
+            'command_line': ['foo'],
+            'name': 'archive.checkpoint',
+            'id': 'id1',
+        },
+        'cache': {},
+        'repository': {
+            'id': 'repo-id',
+        },
+    }
+
+    expected_create_result = {
+        'archive': {
+            'command_line': ['foo'],
+            'name': 'archive',
+            'id': 'id2',
+        },
+        'cache': {},
+        'repository': {'id': 'repo-id', 'label': ''},
+    }
+
+    flexmock(module.logger).answer = lambda message: None
+    flexmock(module.borgmatic.config.validate).should_receive('repositories_match').never()
+    flexmock(module.borgmatic.config.paths).should_receive('Runtime_directory').and_return(
+        flexmock()
+    )
+
+    flexmock(module.borgmatic.borg.create).should_receive('create_archive').and_return(
+        json.dumps(borg_create_result)
+    ).once()
+    flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return(
+        {'dump': mock_dump_process}
+    )
+    flexmock(module.borgmatic.hooks.dispatch).should_receive(
+        'call_hooks_even_if_unconfigured'
+    ).and_return({})
+    flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(())
+    flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').and_return([])
+    flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
+
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+
+    flexmock(module).should_receive('rename_checkpoint_archive').with_args(
+        repository_path='repo',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    ).once()
+
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id2', 'name': 'archive'},
+    )
+
+    create_arguments = flexmock(
+        repository=None,
+        progress=flexmock(),
+        statistics=flexmock(),
+        json=True,
+        comment=None,
+        list_details=flexmock(),
+    )
+
+    assert list(
+        module.run_create(
+            config_filename='test.yaml',
+            repository={'path': 'repo'},
+            config={},
+            config_paths=['/tmp/test.yaml'],
+            local_borg_version=None,
+            create_arguments=create_arguments,
+            global_arguments=global_arguments,
+            dry_run_label='',
+            local_path=None,
+            remote_path=None,
+        )
+    ) == [expected_create_result]
+
+
+def test_rename_checkpoint_archive_renames_archive():
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'archive.checkpoint'}
+    )
+
+    flexmock(module.borgmatic.borg.rename).should_receive('rename_archive').with_args(
+        repository_name='path',
+        old_archive_name='archive.checkpoint',
+        new_archive_name='archive',
+        dry_run=False,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    )
+
+    module.rename_checkpoint_archive(
+        repository_path='path',
+        global_arguments=global_arguments,
+        config={},
+        local_borg_version=None,
+        local_path=None,
+        remote_path=None,
+    )
+
+
+def test_rename_checkpoint_archive_checks_suffix():
+    global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
+    flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
+        {'id': 'id1', 'name': 'unexpected-archive'}
+    )
+
+    with pytest.raises(
+        ValueError,
+        match='Latest archive did not have a .checkpoint suffix. Got: unexpected-archive',
+    ):
+        module.rename_checkpoint_archive(
+            repository_path='path',
+            global_arguments=global_arguments,
+            config={},
+            local_borg_version=None,
+            local_path=None,
+            remote_path=None,
+        )

tests/unit/borg/test_check.py

@@ -911,12 +911,14 @@ def test_check_archives_with_retention_prefix():
 
 
 def test_check_archives_with_extra_borg_options_passes_through_to_borg():
-    config = {'extra_borg_options': {'check': '--extra --options'}}
+    config = {'extra_borg_options': {'check': '--extra --options "value with space"'}}
     flexmock(module).should_receive('make_check_name_flags').with_args(
         {'repository'}, ()
     ).and_return(())
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
-    insert_execute_command_mock(('borg', 'check', '--extra', '--options', 'repo'))
+    insert_execute_command_mock(
+        ('borg', 'check', '--extra', '--options', 'value with space', 'repo')
+    )
 
     module.check_archives(
         repository_path='repo',

tests/unit/borg/test_compact.py

@@ -68,8 +68,15 @@ def test_compact_segments_with_log_debug_calls_borg_with_debug_flag():
     )
 
 
-def test_compact_segments_with_dry_run_skips_borg_call():
+def test_compact_segments_with_dry_run_skips_borg_call_when_feature_unavailable():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.2.3'
+    ).and_return(False)
+    flexmock(module.environment).should_receive('make_environment').never()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').never()
     flexmock(module).should_receive('execute_command').never()
+    flexmock(logging).should_receive('info').with_args('Skipping compact (dry run)').once()
 
     module.compact_segments(
         repository_path='repo',
@@ -80,6 +87,24 @@ def test_compact_segments_with_dry_run_skips_borg_call():
     )
 
 
+def test_compact_segments_with_dry_run_executes_borg_call_when_feature_available():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.4.1'
+    ).and_return(True)
+    flexmock(module.environment).should_receive('make_environment').once()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').once()
+    flexmock(module).should_receive('execute_command').once()
+
+    module.compact_segments(
+        repository_path='repo',
+        config={},
+        local_borg_version='1.4.1',
+        global_arguments=flexmock(),
+        dry_run=True,
+    )
+
+
 def test_compact_segments_with_local_path_calls_borg_via_local_path():
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
     insert_execute_command_mock(('borg1',) + COMPACT_COMMAND[1:] + ('repo',), logging.INFO)
@@ -207,12 +232,14 @@ def test_compact_segments_with_lock_wait_calls_borg_with_lock_wait_flags():
 
 def test_compact_segments_with_extra_borg_options_calls_borg_with_extra_options():
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
-    insert_execute_command_mock(COMPACT_COMMAND + ('--extra', '--options', 'repo'), logging.INFO)
+    insert_execute_command_mock(
+        COMPACT_COMMAND + ('--extra', '--options', 'value with space', 'repo'), logging.INFO
+    )
 
     module.compact_segments(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'compact': '--extra --options'}},
+        config={'extra_borg_options': {'compact': '--extra --options "value with space"'}},
         local_borg_version='1.2.3',
         global_arguments=flexmock(),
     )

tests/unit/borg/test_create.py

@@ -392,6 +392,39 @@ def test_make_base_create_command_includes_dry_run_in_borg_command():
     assert not pattern_file
 
 
+def test_make_base_create_command_includes_comment_in_borg_command():
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
+        '{hostname}'
+    )
+    flexmock(module.feature).should_receive('available').and_return(True)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+    flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+        (f'repo::{DEFAULT_ARCHIVE_NAME}',)
+    )
+
+    (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': ['exclude'],
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+        comment='a comment',
+    )
+
+    assert create_flags == ('borg', 'create', '--comment', 'a comment')
+    assert create_positional_arguments == REPO_ARCHIVE
+    assert not pattern_file
+
+
 def test_make_base_create_command_includes_local_path_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
@@ -786,6 +819,34 @@ def test_make_base_create_command_includes_repository_and_archive_name_format_wi
     assert not pattern_file
 
 
+def test_make_base_create_command_includes_archive_suffix_in_borg_command():
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
+    flexmock(module.borgmatic.borg.flags).should_receive('make_list_filter_flags').and_return('FOO')
+    flexmock(module.flags).should_receive('get_default_archive_name_format').and_return(
+        DEFAULT_ARCHIVE_NAME
+    )
+    flexmock(module.borgmatic.borg.flags).should_receive('make_exclude_flags').and_return(())
+
+    (create_flags, create_positional_arguments, pattern_file) = module.make_base_create_command(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/run/borgmatic',
+        archive_suffix='.checkpoint',
+    )
+
+    assert create_flags == ('borg', 'create')
+    assert create_positional_arguments == (f'repo::{DEFAULT_ARCHIVE_NAME}.checkpoint',)
+    assert not pattern_file
+
+
 def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module.borgmatic.borg.pattern).should_receive('write_patterns_file').and_return(None)
@@ -805,7 +866,7 @@ def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
         config={
             'source_directories': ['foo', 'bar'],
             'repositories': ['repo'],
-            'extra_borg_options': {'create': '--extra --options'},
+            'extra_borg_options': {'create': '--extra --options "value with space"'},
         },
         patterns=[Pattern('foo'), Pattern('bar')],
         local_borg_version='1.2.3',
@@ -813,7 +874,7 @@ def test_make_base_create_command_includes_extra_borg_options_in_borg_command():
         borgmatic_runtime_directory='/run/borgmatic',
     )
 
-    assert create_flags == ('borg', 'create', '--extra', '--options')
+    assert create_flags == ('borg', 'create', '--extra', '--options', 'value with space')
     assert create_positional_arguments == REPO_ARCHIVE
     assert not pattern_file
 
@@ -1414,6 +1475,41 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_flag():
     assert json_output == '[]'
 
 
+def test_create_archive_with_comment_calls_borg_with_comment_flag():
+    flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
+    flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
+    flexmock(module).should_receive('make_base_create_command').and_return(
+        (('borg', 'create', '--comment', 'a comment'), REPO_ARCHIVE, flexmock())
+    )
+    flexmock(module.environment).should_receive('make_environment')
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module).should_receive('execute_command').with_args(
+        ('borg', 'create', '--comment', 'a comment') + REPO_ARCHIVE,
+        output_log_level=logging.INFO,
+        output_file=None,
+        borg_local_path='borg',
+        borg_exit_codes=None,
+        working_directory=None,
+        environment=None,
+    )
+
+    module.create_archive(
+        dry_run=False,
+        repository_path='repo',
+        config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        patterns=[Pattern('foo'), Pattern('bar')],
+        local_borg_version='1.2.3',
+        global_arguments=flexmock(),
+        borgmatic_runtime_directory='/borgmatic/run',
+        json=False,
+        comment='a comment',
+    )
+
+
 def test_create_archive_calls_borg_with_working_directory():
     flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
     flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER

tests/unit/borg/test_environment.py

@@ -90,8 +90,11 @@ def test_make_environment_without_configuration_sets_certain_environment_variabl
     assert environment == {
         'USER': 'root',
         'BORG_EXIT_CODES': 'modern',
-        'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'no',
+        'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'NO',
-        'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'no',
+        'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'NO',
+        'BORG_USE_CHUNKS_ARCHIVE': 'NO',
+        'BORG_DEBUG_PASSPHRASE': 'NO',
+        'BORG_DISPLAY_PASSPHRASE': 'NO',
     }
 
 
@@ -101,6 +104,9 @@ def test_make_environment_without_configuration_passes_through_default_environme
             'USER': 'root',
             'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
             'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+            'BORG_USE_CHUNKS_ARCHIVE': 'yup',
+            'BORG_DEBUG_PASSPHRASE': 'nah',
+            'BORG_DISPLAY_PASSPHRASE': 'yup',
         }
     )
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -113,11 +119,14 @@ def test_make_environment_without_configuration_passes_through_default_environme
         'USER': 'root',
         'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
         'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+        'BORG_USE_CHUNKS_ARCHIVE': 'yup',
+        'BORG_DEBUG_PASSPHRASE': 'nah',
+        'BORG_DISPLAY_PASSPHRASE': 'yup',
         'BORG_EXIT_CODES': 'modern',
     }
 
 
-def test_make_environment_with_relocated_repo_access_true_should_set_environment_yes():
+def test_make_environment_with_relocated_repo_access_true_should_set_environment_YES():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
@@ -125,10 +134,10 @@ def test_make_environment_with_relocated_repo_access_true_should_set_environment
     flexmock(module.os).should_receive('pipe').never()
     environment = module.make_environment({'relocated_repo_access_is_ok': True})
 
-    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'yes'
+    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'YES'
 
 
-def test_make_environment_with_relocated_repo_access_false_should_set_environment_no():
+def test_make_environment_with_relocated_repo_access_false_should_set_environment_NO():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
@@ -136,7 +145,7 @@ def test_make_environment_with_relocated_repo_access_false_should_set_environmen
     flexmock(module.os).should_receive('pipe').never()
     environment = module.make_environment({'relocated_repo_access_is_ok': False})
 
-    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'no'
+    assert environment.get('BORG_RELOCATED_REPO_ACCESS_IS_OK') == 'NO'
 
 
 def test_make_environment_check_i_know_what_i_am_doing_true_should_set_environment_YES():
@@ -161,6 +170,50 @@ def test_make_environment_check_i_know_what_i_am_doing_false_should_set_environm
     assert environment.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING') == 'NO'
 
 
+def test_make_environment_debug_passphrase_true_should_set_environment_YES():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'debug_passphrase': True})
+
+    assert environment.get('BORG_DEBUG_PASSPHRASE') == 'YES'
+
+
+def test_make_environment_debug_passphrase_false_should_set_environment_NO():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'debug_passphrase': False})
+
+    assert environment.get('BORG_DEBUG_PASSPHRASE') == 'NO'
+
+
+def test_make_environment_display_passphrase_true_should_set_environment_YES():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'display_passphrase': True})
+
+    assert environment.get('BORG_DISPLAY_PASSPHRASE') == 'YES'
+
+
+def test_make_environment_display_passphrase_false_should_set_environment_NO():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+    environment = module.make_environment({'display_passphrase': False})
+
+    assert environment.get('BORG_DISPLAY_PASSPHRASE') == 'NO'
+
+
 def test_make_environment_with_integer_variable_value():
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -170,3 +223,17 @@ def test_make_environment_with_integer_variable_value():
     environment = module.make_environment({'borg_files_cache_ttl': 40})
 
     assert environment.get('BORG_FILES_CACHE_TTL') == '40'
+
+
+def test_make_environment_with_use_chunks_archive_should_set_correct_environment_value():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+
+    environment = module.make_environment({'use_chunks_archive': True})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'YES'
+
+    environment = module.make_environment({'use_chunks_archive': False})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'NO'

tests/unit/borg/test_prune.py

@@ -43,6 +43,28 @@ def test_make_prune_flags_returns_flags_from_config():
     assert result == BASE_PRUNE_FLAGS
 
 
+def test_make_prune_flags_with_keep_13weekly_and_keep_3monthly():
+    config = {
+        'keep_13weekly': 4,
+        'keep_3monthly': 5,
+    }
+    flexmock(module.feature).should_receive('available').and_return(True)
+    flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
+
+    result = module.make_prune_flags(
+        config, flexmock(match_archives=None), local_borg_version='1.2.3'
+    )
+
+    expected = (
+        '--keep-13weekly',
+        '4',
+        '--keep-3monthly',
+        '5',
+    )
+
+    assert result == expected
+
+
 def test_make_prune_flags_accepts_prefix_with_placeholders():
     config = {
         'keep_daily': 1,
@@ -450,13 +472,15 @@ def test_prune_archives_with_extra_borg_options_calls_borg_with_extra_options():
     flexmock(module.feature).should_receive('available').with_args(
         module.feature.Feature.NO_PRUNE_STATS, '1.2.3'
     ).and_return(False)
-    insert_execute_command_mock(PRUNE_COMMAND + ('--extra', '--options', 'repo'), logging.INFO)
+    insert_execute_command_mock(
+        PRUNE_COMMAND + ('--extra', '--options', 'value with space', 'repo'), logging.INFO
+    )
 
     prune_arguments = flexmock(statistics=False, list_details=False)
     module.prune_archives(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'prune': '--extra --options'}},
+        config={'extra_borg_options': {'prune': '--extra --options "value with space"'}},
         local_borg_version='1.2.3',
         global_arguments=flexmock(),
         prune_arguments=prune_arguments,

tests/unit/borg/test_rename.py

@@ -0,0 +1,37 @@
+import logging
+
+from flexmock import flexmock
+
+from borgmatic.borg import rename as module
+
+
+def test_rename_archive_calls_borg_rename():
+    environment = flexmock()
+
+    # Note: make_rename_command is tested as integration test.
+    flexmock(module).should_receive('make_rename_command').and_return(('borg', 'fake-command'))
+    flexmock(module.borgmatic.borg.environment).should_receive('make_environment').and_return(
+        environment
+    )
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
+        '/working/dir'
+    )
+    flexmock(module.borgmatic.execute).should_receive('execute_command').with_args(
+        ('borg', 'fake-command'),
+        output_log_level=logging.INFO,
+        environment=environment,
+        working_directory='/working/dir',
+        borg_local_path='borg',
+        borg_exit_codes=None,
+    ).once()
+
+    module.rename_archive(
+        dry_run=False,
+        repository_name='repo',
+        old_archive_name='old_archive',
+        new_archive_name='new_archive',
+        config={},
+        local_borg_version='1.2.3',
+        local_path='borg',
+        remote_path=None,
+    )

tests/unit/borg/test_repo_create.py

@@ -484,7 +484,7 @@ def test_create_repository_with_umask_calls_borg_with_umask_flag():
 def test_create_repository_with_extra_borg_options_calls_borg_with_extra_options():
     insert_repo_info_command_not_found_mock()
     insert_repo_create_command_mock(
-        REPO_CREATE_COMMAND + ('--extra', '--options', '--repo', 'repo')
+        REPO_CREATE_COMMAND + ('--extra', '--options', 'value with space', '--repo', 'repo')
     )
     flexmock(module.feature).should_receive('available').and_return(True)
     flexmock(module.flags).should_receive('make_repository_flags').and_return(
@@ -497,7 +497,7 @@ def test_create_repository_with_extra_borg_options_calls_borg_with_extra_options
     module.create_repository(
         dry_run=False,
         repository_path='repo',
-        config={'extra_borg_options': {'repo-create': '--extra --options'}},
+        config={'extra_borg_options': {'repo-create': '--extra --options "value with space"'}},
         local_borg_version='2.3.4',
         global_arguments=flexmock(),
         encryption_mode='repokey',

tests/unit/borg/test_repo_list.py

@@ -1,4 +1,5 @@
 import argparse
+import json
 import logging
 
 import pytest
@@ -11,7 +12,7 @@ from ..test_verbosity import insert_logging_mock
 BORG_LIST_LATEST_ARGUMENTS = (
     '--last',
     '1',
-    '--short',
+    '--json',
     'repo',
 )
 
@@ -31,8 +32,43 @@ def test_resolve_archive_name_passes_through_non_latest_archive_name():
     )
 
 
-def test_resolve_archive_name_calls_borg_with_flags():
+def test_resolve_archive_name_calls_get_latest_archive():
     expected_archive = 'archive-name'
+
+    repository_path = flexmock()
+    config = flexmock()
+    local_borg_version = flexmock()
+    global_arguments = flexmock()
+    local_path = flexmock()
+    remote_path = flexmock()
+
+    flexmock(module.environment).should_receive('make_environment')
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
+    flexmock(module).should_receive('get_latest_archive').with_args(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    ).and_return({'name': expected_archive})
+
+    assert (
+        module.resolve_archive_name(
+            repository_path,
+            'latest',
+            config,
+            local_borg_version,
+            global_arguments,
+            local_path,
+            remote_path,
+        )
+        == expected_archive
+    )
+
+
+def test_get_latest_archive_calls_borg_with_flags():
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -41,12 +77,11 @@ def test_resolve_archive_name_calls_borg_with_flags():
         borg_exit_codes=None,
         environment=None,
         working_directory=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -55,8 +90,8 @@ def test_resolve_archive_name_calls_borg_with_flags():
     )
 
 
-def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
+def test_get_latest_archive_with_log_info_calls_borg_without_info_flag():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -65,13 +100,12 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
     insert_logging_mock(logging.INFO)
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -80,8 +114,8 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
     )
 
 
-def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
+def test_get_latest_archive_with_log_debug_calls_borg_without_debug_flag():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -90,13 +124,12 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
     insert_logging_mock(logging.DEBUG)
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -105,8 +138,8 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
     )
 
 
-def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
+def test_get_latest_archive_with_local_path_calls_borg_via_local_path():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -115,12 +148,11 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
         working_directory=None,
         borg_local_path='borg1',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -130,8 +162,8 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
     )
 
 
-def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
+def test_get_latest_archive_with_exit_codes_calls_borg_using_them():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     borg_exit_codes = flexmock()
@@ -141,12 +173,11 @@ def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=borg_exit_codes,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'borg_exit_codes': borg_exit_codes},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -155,8 +186,8 @@ def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
     )
 
 
-def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags():
+def test_get_latest_archive_with_remote_path_calls_borg_with_remote_path_flags():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -165,12 +196,11 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -180,8 +210,8 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
     )
 
 
-def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
+def test_get_latest_archive_with_umask_calls_borg_with_umask_flags():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -190,12 +220,11 @@ def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'umask': '077'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -204,7 +233,7 @@ def test_resolve_archive_name_with_umask_calls_borg_with_umask_flags():
     )
 
 
-def test_resolve_archive_name_without_archives_raises():
+def test_get_latest_archive_without_archives_raises():
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
     flexmock(module).should_receive('execute_command_and_capture_output').with_args(
@@ -213,20 +242,19 @@ def test_resolve_archive_name_without_archives_raises():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return('')
+    ).and_return(json.dumps({'archives': []}))
 
     with pytest.raises(ValueError):
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
         )
 
 
-def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
+def test_get_latest_archive_with_log_json_calls_borg_with_log_json_flags():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
 
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
@@ -236,12 +264,11 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'log_json': True},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -250,8 +277,8 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
     )
 
 
-def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
+def test_get_latest_archive_with_lock_wait_calls_borg_with_lock_wait_flags():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
 
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
@@ -261,12 +288,11 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
         working_directory=None,
         borg_local_path='borg',
         borg_exit_codes=None,
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'lock_wait': 'okay'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),
@@ -275,8 +301,8 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
     )
 
 
-def test_resolve_archive_name_calls_borg_with_working_directory():
+def test_get_latest_archive_calls_borg_with_working_directory():
-    expected_archive = 'archive-name'
+    expected_archive = {'name': 'archive-name'}
     flexmock(module.environment).should_receive('make_environment')
     flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
         '/working/dir',
@@ -287,12 +313,11 @@ def test_resolve_archive_name_calls_borg_with_working_directory():
         borg_exit_codes=None,
         environment=None,
         working_directory='/working/dir',
-    ).and_return(expected_archive + '\n')
+    ).and_return(json.dumps({'archives': [expected_archive]}))
 
     assert (
-        module.resolve_archive_name(
+        module.get_latest_archive(
             'repo',
-            'latest',
             config={'working_directory': '/working/dir'},
             local_borg_version='1.2.3',
             global_arguments=flexmock(),

tests/unit/commands/test_borgmatic.py

@@ -28,6 +28,7 @@ def test_get_skip_actions_uses_config_and_arguments(config, arguments, expected_
 
 
 def test_monitoring_hooks_with_monioring_disabled_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(module.DISABLED)
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -41,6 +42,7 @@ def test_monitoring_hooks_with_monioring_disabled_bails():
 
 
 def test_monitoring_hooks_with_non_primary_action_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -54,6 +56,7 @@ def test_monitoring_hooks_with_non_primary_action_bails():
 
 
 def test_monitoring_hooks_pings_monitors():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -117,6 +120,7 @@ def test_monitoring_hooks_pings_monitors():
 
 
 def test_monitoring_hooks_with_start_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -172,6 +176,7 @@ def test_monitoring_hooks_with_start_ping_error_raises():
 
 
 def test_monitoring_hooks_with_log_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -227,6 +232,7 @@ def test_monitoring_hooks_with_log_ping_error_raises():
 
 
 def test_monitoring_hooks_with_finish_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -281,7 +287,8 @@ def test_monitoring_hooks_with_finish_ping_error_raises():
             pass
 
 
-def test_monitoring_with_wrapped_code_error_pings_fail():
+def test_monitoring_hooks_with_wrapped_code_error_pings_fail():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -345,7 +352,8 @@ def test_monitoring_with_wrapped_code_error_pings_fail():
             raise OSError()
 
 
-def test_monitoring_with_fail_ping_error_raise_original_error():
+def test_monitoring_hooks_with_fail_ping_error_raise_original_error():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',

tests/unit/config/test_normalize.py

@@ -334,6 +334,16 @@ def test_normalize_commands_moves_individual_command_hooks_to_unified_commands(
             {'repositories': [{'path': 'rclone:host:repo'}]},
             True,
         ),
+        (
+            {'repositories': ['s3:stuff']},
+            {'repositories': [{'path': 's3:stuff'}]},
+            True,
+        ),
+        (
+            {'repositories': ['b2:stuff']},
+            {'repositories': [{'path': 'b2:stuff'}]},
+            True,
+        ),
         (
             {'repositories': ['file:///repo']},
             {'repositories': [{'path': '/repo'}]},

tests/unit/hooks/data_source/test_btrfs.py

@@ -5,7 +5,31 @@ from borgmatic.borg.pattern import Pattern, Pattern_source, Pattern_style, Patte
 from borgmatic.hooks.data_source import btrfs as module
 
 
-def test_get_subvolume_mount_points_parses_findmnt_output():
+def test_get_contained_subvolume_paths_parses_btrfs_output():
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).with_args(('btrfs', 'subvolume', 'list', '/mnt0'), close_fds=True).and_return(
+        'ID 256 gen 28 top level 5 path @sub\nID 258 gen 17 top level 5 path snap\n\n'
+    )
+
+    assert module.get_contained_subvolume_paths('btrfs', '/mnt0') == (
+        '/mnt0',
+        '/mnt0/@sub',
+        '/mnt0/snap',
+    )
+
+
+def test_get_contained_subvolume_paths_swallows_called_process_error():
+    flexmock(module.borgmatic.execute).should_receive(
+        'execute_command_and_capture_output'
+    ).with_args(('btrfs', 'subvolume', 'list', '/mnt0'), close_fds=True).and_raise(
+        module.subprocess.CalledProcessError(1, 'btrfs')
+    )
+
+    assert module.get_contained_subvolume_paths('btrfs', '/mnt0') == ()
+
+
+def test_get_all_subvolume_paths_parses_findmnt_output():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return(
@@ -22,31 +46,53 @@ def test_get_subvolume_mount_points_parses_findmnt_output():
                  "source": "/dev/loop0",
                  "fstype": "btrfs",
                  "options": "rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/"
+              },
+              {
+                 "target": "/mnt2",
+                 "source": "/dev/loop0",
+                 "fstype": "btrfs",
+                 "options": "rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/"
               }
            ]
         }
         '''
     )
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt0'
+    ).and_return(('/mnt0',))
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt1'
+    ).and_return(('/mnt1', '/mnt1/sub'))
+    flexmock(module).should_receive('get_contained_subvolume_paths').with_args(
+        'btrfs', '/mnt2'
+    ).never()
 
-    assert module.get_subvolume_mount_points('findmnt') == ('/mnt0', '/mnt1')
+    assert module.get_all_subvolume_paths('btrfs', 'findmnt') == (
+        '/mnt0',
+        '/mnt1',
+        '/mnt1/sub',
+        '/mnt2',
+    )
 
 
-def test_get_subvolume_mount_points_with_invalid_findmnt_json_errors():
+def test_get_all_subvolume_paths_with_invalid_findmnt_json_errors():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return('{')
+    flexmock(module).should_receive('get_contained_subvolume_paths').never()
 
     with pytest.raises(ValueError):
-        module.get_subvolume_mount_points('findmnt')
+        module.get_all_subvolume_paths('btrfs', 'findmnt')
 
 
-def test_get_subvolume_mount_points_with_findmnt_json_missing_filesystems_errors():
+def test_get_all_subvolume_paths_with_findmnt_json_missing_filesystems_errors():
     flexmock(module.borgmatic.execute).should_receive(
         'execute_command_and_capture_output'
     ).and_return('{"wtf": "something is wrong here"}')
+    flexmock(module).should_receive('get_contained_subvolume_paths').never()
 
     with pytest.raises(ValueError):
-        module.get_subvolume_mount_points('findmnt')
+        module.get_all_subvolume_paths('btrfs', 'findmnt')
 
 
 def test_get_subvolume_property_with_invalid_btrfs_output_errors():
@@ -82,7 +128,7 @@ def test_get_subvolume_property_passes_through_general_value():
     assert module.get_subvolume_property('btrfs', '/foo', 'thing') == 'value'
 
 
-def test_omit_read_only_subvolume_mount_points_filters_out_read_only():
+def test_omit_read_only_subvolume_paths_filters_out_read_only_subvolumes():
     flexmock(module).should_receive('get_subvolume_property').with_args(
         'btrfs', '/foo', 'ro'
     ).and_return(False)
@@ -93,17 +139,29 @@ def test_omit_read_only_subvolume_mount_points_filters_out_read_only():
         'btrfs', '/baz', 'ro'
     ).and_return(False)
 
-    assert module.omit_read_only_subvolume_mount_points('btrfs', ('/foo', '/bar', '/baz')) == (
+    assert module.omit_read_only_subvolume_paths('btrfs', ('/foo', '/bar', '/baz')) == (
         '/foo',
         '/baz',
     )
 
 
+def test_omit_read_only_subvolume_paths_filters_out_erroring_subvolumes():
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/foo', 'ro'
+    ).and_raise(module.subprocess.CalledProcessError(1, 'btrfs'))
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/bar', 'ro'
+    ).and_return(True)
+    flexmock(module).should_receive('get_subvolume_property').with_args(
+        'btrfs', '/baz', 'ro'
+    ).and_return(False)
+
+    assert module.omit_read_only_subvolume_paths('btrfs', ('/foo', '/bar', '/baz')) == ('/baz',)
+
+
 def test_get_subvolumes_collects_subvolumes_matching_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-        ('/mnt1', '/mnt2')
-    )
 
     contained_pattern = Pattern(
         '/mnt1',
@@ -128,10 +186,8 @@ def test_get_subvolumes_collects_subvolumes_matching_patterns():
 
 
 def test_get_subvolumes_skips_non_root_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-        ('/mnt1', '/mnt2')
-    )
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'
@@ -162,10 +218,8 @@ def test_get_subvolumes_skips_non_root_patterns():
 
 
 def test_get_subvolumes_skips_non_config_patterns():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-        ('/mnt1', '/mnt2')
-    )
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'
@@ -196,10 +250,8 @@ def test_get_subvolumes_skips_non_config_patterns():
 
 
 def test_get_subvolumes_without_patterns_collects_all_subvolumes():
-    flexmock(module).should_receive('get_subvolume_mount_points').and_return(('/mnt1', '/mnt2'))
+    flexmock(module).should_receive('get_all_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-    flexmock(module).should_receive('omit_read_only_subvolume_mount_points').and_return(
+    flexmock(module).should_receive('omit_read_only_subvolume_paths').and_return(('/mnt1', '/mnt2'))
-        ('/mnt1', '/mnt2')
-    )
 
     flexmock(module.borgmatic.hooks.data_source.snapshot).should_receive(
         'get_contained_patterns'

tests/unit/hooks/data_source/test_lvm.py

@@ -214,6 +214,7 @@ def test_snapshot_logical_volume_with_percentage_snapshot_name_uses_lvcreate_ext
             '/dev/snap',
         ),
         output_log_level=object,
+        close_fds=True,
     )
 
     module.snapshot_logical_volume('lvcreate', 'snap', '/dev/snap', '10%ORIGIN')
@@ -233,6 +234,7 @@ def test_snapshot_logical_volume_with_non_percentage_snapshot_name_uses_lvcreate
             '/dev/snap',
         ),
         output_log_level=object,
+        close_fds=True,
     )
 
     module.snapshot_logical_volume('lvcreate', 'snap', '/dev/snap', '10TB')
@@ -554,16 +556,7 @@ def test_dump_data_sources_with_dry_run_skips_snapshots_and_does_not_touch_patte
     )
     flexmock(module.os).should_receive('getpid').and_return(1234)
     flexmock(module).should_receive('snapshot_logical_volume').never()
-    flexmock(module).should_receive('get_snapshots').with_args(
+    flexmock(module).should_receive('get_snapshots').never()
-        'lvs', snapshot_name='lvolume1_borgmatic-1234'
-    ).and_return(
-        (module.Snapshot(name='lvolume1_borgmatic-1234', device_path='/dev/lvolume1_snap'),)
-    )
-    flexmock(module).should_receive('get_snapshots').with_args(
-        'lvs', snapshot_name='lvolume2_borgmatic-1234'
-    ).and_return(
-        (module.Snapshot(name='lvolume2_borgmatic-1234', device_path='/dev/lvolume2_snap'),)
-    )
     flexmock(module).should_receive('mount_snapshot').never()
 
     assert (

tests/unit/hooks/data_source/test_mariadb.py

@@ -167,6 +167,38 @@ def test_database_names_to_dump_queries_mariadb_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mariadb',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mariadb_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -247,9 +279,22 @@ def test_dump_data_sources_dumps_each_database():
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
     ).replace_with(lambda value, config: value)
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
+    flexmock(module).should_receive('database_names_to_dump').with_args(
-        ('bar',)
+        database=databases[0],
-    )
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -285,9 +330,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
+    flexmock(module).should_receive('database_names_to_dump').with_args(
-        ('bar',)
+        database=database,
-    )
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -311,6 +361,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -378,6 +472,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mariadb_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=mariadb.cnf --skip-ssl'}
     flexmock(module).should_receive('parse_extra_options').and_return(
@@ -479,6 +621,48 @@ def test_execute_dump_command_runs_mariadb_dump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mariadb-dump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mariadb_dump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -849,54 +1033,6 @@ def test_execute_dump_command_with_dry_run_skips_mariadb_dump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mariadb_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1168,6 +1304,49 @@ def test_restore_data_source_dump_runs_mariadb_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mariadb', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/data_source/test_mysql.py

@@ -60,6 +60,42 @@ def test_database_names_to_dump_queries_mysql_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mysql',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mysql_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -141,9 +177,22 @@ def test_dump_data_sources_dumps_each_database():
         'resolve_credential'
     ).and_return(None)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
+    flexmock(module).should_receive('database_names_to_dump').with_args(
-        ('bar',)
+        database=databases[0],
-    )
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -179,9 +228,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
+    flexmock(module).should_receive('database_names_to_dump').with_args(
-        ('bar',)
+        database=database,
-    )
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -205,6 +259,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -272,6 +370,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mysql_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=my.cnf --skip-ssl'}
     flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
@@ -375,6 +521,52 @@ def test_execute_dump_command_runs_mysqldump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mysqldump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mysqldump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -761,54 +953,6 @@ def test_execute_dump_command_with_dry_run_skips_mysqldump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mysql_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1092,6 +1236,53 @@ def test_restore_data_source_dump_runs_mysql_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mysql', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/monitoring/test_cronhub.py

@@ -6,7 +6,7 @@ from borgmatic.hooks.monitoring import cronhub as module
 def test_ping_monitor_rewrites_ping_url_for_start_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -22,7 +22,7 @@ def test_ping_monitor_rewrites_ping_url_for_start_state():
 def test_ping_monitor_rewrites_ping_url_and_state_for_start_state():
     hook_config = {'ping_url': 'https://example.com/ping/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -38,7 +38,7 @@ def test_ping_monitor_rewrites_ping_url_and_state_for_start_state():
 def test_ping_monitor_rewrites_ping_url_for_finish_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/finish/abcdef'
+        'https://example.com/finish/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -54,7 +54,7 @@ def test_ping_monitor_rewrites_ping_url_for_finish_state():
 def test_ping_monitor_rewrites_ping_url_for_fail_state():
     hook_config = {'ping_url': 'https://example.com/start/abcdef'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/fail/abcdef'
+        'https://example.com/fail/abcdef', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -105,7 +105,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/start/abcdef'
+        'https://example.com/start/abcdef', timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_cronitor.py

@@ -5,9 +5,9 @@ from borgmatic.hooks.monitoring import cronitor as module
 
 def test_ping_monitor_hits_ping_url_for_start_state():
     hook_config = {'ping_url': 'https://example.com'}
-    flexmock(module.requests).should_receive('get').with_args('https://example.com/run').and_return(
+    flexmock(module.requests).should_receive('get').with_args(
-        flexmock(ok=True)
+        'https://example.com/run', timeout=int
-    )
+    ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
         hook_config,
@@ -22,7 +22,7 @@ def test_ping_monitor_hits_ping_url_for_start_state():
 def test_ping_monitor_hits_ping_url_for_finish_state():
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/complete'
+        'https://example.com/complete', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -38,7 +38,7 @@ def test_ping_monitor_hits_ping_url_for_finish_state():
 def test_ping_monitor_hits_ping_url_for_fail_state():
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('get').with_args(
-        'https://example.com/fail'
+        'https://example.com/fail', timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -88,9 +88,9 @@ def test_ping_monitor_with_other_error_logs_warning():
     response.should_receive('raise_for_status').and_raise(
         module.requests.exceptions.RequestException
     )
-    flexmock(module.requests).should_receive('get').with_args('https://example.com/run').and_return(
+    flexmock(module.requests).should_receive('get').with_args(
-        response
+        'https://example.com/run', timeout=int
-    )
+    ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_healthchecks.py

@@ -79,7 +79,7 @@ def test_ping_monitor_hits_ping_url_for_start_state():
     ).never()
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -100,7 +100,7 @@ def test_ping_monitor_hits_ping_url_for_finish_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=True
+        'https://example.com', data=payload.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -121,7 +121,7 @@ def test_ping_monitor_hits_ping_url_for_fail_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/fail', data=payload.encode('utf'), verify=True
+        'https://example.com/fail', data=payload.encode('utf'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -142,7 +142,7 @@ def test_ping_monitor_hits_ping_url_for_log_state():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/log', data=payload.encode('utf'), verify=True
+        'https://example.com/log', data=payload.encode('utf'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -166,6 +166,7 @@ def test_ping_monitor_with_ping_uuid_hits_corresponding_url():
         f"https://hc-ping.com/{hook_config['ping_url']}",
         data=payload.encode('utf-8'),
         verify=True,
+        timeout=int,
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -186,7 +187,7 @@ def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=False
+        'https://example.com', data=payload.encode('utf-8'), verify=False, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -207,7 +208,7 @@ def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
         'format_buffered_logs_for_payload'
     ).and_return(payload)
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com', data=payload.encode('utf-8'), verify=True
+        'https://example.com', data=payload.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -260,7 +261,7 @@ def test_ping_monitor_hits_ping_url_when_states_matching():
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'states': ['start', 'finish']}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -279,7 +280,7 @@ def test_ping_monitor_adds_create_query_parameter_when_create_slug_true():
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'create_slug': True}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start?create=1', data=''.encode('utf-8'), verify=True
+        'https://example.com/start?create=1', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -298,7 +299,7 @@ def test_ping_monitor_does_not_add_create_query_parameter_when_create_slug_false
     ).never()
     hook_config = {'ping_url': 'https://example.com', 'create_slug': False}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -317,6 +318,7 @@ def test_ping_monitor_does_not_add_create_query_parameter_when_ping_url_is_uuid(
         f"https://hc-ping.com/{hook_config['ping_url']}",
         data=''.encode('utf-8'),
         verify=True,
+        timeout=int,
     ).and_return(flexmock(ok=True))
 
     module.ping_monitor(
@@ -352,7 +354,7 @@ def test_ping_monitor_with_connection_error_logs_warning():
     ).never()
     hook_config = {'ping_url': 'https://example.com'}
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -376,7 +378,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('post').with_args(
-        'https://example.com/start', data=''.encode('utf-8'), verify=True
+        'https://example.com/start', data=''.encode('utf-8'), verify=True, timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_ntfy.py

@@ -43,6 +43,7 @@ def test_ping_monitor_minimal_config_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -67,6 +68,7 @@ def test_ping_monitor_with_access_token_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -93,6 +95,7 @@ def test_ping_monitor_with_username_password_and_access_token_ignores_username_p
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -119,6 +122,7 @@ def test_ping_monitor_with_username_password_hits_hosted_ntfy_on_fail():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=module.requests.auth.HTTPBasicAuth('testuser', 'fakepassword'),
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -140,6 +144,7 @@ def test_ping_monitor_with_password_but_no_username_warns():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -162,6 +167,7 @@ def test_ping_monitor_with_username_but_no_password_warns():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
     flexmock(module.logger).should_receive('warning').once()
 
@@ -218,6 +224,7 @@ def test_ping_monitor_minimal_config_hits_selfhosted_ntfy_on_fail():
         f'{custom_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -253,7 +260,7 @@ def test_ping_monitor_custom_message_hits_hosted_ntfy_on_fail():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.requests).should_receive('post').with_args(
-        f'{default_base_url}/{topic}', headers=custom_message_headers, auth=None
+        f'{default_base_url}/{topic}', headers=custom_message_headers, auth=None, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -275,6 +282,7 @@ def test_ping_monitor_custom_state_hits_hosted_ntfy_on_start():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.START),
         auth=None,
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -296,6 +304,7 @@ def test_ping_monitor_with_connection_error_logs_warning():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -340,6 +349,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         f'{default_base_url}/{topic}',
         headers=return_default_message_headers(borgmatic.hooks.monitoring.monitor.State.FAIL),
         auth=None,
+        timeout=int,
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 

tests/unit/hooks/monitoring/test_pushover.py

@@ -23,6 +23,7 @@ def test_ping_monitor_config_with_minimum_config_fail_state_backup_successfully_
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'fail',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -81,6 +82,7 @@ def test_ping_monitor_start_state_backup_default_message_successfully_send_to_pu
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'start',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -117,6 +119,7 @@ def test_ping_monitor_start_state_backup_custom_message_successfully_send_to_pus
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'custom start message',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -155,6 +158,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -193,6 +197,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -231,6 +236,7 @@ def test_ping_monitor_start_state_backup_default_message_with_priority_emergency
             'retry': 30,
             'expire': 600,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -328,6 +334,7 @@ def test_ping_monitor_start_state_backup_based_on_documentation_advanced_example
             'html': 1,
             'ttl': 10,
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -399,6 +406,7 @@ def test_ping_monitor_fail_state_backup_based_on_documentation_advanced_example_
             'url': 'https://ticketing-system.example.com/login',
             'url_title': 'Login to ticketing system',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -467,6 +475,7 @@ def test_ping_monitor_finish_state_backup_based_on_documentation_advanced_exampl
             'url': 'https://ticketing-system.example.com/login',
             'url_title': 'Login to ticketing system',
         },
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -550,6 +559,7 @@ def test_ping_monitor_push_post_error_bails():
             'user': '983hfe0of902lkjfa2amanfgui',
             'message': 'fail',
         },
+        timeout=int,
     ).and_return(push_response).once()
 
     flexmock(module.logger).should_receive('warning').once()

tests/unit/hooks/monitoring/test_sentry.py

@@ -29,7 +29,8 @@ def test_ping_monitor_constructs_cron_url_and_pings_it(state, configured_states,
         hook_config['states'] = configured_states
 
     flexmock(module.requests).should_receive('post').with_args(
-        f'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status={expected_status}'
+        f'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status={expected_status}',
+        timeout=int,
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -134,7 +135,8 @@ def test_ping_monitor_with_network_error_does_not_raise():
         module.requests.exceptions.ConnectionError
     )
     flexmock(module.requests).should_receive('post').with_args(
-        'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status=in_progress'
+        'https://o294220.ingest.us.sentry.io/api/203069/cron/test/5f80ec/?status=in_progress',
+        timeout=int,
     ).and_return(response).once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_uptimekuma.py

@@ -10,7 +10,7 @@ CUSTOM_PUSH_URL = 'https://uptime.example.com/api/push/efgh5678'
 def test_ping_monitor_hits_default_uptimekuma_on_fail():
     hook_config = {}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{DEFAULT_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{DEFAULT_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -26,7 +26,7 @@ def test_ping_monitor_hits_default_uptimekuma_on_fail():
 def test_ping_monitor_hits_custom_uptimekuma_on_fail():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -42,7 +42,7 @@ def test_ping_monitor_hits_custom_uptimekuma_on_fail():
 def test_ping_monitor_custom_uptimekuma_on_start():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=up&msg=start', verify=True
+        f'{CUSTOM_PUSH_URL}?status=up&msg=start', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -58,7 +58,7 @@ def test_ping_monitor_custom_uptimekuma_on_start():
 def test_ping_monitor_custom_uptimekuma_on_finish():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=up&msg=finish', verify=True
+        f'{CUSTOM_PUSH_URL}?status=up&msg=finish', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -116,7 +116,7 @@ def test_ping_monitor_does_not_hit_custom_uptimekuma_on_finish_dry_run():
 def test_ping_monitor_with_connection_error_logs_warning():
     hook_config = {'push_url': CUSTOM_PUSH_URL}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_raise(module.requests.exceptions.ConnectionError)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -137,7 +137,7 @@ def test_ping_monitor_with_other_error_logs_warning():
         module.requests.exceptions.RequestException
     )
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(response)
     flexmock(module.logger).should_receive('warning').once()
 
@@ -168,7 +168,7 @@ def test_ping_monitor_with_invalid_run_state():
 def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
     hook_config = {'push_url': CUSTOM_PUSH_URL, 'verify_tls': False}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=False
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=False, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(
@@ -184,7 +184,7 @@ def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
 def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
     hook_config = {'push_url': CUSTOM_PUSH_URL, 'verify_tls': True}
     flexmock(module.requests).should_receive('get').with_args(
-        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True
+        f'{CUSTOM_PUSH_URL}?status=down&msg=fail', verify=True, timeout=int
     ).and_return(flexmock(ok=True)).once()
 
     module.ping_monitor(

tests/unit/hooks/monitoring/test_zabbix.py

@@ -74,7 +74,7 @@ def test_send_zabbix_request_with_post_error_bails():
     )
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -89,7 +89,7 @@ def test_send_zabbix_request_with_invalid_json_response_bails():
     response.should_receive('json').and_raise(module.requests.exceptions.JSONDecodeError)
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -103,7 +103,7 @@ def test_send_zabbix_request_with_success_returns_response_result():
     response.should_receive('json').and_return({'result': {'foo': 'bar'}})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) == {'foo': 'bar'}
@@ -117,7 +117,7 @@ def test_send_zabbix_request_with_success_passes_through_missing_result():
     response.should_receive('json').and_return({})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None
@@ -131,7 +131,7 @@ def test_send_zabbix_request_with_error_bails():
     response.should_receive('json').and_return({'result': {'data': [{'error': 'oops'}]}})
 
     flexmock(module.requests).should_receive('post').with_args(
-        server, headers=headers, json=data
+        server, headers=headers, json=data, timeout=int
     ).and_return(response)
 
     assert module.send_zabbix_request(server, headers, data) is None