Bump version for release.

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1081

.gitea/workflows/build.yaml

@@ -17,6 +17,7 @@ jobs:
   docs:
     needs: [test]
     runs-on: host
+    if: gitea.event_name == 'push'
     env:
       IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
 

.github/pull_request_template.md

@@ -0,0 +1,8 @@
+## Hold up!
+
+Thanks for your contribution. Unfortunately, we don't use GitHub pull requests to manage code contributions to this repository (and GitHub doesn't have any way to disable pull requests entirely). Instead, please see:
+
+https://torsion.org/borgmatic/#contributing
+
+... which provides full instructions on how to submit pull requests. You can even use your GitHub account to login.
+

NEWS

@@ -1,3 +1,35 @@
+2.0.5
+ * #1033: Add a "password_transport" option to the MariaDB and MySQL database hooks for customizing
+   how borgmatic transmits passwords to the database client.
+ * #1078: Add "keep_3monthly" and "keep_13weekly" options for customizing "prune" action archive
+   retention.
+ * #1078: Add a "use_chunks_archive" option for controlling whether Borg uses its chunks cache
+   directory.
+ * #1078: For the "compact" action, pass "--dry-run" through to Borg.
+ * #1085: Fix a regression in which the default monitoring verbosity is 0 (warnings only) instead of
+   1 (info about steps borgmatic is taking). This prevented logs from showing up in monitoring
+   services like Healthchecks unless you had an explicit monitoring verbosity set.
+ * Move Mastodon social hosting from Fosstodon to FLOSS.social: https://floss.social/@borgmatic
+ * The borgmatic project no longer accepts pull requests on GitHub. But see
+   https://torsion.org/borgmatic/#contributing for how you can still submit pull requests. You can
+   even use your GitHub account to login.
+
+2.0.4
+ * #1072: Fix path rewriting for non-root patterns in the ZFS, Btrfs, and LVM hooks.
+ * #1073: Clarify the documentation about when an "after: error" command hook runs and how it
+   differs from other hooks:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #1075: Fix an incorrect warning about Borg placeholders being unsupported in a command hook.
+ * #1080: If the exact same "everything" command hook is present in multiple configuration files,
+   only run it once.
+
+2.0.3
+ * #1065: Fix a regression in monitoring hooks in which an error pinged the finish state instead of
+   the fail state.
+ * #1066: Add a "states" option to command hooks, so you can optionally skip an "after" hook if
+   borgmatic encounters an error.
+ * #1071: Fix an error in the LVM hook when removing a snapshot directory.
+
 2.0.2
  * #1035: Document potential performance issues and workarounds with the ZFS, Btrfs, and LVM hooks:
    https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
@@ -46,7 +78,7 @@
  * #837: Add custom command options for the MongoDB hook.
  * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
  * #1020: Document a database use case involving a temporary database client container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #1037: Fix an error with the "extract" action when both a remote repository and a
    "working_directory" are used.
  * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
@@ -296,7 +328,7 @@
    paths when a "working_directory" is set.
  * #906: Add documentation details for how to run custom database dump commands using binaries from
    running containers:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * Fix a regression in which the "color" option had no effect.
  * Add a recent contributors section to the documentation, because credit where credit's due! See:
    https://torsion.org/borgmatic/#recent-contributors
@@ -352,7 +384,7 @@
  * Fix handling of the NO_COLOR environment variable to ignore an empty value.
  * Add documentation about backing up containerized databases by configuring borgmatic to exec into
    a container to run a dump command:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 
 1.8.9
  * #311: Add custom dump/restore command options for MySQL and MariaDB.
@@ -622,7 +654,7 @@
    at the command-line. See the configuration reference for more information:
    https://torsion.org/borgmatic/docs/reference/configuration/
  * #649: Add documentation on backing up a database running in a container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #655: Fix error when databases are configured and a source directory doesn't exist.
  * Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
    To join in the pedantry, refresh your test environment with "tox --recreate".

README.md

@@ -138,7 +138,8 @@ issues.
 
 ### Social
 
-Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
+Follow <a rel="me" href="https://floss.social/@borgmatic">borgmatic on
+Mastodon</a>.
 
 
 ### Chat
@@ -159,21 +160,21 @@ Other questions or comments? Contact
 ### Contributing
 
 borgmatic [source code is
-available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
-on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
+available](https://projects.torsion.org/borgmatic-collective/borgmatic) and also
+has a read-only mirror on
+[GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
 
 borgmatic is licensed under the GNU General Public License version 3 or any
 later version.
 
-If you'd like to contribute to borgmatic development, please feel free to
-submit a [Pull
-Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
+If you'd like to contribute to borgmatic development, please feel free to submit
+a [pull
+request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
 open an
 [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) to
 discuss your idea. Note that you'll need to
 [register](https://projects.torsion.org/user/sign_up?invite_code=borgmatic)
-first. We also accept Pull Requests on GitHub, if that's more your thing. In
-general, contributions are very welcome. We don't bite!
+first. In general, contributions are very welcome. We don't bite!
 
 Also, please check out the [borgmatic development
 how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for

borgmatic/actions/pattern.py

@@ -165,6 +165,29 @@ def expand_patterns(patterns, working_directory=None, skip_paths=None):
     )
 
 
+def get_existent_path_or_parent(path):
+    '''
+    Given a path, return it if it exists. Otherwise, return the longest parent directory of the path
+    that exists. Return None if none of these paths exist.
+
+    This is used below for finding an existent path prefix of pattern's path, which is necessary if
+    the path contain globs or other special characters that we don't want to try to interpret
+    (because we want to leave that responsibility to Borg).
+    '''
+    if path.startswith('/e2e/'):
+        return None
+
+    try:
+        return next(
+            candidate_path
+            for candidate_path in (path,)
+            + tuple(str(parent) for parent in pathlib.PurePath(path).parents)
+            if os.path.exists(candidate_path)
+        )
+    except StopIteration:
+        return None
+
+
 def device_map_patterns(patterns, working_directory=None):
     '''
     Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
@@ -174,23 +197,31 @@ def device_map_patterns(patterns, working_directory=None):
 
     This is handy for determining whether two different pattern paths are on the same filesystem
     (have the same device identifier).
+
+    This function only considers the start of a pattern's path—from the start of the path up until
+    there's a path component with a glob or other non-literal character. If there are no such
+    characters, the whole path is considered. The rationale is that it's not feasible for borgmatic
+    to interpret Borg's patterns to see which actual files (and therefore devices) they map to. So
+    for instance, a pattern with a path of "/var/log/*/data" would end up with its device set to the
+    device of "/var/log"—ignoring the "/*/data" part due to that glob.
+
+    The one exception is that if a regular expression pattern path starts with "^", that will get
+    stripped off for purposes of determining its device.
     '''
     return tuple(
         borgmatic.borg.pattern.Pattern(
             pattern.path,
             pattern.type,
             pattern.style,
-            device=pattern.device
-            or (
-                os.stat(full_path).st_dev
-                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
-                and os.path.exists(full_path)
-                else None
-            ),
+            device=pattern.device or (os.stat(existent_path).st_dev if existent_path else None),
             source=pattern.source,
         )
         for pattern in patterns
-        for full_path in (os.path.join(working_directory or '', pattern.path),)
+        for existent_path in (
+            get_existent_path_or_parent(
+                os.path.join(working_directory or '', pattern.path.lstrip('^'))
+            ),
+        )
     )
 
 

borgmatic/borg/compact.py

@@ -1,7 +1,7 @@
 import logging
 
 import borgmatic.config.paths
-from borgmatic.borg import environment, flags
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
@@ -37,11 +37,16 @@ def compact_segments(
         + (('--threshold', str(threshold)) if threshold else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (
+            ('--dry-run',)
+            if dry_run and feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version)
+            else ()
+        )
         + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    if dry_run:
+    if dry_run and not feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version):
         logging.info('Skipping compact (dry run)')
         return
 

borgmatic/borg/create.py

@@ -69,7 +69,7 @@ def collect_special_file_paths(
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
     # files including any named pipe we've created. And omit "--filter" because that can break the
-    # paths output parsing below such that path lines no longer start with th expected "- ".
+    # paths output parsing below such that path lines no longer start with the expected "- ".
     paths_output = execute_command_and_capture_output(
         flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
         + ('--dry-run', '--list'),

borgmatic/borg/environment.py

@@ -17,6 +17,7 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
 DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
     'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
     'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
 }
 
 DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {

borgmatic/borg/feature.py

@@ -18,6 +18,7 @@ class Feature(Enum):
     EXCLUDED_FILES_MINUS = 12
     ARCHIVE_SERIES = 13
     NO_PRUNE_STATS = 14
+    DRY_RUN_COMPACT = 15
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -35,6 +36,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
     Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
     Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
+    Feature.DRY_RUN_COMPACT: parse('1.4.1'),  # borg compact --dry-run support
 }
 
 

borgmatic/commands/borgmatic.py

@@ -95,7 +95,9 @@ class Monitoring_hooks:
         self.config_filename = config_filename
         self.config = config
         self.dry_run = global_arguments.dry_run
-        self.monitoring_log_level = verbosity_to_log_level(config.get('monitoring_verbosity'))
+        self.monitoring_log_level = verbosity_to_log_level(
+            get_verbosity({config_filename: config}, 'monitoring_verbosity')
+        )
         self.monitoring_hooks_are_activated = (
             using_primary_action and self.monitoring_log_level != DISABLED
         )
@@ -279,8 +281,19 @@ def run_configuration(config_filename, config, config_paths, arguments):
                             encountered_error = error
                             error_repository = repository
 
+                # Re-raise any error, so that the Monitoring_hooks context manager wrapping this
+                # code can see the error and act accordingly. Do this here rather than as soon as
+                # the error is encountered so that an error with one repository doesn't prevent
+                # other repositories from running.
+                if encountered_error:
+                    raise encountered_error
+
     except (OSError, CalledProcessError, ValueError) as error:
-        yield from log_error_records('Error running configuration', error)
+        # No need to repeat logging of the error if it was already logged above.
+        if error_repository:
+            yield from log_error_records('Error running configuration')
+        else:
+            yield from log_error_records('Error running configuration', error)
 
         encountered_error = error
 
@@ -290,7 +303,10 @@ def run_configuration(config_filename, config, config_paths, arguments):
     try:
         command.execute_hooks(
             command.filter_hooks(
-                config.get('commands'), after='error', action_names=arguments.keys()
+                config.get('commands'),
+                after='error',
+                action_names=arguments.keys(),
+                state_names=['fail'],
             ),
             config.get('umask'),
             borgmatic.config.paths.get_working_directory(config),
@@ -879,23 +895,36 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
         return
 
     try:
+        seen_command_hooks = []
+
         for config_filename, config in configs.items():
-            command.execute_hooks(
-                command.filter_hooks(
-                    config.get('commands'), before='everything', action_names=arguments.keys()
+            command_hooks = command.filter_hooks(
+                tuple(
+                    command_hook
+                    for command_hook in config.get('commands', ())
+                    if command_hook not in seen_command_hooks
                 ),
-                config.get('umask'),
-                borgmatic.config.paths.get_working_directory(config),
-                arguments['global'].dry_run,
-                configuration_filename=config_filename,
-                log_file=log_file_path or '',
+                before='everything',
+                action_names=arguments.keys(),
             )
+
+            if command_hooks:
+                command.execute_hooks(
+                    command_hooks,
+                    config.get('umask'),
+                    borgmatic.config.paths.get_working_directory(config),
+                    arguments['global'].dry_run,
+                    configuration_filename=config_filename,
+                    log_file=log_file_path or '',
+                )
+                seen_command_hooks += list(command_hooks)
     except (CalledProcessError, ValueError, OSError) as error:
         yield from log_error_records('Error running before everything hook', error)
         return
 
     # Execute the actions corresponding to each configuration file.
     json_results = []
+    encountered_error = False
 
     for config_filename, config in configs.items():
         with Log_prefix(config_filename):
@@ -906,6 +935,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
             )
 
             if error_logs:
+                encountered_error = True
                 yield from log_error_records('An error occurred')
                 yield from error_logs
             else:
@@ -928,23 +958,37 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                 local_path=get_local_path(configs),
             )
         except (CalledProcessError, OSError) as error:
+            encountered_error = True
             yield from log_error_records('Error unmounting mount point', error)
 
     if json_results:
         sys.stdout.write(json.dumps(json_results))
 
     try:
+        seen_command_hooks = []
+
         for config_filename, config in configs.items():
-            command.execute_hooks(
-                command.filter_hooks(
-                    config.get('commands'), after='everything', action_names=arguments.keys()
+            command_hooks = command.filter_hooks(
+                tuple(
+                    command_hook
+                    for command_hook in config.get('commands', ())
+                    if command_hook not in seen_command_hooks
                 ),
-                config.get('umask'),
-                borgmatic.config.paths.get_working_directory(config),
-                arguments['global'].dry_run,
-                configuration_filename=config_filename,
-                log_file=log_file_path or '',
+                after='everything',
+                action_names=arguments.keys(),
+                state_names=['fail' if encountered_error else 'finish'],
             )
+
+            if command_hooks:
+                command.execute_hooks(
+                    command_hooks,
+                    config.get('umask'),
+                    borgmatic.config.paths.get_working_directory(config),
+                    arguments['global'].dry_run,
+                    configuration_filename=config_filename,
+                    log_file=log_file_path or '',
+                )
+                seen_command_hooks += list(command_hooks)
     except (CalledProcessError, ValueError, OSError) as error:
         yield from log_error_records('Error running after everything hook', error)
 

borgmatic/config/schema.yaml

@@ -389,6 +389,13 @@ properties:
             Path for Borg cache files. Defaults to
             $borg_base_directory/.cache/borg
         example: /path/to/base/cache
+    use_chunks_archive:
+        type: boolean
+        description: |
+            Enables or disables the use of chunks.archive.d for faster cache
+            resyncs in Borg. If true, value is set to "yes" (default) else
+            it's set to "no", reducing disk usage but slowing resyncs.
+        example: true
     borg_files_cache_ttl:
         type: integer
         description: |
@@ -558,6 +565,14 @@ properties:
         type: integer
         description: Number of yearly archives to keep.
         example: 1
+    keep_13weekly:
+        type: integer
+        description: Number of quarterly archives to keep (13 week strategy).
+        example: 13
+    keep_3monthly:
+        type: integer
+        description: Number of quarterly archives to keep (3 month strategy).
+        example: 3
     prefix:
         type: string
         description: |
@@ -1225,6 +1240,26 @@ properties:
                               particular actions listed here. Defaults to
                               running for all actions.
                           example: [create, prune, compact, check]
+                      states:
+                          type: array
+                          items:
+                              type: string
+                              enum:
+                                  - finish
+                                  - fail
+                          description: |
+                              Only trigger the hook if borgmatic encounters one
+                              of the states (execution results) listed here,
+                              where:
+                               * "finish": No errors occurred.
+                               * "fail": An error occurred.
+                              This state is evaluated only for the scope of the
+                              configured "action", "repository", etc., rather
+                              than for the entire borgmatic run. Only available
+                              for "after" hooks. Defaults to running the hook
+                              for all states.
+                          example:
+                              - finish
                       run:
                           type: array
                           items:
@@ -1525,6 +1560,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MariaDB client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mariadb_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |
@@ -1673,6 +1725,23 @@ properties:
                         Defaults to the "password" option. Supports the
                         "{credential ...}" syntax.
                     example: trustsome1
+                password_transport:
+                    type: string
+                    enum:
+                        - pipe
+                        - environment
+                    description: |
+                        How to transmit database passwords from borgmatic to the
+                        MySQL client, one of:
+                         * "pipe": Securely transmit passwords via anonymous
+                           pipe. Only works if the database client is on the
+                           same host as borgmatic. (The server can be
+                           somewhere else.) This is the default value.
+                         * "environment": Transmit passwords via environment
+                           variable. Potentially less secure than a pipe, but
+                           necessary when the database client is elsewhere, e.g.
+                           when "mysql_dump_command" is configured to "exec"
+                           into a container and run a client there.
                 tls:
                     type: boolean
                     description: |

borgmatic/hooks/command.py

@@ -13,6 +13,20 @@ logger = logging.getLogger(__name__)
 
 
 SOFT_FAIL_EXIT_CODE = 75
+BORG_PLACEHOLDERS = {
+    '{hostname}',
+    '{fqdn}',
+    '{reverse-fqdn}',
+    '{now}',
+    '{utcnow}',
+    '{unixtime}',
+    '{user}',
+    '{pid}',
+    '{borgversion}',
+    '{borgmajor}',
+    '{borgminor}',
+    '{borgpatch}',
+}
 
 
 def interpolate_context(hook_description, command, context):
@@ -23,10 +37,13 @@ def interpolate_context(hook_description, command, context):
     for name, value in context.items():
         command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
 
-    for unsupported_variable in re.findall(r'{\w+}', command):
-        logger.warning(
-            f"Variable '{unsupported_variable}' is not supported in {hook_description} hook"
-        )
+    for unsupported_variable in re.findall(r'\{\w+\}', command):
+        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
+        # be a Borg placeholder, as Borg should hopefully consume it.
+        if unsupported_variable not in BORG_PLACEHOLDERS:
+            logger.warning(
+                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook'
+            )
 
     return command
 
@@ -47,21 +64,25 @@ def make_environment(current_environment, sys_module=sys):
     return environment
 
 
-def filter_hooks(command_hooks, before=None, after=None, hook_name=None, action_names=None):
+def filter_hooks(command_hooks, before=None, after=None, action_names=None, state_names=None):
     '''
     Given a sequence of command hook dicts from configuration and one or more filters (before name,
-    after name, calling hook name, or a sequence of action names), filter down the command hooks to
-    just the ones that match the given filters.
+    after name, a sequence of action names, and/or a sequence of execution result state names),
+    filter down the command hooks to just the ones that match the given filters.
     '''
     return tuple(
         hook_config
         for hook_config in command_hooks or ()
         for config_action_names in (hook_config.get('when'),)
+        for config_state_names in (hook_config.get('states'),)
         if before is None or hook_config.get('before') == before
         if after is None or hook_config.get('after') == after
         if action_names is None
         or config_action_names is None
         or set(config_action_names or ()).intersection(set(action_names))
+        if state_names is None
+        or config_state_names is None
+        or set(config_state_names or ()).intersection(set(state_names))
     )
 
 
@@ -143,7 +164,7 @@ class Before_after_hooks:
            before_after='do_stuff',
            umask=config.get('umask'),
            dry_run=dry_run,
-           hook_name='myhook',
+           action_names=['create'],
        ):
             do()
             some()
@@ -160,22 +181,20 @@ class Before_after_hooks:
         umask,
         working_directory,
         dry_run,
-        hook_name=None,
         action_names=None,
         **context,
     ):
         '''
         Given a sequence of command hook configuration dicts, the before/after name, a umask to run
-        commands with, a working directory to run commands with, a dry run flag, the name of the
-        calling hook, a sequence of action names, and any context for the executed commands, save
-        those data points for use below.
+        commands with, a working directory to run commands with, a dry run flag, a sequence of
+        action names, and any context for the executed commands, save those data points for use
+        below.
         '''
         self.command_hooks = command_hooks
         self.before_after = before_after
         self.umask = umask
         self.working_directory = working_directory
         self.dry_run = dry_run
-        self.hook_name = hook_name
         self.action_names = action_names
         self.context = context
 
@@ -188,7 +207,6 @@ class Before_after_hooks:
                 borgmatic.hooks.command.filter_hooks(
                     self.command_hooks,
                     before=self.before_after,
-                    hook_name=self.hook_name,
                     action_names=self.action_names,
                 ),
                 self.umask,
@@ -202,7 +220,7 @@ class Before_after_hooks:
 
             # Trigger the after hook manually, since raising here will prevent it from being run
             # otherwise.
-            self.__exit__(None, None, None)
+            self.__exit__(exception_type=type(error), exception=error, traceback=None)
 
             raise ValueError(f'Error running before {self.before_after} hook: {error}')
 
@@ -215,8 +233,8 @@ class Before_after_hooks:
                 borgmatic.hooks.command.filter_hooks(
                     self.command_hooks,
                     after=self.before_after,
-                    hook_name=self.hook_name,
                     action_names=self.action_names,
+                    state_names=['fail' if exception_type else 'finish'],
                 ),
                 self.umask,
                 self.working_directory,

borgmatic/hooks/data_source/lvm.py

@@ -407,7 +407,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
                 continue
 
         if not dry_run:
-            shutil.rmtree(snapshots_directory)
+            shutil.rmtree(snapshots_directory, ignore_errors=True)
 
     # Delete snapshots.
     lvremove_command = hook_config.get('lvremove_command', 'lvremove')

borgmatic/hooks/data_source/mariadb.py

@@ -120,13 +120,19 @@ def database_names_to_dump(database, config, username, password, environment, dr
         shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mariadb_show_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -184,14 +190,20 @@ def execute_dump_command(
         for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
     )
     extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mariadb_dump_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -255,7 +267,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -383,18 +402,27 @@ def restore_data_source_dump(
         shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
     )
     extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mariadb_restore_command
-        + make_defaults_file_options(username, password, defaults_extra_filename)
+        + (
+            make_defaults_file_options(username, password, defaults_extra_filename)
+            if password_transport == 'pipe'
+            else ()
+        )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MariaDB database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/mysql.py

@@ -45,15 +45,21 @@ def database_names_to_dump(database, config, username, password, environment, dr
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('list_options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     show_command = (
         mysql_show_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--skip-column-names', '--batch')
@@ -109,16 +115,22 @@ def execute_dump_command(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(database.get('options'))
     )
+    password_transport = database.get('password_transport', 'pipe')
     dump_command = (
         mysql_dump_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
         + (('--host', database['hostname']) if 'hostname' in database else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
         + ('--databases',)
@@ -182,7 +194,14 @@ def dump_data_sources(
         password = borgmatic.hooks.credential.parse.resolve_credential(
             database.get('password'), config
         )
-        environment = dict(os.environ)
+        environment = dict(
+            os.environ,
+            **(
+                {'MYSQL_PWD': password}
+                if password and database.get('password_transport') == 'environment'
+                else {}
+            ),
+        )
         dump_database_names = database_names_to_dump(
             database, config, username, password, environment, dry_run
         )
@@ -312,20 +331,29 @@ def restore_data_source_dump(
     extra_options, defaults_extra_filename = (
         borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
     )
+    password_transport = data_source.get('password_transport', 'pipe')
     restore_command = (
         mysql_restore_command
-        + borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
-            username, password, defaults_extra_filename
+        + (
+            borgmatic.hooks.data_source.mariadb.make_defaults_file_options(
+                username, password, defaults_extra_filename
+            )
+            if password_transport == 'pipe'
+            else ()
         )
         + extra_options
         + ('--batch',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
     )
-    environment = dict(os.environ)
+    environment = dict(
+        os.environ,
+        **({'MYSQL_PWD': password} if password and password_transport == 'environment' else {}),
+    )
 
     logger.debug(f"Restoring MySQL database {data_source['name']}{dry_run_label}")
     if dry_run:

borgmatic/hooks/data_source/snapshot.py

@@ -8,9 +8,9 @@ def get_contained_patterns(parent_directory, candidate_patterns):
     '''
     Given a parent directory and a set of candidate patterns potentially inside it, get the subset
     of contained patterns for which the parent directory is actually the parent, a grandparent, the
-    very same directory, etc. The idea is if, say, /var/log and /var/lib are candidate pattern
-    paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at /var, then
-    /var is what we want to snapshot.
+    very same directory, etc. The idea is if, say, "/var/log" and "/var/lib" are candidate pattern
+    paths, but there's a parent directory (logical volume, dataset, subvolume, etc.) at "/var", then
+    "/var" is what we want to snapshot.
 
     If a parent directory and a candidate pattern are on different devices, skip the pattern. That's
     because any snapshot of a parent directory won't actually include "contained" directories if
@@ -18,8 +18,8 @@ def get_contained_patterns(parent_directory, candidate_patterns):
 
     For this function to work, a candidate pattern path can't have any globs or other non-literal
     characters in the initial portion of the path that matches the parent directory. For instance, a
-    parent directory of /var would match a candidate pattern path of /var/log/*/data, but not a
-    pattern path like /v*/log/*/data.
+    parent directory of "/var" would match a candidate pattern path of "/var/log/*/data", but not a
+    pattern path like "/v*/log/*/data".
 
     The one exception is that if a regular expression pattern path starts with "^", that will get
     stripped off for purposes of matching against a parent directory.

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -55,6 +55,25 @@ commands:
           - "echo Backup: start"
 ```
 
+By default, an `after` command hook runs even if an error occurs in the
+corresponding `before` hook or between those two hooks. This allows you to
+perform cleanup steps that correspond to `before` preparation commands—even when
+something goes wrong. You may notice that this is a departure from the way that
+the deprecated `after_*` hooks worked in borgmatic prior to version 2.0.0.
+
+<span class="minilink minilink-addedin">New in version 2.0.3</span> You can
+customize this behavior with the `states` option. For instance, here's an
+example of an `after` hook that only triggers on success and not on error:
+
+```yaml
+commands:
+    - after: action
+      when: [create]
+      states: [finish]
+      run:
+          - echo "After successful create!"
+```
+
 Each command in the `commands:` list has the following options:
 
  * `before` or `after`: Name for the point in borgmatic's execution that the commands should be run before or after, one of:
@@ -64,17 +83,17 @@ Each command in the `commands:` list has the following options:
     * `everything` runs before or after all configuration files. Errors here do not trigger `error` hooks or the `fail` state in monitoring hooks. This replaces the deprecated `before_everything` and `after_everything`.
     * `error` runs after an error occurs—and it's only available for `after`. This replaces the deprecated `on_error` hook.
  * `when`: Only trigger the hook when borgmatic is run with particular actions (`create`, `prune`, etc.) listed here. Defaults to running for all actions.
+ * `states`: <span class="minilink minilink-addedin">New in version 2.0.3</span> Only trigger the hook if borgmatic encounters one of the states (execution results) listed here. This state is evaluated only for the scope of the configured `action`, `repository`, etc., rather than for the entire borgmatic run. Only available for `after` hooks. Defaults to running the hook for all states. One or more of:
+    * `finish`: No errors occurred.
+    * `fail`: An error occurred.
  * `run`: List of one or more shell commands or scripts to run when this command hook is triggered.
 
-An `after` command hook runs even if an error occurs in the corresponding
-`before` hook or between those two hooks. This allows you to perform cleanup
-steps that correspond to `before` preparation commands—even when something goes
-wrong. This is a departure from the way that the deprecated `after_*` hooks
-worked in borgmatic prior to version 2.0.0.
+When command hooks run, they respect the `working_directory` option if it is
+configured, meaning that the hook commands are run in that directory.
 
-Additionally, when command hooks run, they respect the `working_directory`
-option if it is configured, meaning that the hook commands are run in that
-directory.
+<span class="minilink minilink-addedin">New in version 2.0.4</span>If the exact
+same `everything` command hook is present in multiple configuration files,
+borgmatic only runs it once.
 
 
 ### Order of execution
@@ -97,11 +116,20 @@ borgmatic for the `create` and `prune` actions. Here's the order of execution:
             * Run `after: action` hooks for `prune`.
         * Run `after: repository` hooks (for the first repository).
     * Run `after: configuration` hooks (from the first configuration file).
+    * Run `after: error` hooks (if an error occurs).
  * Run `after: everything` hooks (from all configuration files).
 
 This same order of execution extends to multiple repositories and/or
 configuration files.
 
+Based on the above, you can see the difference between, say, an `after: action`
+hook with `states: [fail]` and an `after: error` hook. The `after: action hook`
+runs immediately after the create action fails for a particular repository—so
+before any subsequent actions for that repository or other repositories even
+have a chance to run. Whereas the `after: error` hook doesn't run until all
+actions for—and repositories in—a configuration file have had a chance to
+execute.
+
 
 ### Deprecated command hooks
 

docs/how-to/backup-your-databases.md

@@ -191,7 +191,7 @@ mysql_databases:
       format: sql
 ```
 
-### Containers
+### Database containers
 
 If your database server is running within a container and borgmatic is too, no
 problem—configure borgmatic to connect to the container's name on its exposed
@@ -209,6 +209,9 @@ postgresql_databases:
 <span class="minilink minilink-addedin">Prior to version 1.8.0</span> Put
 these options in the `hooks:` section of your configuration.
 
+
+#### Database client on the host
+
 But what if borgmatic is running on the host? You can still connect to a
 database server container if its ports are properly exposed to the host. For
 instance, when running the database container, you can specify `--publish
@@ -228,17 +231,19 @@ And then you can configure borgmatic running on the host to connect to the
 database:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: 127.0.0.1
-          port: 5433
-          username: postgres
-          password: trustsome1
+postgresql_databases:
+    - name: users
+      hostname: 127.0.0.1
+      port: 5433
+      username: postgres
+      password: trustsome1
 ```
 
 Alter the ports in these examples to suit your particular database system.
 
+
+#### Database client in a running container
+
 Normally, borgmatic dumps a database by running a database dump command (e.g.
 `pg_dump`) on the host or wherever borgmatic is running, and this command
 connects to your containerized database via the given `hostname` and `port`. But
@@ -250,14 +255,13 @@ to run the dump command.
 For instance, if using Docker and PostgreSQL, something like this might work:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: 127.0.0.1
-          port: 5433
-          username: postgres
-          password: trustsome1
-          pg_dump_command: docker exec my_pg_container pg_dump
+postgresql_databases:
+    - name: users
+      hostname: 127.0.0.1
+      port: 5433
+      username: postgres
+      password: trustsome1
+      pg_dump_command: docker exec my_pg_container pg_dump
 ```
 
 ... where `my_pg_container` is the name of your running database container.
@@ -282,19 +286,43 @@ services:
       - /run/user/1000:/run/user/1000
 ```
 
+And here's an example of using a MariaDB database client within a running Docker
+container:
+
+```yaml
+mariadb_databases:
+    - name: users
+      hostname: 127.0.0.1
+      username: example
+      password: trustsome1
+      password_transport: environment
+      mariadb_dump_command: docker exec --env MYSQL_PWD my_mariadb_container mariadb-dump
+```
+
+The `password_transport: environment` option tells borgmatic to transmit the
+password via environment variable instead of the default behavior of using an
+anonymous pipe. The environment variable transport is potentially less secure
+than the pipe, but it's necessary when the database client is running in a
+container separate from borgmatic.
+
+A similar approach can work with MySQL, using `mysql_dump_command` instead of
+`mariadb_dump_command` to run `mysqldump` in a container.
+
+
+#### Database client in a temporary container
+
 Another variation: If you're running borgmatic on the host but want to spin up a
 temporary `pg_dump` container whenever borgmatic dumps a database, for
 instance to make use of a `pg_dump` version not present on the host, try
 something like this:
 
 ```yaml
-hooks:
-    postgresql_databases:
-        - name: users
-          hostname: your-database-hostname
-          username: postgres
-          password: trustsome1
-          pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
+postgresql_databases:
+    - name: users
+      hostname: your-database-hostname
+      username: postgres
+      password: trustsome1
+      pg_dump_command: docker run --rm --env PGPASSWORD postgres:17-alpine pg_dump
 ```
 
 The `--env PGPASSWORD` is necessary here for borgmatic to provide your database

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "borgmatic"
-version = "2.0.2"
+version = "2.0.5"
 authors = [
   { name="Dan Helfman", email="witten@torsion.org" },
 ]

tests/end-to-end/commands/fake_btrfs.py

@@ -53,7 +53,7 @@ def save_snapshots(snapshot_paths):
 
 
 def print_subvolume_list(arguments, snapshot_paths):
-    assert arguments.subvolume_path == '/mnt/subvolume'
+    assert arguments.subvolume_path == '/e2e/mnt/subvolume'
 
     if not arguments.snapshots_only:
         for line in BUILTIN_SUBVOLUME_LIST_LINES:

tests/end-to-end/commands/fake_findmnt.py

@@ -14,7 +14,7 @@ def parse_arguments(*unparsed_arguments):
 BUILTIN_FILESYSTEM_MOUNT_OUTPUT = '''{
        "filesystems": [
           {
-             "target": "/mnt/subvolume",
+             "target": "/e2e/mnt/subvolume",
              "source": "/dev/loop0",
              "fstype": "btrfs",
              "options": "rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/"

tests/end-to-end/commands/fake_lsblk.py

@@ -20,7 +20,7 @@ BUILTIN_BLOCK_DEVICES = {
         {
             'name': 'vgroup-lvolume',
             'path': '/dev/mapper/vgroup-lvolume',
-            'mountpoint': '/mnt/lvolume',
+            'mountpoint': '/e2e/mnt/lvolume',
             'type': 'lvm',
         },
         {

tests/end-to-end/commands/fake_zfs.py

@@ -23,20 +23,20 @@ def parse_arguments(*unparsed_arguments):
 
 BUILTIN_DATASETS = (
     {
-        'name': 'pool',
+        'name': 'e2e/pool',
         'used': '256K',
         'avail': '23.7M',
         'refer': '25K',
         'canmount': 'on',
-        'mountpoint': '/pool',
+        'mountpoint': '/e2e/pool',
     },
     {
-        'name': 'pool/dataset',
+        'name': 'e2e/pool/dataset',
         'used': '256K',
         'avail': '23.7M',
         'refer': '25K',
         'canmount': 'on',
-        'mountpoint': '/pool/dataset',
+        'mountpoint': '/e2e/pool/dataset',
     },
 )
 

tests/end-to-end/hooks/data_source/test_btrfs.py

@@ -16,10 +16,10 @@ def generate_configuration(config_path, repository_path):
         open(config_path)
         .read()
         .replace('ssh://user@backupserver/./sourcehostname.borg', repository_path)
-        .replace('- path: /mnt/backup', '')
+        .replace('- path: /e2e/mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /mnt/subvolume/subdir')
+        .replace('- /etc', '- /e2e/mnt/subvolume/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'btrfs:\n'
@@ -51,11 +51,11 @@ def test_btrfs_create_and_list():
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'mnt/subvolume/subdir/file.txt' in output
+        assert 'e2e/mnt/subvolume/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not subprocess.check_output(
-            'python3 /app/tests/end-to-end/commands/fake_btrfs.py subvolume list -s /mnt/subvolume'.split(
+            'python3 /app/tests/end-to-end/commands/fake_btrfs.py subvolume list -s /e2e/mnt/subvolume'.split(
                 ' '
             )
         )

tests/end-to-end/hooks/data_source/test_lvm.py

@@ -17,10 +17,10 @@ def generate_configuration(config_path, repository_path):
         open(config_path)
         .read()
         .replace('ssh://user@backupserver/./sourcehostname.borg', repository_path)
-        .replace('- path: /mnt/backup', '')
+        .replace('- path: /e2e/mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /mnt/lvolume/subdir')
+        .replace('- /etc', '- /e2e/mnt/lvolume/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'lvm:\n'
@@ -56,7 +56,7 @@ def test_lvm_create_and_list():
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'mnt/lvolume/subdir/file.txt' in output
+        assert 'e2e/mnt/lvolume/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not json.loads(

tests/end-to-end/hooks/data_source/test_zfs.py

@@ -19,7 +19,7 @@ def generate_configuration(config_path, repository_path):
         .replace('- path: /mnt/backup', '')
         .replace('label: local', '')
         .replace('- /home', f'- {config_path}')
-        .replace('- /etc', '- /pool/dataset/subdir')
+        .replace('- /etc', '- /e2e/pool/dataset/subdir')
         .replace('- /var/log/syslog*', '')
         + 'encryption_passphrase: "test"\n'
         + 'zfs:\n'
@@ -45,14 +45,14 @@ def test_zfs_create_and_list():
         )
 
         # Run a create action to exercise ZFS snapshotting and backup.
-        subprocess.check_call(f'borgmatic --config {config_path} create'.split(' '))
+        subprocess.check_call(f'borgmatic -v 2 --config {config_path} create'.split(' '))
 
         # List the resulting archive and assert that the snapshotted files are there.
         output = subprocess.check_output(
             f'borgmatic --config {config_path} list --archive latest'.split(' ')
         ).decode(sys.stdout.encoding)
 
-        assert 'pool/dataset/subdir/file.txt' in output
+        assert 'e2e/pool/dataset/subdir/file.txt' in output
 
         # Assert that the snapshot has been deleted.
         assert not subprocess.check_output(

tests/integration/commands/test_borgmatic.py

@@ -12,3 +12,59 @@ def test_borgmatic_version_matches_news_version():
     news_version = open('NEWS').readline()
 
     assert borgmatic_version == news_version
+
+
+def test_run_configuration_without_error_pings_monitoring_hooks_start_and_finish():
+    config = {'repositories': [{'path': 'foo'}]}
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
+    flexmock(module).should_receive('run_actions').and_return([])
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.START,
+        object,
+        object,
+    ).once()
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.FINISH,
+        object,
+        object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
+
+
+def test_run_configuration_with_action_error_pings_monioring_hooks_start_and_fail():
+    config = {'repositories': [{'path': 'foo'}]}
+    arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
+    flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
+    flexmock(module).should_receive('run_actions').and_raise(OSError)
+    flexmock(module.dispatch).should_receive('call_hooks')
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.START,
+        object,
+        object,
+    ).once()
+    flexmock(module.dispatch).should_receive('call_hooks').with_args(
+        'ping_monitor',
+        config,
+        module.dispatch.Hook_type.MONITORING,
+        'test.yaml',
+        module.monitor.State.FAIL,
+        object,
+        object,
+    ).once()
+
+    list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))

tests/unit/actions/test_pattern.py

@@ -245,36 +245,69 @@ def test_expand_patterns_expands_only_tildes_in_non_root_patterns():
     assert paths == (Pattern('/root/bar/*', Pattern_type.INCLUDE),)
 
 
-def test_device_map_patterns_gives_device_id_per_path():
+def test_get_existent_path_or_parent_passes_through_existent_path():
     flexmock(module.os.path).should_receive('exists').and_return(True)
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz') == '/foo/bar/baz'
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_none():
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz') is None
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_existent_parent():
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar/baz*').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar').and_return(True)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo').never()
+    flexmock(module.os.path).should_receive('exists').with_args('/').never()
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz*') == '/foo/bar'
+
+
+def test_get_existent_path_or_parent_with_non_existent_path_returns_existent_grandparent():
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar/baz*').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo/bar').and_return(False)
+    flexmock(module.os.path).should_receive('exists').with_args('/foo').and_return(True)
+    flexmock(module.os.path).should_receive('exists').with_args('/').never()
+
+    assert module.get_existent_path_or_parent('/foo/bar/baz*') == '/foo'
+
+
+def test_get_existent_path_or_parent_with_end_to_end_test_prefix_returns_none():
+    flexmock(module.os.path).should_receive('exists').never()
+
+    assert module.get_existent_path_or_parent('/e2e/foo/bar/baz') is None
+
+
+def test_device_map_patterns_gives_device_id_per_path():
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
     flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
     flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
 
-    device_map = module.device_map_patterns((Pattern('/foo'), Pattern('/bar')))
-
-    assert device_map == (
-        Pattern('/foo', device=55),
-        Pattern('/bar', device=66),
-    )
-
-
-def test_device_map_patterns_only_considers_root_patterns():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
-    flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
-    flexmock(module.os).should_receive('stat').with_args('/bar*').never()
-
     device_map = module.device_map_patterns(
-        (Pattern('/foo'), Pattern('/bar*', Pattern_type.INCLUDE))
+        (
+            Pattern('/foo'),
+            Pattern('^/bar', type=Pattern_type.INCLUDE, style=Pattern_style.REGULAR_EXPRESSION),
+        )
     )
 
     assert device_map == (
         Pattern('/foo', device=55),
-        Pattern('/bar*', Pattern_type.INCLUDE),
+        Pattern(
+            '^/bar', type=Pattern_type.INCLUDE, style=Pattern_style.REGULAR_EXPRESSION, device=66
+        ),
     )
 
 
 def test_device_map_patterns_with_missing_path_does_not_error():
-    flexmock(module.os.path).should_receive('exists').and_return(True).and_return(False)
+    flexmock(module).should_receive('get_existent_path_or_parent').with_args('/foo').and_return(
+        '/foo'
+    )
+    flexmock(module).should_receive('get_existent_path_or_parent').with_args('/bar').and_return(
+        None
+    )
     flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
     flexmock(module.os).should_receive('stat').with_args('/bar').never()
 
@@ -287,7 +320,7 @@ def test_device_map_patterns_with_missing_path_does_not_error():
 
 
 def test_device_map_patterns_uses_working_directory_to_construct_path():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
     flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
     flexmock(module.os).should_receive('stat').with_args('/working/dir/bar').and_return(
         flexmock(st_dev=66)
@@ -304,7 +337,7 @@ def test_device_map_patterns_uses_working_directory_to_construct_path():
 
 
 def test_device_map_patterns_with_existing_device_id_does_not_overwrite_it():
-    flexmock(module.os.path).should_receive('exists').and_return(True)
+    flexmock(module).should_receive('get_existent_path_or_parent').replace_with(lambda path: path)
     flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
     flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=100))
 

tests/unit/borg/test_compact.py

@@ -68,8 +68,15 @@ def test_compact_segments_with_log_debug_calls_borg_with_debug_flag():
     )
 
 
-def test_compact_segments_with_dry_run_skips_borg_call():
+def test_compact_segments_with_dry_run_skips_borg_call_when_feature_unavailable():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.2.3'
+    ).and_return(False)
+    flexmock(module.environment).should_receive('make_environment').never()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').never()
     flexmock(module).should_receive('execute_command').never()
+    flexmock(logging).should_receive('info').with_args('Skipping compact (dry run)').once()
 
     module.compact_segments(
         repository_path='repo',
@@ -80,6 +87,24 @@ def test_compact_segments_with_dry_run_skips_borg_call():
     )
 
 
+def test_compact_segments_with_dry_run_executes_borg_call_when_feature_available():
+    flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
+    flexmock(module.feature).should_receive('available').with_args(
+        module.feature.Feature.DRY_RUN_COMPACT, '1.4.1'
+    ).and_return(True)
+    flexmock(module.environment).should_receive('make_environment').once()
+    flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').once()
+    flexmock(module).should_receive('execute_command').once()
+
+    module.compact_segments(
+        repository_path='repo',
+        config={},
+        local_borg_version='1.4.1',
+        global_arguments=flexmock(),
+        dry_run=True,
+    )
+
+
 def test_compact_segments_with_local_path_calls_borg_via_local_path():
     flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
     insert_execute_command_mock(('borg1',) + COMPACT_COMMAND[1:] + ('repo',), logging.INFO)

tests/unit/borg/test_environment.py

@@ -92,6 +92,7 @@ def test_make_environment_without_configuration_sets_certain_environment_variabl
         'BORG_EXIT_CODES': 'modern',
         'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'no',
         'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'no',
+        'BORG_USE_CHUNKS_ARCHIVE': 'no',
     }
 
 
@@ -101,6 +102,7 @@ def test_make_environment_without_configuration_passes_through_default_environme
             'USER': 'root',
             'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
             'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+            'BORG_USE_CHUNKS_ARCHIVE': 'yup',
         }
     )
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -113,6 +115,7 @@ def test_make_environment_without_configuration_passes_through_default_environme
         'USER': 'root',
         'BORG_RELOCATED_REPO_ACCESS_IS_OK': 'yup',
         'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK': 'nah',
+        'BORG_USE_CHUNKS_ARCHIVE': 'yup',
         'BORG_EXIT_CODES': 'modern',
     }
 
@@ -170,3 +173,17 @@ def test_make_environment_with_integer_variable_value():
     environment = module.make_environment({'borg_files_cache_ttl': 40})
 
     assert environment.get('BORG_FILES_CACHE_TTL') == '40'
+
+
+def test_make_environment_with_use_chunks_archive_should_set_correct_environment_value():
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).and_return(None)
+    flexmock(module.os).should_receive('pipe').never()
+
+    environment = module.make_environment({'use_chunks_archive': True})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'yes'
+
+    environment = module.make_environment({'use_chunks_archive': False})
+    assert environment.get('BORG_USE_CHUNKS_ARCHIVE') == 'no'

tests/unit/borg/test_prune.py

@@ -43,6 +43,28 @@ def test_make_prune_flags_returns_flags_from_config():
     assert result == BASE_PRUNE_FLAGS
 
 
+def test_make_prune_flags_with_keep_13weekly_and_keep_3monthly():
+    config = {
+        'keep_13weekly': 4,
+        'keep_3monthly': 5,
+    }
+    flexmock(module.feature).should_receive('available').and_return(True)
+    flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
+
+    result = module.make_prune_flags(
+        config, flexmock(match_archives=None), local_borg_version='1.2.3'
+    )
+
+    expected = (
+        '--keep-13weekly',
+        '4',
+        '--keep-3monthly',
+        '5',
+    )
+
+    assert result == expected
+
+
 def test_make_prune_flags_accepts_prefix_with_placeholders():
     config = {
         'keep_daily': 1,

tests/unit/commands/test_borgmatic.py

@@ -28,6 +28,7 @@ def test_get_skip_actions_uses_config_and_arguments(config, arguments, expected_
 
 
 def test_monitoring_hooks_with_monioring_disabled_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(module.DISABLED)
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -41,6 +42,7 @@ def test_monitoring_hooks_with_monioring_disabled_bails():
 
 
 def test_monitoring_hooks_with_non_primary_action_bails():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').never()
 
@@ -54,6 +56,7 @@ def test_monitoring_hooks_with_non_primary_action_bails():
 
 
 def test_monitoring_hooks_pings_monitors():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -117,6 +120,7 @@ def test_monitoring_hooks_pings_monitors():
 
 
 def test_monitoring_hooks_with_start_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -172,6 +176,7 @@ def test_monitoring_hooks_with_start_ping_error_raises():
 
 
 def test_monitoring_hooks_with_log_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -227,6 +232,7 @@ def test_monitoring_hooks_with_log_ping_error_raises():
 
 
 def test_monitoring_hooks_with_finish_ping_error_raises():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -281,7 +287,8 @@ def test_monitoring_hooks_with_finish_ping_error_raises():
             pass
 
 
-def test_monitoring_with_wrapped_code_error_pings_fail():
+def test_monitoring_hooks_with_wrapped_code_error_pings_fail():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -345,7 +352,8 @@ def test_monitoring_with_wrapped_code_error_pings_fail():
             raise OSError()
 
 
-def test_monitoring_with_fail_ping_error_raise_original_error():
+def test_monitoring_hooks_with_fail_ping_error_raise_original_error():
+    flexmock(module).should_receive('get_verbosity').and_return(module.logging.INFO)
     flexmock(module).should_receive('verbosity_to_log_level').and_return(flexmock())
     flexmock(module.dispatch).should_receive('call_hooks').with_args(
         'initialize_monitor',
@@ -465,8 +473,10 @@ def test_run_configuration_logs_actions_error():
     flexmock(module).should_receive('Monitoring_hooks').and_return(flexmock())
     flexmock(module.command).should_receive('Before_after_hooks').and_return(flexmock())
     flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
-    expected_results = [flexmock()]
-    flexmock(module).should_receive('log_error_records').and_return(expected_results)
+    expected_results = [flexmock(), flexmock()]
+    flexmock(module).should_receive('log_error_records').and_return(
+        expected_results[:1]
+    ).and_return(expected_results[1:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     flexmock(module.command).should_receive('filter_hooks')
@@ -532,7 +542,7 @@ def test_run_configuration_logs_on_error_hook_error():
     expected_results = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').and_return(
         expected_results[:1]
-    ).and_return(expected_results[1:])
+    ).and_return(expected_results[1:2]).and_return(expected_results[2:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     config = {'repositories': [{'path': 'foo'}]}
@@ -601,8 +611,10 @@ def test_run_configuration_bails_for_on_error_hook_soft_failure():
     error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks').and_raise(error)
-    expected_results = [flexmock()]
-    flexmock(module).should_receive('log_error_records').and_return(expected_results)
+    expected_results = [flexmock(), flexmock()]
+    flexmock(module).should_receive('log_error_records').and_return(
+        expected_results[:1]
+    ).and_return(expected_results[1:])
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError)
     config = {'repositories': [{'path': 'foo'}]}
@@ -654,11 +666,14 @@ def test_run_configuration_retries_hard_error():
         levelno=logging.WARNING,
         log_command_error_output=True,
     ).and_return([flexmock()])
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository',
         OSError,
-    ).and_return(error_logs)
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {'repositories': [{'path': 'foo'}], 'retries': 1}
@@ -680,13 +695,16 @@ def test_run_configuration_retries_repositories_in_order():
     flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_actions').and_raise(OSError).times(2)
-    expected_results = [flexmock(), flexmock()]
+    expected_results = [flexmock(), flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
     ).and_return(expected_results[:1]).ordered()
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(expected_results[1:]).ordered()
+    ).and_return(expected_results[1:2]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(expected_results[2:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
@@ -728,6 +746,10 @@ def test_run_configuration_retries_round_robin():
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
     ).and_return(bar_error_logs).ordered()
+    config_error_logs = [flexmock()]
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(config_error_logs).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -741,7 +763,7 @@ def test_run_configuration_retries_round_robin():
 
     results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
 
-    assert results == foo_error_logs + bar_error_logs
+    assert results == foo_error_logs + bar_error_logs + config_error_logs
 
 
 def test_run_configuration_with_one_retry():
@@ -766,10 +788,13 @@ def test_run_configuration_with_one_retry():
         levelno=logging.WARNING,
         log_command_error_output=True,
     ).and_return(flexmock()).ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -818,10 +843,13 @@ def test_run_configuration_with_retry_wait_does_backoff_after_each_retry():
     ).and_return([flexmock()]).ordered()
 
     flexmock(time).should_receive('sleep').with_args(30).and_return().ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -867,10 +895,13 @@ def test_run_configuration_with_multiple_repositories_retries_with_timeout():
 
     # Sleep before retrying bar (and failing)
     flexmock(time).should_receive('sleep').with_args(10).and_return().ordered()
-    error_logs = [flexmock()]
+    error_logs = [flexmock(), flexmock()]
     flexmock(module).should_receive('log_error_records').with_args(
         'Error running actions for repository', OSError
-    ).and_return(error_logs).ordered()
+    ).and_return(error_logs[:1]).ordered()
+    flexmock(module).should_receive('log_error_records').with_args(
+        'Error running configuration',
+    ).and_return(error_logs[1:]).ordered()
     flexmock(module.command).should_receive('filter_hooks')
     flexmock(module.command).should_receive('execute_hooks')
     config = {
@@ -1909,7 +1940,12 @@ def test_collect_highlander_action_summary_logs_error_on_run_validate_failure():
 
 def test_collect_configuration_run_summary_logs_info_for_success():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -1928,9 +1964,17 @@ def test_collect_configuration_run_summary_logs_info_for_success():
 
 
 def test_collect_configuration_run_summary_executes_hooks_for_create():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
-    flexmock(module.command).should_receive('execute_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook])
+    flexmock(module.command).should_receive('execute_hooks').twice()
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
     arguments = {
@@ -1940,7 +1984,7 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -1950,9 +1994,51 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
     assert {log.levelno for log in logs} == {logging.INFO}
 
 
+def test_collect_configuration_run_summary_deduplicates_everything_hooks_across_config_files():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, before='everything', action_names=object
+    ).and_return([before_everything_hook]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        (after_everything_hook,), before='everything', action_names=object
+    ).and_return([]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        command_hooks, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook]).once()
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        (before_everything_hook,), after='everything', action_names=object, state_names=['finish']
+    ).and_return([]).once()
+    flexmock(module.command).should_receive('execute_hooks').twice()
+    flexmock(module).should_receive('Log_prefix').and_return(flexmock())
+    flexmock(module).should_receive('run_configuration').and_return([])
+    arguments = {
+        'create': flexmock(),
+        'global': flexmock(monitoring_verbosity=1, dry_run=False),
+    }
+
+    logs = tuple(
+        module.collect_configuration_run_summary_logs(
+            {'test.yaml': {'commands': command_hooks}, 'other.yaml': {'commands': command_hooks}},
+            config_paths=['/tmp/test.yaml', '/tmp/other.yaml'],
+            arguments=arguments,
+            log_file_path=None,
+        )
+    )
+
+    assert {log.levelno for log in logs} == {logging.INFO}
+
+
 def test_collect_configuration_run_summary_logs_info_for_success_with_extract():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -1995,7 +2081,12 @@ def test_collect_configuration_run_summary_logs_extract_with_repository_error():
 
 def test_collect_configuration_run_summary_logs_info_for_success_with_mount():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2041,7 +2132,12 @@ def test_collect_configuration_run_summary_logs_mount_with_repository_error():
 
 def test_collect_configuration_run_summary_logs_missing_configs_error():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     arguments = {'global': flexmock(config_paths=[])}
     expected_logs = (flexmock(),)
@@ -2056,9 +2152,17 @@ def test_collect_configuration_run_summary_logs_missing_configs_error():
     assert logs == expected_logs
 
 
-def test_collect_configuration_run_summary_logs_pre_hook_error():
+def test_collect_configuration_run_summary_logs_before_hook_error():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    ).and_return([after_everything_hook])
     flexmock(module.command).should_receive('execute_hooks').and_raise(ValueError)
     expected_logs = (flexmock(),)
     flexmock(module).should_receive('log_error_records').and_return(expected_logs)
@@ -2069,7 +2173,7 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -2079,9 +2183,17 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
     assert logs == expected_logs
 
 
-def test_collect_configuration_run_summary_logs_post_hook_error():
+def test_collect_configuration_run_summary_logs_after_hook_error():
+    before_everything_hook = {'before': 'everything', 'run': ['echo hi']}
+    after_everything_hook = {'after': 'everything', 'run': ['echo hi']}
+    command_hooks = (before_everything_hook, after_everything_hook)
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    ).and_return([before_everything_hook])
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    ).and_return([after_everything_hook])
     flexmock(module.command).should_receive('execute_hooks').and_return(None).and_raise(ValueError)
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2094,7 +2206,7 @@ def test_collect_configuration_run_summary_logs_post_hook_error():
 
     logs = tuple(
         module.collect_configuration_run_summary_logs(
-            {'test.yaml': {}},
+            {'test.yaml': {'commands': command_hooks}},
             config_paths=['/tmp/test.yaml'],
             arguments=arguments,
             log_file_path=None,
@@ -2129,7 +2241,12 @@ def test_collect_configuration_run_summary_logs_for_list_with_archive_and_reposi
 
 def test_collect_configuration_run_summary_logs_info_for_success_with_list():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2152,7 +2269,12 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_list():
 
 def test_collect_configuration_run_summary_logs_run_configuration_error_logs():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return(
@@ -2175,7 +2297,12 @@ def test_collect_configuration_run_summary_logs_run_configuration_error_logs():
 
 def test_collect_configuration_run_summary_logs_run_umount_error():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['fail']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return([])
@@ -2202,7 +2329,12 @@ def test_collect_configuration_run_summary_logs_run_umount_error():
 
 def test_collect_configuration_run_summary_logs_outputs_merged_json_results():
     flexmock(module.validate).should_receive('guard_configuration_contains_repository')
-    flexmock(module.command).should_receive('filter_hooks')
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, before='everything', action_names=object
+    )
+    flexmock(module.command).should_receive('filter_hooks').with_args(
+        object, after='everything', action_names=object, state_names=['finish']
+    )
     flexmock(module.command).should_receive('execute_hooks')
     flexmock(module).should_receive('Log_prefix').and_return(flexmock())
     flexmock(module).should_receive('run_configuration').and_return(['foo', 'bar']).and_return(

tests/unit/hooks/data_source/test_mariadb.py

@@ -167,6 +167,38 @@ def test_database_names_to_dump_queries_mariadb_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mariadb',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mariadb_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -247,9 +279,22 @@ def test_dump_data_sources_dumps_each_database():
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
         'resolve_credential'
     ).replace_with(lambda value, config: value)
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[0],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -285,9 +330,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -311,6 +361,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -378,6 +472,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mariadb_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=mariadb.cnf --skip-ssl'}
     flexmock(module).should_receive('parse_extra_options').and_return(
@@ -479,6 +621,48 @@ def test_execute_dump_command_runs_mariadb_dump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mariadb-dump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mariadb_dump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -849,54 +1033,6 @@ def test_execute_dump_command_with_dry_run_skips_mariadb_dump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mariadb_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1168,6 +1304,49 @@ def test_restore_data_source_dump_runs_mariadb_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module).should_receive('parse_extra_options').and_return((), None)
+    flexmock(module).should_receive('make_defaults_file_options').never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mariadb', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/data_source/test_mysql.py

@@ -60,6 +60,42 @@ def test_database_names_to_dump_queries_mysql_for_database_names():
     assert names == ('foo', 'bar')
 
 
+def test_database_names_to_dump_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    environment = flexmock()
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module).should_receive('execute_command_and_capture_output').with_args(
+        (
+            'mysql',
+            '--user',
+            'root',
+            '--skip-column-names',
+            '--batch',
+            '--execute',
+            'show schemas',
+        ),
+        environment=environment,
+    ).and_return('foo\nbar\nmysql\n').once()
+
+    names = module.database_names_to_dump(
+        {'name': 'all', 'password_transport': 'environment'},
+        {},
+        'root',
+        'trustsome1',
+        environment,
+        dry_run=False,
+    )
+
+    assert names == ('foo', 'bar')
+
+
 def test_database_names_to_dump_runs_mysql_with_tls():
     environment = flexmock()
     flexmock(module.borgmatic.hooks.credential.parse).should_receive(
@@ -141,9 +177,22 @@ def test_dump_data_sources_dumps_each_database():
         'resolve_credential'
     ).and_return(None)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[0],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=databases[1],
+        config={},
+        username=None,
+        password=None,
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('bar',))
 
     for name, process in zip(('foo', 'bar'), processes):
         flexmock(module).should_receive('execute_dump_command').with_args(
@@ -179,9 +228,14 @@ def test_dump_data_sources_dumps_with_password():
         'resolve_credential'
     ).replace_with(lambda value, config: value)
     flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module).should_receive('database_names_to_dump').and_return(('foo',)).and_return(
-        ('bar',)
-    )
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root'},
+        dry_run=False,
+    ).and_return(('foo',))
 
     flexmock(module).should_receive('execute_dump_command').with_args(
         database=database,
@@ -205,6 +259,50 @@ def test_dump_data_sources_dumps_with_password():
     ) == [process]
 
 
+def test_dump_data_sources_dumps_with_environment_password_transport_passes_password_environment_variable():
+    database = {
+        'name': 'foo',
+        'username': 'root',
+        'password': 'trustsome1',
+        'password_transport': 'environment',
+    }
+    process = flexmock()
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module).should_receive('database_names_to_dump').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=False,
+    ).and_return(('foo',))
+
+    flexmock(module).should_receive('execute_dump_command').with_args(
+        database=database,
+        config={},
+        username='root',
+        password='trustsome1',
+        dump_path=object,
+        database_names=('foo',),
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+        dry_run=object,
+        dry_run_label=object,
+    ).and_return(process).once()
+
+    assert module.dump_data_sources(
+        [database],
+        {},
+        config_paths=('test.yaml',),
+        borgmatic_runtime_directory='/run/borgmatic',
+        patterns=[],
+        dry_run=False,
+    ) == [process]
+
+
 def test_dump_data_sources_dumps_all_databases_at_once():
     databases = [{'name': 'all'}]
     process = flexmock()
@@ -272,6 +370,54 @@ def test_dump_data_sources_dumps_all_databases_separately_when_format_configured
     )
 
 
+def test_dump_data_sources_errors_for_missing_all_databases():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    with pytest.raises(ValueError):
+        assert module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=False,
+        )
+
+
+def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
+    databases = [{'name': 'all'}]
+    flexmock(module).should_receive('make_dump_path').and_return('')
+    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
+        'databases/localhost/all'
+    )
+    flexmock(module).should_receive('database_names_to_dump').and_return(())
+
+    assert (
+        module.dump_data_sources(
+            databases,
+            {},
+            config_paths=('test.yaml',),
+            borgmatic_runtime_directory='/run/borgmatic',
+            patterns=[],
+            dry_run=True,
+        )
+        == []
+    )
+
+
 def test_database_names_to_dump_runs_mysql_with_list_options():
     database = {'name': 'all', 'list_options': '--defaults-extra-file=my.cnf --skip-ssl'}
     flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
@@ -375,6 +521,52 @@ def test_execute_dump_command_runs_mysqldump():
     )
 
 
+def test_execute_dump_command_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    process = flexmock()
+    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
+    flexmock(module.os.path).should_receive('exists').and_return(False)
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.dump).should_receive('create_named_pipe_for_dump')
+
+    flexmock(module).should_receive('execute_command').with_args(
+        (
+            'mysqldump',
+            '--add-drop-database',
+            '--user',
+            'root',
+            '--databases',
+            'foo',
+            '--result-file',
+            'dump',
+        ),
+        environment=None,
+        run_to_completion=False,
+    ).and_return(process).once()
+
+    assert (
+        module.execute_dump_command(
+            database={'name': 'foo', 'password_transport': 'environment'},
+            config={},
+            username='root',
+            password='trustsome1',
+            dump_path=flexmock(),
+            database_names=('foo',),
+            environment=None,
+            dry_run=False,
+            dry_run_label='',
+        )
+        == process
+    )
+
+
 def test_execute_dump_command_runs_mysqldump_without_add_drop_database():
     process = flexmock()
     flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -761,54 +953,6 @@ def test_execute_dump_command_with_dry_run_skips_mysqldump():
     )
 
 
-def test_dump_data_sources_errors_for_missing_all_databases():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    with pytest.raises(ValueError):
-        assert module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=False,
-        )
-
-
-def test_dump_data_sources_does_not_error_for_missing_all_databases_with_dry_run():
-    databases = [{'name': 'all'}]
-    flexmock(module).should_receive('make_dump_path').and_return('')
-    flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
-    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
-        'resolve_credential'
-    ).replace_with(lambda value, config: value)
-    flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
-        'databases/localhost/all'
-    )
-    flexmock(module).should_receive('database_names_to_dump').and_return(())
-
-    assert (
-        module.dump_data_sources(
-            databases,
-            {},
-            config_paths=('test.yaml',),
-            borgmatic_runtime_directory='/run/borgmatic',
-            patterns=[],
-            dry_run=True,
-        )
-        == []
-    )
-
-
 def test_restore_data_source_dump_runs_mysql_to_restore():
     hook_config = [{'name': 'foo'}, {'name': 'bar'}]
     extract_process = flexmock(stdout=flexmock())
@@ -1092,6 +1236,53 @@ def test_restore_data_source_dump_runs_mysql_with_username_and_password():
     )
 
 
+def test_restore_data_source_with_environment_password_transport_skips_defaults_file_and_passes_user_flag():
+    hook_config = [
+        {
+            'name': 'foo',
+            'username': 'root',
+            'password': 'trustsome1',
+            'password_transport': 'environment',
+        }
+    ]
+    extract_process = flexmock(stdout=flexmock())
+
+    flexmock(module.borgmatic.hooks.credential.parse).should_receive(
+        'resolve_credential'
+    ).replace_with(lambda value, config: value)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'parse_extra_options'
+    ).and_return((), None)
+    flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
+        'make_defaults_file_options'
+    ).never()
+    flexmock(module.os).should_receive('environ').and_return(
+        {'USER': 'root', 'MYSQL_PWD': 'trustsome1'}
+    )
+    flexmock(module).should_receive('execute_command_with_processes').with_args(
+        ('mysql', '--batch', '--user', 'root'),
+        processes=[extract_process],
+        output_log_level=logging.DEBUG,
+        input_file=extract_process.stdout,
+        environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
+    ).once()
+
+    module.restore_data_source_dump(
+        hook_config,
+        {},
+        data_source=hook_config[0],
+        dry_run=False,
+        extract_process=extract_process,
+        connection_params={
+            'hostname': None,
+            'port': None,
+            'username': None,
+            'password': None,
+        },
+        borgmatic_runtime_directory='/run/borgmatic',
+    )
+
+
 def test_restore_data_source_dump_with_connection_params_uses_connection_params_for_restore():
     hook_config = [
         {

tests/unit/hooks/test_command.py

@@ -11,8 +11,16 @@ def test_interpolate_context_passes_through_command_without_variable():
     assert module.interpolate_context('pre-backup', 'ls', {'foo': 'bar'}) == 'ls'
 
 
-def test_interpolate_context_passes_through_command_with_unknown_variable():
+def test_interpolate_context_warns_and_passes_through_command_with_unknown_variable():
     command = 'ls {baz}'  # noqa: FS003
+    flexmock(module.logger).should_receive('warning').once()
+
+    assert module.interpolate_context('pre-backup', command, {'foo': 'bar'}) == command
+
+
+def test_interpolate_context_does_not_warn_and_passes_through_command_with_unknown_variable_matching_borg_placeholder():
+    command = 'ls {hostname}'  # noqa: FS003
+    flexmock(module.logger).should_receive('warning').never()
 
     assert module.interpolate_context('pre-backup', command, {'foo': 'bar'}) == command
 
@@ -167,6 +175,44 @@ def test_make_environment_with_pyinstaller_and_LD_LIBRARY_PATH_ORIG_copies_it_in
                 },
             ),
         ),
+        (
+            (
+                {
+                    'before': 'action',
+                    'states': ['finish'],  # Not actually valid; only valid for "after".
+                    'run': ['foo'],
+                },
+                {
+                    'after': 'action',
+                    'run': ['bar'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['finish'],
+                    'run': ['baz'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['fail'],
+                    'run': ['quux'],
+                },
+            ),
+            {
+                'after': 'action',
+                'state_names': ['finish'],
+            },
+            (
+                {
+                    'after': 'action',
+                    'run': ['bar'],
+                },
+                {
+                    'after': 'action',
+                    'states': ['finish'],
+                    'run': ['baz'],
+                },
+            ),
+        ),
     ),
 )
 def test_filter_hooks(hooks, filters, expected_hooks):
@@ -336,14 +382,13 @@ def test_before_after_hooks_calls_command_hooks():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').twice()
 
@@ -353,7 +398,6 @@ def test_before_after_hooks_calls_command_hooks():
         umask=1234,
         working_directory='/working',
         dry_run=False,
-        hook_name='myhook',
         action_names=['create'],
         context1='stuff',
         context2='such',
@@ -369,14 +413,13 @@ def test_before_after_hooks_with_before_error_runs_after_hook_and_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['fail'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_raise(OSError).and_return(None)
     flexmock(module).should_receive('considered_soft_failure').and_return(False)
@@ -388,7 +431,6 @@ def test_before_after_hooks_with_before_error_runs_after_hook_and_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',
@@ -404,14 +446,13 @@ def test_before_after_hooks_with_before_soft_failure_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).never()
     flexmock(module).should_receive('execute_hooks').and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(True)
@@ -423,7 +464,6 @@ def test_before_after_hooks_with_before_soft_failure_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',
@@ -431,6 +471,38 @@ def test_before_after_hooks_with_before_soft_failure_raises():
             pass
 
 
+def test_before_after_hooks_with_wrapped_code_error_runs_after_hook_and_raises():
+    commands = [
+        {'before': 'repository', 'run': ['foo', 'bar']},
+        {'after': 'repository', 'run': ['baz']},
+    ]
+    flexmock(module).should_receive('filter_hooks').with_args(
+        commands,
+        before='action',
+        action_names=['create'],
+    ).and_return(flexmock()).once()
+    flexmock(module).should_receive('filter_hooks').with_args(
+        commands,
+        after='action',
+        action_names=['create'],
+        state_names=['fail'],
+    ).and_return(flexmock()).once()
+    flexmock(module).should_receive('execute_hooks').twice()
+
+    with pytest.raises(ValueError):
+        with module.Before_after_hooks(
+            command_hooks=commands,
+            before_after='action',
+            umask=1234,
+            working_directory='/working',
+            dry_run=False,
+            action_names=['create'],
+            context1='stuff',
+            context2='such',
+        ):
+            raise ValueError()
+
+
 def test_before_after_hooks_with_after_error_raises():
     commands = [
         {'before': 'repository', 'run': ['foo', 'bar']},
@@ -439,14 +511,13 @@ def test_before_after_hooks_with_after_error_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_return(None).and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(False)
@@ -458,7 +529,6 @@ def test_before_after_hooks_with_after_error_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',
@@ -474,14 +544,13 @@ def test_before_after_hooks_with_after_soft_failure_raises():
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         before='action',
-        hook_name='myhook',
         action_names=['create'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('filter_hooks').with_args(
         commands,
         after='action',
-        hook_name='myhook',
         action_names=['create'],
+        state_names=['finish'],
     ).and_return(flexmock()).once()
     flexmock(module).should_receive('execute_hooks').and_return(None).and_raise(OSError)
     flexmock(module).should_receive('considered_soft_failure').and_return(True)
@@ -493,7 +562,6 @@ def test_before_after_hooks_with_after_soft_failure_raises():
             umask=1234,
             working_directory='/working',
             dry_run=False,
-            hook_name='myhook',
             action_names=['create'],
             context1='stuff',
             context2='such',