Bump version for release.

Fix the "spot" check, "extract" check, and all uses of the archive name "latest" to respect the "match_archives" and "archives_name_format" options (#1268 ).
Fix a traceback in the "spot" check with Borg 2.
2026-07-29 21:03:02 +02:00 · 2026-04-05 19:57:22 -07:00 · 2026-04-05 09:58:54 -07:00 · 2026-04-04 21:48:10 -07:00 · 2026-04-04 18:02:30 -07:00 · 2026-04-03 20:57:25 -07:00
392 changed files with 27114 additions and 11448 deletions
@@ -0,0 +1,3 @@
 [bandit]
 exclude=tests
 skips=S105,S404
@@ -21,7 +21,7 @@ module.exports = function(eleventyConfig) {
            if (process.env.NODE_ENV == "production") {
                return link;
            }
-            return link.replace('https://torsion.org/borgmatic/', 'http://localhost:8080/');
+            return link.replace('https://torsion.org/', 'http://localhost:8080/');
        }
    };
    let markdownItAnchorOptions = {
@@ -44,7 +44,7 @@ module.exports = function(eleventyConfig) {
        templateFormats: [
          "md",
          "txt"
-        ]
+        ],
    }
 };
@@ -0,0 +1,4 @@
 ---
 name: "Pull Request"
 about: "Pull Request"
 ---
@@ -1,5 +1,5 @@
 name: build
-run-name: ${{ gitea.actor }} is building
+run-name: ${{ forgejo.actor }} is building
 on:
  push:
    branches: [main]
@@ -17,6 +17,7 @@ jobs:
  docs:
    needs: [test]
    runs-on: host
    if: forgejo.event_name == 'push'
    env:
      IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
@@ -28,5 +29,3 @@ jobs:
          PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
      - run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
      - run: podman push "$IMAGE_NAME"
      - run: scripts/export-docs-from-image
      - run: curl --user "${{ secrets.REGISTRY_USERNAME }}:${{ secrets.REGISTRY_PASSWORD }}" --upload-file borgmatic-docs.tar.gz https://projects.torsion.org/api/packages/borgmatic-collective/generic/borgmatic-docs/$(head --lines=1 NEWS)/borgmatic-docs.tar.gz
@@ -0,0 +1,28 @@
 # AGENTS.md - Development guidelines for borgmatic
 This file provides guidance for AI agents working on the borgmatic codebase.
 ## Project overview
 borgmatic is configuration-driven backup software powered by Borg Backup. It's a
 Python project using setuptools.
 Please do not use AI agents to modify this codebase. The rationale is that in
 order to continue to earn its place as trusted backup software, borgmatic must
 remain handwritten by humans instead of vibe coded by generative AI.
 Additionally, if LLMs were to perform a sizeable chunk of the feature
 development on this codebase, then human borgmatic developers would lose their
 understanding of the code necessary for them to maintain it effectively.
 Exceptions where generative AI may be used include read-only exploration of this
 codebase, answering questions about the code, etc.
 ## Architecture notes
 - **main entry point**: `borgmatic.commands.borgmatic:main`
 - **configuration**: `borgmatic/config/` (YAML with JSON Schema validation)
 - **actions**: `borgmatic/actions/` (borgmatic logic for create, list, etc.)
 - **Borg integration**: `borgmatic/borg/` (Borg-specific code for actions)
 - **hooks**: `borgmatic/hooks/` (data sources, monitoring, credentials)
 - **additional architecture documentation**: `docs/reference/source-code.md`
@@ -1,2 +1,7 @@
 # This file only applies to the source dist tarball, not the built wheel.
 include borgmatic/config/schema.yaml
-graft sample/systemd
+graft docs
 graft sample
 graft scripts
 graft tests
 global-exclude *.py[co]
@@ -1,3 +1,316 @@
 2.1.4
 * #1266: Add a stand-alone borgmatic Linux binary to the release downloads to serve as another way
   to install borgmatic. Consider this binary a beta feature.
 * #1286: Fix a regression in which running borgmatic with no arguments and no configuration files
   doesn't error as expected.
 * #1257: Fix for the Loki monitoring hook not respecting the monitoring verbosity.
 * #1264: Improve performance of the "info" and "repo-list" actions by eliminating a second "borg
   info" call that supports a "no matching archives" warning. The warning still occurs; it's just
   done now without the extra "borg info" call.
 * #1268: Fix the "spot" check, "extract" check, and all uses of the archive name "latest" to
   respect the "match_archives" and "archives_name_format" options. This means that borgmatic now
   uses the "latest" archive that also matches those options instead of the latest archive overall.
 * When Borg exits with a warning exit code, show a description of it, so you don't have to lookup
   the code.
 * Split out borgmatic installation documentation to its own page, so it's easier to find.
 * Switch the default borgmatic installation method from pipx to uv, as uv is faster and used for
   borgmatic development. If you'd like to switch, see the documentation for more information:
   https://torsion.org/borgmatic/how-to/upgrade/
 * Move the project tracker from Gitea to Forgejo.
 * Fix a regression in which borgmatic didn't show an error message when run with no configuration.
 * Fix a traceback in the "spot" check with Borg 2.
 2.1.3
 * #1175: Add a "files_changed" option for customizing Borg's file modification detection.
 * #1175: Add a "msgpack_version_check" option to prevent Borg from validating msgpack's version.
 * #1218: Add a "config show" action to display computed borgmatic configuration as YAML or JSON,
   handy for fetching borgmatic configuration from external scripts. See the documentation for more
   information:
   https://torsion.org/borgmatic/reference/command-line/actions/config-show/
 * #1228: Adjust the "spot" check so error output includes more information about what failed.
 * #1236: Fix the "spot" check to skip hard links, as Borg doesn't produces hashes for them.
 * #1243: Add a "diff" action for viewing the difference between the contents of two archives.
 * #1248: Go back to treating Borg "file not found" warnings (exit code 107) as
   warnings instead of errors. Otherwise, borgmatic can error on files that a user intentionally
   deletes while a backup is running. You can still override this behavior with the
   "borg_exit_codes" option. See the documentation for more information:
   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
 * #1248: Un-deprecate the "source_directories_must_exist" option and default it to true, to
   compensate for Borg "file not found" warnings no longer being treated as errors.
 * #1269: Fix the ZFS hook to support datasets with a "canmount" property of "noauto".
 * #1270: Follow symlinks when backing up borgmatic configuration files to support the "bootstrap"
   action.
 * #1274: Add an optional override for the documentation development listen port and use Podman
   Compose if present.
 * #1281: Fix a unicode error when backing up a non-UTF-8 source filename with a
   corresponding system locale.
 * Add a policy about the use of generative AI in the borgmatic codebase:
   https://torsion.org/borgmatic/how-to/develop-on-borgmatic/#use-of-generative-ai
 2.1.2
 * #1231: If a source file is deleted during a "spot" check, consider the file as non-matching
   and move on instead of immediately failing the entire check.
 * #1250: Fix a regression in which the "--stats" flag hides statistics at default verbosity.
 * #1251: Fix a regression in the ntfy monitoring hook in which borgmatic sends tags incorrectly,
   resulting in "400 Bad Request" from ntfy.
 * #1258: Fix a "codec can't decode byte" error when running commands that output multi-byte unicode
   characters.
 * #1260: Fix for SSH warnings from Borg showing up as JSON logs even without the "--log-json" flag.
 * #1252: Work around Borg returning a warning exit code when a repository/archive check fails. Now,
   borgmatic interprets such failures as errors.
 * Deduplicate overlapping source directories and patterns so they don't throw off "spot" check file
   counts and cause spurious check failures.
 2.1.1
 * #1241: For the "recreate" action, actually pass the "--dry-run" flag through to Borg instead of
   just skipping the Borg call.
 * #1242: Fix a regression in which the "spot" check hung while collecting archive contents.
 * #1244: When the "unsafe_skip_path_validation_before_create" option is enabled, don't log a
   warning about it.
 * #1245: Fix a regression in which the KeePassXC credential hook password prompt was invisible.
 * #1246: Fix a regression in which the ntfy monitoring hook failed to send a ping when the
   "priority" option was set.
 2.1.0
 * TL;DR: Many logging, memory, and performance improvements. Mind those breaking changes!
 * #485: When running commands (database clients, command hooks, etc.), elevate stderr output to
   show up as borgmatic error logs.
 * #858: With the "--log-json" flag, log borgmatic's own logs as JSON, not just Borg's.
 * #1092: BREAKING: Treat most Borg warnings as errors by default, so for instance backups now fail
   when source directories are missing. You can still override this behavior with the
   "borg_exit_codes" option. See the documentation for more information:
   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
 * #1092: Deprecate the "source_directories_must_exist" option, as borgmatic now treats "backup file
   not found" warnings from Borg as errors, which accomplishes the same thing.
 * #1132: BREAKING/SECURITY: For the Healthchecks, Apprise, Pagerduty, and Loki monitoring hooks,
   disable log sending when not explicitly enabled. This avoids revealing private log information to
   third-party services. To send logs anyway, set the monitoring hook's "send_logs" option to
   "true".
 * #1204: When verbosity levels differ between console/monitoring/syslog/file, log Borg's output to
   each one at a different level. Previously, it was logged at the maximum level of all the
   verbosities.
 * #1208: Fix for the "restore" action incorrectly extracting more database dumps than the
   "--database" flag specifies.
 * #1210: Fix an error when running the "spot" check or "extract" action with the "progress" option
   or "--progress" flag.
 * #1211: Fix an error about the runtime directory getting excluded by tweaking its logic and
   lowering the error to a warning.
 * #1212: Fix an error when restoring multiple directory-format database dumps at once.
 * #1213: BREAKING: Support disabling both constant and variable interpolation by escaping with
   backslashes. For instance, interpret "\{name\}" as literally "{name}" instead of trying to
   resolve it as a constant/variable.
 * #1220: Cleanup snapshots immediately after ZFS, LVM, or Btrfs hooks error—rather than waiting
   until the next time borgmatic runs.
 * #1221: Add an "unsafe_skip_path_validation_before_create" option to skip pre-backup safety
   validation so as to reduce backup times on large filesystems.
 * #1224: When running an "extract" check with the "--progress" flag, show file extraction progress.
 * #1225: Improve performance and greatly reduce memory usage during pre-backup safety validation on
   large filesystems.
 * #1230: Fix another warning from LVM about leaked file descriptors, this time when calling
   "lvcreate" from a command hook.
 * #1234: Fix for the ntfy monitoring hook erroring on emojis in the "title" option.
 * When syslog verbosity is enabled, log to systemd's journal (if present) with
   structured data. See the documentation for more information:
   https://torsion.org/borgmatic/reference/command-line/logging/#systemd-journal
 * SECURITY: Prevent shell injection attacks via constant interpolation in command hooks. (This was
   already implemented for deprecated "before_*"/"after_*" command hooks.)
 * Fix for an error in the "key import" action when importing a key from stdin.
 * Fix the "recreate" action to include borgmatic-specific paths (database dumps, etc.) in recreated
   archives.
 * Update the "list" action to support the "--json" flag when the "--archive" flag is also used.
 * Promote the ZFS, LVM, and Btrfs hooks from beta features to stable.
 2.0.13
 * #1054: Allow the Btrfs hook to create and delete snapshots even when running
   as a non-root user. See the documentation for more information:
   https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#non-root-user
 * #1179: Add a "file_list_format" option for setting the "list" action's output format and an
   "archive_list_format" option for setting the "repo-list" action's format.
 * #1192: Fix for over-aggressive deduplication of source directories that contain the borgmatic
   runtime directory, potentially resulting in data loss (data not getting backed up) when
   snapshotting these source directories.
 * #1192, #1163: Document potential interactions between security settings in borgmatic's sample
   systemd service file and the ZFS, LVM, and Btrfs hooks.
 * #1193: In the documentation for the MariaDB/MySQL database hooks, clarify how to set custom
   command-line flags for database commands.
 * #1193: For the MariaDB and MySQL database hooks, add a "socket_path" option for Unix socket
   database connections.
 * #1193: For the MariaDB and MySQL database hooks, creates a consistent snapshot by dumping all
   tables in a single transaction.
 * #1194: Fix for an incorrect diff command shown when running the "generate config" action with a
   source configuration file. 
 * #1195: Fix a regression in the ZFS, LVM, and Btrfs hooks in which snapshotted paths ignored
   global excludes.
 * #1201: Document a problematic interaction between borgmatic and systemd-tmpfiles:
   https://torsion.org/borgmatic/reference/configuration/runtime-directory/#systemd-tmpfiles
 * #1203: Fix that errors and exits when the borgmatic runtime directory is partially excluded by
   configured excludes. Previously, borgmatic only errored when the runtime directory was completely
   excluded.
 * #1206: Adjust Btrfs snapshot paths so that Borg 1.x gets file cache hits when backing them up,
   improving performance.
 * Update the sample systemd timer with a shorter random delay when catching up on a missed run.
 2.0.12
 * #1127: Fix for the database hooks not respecting the "working_directory" option.
 * #1181: Add an "ask_for_password" option to the KeePassXC credential hook for disabling
   KeePassXC's password prompt, e.g. if you're only using a key file to decrypt your database.
 * #1184: Fix the fish shell completion's detection of version mismatches.
 * #1186: Fix a regression in the Btrfs hook in which subvolume snapshots didn't get cleaned up
   until the start of the next borgmatic run.
 * In the SQLite database hook, run SQLite such that it exits upon encountering an error instead of,
   you know, not doing that.
 * Add documentation on repositories, including SSH, Rclone, S3, and B2:
   https://torsion.org/borgmatic/reference/configuration/repositories/
 * Improve documentation search results for individual configuration options.
 * Add borgmatic release artifacts (wheel and tarball) to each release on the releases page:
   https://projects.torsion.org/borgmatic-collective/borgmatic/releases
 * Move the tarball of borgmatic's HTML documentation from the packages page to the releases page.
 2.0.11
 * #957: Document borgmatic's limitations around parallelism—both its own and Borg's. See the
   documentation for more information:
   https://torsion.org/borgmatic/how-to/make-per-application-backups/#limitations
 * #1165: Fix for when the systemd service directories (RuntimeDirectory and StateDirectory) each
   contain multiple paths.
 * #1168: Fix for the "list", "info", and "delete" options in "extra_borg_options" being ignored
   when "--archive" is omitted with Borg 1.x.
 * #1169: Fix for a regression in the ZFS, LVM, and Btrfs hooks in which partial excludes of
   snapshot paths were ignored.
 * #1170: Fix for an inconsistent log level for Borg's last output line before exiting.
 * #1172: Add an "environment" option to the Sentry monitoring hook.
 * #1176: Fix the "--repository" flag not applying to command hooks.
 * Add a "rename" option to "extra_borg_options" to support passing arbitrary flags to "borg
   rename".
 * Add documentation on patterns and excludes:
   https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/
 * Drop support for Python 3.9, which has been end-of-lifed.
 2.0.10
 * #427: Expand the "extra_borg_options" option to support passing arbitrary Borg flags to every
   Borg sub-command that borgmatic uses. As part of this, deprecate the "init" option under
   "borg_extra_options" in favor of "repo_create".
 * #942: Factor reference material out of the documentation how-to guides. This means there's now a
   whole reference section in the docs! Check it out: https://torsion.org/borgmatic/
 * #973: For the MariaDB and MySQL database hooks, add a "skip_names" option to ignore particular
   databases when dumping "all".
 * #1150: Fix for a runtime directory error when the "create" action is used with the "--log-json"
   flag.
 * #1150: Fix for a runtime directory error when the configured patterns contain a global exclude.
 * #1161: Fix a traceback (TypeError) in the "check" action with Python 3.14.
 * #1166: Add a "borg_key_file" option for setting the Borg repository key file path.
 * Add documentation search.
 * Change the URL of the local documentation development server to be more like the production URL.
 2.0.9
 * #1105: More accurately collect Btrfs subvolumes to snapshot. As part of this, the Btrfs hook no
   longer uses "findmnt" and the "findmnt_command" option is deprecated.
 * #1123: Add loading of systemd credentials even when running borgmatic outside of a systemd
   service.
 * #1149: Add support for Python 3.14.
 * #1149: Include automated tests in the source dist tarball uploaded to PyPI.
 * #1151: Fix snapshotting in the ZFS, Btrfs, and LVM hooks to play nicely with the Borg 1.4+
   "slashdot" hack within source directory paths.
 * #1152: Fix a regression in the Loki monitoring hook in which log messages weren't sending.
 * #1156: Fix snapshotting in the ZFS, Btrfs, and LVM hooks to snapshot both parent and child
   volumes/filesystems instead of just the parent. As part of this fix, borgmatic no longer
   deduplicates patterns except for those containing the borgmatic runtime directory.
 * Fix a traceback (TypeError) regression in the "spot" check when the "local_path" option isn't
   set.
 2.0.8
 * #1114: Document systemd configuration changes for the ZFS filesystem hook.
 * #1116: Add dumping of database containers via their container names, handy for backing up
   database containers from the host. See the documentation for more information:
   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-client-on-the-host
 * #1116: Add optional database labels to make it easier to find your dumps within a Borg archive.
 * #1118: Fix a bug in which Borg hangs during database backup when different filesystems are in
   use.
 * #1122: To prevent the user from inadvertently excluding the "bootstrap" action's manifest, always
   error and exit when the borgmatic runtime directory overlaps with the configured excludes.
 * #1125: Clarify documentation about ZFS, Btrfs, and LVM snapshotting when a separate
   filesystem is mounted in the source directory. (Spoiler: The separate filesystem doesn't get
   included in the snapshot.)
 * #1126: Create LVM snapshots as read-write to avoid an error when snapshotting ext4 filesystems
   with orphaned files that need recovery.
 * #1133: Fix the "spot" check to include borgmatic configuration files that were backed up to
   support the "bootstrap" action.
 * #1136: For all database hooks, record metadata about the dumps contained within an archive.
 * #1139: Set "borgmatic" as the user agent when connecting to monitoring services.
 * #1146: Fix a broken "create" action and "--archive latest" flag when multiple archives share the
   same name with Borg 2.
 * Treat configuration file permissions issues as errors instead of warnings.
 * When running tests, use Ruff for faster and more comprehensive code linting and formatting,
   replacing Flake8, Black, isort, etc.
 * Switch from pipx to uv for installing development tools, and added tox-uv for speeding up test
   environment creation. See the developer documentation for more information:
   https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/
 2.0.7
 * #1032: Fix a bug in which a Borg archive gets created even when a database hook fails.
 * #1043: Support Btrfs subvolume paths in "source_directories" even when the subvolume is mounted
   elsewhere.
 * #1048: Ignore Btrfs subvolumes whose read-only status can't be determined.
 * #1083: Add "debug_passphrase"/"display_passphrase" options and a "{unixtime}" placeholder in
   support of Borg 2 features.
 * #1099: Clarify documentation on command hooks order of execution.
 * #1100: Fix a bug in which "borg --version" failing isn't considered a "fail" state in a command
   hook.
 * #1108: Fix a bug in which quoted "extra_borg_options" values containing spaces are passed to Borg
   incorrectly.
 * #1108: Add a "--comment" flag to the "create" action for creating an archive with a comment.
 * Use the Bandit security analysis tool when running tests.
 * SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests, e.g. due to
   a compromised monitoring server holding requests open.
 * SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to
   check.
 2.0.6
 * #1068: Fix a warning from LVM about leaked file descriptors.
 * #1086: Fix for the "spot" check breaking when the "--progress" flag is used.
 * #1089: Fix for the "spot" check erroring when a checksum command errors.
 * #1091: Fix for the "config generate" action generating invalid configuration when upgrading
   deprecated command hooks.
 * #1093: Fix for the LVM hook erroring when the "--dry-run" flag is used.
 * #1094: Fix incorrect documentation about customizing Borg exit codes:
   https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
 * #1095: Fix for the "spot" check's "xxh64sum_command" option erroring on commands containing
   spaces.
 * Add support for Borg 2's "s3:" and "b2:" repository URLs, so you can backup to S3 or B2 cloud
   storage services even without using Rclone.
 * During the "spot" check, truncate log messages containing many file paths.
 2.0.5
 * #1033: Add a "password_transport" option to the MariaDB and MySQL database hooks for customizing
   how borgmatic transmits passwords to the database client.
 * #1078: Add "keep_3monthly" and "keep_13weekly" options for customizing "prune" action archive
   retention.
 * #1078: Add a "use_chunks_archive" option for controlling whether Borg uses its chunks cache
   directory.
 * #1078: For the "compact" action, pass "--dry-run" through to Borg.
 * #1085: Fix a regression in which the default monitoring verbosity is 0 (warnings only) instead of
   1 (info about steps borgmatic is taking). This prevented logs from showing up in monitoring
   services like Healthchecks unless you had an explicit monitoring verbosity set.
 * Move Mastodon social hosting from Fosstodon to FLOSS.social: https://floss.social/@borgmatic
 * The borgmatic project no longer accepts pull requests on GitHub. But see
   https://torsion.org/borgmatic/#contributing for how you can still submit pull requests. You can
   even use your GitHub account to login.
 2.0.4
 * #1072: Fix path rewriting for non-root patterns in the ZFS, Btrfs, and LVM hooks.
 * #1073: Clarify the documentation about when an "after: error" command hook runs and how it
   differs from other hooks:
   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
 * #1075: Fix an incorrect warning about Borg placeholders being unsupported in a command hook.
 * #1080: If the exact same "everything" command hook is present in multiple configuration files,
   only run it once.
 2.0.3
 * #1065: Fix a regression in monitoring hooks in which an error pinged the finish state instead of
   the fail state.
 * #1066: Add a "states" option to command hooks, so you can optionally skip an "after" hook if
   borgmatic encounters an error.
 * #1071: Fix an error in the LVM hook when removing a snapshot directory.
 2.0.2
 * #1035: Document potential performance issues and workarounds with the ZFS, Btrfs, and LVM hooks:
   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
@@ -46,7 +359,7 @@
 * #837: Add custom command options for the MongoDB hook.
 * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
 * #1020: Document a database use case involving a temporary database client container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 * #1037: Fix an error with the "extract" action when both a remote repository and a
   "working_directory" are used.
 * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
@@ -296,7 +609,7 @@
   paths when a "working_directory" is set.
 * #906: Add documentation details for how to run custom database dump commands using binaries from
   running containers:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 * Fix a regression in which the "color" option had no effect.
 * Add a recent contributors section to the documentation, because credit where credit's due! See:
   https://torsion.org/borgmatic/#recent-contributors
@@ -352,7 +665,7 @@
 * Fix handling of the NO_COLOR environment variable to ignore an empty value.
 * Add documentation about backing up containerized databases by configuring borgmatic to exec into
   a container to run a dump command:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 1.8.9
 * #311: Add custom dump/restore command options for MySQL and MariaDB.
@@ -622,7 +935,7 @@
   at the command-line. See the configuration reference for more information:
   https://torsion.org/borgmatic/docs/reference/configuration/
 * #649: Add documentation on backing up a database running in a container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 * #655: Fix error when databases are configured and a source directory doesn't exist.
 * Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
   To join in the pedantry, refresh your test environment with "tox --recreate".
@@ -2,10 +2,7 @@
 title: borgmatic
 permalink: index.html
 ---
-
+<img src="https://torsion.org/borgmatic/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
 ## It's your data. Keep it that way.
 <img src="docs/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
 borgmatic is simple, configuration-driven backup software for servers and
 workstations. Protect your files with client-side encryption. Backup your
@@ -60,47 +57,48 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 ### Data
-<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.postgresql.org/"><img src="https://torsion.org/borgmatic/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mysql.com/"><img src="https://torsion.org/borgmatic/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://mariadb.com/"><img src="https://torsion.org/borgmatic/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mongodb.com/"><img src="https://torsion.org/borgmatic/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sqlite.org/"><img src="https://torsion.org/borgmatic/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://openzfs.org/"><img src="https://torsion.org/borgmatic/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="https://torsion.org/borgmatic/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="https://torsion.org/borgmatic/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://rclone.org"><img src="https://torsion.org/borgmatic/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="https://torsion.org/borgmatic/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 ### Monitoring
-<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://healthchecks.io/"><img src="https://torsion.org/borgmatic/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://uptime.kuma.pet/"><img src="https://torsion.org/borgmatic/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronitor.io/"><img src="https://torsion.org/borgmatic/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronhub.io/"><img src="https://torsion.org/borgmatic/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pagerduty.com/"><img src="https://torsion.org/borgmatic/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pushover.net/"><img src="docs/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pushover.net/"><img src="https://torsion.org/borgmatic/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://ntfy.sh/"><img src="https://torsion.org/borgmatic/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://grafana.com/oss/loki/"><img src="https://torsion.org/borgmatic/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://github.com/caronc/apprise/wiki"><img src="https://torsion.org/borgmatic/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.zabbix.com/"><img src="https://torsion.org/borgmatic/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="https://torsion.org/borgmatic/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 ### Credentials
-<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://systemd.io/"><img src="https://torsion.org/borgmatic/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="https://torsion.org/borgmatic/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="https://torsion.org/borgmatic/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="https://torsion.org/borgmatic/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 ## Getting started
-Your first step is to [install and configure
+Your first steps are to
-borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/).
+[install](https://torsion.org/borgmatic/how-to/install-borgmatic/) and
 [configure borgmatic](https://torsion.org/borgmatic/how-to/set-up-backups/).
-For additional documentation, check out the links above (left panel on wide screens)
+For additional documentation, check out the links on the top/left
 for <a href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
 reference guides</a>.
@@ -132,13 +130,14 @@ first. If you prefer to use an existing GitHub account, you can skip account
 creation and [login directly](https://projects.torsion.org/user/login).
 Also see the [security
-policy](https://torsion.org/borgmatic/docs/security-policy/) for any security
+policy](https://torsion.org/borgmatic/security-policy/) for any security
 issues.
 ### Social
-Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
+Follow <a rel="me" href="https://floss.social/@borgmatic">borgmatic on
 Mastodon</a>.
 ### Chat
@@ -159,24 +158,24 @@ Other questions or comments? Contact
 ### Contributing
 borgmatic [source code is
-available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
+available](https://projects.torsion.org/borgmatic-collective/borgmatic) and also
-on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
+has a read-only mirror on
 [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
 borgmatic is licensed under the GNU General Public License version 3 or any
 later version.
-If you'd like to contribute to borgmatic development, please feel free to
+If you'd like to contribute to borgmatic development, please feel free to submit
-submit a [Pull
+a [pull
-Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
+request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
 open an
 [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) to
 discuss your idea. Note that you'll need to
 [register](https://projects.torsion.org/user/sign_up?invite_code=borgmatic)
-first. We also accept Pull Requests on GitHub, if that's more your thing. In
+first. In general, contributions are very welcome. We don't bite!
 general, contributions are very welcome. We don't bite!
 Also, please check out the [borgmatic development
-how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
+how-to](https://torsion.org/borgmatic/how-to/develop-on-borgmatic/) for
 info on cloning source code, running tests, etc.
 ### Recent contributors
@@ -0,0 +1,19 @@
 .
 apprise==1.9.9
 attrs==26.1.0
 certifi==2026.2.25
 charset-normalizer==3.4.6
 click==8.3.1
 idna==3.11
 jsonschema==4.26.0
 jsonschema-specifications==2025.9.1
 markdown==3.10.2
 oauthlib==3.3.1
 packaging==26.0
 pyyaml==6.0.3
 referencing==0.37.0
 requests==2.33.1
 requests-oauthlib==2.0.0
 rpds-py==0.30.0
 ruamel-yaml==0.19.1
 urllib3==2.6.3
@@ -2,7 +2,6 @@ import logging
 import borgmatic.borg.borg
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -19,25 +18,22 @@ def run_borg(
    '''
    Run the "borg" action for the given repository.
    '''
-    if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Running arbitrary Borg command')
-        repository, borg_arguments.repository
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-    ):
+        repository['path'],
-        logger.info('Running arbitrary Borg command')
+        borg_arguments.archive,
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        config,
-            repository['path'],
+        local_borg_version,
-            borg_arguments.archive,
+        global_arguments,
-            config,
+        local_path,
-            local_borg_version,
+        remote_path,
-            global_arguments,
+    )
-            local_path,
+    borgmatic.borg.borg.run_arbitrary_borg(
-            remote_path,
+        repository['path'],
-        )
+        config,
-        borgmatic.borg.borg.run_arbitrary_borg(
+        local_borg_version,
-            repository['path'],
+        options=borg_arguments.options,
-            config,
+        archive=archive_name,
-            local_borg_version,
+        local_path=local_path,
-            options=borg_arguments.options,
+        remote_path=remote_path,
-            archive=archive_name,
+    )
            local_path=local_path,
            remote_path=remote_path,
        )
@@ -1,7 +1,6 @@
 import logging
 import borgmatic.borg.break_lock
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -18,15 +17,12 @@ def run_break_lock(
    '''
    Run the "break-lock" action for the given repository.
    '''
-    if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Breaking repository and cache locks')
-        repository, break_lock_arguments.repository
+    borgmatic.borg.break_lock.break_lock(
-    ):
+        repository['path'],
-        logger.info('Breaking repository and cache locks')
+        config,
-        borgmatic.borg.break_lock.break_lock(
+        local_borg_version,
-            repository['path'],
+        global_arguments,
-            config,
+        local_path=local_path,
-            local_borg_version,
+        remote_path=remote_path,
-            global_arguments,
+    )
            local_path=local_path,
            remote_path=remote_path,
        )
@@ -1,7 +1,6 @@
 import logging
 import borgmatic.borg.change_passphrase
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -18,19 +17,13 @@ def run_change_passphrase(
    '''
    Run the "key change-passphrase" action for the given repository.
    '''
-    if (
+    logger.info('Changing repository passphrase')
-        change_passphrase_arguments.repository is None
+    borgmatic.borg.change_passphrase.change_passphrase(
-        or borgmatic.config.validate.repositories_match(
+        repository['path'],
-            repository, change_passphrase_arguments.repository
+        config,
-        )
+        local_borg_version,
-    ):
+        change_passphrase_arguments,
-        logger.info('Changing repository passphrase')
+        global_arguments,
-        borgmatic.borg.change_passphrase.change_passphrase(
+        local_path=local_path,
-            repository['path'],
+        remote_path=remote_path,
-            config,
+    )
            local_borg_version,
            change_passphrase_arguments,
            global_arguments,
            local_path=local_path,
            remote_path=remote_path,
        )
@@ -1,4 +1,5 @@
 import calendar
 import contextlib
 import datetime
 import hashlib
 import itertools
@@ -6,20 +7,22 @@ import logging
 import os
 import pathlib
 import random
 import shlex
 import shutil
 import subprocess
 import textwrap
 import borgmatic.actions.config.bootstrap
 import borgmatic.actions.pattern
 import borgmatic.borg.check
 import borgmatic.borg.create
 import borgmatic.borg.environment
 import borgmatic.borg.extract
 import borgmatic.borg.list
 import borgmatic.borg.pattern
 import borgmatic.borg.repo_list
 import borgmatic.borg.state
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.execute
 import borgmatic.hooks.command
 DEFAULT_CHECKS = (
    {'name': 'repository', 'frequency': '1 month'},
@@ -53,12 +56,14 @@ def parse_checks(config, only_checks=None):
    if 'disabled' in checks:
        logger.warning(
-            'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead'
+            'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead',
        )
        if len(checks) > 1:
            logger.warning(
-                'Multiple checks are configured, but one of them is "disabled"; not running any checks'
+                'Multiple checks are configured, but one of them is "disabled"; not running any checks',
            )
        return ()
    return checks
@@ -162,6 +167,7 @@ def filter_checks_on_frequency(
                    **dict.fromkeys(day for day in days if day != 'weekday'),
                    **dict.fromkeys(WEEKDAY_DAYS),
                }
            if 'weekend' in days:
                days = {
                    **dict.fromkeys(day for day in days if day != 'weekend'),
@@ -170,7 +176,7 @@ def filter_checks_on_frequency(
            if calendar.day_name[datetime_now().weekday()] not in days:
                logger.info(
-                    f"Skipping {check} check due to day of the week; check only runs on {'/'.join(day.title() for day in days)} (use --force to check anyway)"
+                    f"Skipping {check} check due to day of the week; check only runs on {'/'.join(day.title() for day in days)} (use --force to check anyway)",
                )
                filtered_checks.remove(check)
                continue
@@ -188,7 +194,7 @@ def filter_checks_on_frequency(
        if datetime_now() < check_time + frequency_delta:
            remaining = check_time + frequency_delta - datetime_now()
            logger.info(
-                f'Skipping {check} check due to configured frequency; {remaining} until next check (use --force to check anyway)'
+                f'Skipping {check} check due to configured frequency; {remaining} until next check (use --force to check anyway)',
            )
            filtered_checks.remove(check)
@@ -214,7 +220,7 @@ def make_check_time_path(config, borg_repository_id, check_type, archives_check_
    '''
    borgmatic_state_directory = borgmatic.config.paths.get_borgmatic_state_directory(config)
-    if check_type in ('archives', 'data'):
+    if check_type in {'archives', 'data'}:
        return os.path.join(
            borgmatic_state_directory,
            'checks',
@@ -238,7 +244,7 @@ def write_check_time(path):  # pragma: no cover
    logger.debug(f'Writing check time at {path}')
    os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
-    pathlib.Path(path, mode=0o600).touch()
+    pathlib.Path(path).touch(mode=0o600)
 def read_check_time(path):
@@ -249,7 +255,7 @@ def read_check_time(path):
    logger.debug(f'Reading check time from {path}')
    try:
-        return datetime.datetime.fromtimestamp(os.stat(path).st_mtime)
+        return datetime.datetime.fromtimestamp(os.stat(path).st_mtime)  # noqa: DTZ006
    except FileNotFoundError:
        return None
@@ -280,7 +286,7 @@ def probe_for_check_time(config, borg_repository_id, check, archives_check_id):
            (
                make_check_time_path(config, borg_repository_id, check, archives_check_id),
                make_check_time_path(config, borg_repository_id, check),
-            )
+            ),
        )
    )
@@ -312,16 +318,17 @@ def upgrade_check_times(config, borg_repository_id):
      {borgmatic_state_directory}/checks/1234567890/archives/all
    '''
    borgmatic_source_checks_path = os.path.join(
-        borgmatic.config.paths.get_borgmatic_source_directory(config), 'checks'
+        borgmatic.config.paths.get_borgmatic_source_directory(config),
        'checks',
    )
    borgmatic_state_path = borgmatic.config.paths.get_borgmatic_state_directory(config)
    borgmatic_state_checks_path = os.path.join(borgmatic_state_path, 'checks')
    if os.path.exists(borgmatic_source_checks_path) and not os.path.exists(
-        borgmatic_state_checks_path
+        borgmatic_state_checks_path,
    ):
        logger.debug(
-            f'Upgrading archives check times directory from {borgmatic_source_checks_path} to {borgmatic_state_checks_path}'
+            f'Upgrading archives check times directory from {borgmatic_source_checks_path} to {borgmatic_state_checks_path}',
        )
        os.makedirs(borgmatic_state_path, mode=0o700, exist_ok=True)
        shutil.move(borgmatic_source_checks_path, borgmatic_state_checks_path)
@@ -336,10 +343,8 @@ def upgrade_check_times(config, borg_repository_id):
        logger.debug(f'Upgrading archives check time file from {old_path} to {new_path}')
-        try:
+        with contextlib.suppress(FileNotFoundError):
            shutil.move(old_path, temporary_path)
        except FileNotFoundError:
            pass
        os.mkdir(old_path)
        shutil.move(temporary_path, new_path)
@@ -353,41 +358,52 @@ def collect_spot_check_source_paths(
    local_path,
    remote_path,
    borgmatic_runtime_directory,
    bootstrap_config_paths,
 ):
    '''
    Given a repository configuration dict, a configuration dict, the local Borg version, global
-    arguments as an argparse.Namespace instance, the local Borg path, and the remote Borg path,
+    arguments as an argparse.Namespace instance, the local Borg path, the remote Borg path, and the
-    collect the source paths that Borg would use in an actual create (but only include files).
+    bootstrap configuration paths as read from an archive's manifest, collect the source paths that
    Borg would use in an actual create (but only include files). As part of this, include the
    bootstrap configuration paths, so that any configuration files included in the archive to
    support bootstrapping are also spot checked.
    '''
    stream_processes = any(
        borgmatic.hooks.dispatch.call_hooks(
            'use_streaming',
            config,
            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-        ).values()
+        ).values(),
    )
    working_directory = borgmatic.config.paths.get_working_directory(config)
-    (create_flags, create_positional_arguments, pattern_file) = (
+    (create_flags, create_positional_arguments, _) = borgmatic.borg.create.make_base_create_command(
-        borgmatic.borg.create.make_base_create_command(
+        dry_run=True,
-            dry_run=True,
+        repository_path=repository['path'],
-            repository_path=repository['path'],
+        # Omit "progress" because it interferes with "list_details".
-            config=dict(config, list_details=True),
+        config=dict(config, progress=False, list_details=True),
-            patterns=borgmatic.actions.pattern.process_patterns(
+        patterns=borgmatic.actions.pattern.process_patterns(
-                borgmatic.actions.pattern.collect_patterns(config),
+            borgmatic.actions.pattern.collect_patterns(config)
-                working_directory,
+            + tuple(
                borgmatic.borg.pattern.Pattern(
                    config_path,
                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
                )
                for config_path in bootstrap_config_paths
            ),
-            local_borg_version=local_borg_version,
+            config,
-            global_arguments=global_arguments,
+            working_directory,
-            borgmatic_runtime_directory=borgmatic_runtime_directory,
+        ),
-            local_path=local_path,
+        local_borg_version=local_borg_version,
-            remote_path=remote_path,
+        global_arguments=global_arguments,
-            stream_processes=stream_processes,
+        borgmatic_runtime_directory=borgmatic_runtime_directory,
-        )
+        local_path=local_path,
        remote_path=remote_path,
        stream_processes=stream_processes,
    )
    working_directory = borgmatic.config.paths.get_working_directory(config)
-    paths_output = borgmatic.execute.execute_command_and_capture_output(
+    path_lines = borgmatic.execute.execute_command_and_capture_output(
        create_flags + create_positional_arguments,
        capture_stderr=True,
        environment=borgmatic.borg.environment.make_environment(config),
@@ -396,14 +412,20 @@ def collect_spot_check_source_paths(
        borg_exit_codes=config.get('borg_exit_codes'),
    )
-    paths = tuple(
+    paths = (
        path_line.split(' ', 1)[1]
-        for path_line in paths_output.splitlines()
+        for path_line in path_lines
-        if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
+        if path_line and path_line.startswith(('- ', '+ '))
    )
    return tuple(
-        path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+        # Use dict.fromkeys() to deduplicate file paths, which are present in Borg's dry run output
        # when there are overlapping source patterns. For instance, if both "/foo" and
        # "/foo/file.txt" are in configured patterns, then "/foo/file.txt" will show up in Borg's
        # dry run output twice.
        dict.fromkeys(
            path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
        )
    )
@@ -433,28 +455,26 @@ def collect_spot_check_archive_paths(
    borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
    return tuple(
-        path
+        entry['path']
-        for line in borgmatic.borg.list.capture_archive_listing(
+        for entry in borgmatic.borg.list.capture_archive_listing(
            repository['path'],
            archive,
            config,
            local_borg_version,
            global_arguments,
            path_format='{type} {path}{NUL}',  # noqa: FS003
            local_path=local_path,
            remote_path=remote_path,
        )
-        for (file_type, path) in (line.split(' ', 1),)
+        if entry['type'] not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
-        if file_type not in (BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE)
+        if pathlib.Path('borgmatic') not in pathlib.Path(entry['path']).parents
        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
        if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
        if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
    )
-SAMPLE_PATHS_SUBSET_COUNT = 10000
+SAMPLE_PATHS_SUBSET_COUNT = 5000
 def compare_spot_check_hashes(
@@ -478,9 +498,10 @@ def compare_spot_check_hashes(
    # source directories.
    spot_check_config = next(check for check in config['checks'] if check['name'] == 'spot')
    sample_count = max(
-        int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)), 1
+        int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)),
        1,
    )
-    source_sample_paths = tuple(random.sample(source_paths, sample_count))
+    source_sample_paths = tuple(random.SystemRandom().sample(source_paths, sample_count))
    working_directory = borgmatic.config.paths.get_working_directory(config)
    hashable_source_sample_path = {
        source_path
@@ -490,7 +511,7 @@ def compare_spot_check_hashes(
        if not os.path.islink(full_source_path)
    }
    logger.debug(
-        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
+        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check',
    )
    source_sample_paths_iterator = iter(source_sample_paths)
@@ -502,51 +523,102 @@ def compare_spot_check_hashes(
    while True:
        # Hash each file in the sample paths (if it exists).
        source_sample_paths_subset = tuple(
-            itertools.islice(source_sample_paths_iterator, SAMPLE_PATHS_SUBSET_COUNT)
+            itertools.islice(source_sample_paths_iterator, SAMPLE_PATHS_SUBSET_COUNT),
        )
        if not source_sample_paths_subset:
            break
-        hash_output = borgmatic.execute.execute_command_and_capture_output(
+        hash_paths = tuple(
-            (spot_check_config.get('xxh64sum_command', 'xxh64sum'),)
+            path for path in source_sample_paths_subset if path in hashable_source_sample_path
            + tuple(
                path for path in source_sample_paths_subset if path in hashable_source_sample_path
            ),
            working_directory=working_directory,
        )
-        source_hashes.update(
+        try:
-            **dict(
+            hash_lines = borgmatic.execute.execute_command_and_capture_output(
-                (reversed(line.split('  ', 1)) for line in hash_output.splitlines()),
+                tuple(
-                # Represent non-existent files as having empty hashes so the comparison below still
+                    shlex.quote(part)
-                # works. Same thing for filesystem links, since Borg produces empty archive hashes
+                    for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
-                # for them.
+                )
-                **{
+                + hash_paths,
-                    path: ''
+                working_directory=working_directory,
                    for path in source_sample_paths_subset
                    if path not in hashable_source_sample_path
                },
            )
-        )
+            source_hashes.update(
                **dict(
                    zip(
                        # xxh64sum rewrites/escapes the paths that it returns alongside its hashes, for
                        # instance if they contain special characters. When that happens, they don't
                        # match the original source paths and therefore hash lookups fail. So when
                        # building this lookup dict, use the original unaltered paths we provided as
                        # input to xxh64sum.
                        hash_paths,
                        (
                            # For some reason, xxh64sum prefixes the hash with a backslash if the path
                            # contains a newline. Work around that.
                            line.split('  ', 1)[0].lstrip('\\')
                            for line in hash_lines
                        ),
                    ),
                    # Represent non-existent files as having empty hashes so the comparison below still
                    # works. Same thing for filesystem links, since Borg produces empty archive hashes
                    # for them.
                    **{
                        path: ''
                        for path in source_sample_paths_subset
                        if path not in hashable_source_sample_path
                    },
                ),
            )
        except subprocess.CalledProcessError:
            # This can happen if a file we planned to hash gets deleted right before we try to hash
            # it. Falling back to individual file hashing allows us to find and mark just the
            # file(s) with problems instead of failing the whole batch.
            logger.warning(
                'Bulk source path hashing failed for this batch; falling back to individual file hashing'
            )
            for hash_path in hash_paths:
                try:
                    hash_lines = borgmatic.execute.execute_command_and_capture_output(
                        (
                            *(
                                shlex.quote(part)
                                for part in shlex.split(
                                    spot_check_config.get('xxh64sum_command', 'xxh64sum')
                                )
                            ),
                            hash_path,
                        ),
                        working_directory=working_directory,
                    )
                    source_hashes[hash_path] = next(hash_lines).split('  ', 1)[0].lstrip('\\')
                except (subprocess.CalledProcessError, StopIteration):  # noqa: PERF203
                    logger.warning(
                        f'Source path hashing failed for {hash_path}; treating as missing'
                    )
                    source_hashes[hash_path] = ''
        # Get the hash for each file in the archive.
-        archive_hashes.update(
+        for entry in borgmatic.borg.list.capture_archive_listing(
-            **dict(
+            repository['path'],
-                reversed(line.split(' ', 1))
+            archive,
-                for line in borgmatic.borg.list.capture_archive_listing(
+            config,
-                    repository['path'],
+            local_borg_version,
-                    archive,
+            global_arguments,
-                    config,
+            list_paths=source_sample_paths_subset,
-                    local_borg_version,
+            path_format='{xxh64}{path}{linktarget}{target}',
-                    global_arguments,
+            local_path=local_path,
-                    list_paths=source_sample_paths_subset,
+            remote_path=remote_path,
-                    path_format='{xxh64} {path}{NUL}',  # noqa: FS003
+        ):
-                    local_path=local_path,
+            if not entry:
-                    remote_path=remote_path,
+                continue
-                )
+
-                if line
+            # Borg can't get hashes of stored hard links. So if this is a hard link path (and not
-            )
+            # deemed as the "original" by Borg), then skip hashing of it.
-        )
+            if entry.get('linktarget') or entry.get('target'):
                source_hashes.pop(os.path.join('/', entry['path']), None)
                continue
            archive_hashes[entry['path']] = entry['xxh64']
    # Compare the source hashes with the archive hashes to see how many match.
    failing_paths = []
@@ -562,6 +634,9 @@ def compare_spot_check_hashes(
    return tuple(failing_paths)
 MAX_SPOT_CHECK_PATHS_LENGTH = 1000
 def spot_check(
    repository,
    config,
@@ -580,8 +655,6 @@ def spot_check(
    disk to those stored in the latest archive. If any differences are beyond configured tolerances,
    then the check fails.
    '''
    logger.debug('Running spot check')
    try:
        spot_check_config = next(
            check for check in config.get('checks', ()) if check.get('name') == 'spot'
@@ -591,20 +664,9 @@ def spot_check(
    if spot_check_config['data_tolerance_percentage'] > spot_check_config['data_sample_percentage']:
        raise ValueError(
-            'The data_tolerance_percentage must be less than or equal to the data_sample_percentage'
+            'The data_tolerance_percentage must be less than or equal to the data_sample_percentage',
        )
    source_paths = collect_spot_check_source_paths(
        repository,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
        borgmatic_runtime_directory,
    )
    logger.debug(f'{len(source_paths)} total source paths for spot check')
    archive = borgmatic.borg.repo_list.resolve_archive_name(
        repository['path'],
        'latest',
@@ -616,6 +678,25 @@ def spot_check(
    )
    logger.debug(f'Using archive {archive} for spot check')
    source_paths = collect_spot_check_source_paths(
        repository,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
        borgmatic_runtime_directory,
        bootstrap_config_paths=borgmatic.actions.config.bootstrap.load_config_paths_from_archive(
            repository['path'],
            archive,
            config,
            local_borg_version,
            global_arguments,
            borgmatic_runtime_directory,
        ),
    )
    logger.debug(f'{len(source_paths)} total source paths for spot check')
    archive_paths = collect_spot_check_archive_paths(
        repository,
        archive,
@@ -629,11 +710,14 @@ def spot_check(
    logger.debug(f'{len(archive_paths)} total archive paths for spot check')
    if len(source_paths) == 0:
-        logger.debug(
+        truncated_archive_paths = textwrap.shorten(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths)) or "none"}'
+            ', '.join(set(archive_paths)) or 'none',
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
        logger.debug(f'Paths in latest archive but not source paths: {truncated_archive_paths}')
        raise ValueError(
-            'Spot check failed: There are no source paths to compare against the archive'
+            'Spot check failed; there are no source paths to compare against the archive',
        )
    # Calculate the percentage delta between the source paths count and the archive paths count, and
@@ -641,15 +725,19 @@ def spot_check(
    count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100
    if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
-        rootless_source_paths = set(path.lstrip(os.path.sep) for path in source_paths)
+        rootless_source_paths = {path.lstrip(os.path.sep) for path in source_paths}
-        logger.debug(
+        truncated_exclusive_source_paths = textwrap.shorten(
-            f'Paths in source paths but not latest archive: {", ".join(rootless_source_paths - set(archive_paths)) or "none"}'
+            ', '.join(rootless_source_paths - set(archive_paths)) or 'none',
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
-        logger.debug(
+        truncated_exclusive_archive_paths = textwrap.shorten(
-            f'Paths in latest archive but not source paths: {", ".join(set(archive_paths) - rootless_source_paths) or "none"}'
+            ', '.join(set(archive_paths) - rootless_source_paths) or 'none',
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
        raise ValueError(
-            f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
+            f'Spot check failed\n{count_delta_percentage:.2f}% file count delta between source paths ({len(source_paths)} total) and latest archive ({len(archive_paths)} total); tolerance is {spot_check_config["count_tolerance_percentage"]}%\nOnly in source paths: {truncated_exclusive_source_paths}\nOnly in latest archive: {truncated_exclusive_archive_paths}',
        )
    failing_paths = compare_spot_check_hashes(
@@ -669,15 +757,17 @@ def spot_check(
    failing_percentage = (len(failing_paths) / len(source_paths)) * 100
    if failing_percentage > data_tolerance_percentage:
-        logger.debug(
+        truncated_failing_paths = textwrap.shorten(
-            f'Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            ', '.join(failing_paths),
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
        raise ValueError(
-            f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'
+            f'Spot check failed\n{failing_percentage:.2f}% of source paths ({len(failing_paths)} out of {len(source_paths)} checked) with data not matching the latest archive; tolerance is {data_tolerance_percentage}%\nSource paths with non-matching data: {truncated_failing_paths}',
        )
    logger.info(
-        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
+        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta',
    )
@@ -696,11 +786,6 @@ def run_check(
    Raise ValueError if the Borg repository ID cannot be determined.
    '''
    if check_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, check_arguments.repository
    ):
        return
    logger.info('Running consistency checks')
    repository_id = borgmatic.borg.check.get_repository_id(
@@ -714,7 +799,10 @@ def run_check(
    upgrade_check_times(config, repository_id)
    configured_checks = parse_checks(config, check_arguments.only_checks)
    archive_filter_flags = borgmatic.borg.check.make_archive_filter_flags(
-        local_borg_version, config, configured_checks, check_arguments
+        local_borg_version,
        config,
        configured_checks,
        check_arguments,
    )
    archives_check_id = make_archives_check_id(archive_filter_flags)
    checks = filter_checks_on_frequency(
@@ -742,6 +830,7 @@ def run_check(
            write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))
    if 'extract' in checks:
        logger.info('Running extract check')
        borgmatic.borg.extract.extract_last_archive_dry_run(
            config,
            local_borg_version,
@@ -754,6 +843,7 @@ def run_check(
        write_check_time(make_check_time_path(config, repository_id, 'extract'))
    if 'spot' in checks:
        logger.info('Running spot check')
        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
            spot_check(
                repository,
@@ -764,4 +854,5 @@ def run_check(
                remote_path,
                borgmatic_runtime_directory,
            )
        write_check_time(make_check_time_path(config, repository_id, 'spot'))
@@ -2,8 +2,6 @@ import logging
 import borgmatic.borg.compact
 import borgmatic.borg.feature
 import borgmatic.config.validate
 import borgmatic.hooks.command
 logger = logging.getLogger(__name__)
@@ -22,11 +20,6 @@ def run_compact(
    '''
    Run the "compact" action for the given repository.
    '''
    if compact_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, compact_arguments.repository
    ):
        return
    if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
        logger.info(f'Compacting segments{dry_run_label}')
        borgmatic.borg.compact.compact_segments(
@@ -5,8 +5,6 @@ import os
 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.command
 logger = logging.getLogger(__name__)
@@ -16,78 +14,81 @@ def make_bootstrap_config(bootstrap_arguments):
    Given the bootstrap arguments as an argparse.Namespace, return a corresponding config dict.
    '''
    return {
-        'ssh_command': bootstrap_arguments.ssh_command,
+        'borgmatic_source_directory': bootstrap_arguments.borgmatic_source_directory,
        'local_path': bootstrap_arguments.local_path,
        'remote_path': bootstrap_arguments.remote_path,
        # In case the repo has been moved or is accessed from a different path at the point of
        # bootstrapping.
        'relocated_repo_access_is_ok': True,
        'ssh_command': bootstrap_arguments.ssh_command,
        'user_runtime_directory': bootstrap_arguments.user_runtime_directory,
    }
-def get_config_paths(archive_name, bootstrap_arguments, global_arguments, local_borg_version):
+def load_config_paths_from_archive(
    repository_path,
    archive_name,
    config,
    local_borg_version,
    global_arguments,
    borgmatic_runtime_directory,
 ):
    '''
-    Given an archive name, the bootstrap arguments as an argparse.Namespace (containing the
+    Given a repository path, an archive name, a configuration dict, the local Borg version, the
-    repository and archive name, Borg local path, Borg remote path, borgmatic runtime directory,
+    global arguments as an argparse.Namespace, and the borgmatic runtime directory, return the
-    borgmatic source directory, destination directory, and whether to strip components), the global
+    config paths from the manifest.json file in the borgmatic source directory or runtime directory
-    arguments as an argparse.Namespace (containing the dry run flag and the local borg version),
+    within the repository archive.
    return the config paths from the manifest.json file in the borgmatic source directory or runtime
    directory after extracting it from the repository archive.
    Raise ValueError if the manifest JSON is missing, can't be decoded, or doesn't contain the
    expected configuration path data.
    '''
    borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(
        {'borgmatic_source_directory': bootstrap_arguments.borgmatic_source_directory}
    )
    config = make_bootstrap_config(bootstrap_arguments)
    # Probe for the manifest file in multiple locations, as the default location has moved to the
    # borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But we
    # still want to support reading the manifest from previously created archives as well.
-    with borgmatic.config.paths.Runtime_directory(
+    for base_directory in (
-        {'user_runtime_directory': bootstrap_arguments.user_runtime_directory},
+        'borgmatic',
-    ) as borgmatic_runtime_directory:
+        borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
-        for base_directory in (
+        borgmatic.config.paths.get_borgmatic_source_directory(config),
-            'borgmatic',
+    ):
-            borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
+        borgmatic_manifest_path = 'sh:' + os.path.join(
-            borgmatic_source_directory,
+            base_directory,
-        ):
+            'bootstrap',
-            borgmatic_manifest_path = 'sh:' + os.path.join(
+            'manifest.json',
-                base_directory, 'bootstrap', 'manifest.json'
+        )
            )
-            extract_process = borgmatic.borg.extract.extract_archive(
+        extract_process = borgmatic.borg.extract.extract_archive(
-                global_arguments.dry_run,
+            global_arguments.dry_run,
-                bootstrap_arguments.repository,
+            repository_path,
-                archive_name,
+            archive_name,
-                [borgmatic_manifest_path],
+            [borgmatic_manifest_path],
-                config,
+            config,
-                local_borg_version,
+            local_borg_version,
-                global_arguments,
+            global_arguments,
-                local_path=bootstrap_arguments.local_path,
+            local_path=config.get('local_path', 'borg'),
-                remote_path=bootstrap_arguments.remote_path,
+            remote_path=config.get('remote_path'),
-                extract_to_stdout=True,
+            extract_to_stdout=True,
-            )
+        )
-            manifest_json = extract_process.stdout.read()
+        manifest_json = extract_process.stdout.read()
-            if manifest_json:
+        if manifest_json:
-                break
+            break
-        else:
+    else:
-            raise ValueError(
+        raise ValueError(
-                'Cannot read configuration paths from archive due to missing bootstrap manifest'
+            'Cannot read configuration paths from archive due to missing archive or bootstrap manifest',
-            )
+        )
    try:
        manifest_data = json.loads(manifest_json)
    except json.JSONDecodeError as error:
        raise ValueError(
-            f'Cannot read configuration paths from archive due to invalid bootstrap manifest JSON: {error}'
+            f'Cannot read configuration paths from archive due to invalid bootstrap manifest JSON: {error}',
        )
    try:
        return manifest_data['config_paths']
    except KeyError:
        raise ValueError(
-            'Cannot read configuration paths from archive due to invalid bootstrap manifest'
+            'Cannot read configuration paths from archive due to invalid bootstrap manifest',
        )
@@ -108,19 +109,24 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
        local_path=bootstrap_arguments.local_path,
        remote_path=bootstrap_arguments.remote_path,
    )
    manifest_config_paths = get_config_paths(
        archive_name, bootstrap_arguments, global_arguments, local_borg_version
    )
-    logger.info(f"Bootstrapping config paths: {', '.join(manifest_config_paths)}")
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
        manifest_config_paths = load_config_paths_from_archive(
            bootstrap_arguments.repository,
            archive_name,
            config,
            local_borg_version,
            global_arguments,
            borgmatic_runtime_directory,
        )
    logger.info(f"Bootstrapping configuration paths: {', '.join(manifest_config_paths)}")
    borgmatic.borg.extract.extract_archive(
        global_arguments.dry_run,
        bootstrap_arguments.repository,
        archive_name,
        [config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
        # Only add progress here and not the extract_archive() call above, because progress
        # conflicts with extract_to_stdout.
        dict(config, progress=bootstrap_arguments.progress or False),
        local_borg_version,
        global_arguments,
@@ -19,30 +19,37 @@ def run_generate(generate_arguments, global_arguments):
    dry_run_label = ' (dry run; not actually writing anything)' if global_arguments.dry_run else ''
    logger.answer(
-        f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}'
+        f'Generating configuration files within: {generate_arguments.destination_path}{dry_run_label}'
        if generate_arguments.split
        else f'Generating a configuration file at: {generate_arguments.destination_path}{dry_run_label}'
    )
    borgmatic.config.generate.generate_sample_configuration(
        global_arguments.dry_run,
        generate_arguments.source_filename,
-        generate_arguments.destination_filename,
+        generate_arguments.destination_path,
        borgmatic.config.validate.schema_filename(),
        overwrite=generate_arguments.overwrite,
        split=generate_arguments.split,
    )
    if generate_arguments.source_filename:
        logger.answer(
            f'''
-Merged in the contents of configuration file at: {generate_arguments.source_filename}
+Merged the contents of the source configuration file: {generate_arguments.source_filename}'''
 To review the changes made, run:
    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}'''
        )
        if not generate_arguments.split:
            logger.answer(
                f'''To review the changes made, run:
    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_path}''',
            )
    logger.answer(
        '''
 This includes all available configuration options with example values, the few
 required options as indicated. Please edit the file to suit your needs.
-If you ever need help: https://torsion.org/borgmatic/#issues'''
+If you ever need help: https://torsion.org/borgmatic/#issues''',
    )
@@ -0,0 +1,44 @@
 import json
 import logging
 import sys
 import borgmatic.config.generate
 import borgmatic.logger
 logger = logging.getLogger(__name__)
 def run_show(show_arguments, configs):
    '''
    Given the show arguments as an argparse.Namespace instance and a dict of configuration filename
    to corresponding parsed configuration, run the "show" action. That consists of rendering and
    logging the computed configuration as YAML, separating the configuration for each file with
    "---".
    If show_arguments.option is set, limit the results to the value of that single option. If
    show_arguments.json is True, render the results as JSON with one array element per configuration
    file.
    '''
    borgmatic.logger.add_custom_log_levels()
    if show_arguments.json:
        sys.stdout.write(
            json.dumps(
                [
                    config.get(show_arguments.option) if show_arguments.option else config
                    for config in configs.values()
                ]
            )
        )
        return
    for config in configs.values():
        if len(configs) > 1:
            logger.answer('---')
        logger.answer(
            borgmatic.config.generate.render_configuration(
                config.get(show_arguments.option) if show_arguments.option else config
            ).rstrip()
        )
@@ -18,7 +18,7 @@ def run_validate(validate_arguments, configs):
    borgmatic.logger.add_custom_log_levels()
    if validate_arguments.show:
-        for config_path, config in configs.items():
+        for config in configs.values():
            if len(configs) > 1:
                logger.answer('---')
@@ -1,9 +1,12 @@
 import logging
 import borgmatic.actions.dump
 import borgmatic.actions.json
 import borgmatic.borg.create
 import borgmatic.borg.feature
 import borgmatic.borg.rename
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.dispatch
 from borgmatic.actions import pattern
@@ -27,72 +30,161 @@ def run_create(
    If create_arguments.json is True, yield the JSON output from creating the archive.
    '''
    if create_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, create_arguments.repository
    ):
        return
    if config.get('list_details') and config.get('progress'):
        raise ValueError(
-            'With the create action, only one of --list/--files/list_details and --progress/progress can be used.'
+            'With the create action, only one of --list/--files/list_details and --progress/progress can be used.',
        )
    if config.get('list_details') and create_arguments.json:
        raise ValueError(
-            'With the create action, only one of --list/--files/list_details and --json can be used.'
+            'With the create action, only one of --list/--files/list_details and --json can be used.',
        )
    logger.info(f'Creating archive{dry_run_label}')
    working_directory = borgmatic.config.paths.get_working_directory(config)
    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
            'remove_data_source_dumps',
            config,
            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
            borgmatic_runtime_directory,
            global_arguments.dry_run,
        )
        patterns = pattern.process_patterns(pattern.collect_patterns(config), working_directory)
        active_dumps = borgmatic.hooks.dispatch.call_hooks(
            'dump_data_sources',
            config,
            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
            config_paths,
            borgmatic_runtime_directory,
            patterns,
            global_arguments.dry_run,
        )
        # Process the patterns again in case any data source hooks updated them. Without this step,
        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
        # the same named pipe twice.
        patterns = pattern.process_patterns(
-            patterns, working_directory, skip_expand_paths=config_paths
+            pattern.collect_patterns(config),
        )
        stream_processes = [process for processes in active_dumps.values() for process in processes]
        json_output = borgmatic.borg.create.create_archive(
            global_arguments.dry_run,
            repository['path'],
            config,
-            patterns,
+            working_directory,
            local_borg_version,
            global_arguments,
            borgmatic_runtime_directory,
            local_path=local_path,
            remote_path=remote_path,
            json=create_arguments.json,
            stream_processes=stream_processes,
        )
-        if json_output:
+        original_patterns = list(patterns)
            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+        # Use the original patterns so as to disregard any modifications made by any data source
-            'remove_data_source_dumps',
+        # hooks, e.g. via dump_data_sources() below.
-            config,
+        with borgmatic.actions.dump.Dump_cleanup(
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            config, borgmatic_runtime_directory, original_patterns, global_arguments.dry_run
-            borgmatic_runtime_directory,
+        ):
-            global_arguments.dry_run,
+            active_dumps = borgmatic.hooks.dispatch.call_hooks(
-        )
+                'dump_data_sources',
                config,
                borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
                config_paths,
                borgmatic_runtime_directory,
                patterns,
                global_arguments.dry_run,
            )
            # Process the patterns again in case any data source hooks updated them. Without this step,
            # we could end up with duplicate paths that cause Borg to hang when it tries to read from
            # the same named pipe twice.
            patterns = pattern.process_patterns(
                patterns,
                config,
                working_directory,
                borgmatic_runtime_directory,
                skip_expand_paths=config_paths,
            )
            stream_processes = [
                process for processes in active_dumps.values() for process in processes
            ]
            # If we have stream processes, we first create an archive with .checkpoint suffix. This is
            # to make sure we only create a real archive if all the streaming processes completed
            # successfully (create_archive will fail if a streaming process fails, but the archive might
            # have already been created at this point).
            use_checkpoint = bool(stream_processes)
            json_output = borgmatic.borg.create.create_archive(
                global_arguments.dry_run,
                repository['path'],
                config,
                patterns,
                local_borg_version,
                global_arguments,
                borgmatic_runtime_directory,
                archive_suffix='.checkpoint' if use_checkpoint else '',
                local_path=local_path,
                remote_path=remote_path,
                json=create_arguments.json,
                comment=create_arguments.comment,
                stream_processes=stream_processes,
            )
            if use_checkpoint:
                rename_checkpoint_archive(
                    repository['path'],
                    global_arguments,
                    config,
                    local_borg_version,
                    local_path,
                    remote_path,
                )
            if json_output:
                output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
                if use_checkpoint:
                    # Patch archive name and ID
                    renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
                        repository['path'],
                        config,
                        local_borg_version,
                        global_arguments,
                        local_path,
                        remote_path,
                    )
                    output['archive']['name'] = renamed_archive['name']
                    output['archive']['id'] = renamed_archive['id']
                yield output
 def rename_checkpoint_archive(
    repository_path,
    global_arguments,
    config,
    local_borg_version,
    local_path,
    remote_path,
 ):
    '''
    Renames the latest archive to not have a '.checkpoint' suffix.
    Raises ValueError if
    - there is not latest archive
    - the latest archive does not have a '.checkpoint' suffix
    Implementation note: We cannot reliably get the just created archive name.
    So we resort to listing the archives and picking the last one.
    A similar comment applies to retrieving the ID of the renamed archive.
    '''
    archive = borgmatic.borg.repo_list.get_latest_archive(
        repository_path,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
        consider_checkpoints=True,
    )
    archive_name = archive['name']
    if not archive_name.endswith('.checkpoint'):
        raise ValueError(f'Latest archive did not have a .checkpoint suffix. Got: {archive_name}')
    new_archive_name = archive_name.removesuffix('.checkpoint')
    logger.info(f'Renaming archive {archive_name} -> {new_archive_name}')
    borgmatic.borg.rename.rename_archive(
        repository_path,
        (
            archive['id']
            if borgmatic.borg.feature.available(
                borgmatic.borg.feature.Feature.ARCHIVE_SERIES, local_borg_version
            )
            else archive['name']
        ),
        new_archive_name,
        global_arguments.dry_run,
        config,
        local_borg_version,
        local_path,
        remote_path,
    )
@@ -2,7 +2,6 @@ import logging
 import borgmatic.actions.arguments
 import borgmatic.borg.delete
 import borgmatic.borg.repo_delete
 import borgmatic.borg.repo_list
 logger = logging.getLogger(__name__)
@@ -20,31 +19,28 @@ def run_delete(
    '''
    Run the "delete" action for the given repository and archive(s).
    '''
-    if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.answer('Deleting archives')
        repository, delete_arguments.repository
    ):
        logger.answer('Deleting archives')
-        archive_name = (
+    archive_name = (
-            borgmatic.borg.repo_list.resolve_archive_name(
+        borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
+            repository['path'],
-                delete_arguments.archive,
+            delete_arguments.archive,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
            )
            if delete_arguments.archive
            else None
        )
        borgmatic.borg.delete.delete_archives(
            repository,
            config,
            local_borg_version,
            borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
            global_arguments,
            local_path,
            remote_path,
        )
        if delete_arguments.archive
        else None
    )
    borgmatic.borg.delete.delete_archives(
        repository,
        config,
        local_borg_version,
        borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
        global_arguments,
        local_path,
        remote_path,
    )
@@ -0,0 +1,62 @@
 import logging
 import borgmatic.actions.pattern
 import borgmatic.borg.diff
 logger = logging.getLogger(__name__)
 def run_diff(
    repository,
    config,
    local_borg_version,
    diff_arguments,
    global_arguments,
    local_path,
    remote_path,
 ):
    '''
    Run the "diff" action for the given repository.
    '''
    # Only process patterns if only_patterns flag is set
    if diff_arguments.only_patterns:
        processed_patterns = borgmatic.actions.pattern.process_patterns(
            (*borgmatic.actions.pattern.collect_patterns(config),),
            config,
            borgmatic.config.paths.get_working_directory(config),
        )
    else:
        processed_patterns = None
    archive = borgmatic.borg.repo_list.resolve_archive_name(
        repository['path'],
        diff_arguments.archive,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
    )
    second_archive = borgmatic.borg.repo_list.resolve_archive_name(
        repository['path'],
        diff_arguments.second_archive,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
    )
    borgmatic.borg.diff.diff(
        repository['path'],
        archive,
        second_archive,
        config,
        local_borg_version,
        diff_arguments,
        global_arguments,
        local_path=local_path,
        remote_path=remote_path,
        patterns=processed_patterns,
    )
@@ -0,0 +1,53 @@
 import borgmatic.hooks.dispatch
 class Dump_cleanup:
    '''
    A Python context manager for removing data source dumps before and after wrapped code. Besides
    doing this for general filesystem cleanliness reasons, leaving old dumps lying around can
    prevent future dumping from working (in the case of filesystem snapshots) or cause Borg hangs
    (in the case of database dump named pipes).
    Example use as a context manager:
        with borgmatic.actions.dump.Dump_cleanup(
            config, borgmatic_runtime_directory, patterns, dry_run,
        ):
            do_something_like_perform_a_dump_or_restore()
    '''
    def __init__(self, config, borgmatic_runtime_directory, patterns, dry_run):
        '''
        Given a configuration dict, the borgmatic runtime directory, the configured patterns, and
        whether this is a dry-run, store these values for use below.
        '''
        self.config = config
        self.borgmatic_runtime_directory = borgmatic_runtime_directory
        self.patterns = patterns
        self.dry_run = dry_run
    def __enter__(self):
        '''
        Remove all data source dumps that exist prior to the wrapped code running.
        '''
        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
            'remove_data_source_dumps',
            self.config,
            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
            self.borgmatic_runtime_directory,
            self.patterns,
            self.dry_run,
        )
    def __exit__(self, exception_type, exception, traceback):
        '''
        Remove all data source dumps, including any created by the wrapped code.
        '''
        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
            'remove_data_source_dumps',
            self.config,
            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
            self.borgmatic_runtime_directory,
            self.patterns,
            self.dry_run,
        )
@@ -1,7 +1,6 @@
 import logging
 import borgmatic.borg.export_key
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -18,16 +17,13 @@ def run_export_key(
    '''
    Run the "key export" action for the given repository.
    '''
-    if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Exporting repository key')
-        repository, export_arguments.repository
+    borgmatic.borg.export_key.export_key(
-    ):
+        repository['path'],
-        logger.info('Exporting repository key')
+        config,
-        borgmatic.borg.export_key.export_key(
+        local_borg_version,
-            repository['path'],
+        export_arguments,
-            config,
+        global_arguments,
-            local_borg_version,
+        local_path=local_path,
-            export_arguments,
+        remote_path=remote_path,
-            global_arguments,
+    )
            local_path=local_path,
            remote_path=remote_path,
        )
@@ -2,7 +2,6 @@ import logging
 import borgmatic.borg.export_tar
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -19,29 +18,26 @@ def run_export_tar(
    '''
    Run the "export-tar" action for the given repository.
    '''
-    if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
-        repository, export_tar_arguments.repository
+    borgmatic.borg.export_tar.export_tar_archive(
-    ):
+        global_arguments.dry_run,
-        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
+        repository['path'],
-        borgmatic.borg.export_tar.export_tar_archive(
+        borgmatic.borg.repo_list.resolve_archive_name(
            global_arguments.dry_run,
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            export_tar_arguments.archive,
                repository['path'],
                export_tar_arguments.archive,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
            ),
            export_tar_arguments.paths,
            export_tar_arguments.destination,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-            tar_filter=export_tar_arguments.tar_filter,
+        ),
-            strip_components=export_tar_arguments.strip_components,
+        export_tar_arguments.paths,
-        )
+        export_tar_arguments.destination,
        config,
        local_borg_version,
        global_arguments,
        local_path=local_path,
        remote_path=remote_path,
        tar_filter=export_tar_arguments.tar_filter,
        strip_components=export_tar_arguments.strip_components,
    )
@@ -2,8 +2,6 @@ import logging
 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 import borgmatic.hooks.command
 logger = logging.getLogger(__name__)
@@ -21,28 +19,25 @@ def run_extract(
    '''
    Run the "extract" action for the given repository.
    '''
-    if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info(f'Extracting archive {extract_arguments.archive}')
-        repository, extract_arguments.repository
+    borgmatic.borg.extract.extract_archive(
-    ):
+        global_arguments.dry_run,
-        logger.info(f'Extracting archive {extract_arguments.archive}')
+        repository['path'],
-        borgmatic.borg.extract.extract_archive(
+        borgmatic.borg.repo_list.resolve_archive_name(
            global_arguments.dry_run,
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            extract_arguments.archive,
                repository['path'],
                extract_arguments.archive,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
            ),
            extract_arguments.paths,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-            destination_path=extract_arguments.destination,
+        ),
-            strip_components=extract_arguments.strip_components,
+        extract_arguments.paths,
-        )
+        config,
        local_borg_version,
        global_arguments,
        local_path=local_path,
        remote_path=remote_path,
        destination_path=extract_arguments.destination,
        strip_components=extract_arguments.strip_components,
    )
@@ -1,7 +1,6 @@
 import logging
 import borgmatic.borg.import_key
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -18,16 +17,13 @@ def run_import_key(
    '''
    Run the "key import" action for the given repository.
    '''
-    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Importing repository key')
-        repository, import_arguments.repository
+    borgmatic.borg.import_key.import_key(
-    ):
+        repository['path'],
-        logger.info('Importing repository key')
+        config,
-        borgmatic.borg.import_key.import_key(
+        local_borg_version,
-            repository['path'],
+        import_arguments,
-            config,
+        global_arguments,
-            local_borg_version,
+        local_path=local_path,
-            import_arguments,
+        remote_path=remote_path,
-            global_arguments,
+    )
            local_path=local_path,
            remote_path=remote_path,
        )
@@ -4,7 +4,6 @@ import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.info
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -23,28 +22,26 @@ def run_info(
    If info_arguments.json is True, yield the JSON output from the info for the archive.
    '''
-    if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not info_arguments.json:
-        repository, info_arguments.repository
+        logger.answer('Displaying archive summary information')
-    ):
+
-        if not info_arguments.json:
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            logger.answer('Displaying archive summary information')
+        repository['path'],
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        info_arguments.archive,
-            repository['path'],
+        config,
-            info_arguments.archive,
+        local_borg_version,
-            config,
+        global_arguments,
-            local_borg_version,
+        local_path,
-            global_arguments,
+        remote_path,
-            local_path,
+    )
-            remote_path,
+    json_output = borgmatic.borg.info.display_archives_info(
-        )
+        repository['path'],
-        json_output = borgmatic.borg.info.display_archives_info(
+        config,
-            repository['path'],
+        local_borg_version,
-            config,
+        borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
-            local_borg_version,
+        global_arguments,
-            borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
+        local_path,
-            global_arguments,
+        remote_path,
-            local_path,
+    )
-            remote_path,
+    if json_output:
-        )
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
        if json_output:
            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -3,7 +3,6 @@ import logging
 import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -22,32 +21,29 @@ def run_list(
    If list_arguments.json is True, yield the JSON output from listing the archive.
    '''
-    if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not list_arguments.json:
-        repository, list_arguments.repository
+        if list_arguments.find_paths:  # pragma: no cover
-    ):
+            logger.answer('Searching archives')
-        if not list_arguments.json:
+        elif not list_arguments.archive:  # pragma: no cover
-            if list_arguments.find_paths:  # pragma: no cover
+            logger.answer('Listing archives')
                logger.answer('Searching archives')
            elif not list_arguments.archive:  # pragma: no cover
                logger.answer('Listing archives')
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
+        repository['path'],
-            list_arguments.archive,
+        list_arguments.archive,
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        json_output = borgmatic.borg.list.list_archive(
+    json_output = borgmatic.borg.list.list_archive(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
+        borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -2,7 +2,6 @@ import logging
 import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -19,29 +18,26 @@ def run_mount(
    '''
    Run the "mount" action for the given repository.
    '''
-    if mount_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if mount_arguments.archive:
-        repository, mount_arguments.repository
+        logger.info(f'Mounting archive {mount_arguments.archive}')
-    ):
+    else:  # pragma: nocover
-        if mount_arguments.archive:
+        logger.info('Mounting repository')
            logger.info(f'Mounting archive {mount_arguments.archive}')
        else:  # pragma: nocover
            logger.info('Mounting repository')
-        borgmatic.borg.mount.mount_archive(
+    borgmatic.borg.mount.mount_archive(
        repository['path'],
        borgmatic.borg.repo_list.resolve_archive_name(
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            mount_arguments.archive,
                repository['path'],
                mount_arguments.archive,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
            ),
            mount_arguments,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-        )
+        ),
        mount_arguments,
        config,
        local_borg_version,
        global_arguments,
        local_path=local_path,
        remote_path=remote_path,
    )
@@ -47,7 +47,8 @@ def collect_patterns(config):
        return (
            tuple(
                borgmatic.borg.pattern.Pattern(
-                    source_directory, source=borgmatic.borg.pattern.Pattern_source.CONFIG
+                    source_directory,
                    source=borgmatic.borg.pattern.Pattern_source.CONFIG,
                )
                for source_directory in config.get('source_directories', ())
            )
@@ -67,7 +68,7 @@ def collect_patterns(config):
            + tuple(
                parse_pattern(pattern_line.strip())
                for filename in config.get('patterns_from', ())
-                for pattern_line in open(filename).readlines()
+                for pattern_line in open(filename, encoding='utf-8')
                if not pattern_line.lstrip().startswith('#')
                if pattern_line.strip()
            )
@@ -77,7 +78,7 @@ def collect_patterns(config):
                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
                )
                for filename in config.get('exclude_from', ())
-                for exclude_line in open(filename).readlines()
+                for exclude_line in open(filename, encoding='utf-8')
                if not exclude_line.lstrip().startswith('#')
                if exclude_line.strip()
            )
@@ -112,9 +113,8 @@ def expand_directory(directory, working_directory):
            glob_path
            # If these are equal, that means we didn't add any working directory prefix above.
            if normalized_directory == expanded_directory
-            # Remove the working directory prefix that we added above in order to make glob() work.
+            # Remove the working directory prefix added above in order to make glob() work. We
-            # We can't use os.path.relpath() here because it collapses any use of Borg's slashdot
+            # can't use os.path.relpath() here because it collapses any use of Borg's slashdot hack.
            # hack.
            else glob_path.removeprefix(working_directory_prefix)
        )
        for glob_path in glob_paths
@@ -161,10 +161,35 @@ def expand_patterns(patterns, working_directory=None, skip_paths=None):
                )
            )
            for pattern in patterns
-        )
+        ),
    )
 def get_existent_path_or_parent(path):
    '''
    Given a path, return it if it exists. Otherwise, return the longest parent directory of the path
    that exists. Return None if none of these paths exist.
    This is used below for finding an existent path prefix of pattern's path, which is necessary if
    the path contain globs or other special characters that we don't want to try to interpret
    (because we want to leave that responsibility to Borg).
    '''
    if path.startswith('/e2e/'):
        return None
    try:
        return next(
            candidate_path
            for candidate_path in (
                path,
                *tuple(str(parent) for parent in pathlib.PurePath(path).parents),
            )
            if os.path.exists(candidate_path)
        )
    except StopIteration:
        return None
 def device_map_patterns(patterns, working_directory=None):
    '''
    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
@@ -174,63 +199,97 @@ def device_map_patterns(patterns, working_directory=None):
    This is handy for determining whether two different pattern paths are on the same filesystem
    (have the same device identifier).
    This function only considers the start of a pattern's path—from the start of the path up until
    there's a path component with a glob or other non-literal character. If there are no such
    characters, the whole path is considered. The rationale is that it's not feasible for borgmatic
    to interpret Borg's patterns to see which actual files (and therefore devices) they map to. So
    for instance, a pattern with a path of "/var/log/*/data" would end up with its device set to the
    device of "/var/log"—ignoring the "/*/data" part due to that glob.
    The one exception is that if a regular expression pattern path starts with "^", that will get
    stripped off for purposes of determining its device.
    '''
    return tuple(
        borgmatic.borg.pattern.Pattern(
            pattern.path,
            pattern.type,
            pattern.style,
-            device=pattern.device
+            device=pattern.device or (os.stat(existent_path).st_dev if existent_path else None),
            or (
                os.stat(full_path).st_dev
                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
                and os.path.exists(full_path)
                else None
            ),
            source=pattern.source,
        )
        for pattern in patterns
-        for full_path in (os.path.join(working_directory or '', pattern.path),)
+        for existent_path in (
            get_existent_path_or_parent(
                os.path.join(working_directory or '', pattern.path.lstrip('^')),
            ),
        )
    )
-def deduplicate_patterns(patterns):
+def deduplicate_runtime_directory_patterns(patterns, config, borgmatic_runtime_directory=None):
    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances, return them with all duplicate
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, the borgmatic runtime directory,
-    root child patterns removed. For instance, if two root patterns are given with paths "/foo" and
+    and a configuration dict, return them without any duplicate root child patterns that contain the
-    "/foo/bar", return just the one with "/foo". Non-root patterns are passed through without
+    runtime directory. For instance, if two root patterns are given with paths "/foo" and
    "/foo/bar", and the runtime directory is "/foo/bar", return just the "/foo" pattern. Non-root
    patterns and patterns not containing the runtime directory are passed through without
    modification.
-    The one exception to deduplication is two paths are on different filesystems (devices). In that
+    One exception to deduplication is if two paths are on different filesystems (devices) and
-    case, they won't get deduplicated, in case they both need to be passed to Borg (e.g. the
+    "one_file_system" is True in the given configuration. In that case, the paths won't get
-    one_file_system option is true).
+    deduplicated, because Borg won't cross filesystem boundaries when "one_file_system" is True.
-    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    The idea is that if Borg is given a root parent pattern containing the borgmatic runtime
-    child patterns, because it will naturally spider the contents of the parent pattern's path. And
+    directory, then Borg doesn't also need to be given child patterns, because it will naturally
-    there are cases where Borg coming across the same file twice will result in duplicate reads and
+    spider the contents of the parent pattern's path. Additionally, there are cases where Borg
-    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
+    coming across the same file twice will result in duplicate reads and even hangs, e.g. when a
-    Borg.
+    database hook in the borgmatic runtime directory is using a named pipe for streaming database
    dumps to Borg.
    This deduplication is limited to the borgmatic runtime directory (where borgmatic's named pipes
    exist), because there are other legitimate use cases for parent and child patterns to both exist
    in patterns. For instance, with some snapshotted filesystems, snapshots don't traverse from a
    parent filesystem to a child and therefore both need to remain in patterns.
    And for the case of named pipes outside of the borgmatic runtime directory, there is code
    elsewhere (in the "create" action) that auto-excludes special files to prevent Borg hangs.
    '''
    if borgmatic_runtime_directory is None:
        return patterns
    deduplicated = {}  # Use just the keys as an ordered set.
    runtime_directory_parents = set(pathlib.PurePath(borgmatic_runtime_directory).parents).union(
        {pathlib.PurePath(borgmatic_runtime_directory)}
    )
    for pattern in patterns:
        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
            deduplicated[pattern] = True
            continue
-        parents = pathlib.PurePath(pattern.path).parents
+        pattern_parents = pathlib.PurePath(pattern.path).parents
-        # If another directory in the given list is a parent of current directory (even n levels up)
+        # If:
-        # and both are on the same filesystem, then the current directory is a duplicate.
+        #
        #   1. another pattern is a parent of the current pattern (even n levels up),
        #   2. both patterns are parents of the runtime directory (even n levels up),
        #   3. and both patterns are on the same filesystem (or one_file_system is not set)
        #
        # ... then consider the current pattern as a duplicate.
        for other_pattern in patterns:
            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
                continue
-            if any(
+            device_matches = pattern.device is not None and (
-                pathlib.PurePath(other_pattern.path) == parent
+                other_pattern.device == pattern.device or config.get('one_file_system') is not True
-                and pattern.device is not None
+            )
-                and other_pattern.device == pattern.device
+
-                for parent in parents
+            if (
                pathlib.PurePath(other_pattern.path) in pattern_parents
                and pathlib.PurePosixPath(other_pattern.path) in runtime_directory_parents
                and pathlib.PurePosixPath(pattern.path) in runtime_directory_parents
                and device_matches
            ):
                break
        else:
@@ -239,23 +298,30 @@ def deduplicate_patterns(patterns):
    return tuple(deduplicated.keys())
-def process_patterns(patterns, working_directory, skip_expand_paths=None):
+def process_patterns(
    patterns, config, working_directory, borgmatic_runtime_directory=None, skip_expand_paths=None
 ):
    '''
-    Given a sequence of Borg patterns and a configured working directory, expand and deduplicate any
+    Given a sequence of Borg patterns, a configuration dict, a configured working directory, the
-    "root" patterns, returning the resulting root and non-root patterns as a list.
+    borgmatic runtime directory, and a sequence of paths to skip path expansion for, expand and
    deduplicate any "root" patterns, returning the resulting root and non-root patterns as a list.
-    If any paths are given to skip, don't expand them.
+    If the borgmatic runtime directory is None, then don't deduplicate patterns. Deduplication is
    really only necessary for the "create" action when the runtime directory might contain named
    pipes for database dumps.
    '''
    skip_paths = set(skip_expand_paths or ())
    return list(
-        deduplicate_patterns(
+        deduplicate_runtime_directory_patterns(
            device_map_patterns(
                expand_patterns(
                    patterns,
                    working_directory=working_directory,
                    skip_paths=skip_paths,
-                )
+                ),
-            )
+            ),
-        )
+            config,
            borgmatic_runtime_directory,
        ),
    )
@@ -1,8 +1,6 @@
 import logging
 import borgmatic.borg.prune
 import borgmatic.config.validate
 import borgmatic.hooks.command
 logger = logging.getLogger(__name__)
@@ -21,11 +19,6 @@ def run_prune(
    '''
    Run the "prune" action for the given repository.
    '''
    if prune_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, prune_arguments.repository
    ):
        return
    logger.info(f'Pruning archives{dry_run_label}')
    borgmatic.borg.prune.prune_archives(
        global_arguments.dry_run,
@@ -1,11 +1,10 @@
 import logging
 import subprocess
-import borgmatic.borg.info
+import borgmatic.actions.pattern
 import borgmatic.borg.pattern
 import borgmatic.borg.recreate
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 from borgmatic.actions.pattern import collect_patterns, process_patterns
 logger = logging.getLogger(__name__)
@@ -19,66 +18,75 @@ def run_recreate(
    local_borg_version,
    recreate_arguments,
    global_arguments,
    dry_run_label,
    local_path,
    remote_path,
 ):
    '''
    Run the "recreate" action for the given repository.
    '''
-    if recreate_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if recreate_arguments.archive:
-        repository, recreate_arguments.repository
+        logger.answer(f'Recreating archive {recreate_arguments.archive}{dry_run_label}')
-    ):
+    else:
-        if recreate_arguments.archive:
+        logger.answer(f'Recreating repository{dry_run_label}')
            logger.answer(f'Recreating archive {recreate_arguments.archive}')
        else:
            logger.answer('Recreating repository')
-        # Collect and process patterns.
+    # Collect and process patterns.
-        processed_patterns = process_patterns(
+    processed_patterns = borgmatic.actions.pattern.process_patterns(
-            collect_patterns(config), borgmatic.config.paths.get_working_directory(config)
+        (
            *borgmatic.actions.pattern.collect_patterns(config),
            # Also add borgmatic-specific paths, so they don't get excluded from the recreated
            # archive. Note that this doesn't currently work for archives created with Borg 1.2 or
            # below.
            borgmatic.borg.pattern.Pattern(
                '/borgmatic', source=borgmatic.borg.pattern.Pattern_source.INTERNAL
            ),
        ),
        config,
        borgmatic.config.paths.get_working_directory(config),
    )
    archive = borgmatic.borg.repo_list.resolve_archive_name(
        repository['path'],
        recreate_arguments.archive,
        config,
        local_borg_version,
        global_arguments,
        local_path,
        remote_path,
    )
    if archive and archive.endswith('.recreate'):
        if recreate_arguments.archive == 'latest':
            raise ValueError(
                f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
            )
        raise ValueError(
            f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
        )
-        archive = borgmatic.borg.repo_list.resolve_archive_name(
+    try:
        borgmatic.borg.recreate.recreate_archive(
            repository['path'],
-            recreate_arguments.archive,
+            archive,
            config,
            local_borg_version,
            recreate_arguments,
            global_arguments,
-            local_path,
+            local_path=local_path,
-            remote_path,
+            remote_path=remote_path,
            patterns=processed_patterns,
        )
-
+    except subprocess.CalledProcessError as error:
-        if archive and archive.endswith('.recreate'):
+        if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
-            if recreate_arguments.archive == 'latest':
+            if recreate_arguments.target:
                raise ValueError(
-                    f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.'
+                    f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
                )
            else:
                raise ValueError(
                    f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.'
                )
-        try:
+            if archive:
-            borgmatic.borg.recreate.recreate_archive(
+                raise ValueError(
-                repository['path'],
+                    f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
-                archive,
+                )
                config,
                local_borg_version,
                recreate_arguments,
                global_arguments,
                local_path=local_path,
                remote_path=remote_path,
                patterns=processed_patterns,
            )
        except subprocess.CalledProcessError as error:
            if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
                if recreate_arguments.target:
                    raise ValueError(
                        f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.'
                    )
                elif archive:
                    raise ValueError(
                        f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.'
                    )
-            raise
+        raise
@@ -1,7 +1,6 @@
 import logging
 import borgmatic.borg.repo_create
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -18,18 +17,13 @@ def run_repo_create(
    '''
    Run the "repo-create" action for the given repository.
    '''
    if repo_create_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, repo_create_arguments.repository
    ):
        return
    logger.info('Creating repository')
    encryption_mode = repo_create_arguments.encryption_mode or repository.get('encryption')
    if not encryption_mode:
        raise ValueError(
-            'With the repo-create action, either the --encryption flag or the repository encryption option is required.'
+            'With the repo-create action, either the --encryption flag or the repository encryption option is required.',
        )
    borgmatic.borg.repo_create.create_repository(
@@ -17,19 +17,16 @@ def run_repo_delete(
    '''
    Run the "repo-delete" action for the given repository.
    '''
-    if repo_delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.answer(
-        repository, repo_delete_arguments.repository
+        'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
-    ):
+    )
        logger.answer(
            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else '')
        )
-        borgmatic.borg.repo_delete.delete_repository(
+    borgmatic.borg.repo_delete.delete_repository(
-            repository,
+        repository,
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_delete_arguments,
+        repo_delete_arguments,
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
@@ -2,7 +2,6 @@ import logging
 import borgmatic.actions.json
 import borgmatic.borg.repo_info
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -21,20 +20,17 @@ def run_repo_info(
    If repo_info_arguments.json is True, yield the JSON output from the info for the repository.
    '''
-    if repo_info_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not repo_info_arguments.json:
-        repository, repo_info_arguments.repository
+        logger.answer('Displaying repository summary information')
    ):
        if not repo_info_arguments.json:
            logger.answer('Displaying repository summary information')
-        json_output = borgmatic.borg.repo_info.display_repository_info(
+    json_output = borgmatic.borg.repo_info.display_repository_info(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_info_arguments=repo_info_arguments,
+        repo_info_arguments=repo_info_arguments,
-            global_arguments=global_arguments,
+        global_arguments=global_arguments,
-            local_path=local_path,
+        local_path=local_path,
-            remote_path=remote_path,
+        remote_path=remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -2,7 +2,6 @@ import logging
 import borgmatic.actions.json
 import borgmatic.borg.repo_list
 import borgmatic.config.validate
 logger = logging.getLogger(__name__)
@@ -21,20 +20,17 @@ def run_repo_list(
    If repo_list_arguments.json is True, yield the JSON output from listing the repository.
    '''
-    if repo_list_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not repo_list_arguments.json:
-        repository, repo_list_arguments.repository
+        logger.answer('Listing repository')
    ):
        if not repo_list_arguments.json:
            logger.answer('Listing repository')
-        json_output = borgmatic.borg.repo_list.list_repository(
+    json_output = borgmatic.borg.repo_list.list_repository(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_list_arguments=repo_list_arguments,
+        repo_list_arguments=repo_list_arguments,
-            global_arguments=global_arguments,
+        global_arguments=global_arguments,
-            local_path=local_path,
+        local_path=local_path,
-            remote_path=remote_path,
+        remote_path=remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -1,16 +1,17 @@
 import collections
 import locale
 import logging
 import os
 import pathlib
 import shutil
 import tempfile
 import borgmatic.actions.dump
 import borgmatic.actions.pattern
 import borgmatic.borg.extract
 import borgmatic.borg.list
 import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch
@@ -22,8 +23,8 @@ UNSPECIFIED = object()
 Dump = collections.namedtuple(
    'Dump',
-    ('hook_name', 'data_source_name', 'hostname', 'port'),
+    ('hook_name', 'data_source_name', 'hostname', 'port', 'label', 'container'),
-    defaults=('localhost', None),
+    defaults=(None, None, None, None),
 )
@@ -33,17 +34,24 @@ def dumps_match(first, second, default_port=None):
    indicates that the field should match any value. If a default port is given, then consider any
    dump having that port to match with a dump having a None port.
    '''
-    for field_name in first._fields:
+    # label kinda counts as an unique id, if they match ignore host/container/port
    if first.label not in {None, UNSPECIFIED} and first.label == second.label:
        field_list = ('hook_name', 'data_source_name')
    else:
        field_list = Dump._fields
    for field_name in field_list:
        first_value = getattr(first, field_name)
        second_value = getattr(second, field_name)
        if default_port is not None and field_name == 'port':
            if first_value == default_port and second_value is None:
                continue
            if second_value == default_port and first_value is None:
                continue
-        if first_value == UNSPECIFIED or second_value == UNSPECIFIED:
+        if first_value == UNSPECIFIED or second_value == UNSPECIFIED:  # noqa: PLR1714
            continue
        if first_value != second_value:
@@ -56,16 +64,19 @@ def render_dump_metadata(dump):
    '''
    Given a Dump instance, make a display string describing it for use in log messages.
    '''
    label = dump.label or UNSPECIFIED
    name = 'unspecified' if dump.data_source_name is UNSPECIFIED else dump.data_source_name
-    hostname = dump.hostname or UNSPECIFIED
+    host = dump.container or dump.hostname or UNSPECIFIED
    port = None if dump.port is UNSPECIFIED else dump.port
-    if port:
+    if label is not UNSPECIFIED:
-        metadata = f'{name}@:{port}' if hostname is UNSPECIFIED else f'{name}@{hostname}:{port}'
+        metadata = f'{name}@{label}'
    elif port:
        metadata = f'{name}@:{port}' if host is UNSPECIFIED else f'{name}@{host}:{port}'
    else:
-        metadata = f'{name}' if hostname is UNSPECIFIED else f'{name}@{hostname}'
+        metadata = f'{name}' if host is UNSPECIFIED else f'{name}@{host}'
-    if dump.hook_name not in (None, UNSPECIFIED):
+    if dump.hook_name not in {None, UNSPECIFIED}:
        return f'{metadata} ({dump.hook_name})'
    return metadata
@@ -98,8 +109,10 @@ def get_configured_data_source(config, restore_dump):
            Dump(
                hook_name,
                hook_data_source.get('name'),
-                hook_data_source.get('hostname', 'localhost'),
+                hook_data_source.get('hostname'),
                hook_data_source.get('port'),
                hook_data_source.get('label') or UNSPECIFIED,
                hook_data_source.get('container'),
            ),
            restore_dump,
            default_port,
@@ -111,20 +124,21 @@ def get_configured_data_source(config, restore_dump):
    if len(matching_dumps) > 1:
        raise ValueError(
-            f'Cannot restore data source {render_dump_metadata(restore_dump)} because there are multiple matching data sources configured'
+            f'Cannot restore data source {render_dump_metadata(restore_dump)} because there are multiple matching data sources configured',
        )
    return matching_dumps[0]
 def strip_path_prefix_from_extracted_dump_destination(
-    destination_path, borgmatic_runtime_directory
+    extract_path,
    borgmatic_runtime_directory,
 ):
    '''
    Directory-format dump files get extracted into a temporary directory containing a path prefix
-    that depends how the files were stored in the archive. So, given the destination path where the
+    that depends how the files were stored in the archive. So, given the path where the dump was
-    dump was extracted and the borgmatic runtime directory, move the dump files such that the
+    extracted and the borgmatic runtime directory, move the dump files such that the restore doesn't
-    restore doesn't have to deal with that varying path prefix.
+    have to deal with that varying path prefix.
    For instance, if the dump was extracted to:
@@ -138,15 +152,16 @@ def strip_path_prefix_from_extracted_dump_destination(
      /run/user/0/borgmatic/postgresql_databases/test/...
    '''
-    for subdirectory_path, _, _ in os.walk(destination_path):
+    for subdirectory_path, _, _ in os.walk(extract_path):
        databases_directory = os.path.basename(subdirectory_path)
        if not databases_directory.endswith('_databases'):
            continue
-        shutil.move(
+        destination_path = os.path.join(borgmatic_runtime_directory, databases_directory)
-            subdirectory_path, os.path.join(borgmatic_runtime_directory, databases_directory)
+        shutil.rmtree(destination_path, ignore_errors=True)
-        )
+        shutil.move(subdirectory_path, destination_path)
        break
@@ -169,7 +184,14 @@ def restore_single_dump(
    that data source from the archive.
    '''
    dump_metadata = render_dump_metadata(
-        Dump(hook_name, data_source['name'], data_source.get('hostname'), data_source.get('port'))
+        Dump(
            hook_name,
            data_source['name'],
            data_source.get('hostname'),
            data_source.get('port'),
            data_source.get('label') or UNSPECIFIED,
            data_source.get('container'),
        ),
    )
    logger.info(f'Restoring data source {dump_metadata}')
@@ -180,6 +202,10 @@ def restore_single_dump(
        borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
        borgmatic_runtime_directory,
        data_source['name'],
        data_source.get('hostname'),
        data_source.get('port'),
        data_source.get('container'),
        data_source.get('label'),
    )[hook_name.split('_databases', 1)[0]]
    destination_path = (
@@ -197,8 +223,8 @@ def restore_single_dump(
            archive=archive_name,
            paths=[
                borgmatic.hooks.data_source.dump.convert_glob_patterns_to_borg_pattern(
-                    dump_patterns
+                    dump_patterns,
-                )
+                ),
            ],
            config=config,
            local_borg_version=local_borg_version,
@@ -213,7 +239,8 @@ def restore_single_dump(
        if destination_path and not global_arguments.dry_run:
            strip_path_prefix_from_extracted_dump_destination(
-                destination_path, borgmatic_runtime_directory
+                destination_path,
                borgmatic_runtime_directory,
            )
    finally:
        if destination_path and not global_arguments.dry_run:
@@ -246,16 +273,12 @@ def collect_dumps_from_archive(
    Given a local or remote repository path, a resolved archive name, a configuration dict, the
    local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
    borgmatic runtime directory, query the archive for the names of data sources dumps it contains
-    and return them as a set of Dump instances.
+    and return them as a tuple of Dump instances.
    '''
-    borgmatic_source_directory = str(
+    dumps_from_archive = {}  # Use a dict as an ordered set.
        pathlib.Path(borgmatic.config.paths.get_borgmatic_source_directory(config))
    )
-    # Probe for the data source dumps in multiple locations, as the default location has moved to
+    # There is (at most) one dump metadata file per data source hook. Load each.
-    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
+    for dumps_metadata_entry in borgmatic.borg.list.capture_archive_listing(
    # we still want to support reading dumps from previously created archives as well.
    dump_paths = borgmatic.borg.list.capture_archive_listing(
        repository,
        archive,
        config,
@@ -264,7 +287,69 @@ def collect_dumps_from_archive(
        list_paths=[
            'sh:'
            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
-                base_directory, '*_databases/*/*'
+                base_directory,
                '*_databases/dumps.json',
            )
            # Probe for dump metadata files in multiple locations, as the default location is
            # "/borgmatic/*_databases/dumps.json" with Borg 1.4+, but instead begins with the
            # borgmatic runtime directory for older versions of Borg.
            for base_directory in (
                'borgmatic',
                borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
            )
        ],
        local_path=local_path,
        remote_path=remote_path,
    ):
        if not dumps_metadata_entry.get('path'):
            continue
        for dump in borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
            borgmatic.borg.extract.extract_archive(
                global_arguments.dry_run,
                repository,
                archive,
                [dumps_metadata_entry['path']],
                config,
                local_borg_version,
                global_arguments,
                local_path=local_path,
                remote_path=remote_path,
                extract_to_stdout=True,
            )
            .stdout.read()
            .decode(locale.getpreferredencoding()),
            dumps_metadata_entry['path'],
        ):
            dumps_from_archive[dump] = None
    # If we've successfully loaded any dumps metadata, we're done.
    if dumps_from_archive:
        logger.debug('Collecting database dumps from archive data source dumps metadata files')
        return tuple(dumps_from_archive.keys())
    # No dumps metadata files were found, so for backwards compatibility, fall back to parsing the
    # paths of dumps found in the archive to get their respective dump metadata.
    logger.debug('Collecting database dumps from archive data source dump paths (fallback)')
    borgmatic_source_directory = str(
        pathlib.Path(borgmatic.config.paths.get_borgmatic_source_directory(config)),
    )
    # Probe for the data source dumps in multiple locations, as the default location has moved to
    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
    # we still want to support reading dumps from previously created archives as well.
    dump_entries = borgmatic.borg.list.capture_archive_listing(
        repository,
        archive,
        config,
        local_borg_version,
        global_arguments,
        list_paths=[
            'sh:'
            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
                base_directory,
                '*_databases/*/*',
            )
            for base_directory in (
                'borgmatic',
@@ -276,10 +361,9 @@ def collect_dumps_from_archive(
        remote_path=remote_path,
    )
-    # Parse the paths of dumps found in the archive to get their respective dump metadata.
+    for dump_entry in dump_entries:
-    dumps_from_archive = set()
+        dump_path = dump_entry.get('path')
    for dump_path in dump_paths:
        if not dump_path:
            continue
@@ -291,7 +375,8 @@ def collect_dumps_from_archive(
        ):
            try:
                (hook_name, host_and_port, data_source_name) = dump_path.split(
-                    base_directory + os.path.sep, 1
+                    base_directory + os.path.sep,
                    1,
                )[1].split(os.path.sep)[0:3]
            except (ValueError, IndexError):
                continue
@@ -308,29 +393,34 @@ def collect_dumps_from_archive(
            except (ValueError, TypeError):
                port = None
-            dumps_from_archive.add(Dump(hook_name, data_source_name, hostname, port))
+            dumps_from_archive[
                Dump(
                    hook_name, data_source_name, None if hostname == 'localhost' else hostname, port
                )
            ] = None
            # We've successfully parsed the dump path, so need to probe any further.
            break
        else:
            logger.warning(
-                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
+                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}',
            )
-    return dumps_from_archive
+    return tuple(dumps_from_archive.keys())
 def get_dumps_to_restore(restore_arguments, dumps_from_archive):
    '''
    Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
-    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    a tuple of Dump instances representing the dumps found in an archive, return a tuple of specific
    Dump instances from the archive to restore. As part of this, replace any Dump having a data
    source name of "all" with multiple named Dump instances as appropriate.
    Raise ValueError if any of the requested data source names cannot be found in the archive or if
    there are multiple archive dump matches for a given requested dump.
    '''
-    requested_dumps = (
+    requested_dumps = tuple(
        # Use a dict comprehension as an ordered set.
        {
            Dump(
                hook_name=(
@@ -345,28 +435,35 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                data_source_name=name,
                hostname=restore_arguments.original_hostname or UNSPECIFIED,
                port=restore_arguments.original_port,
-            )
+                label=restore_arguments.original_label or UNSPECIFIED,
                container=restore_arguments.original_container or UNSPECIFIED,
            ): None
            for name in restore_arguments.data_sources or (UNSPECIFIED,)
-        }
+        }.keys()
        if restore_arguments.hook
        or restore_arguments.data_sources
        or restore_arguments.original_hostname
        or restore_arguments.original_port
-        else {
+        or restore_arguments.original_label
        or restore_arguments.original_container
        else (
            Dump(
                hook_name=UNSPECIFIED,
                data_source_name='all',
                hostname=UNSPECIFIED,
                port=UNSPECIFIED,
-            )
+                label=UNSPECIFIED,
-        }
+                container=UNSPECIFIED,
            ),
        )
    )
    missing_dumps = set()
-    dumps_to_restore = set()
+    dumps_to_restore = {}  # Use a dict as an ordered set.
    # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
    if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
-        dumps_to_restore.update(dumps_from_archive)
+        for dump in dumps_from_archive:
            dumps_to_restore[dump] = None
    # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
    for requested_dump in requested_dumps:
@@ -382,10 +479,10 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
        if len(matching_dumps) == 0:
            missing_dumps.add(requested_dump)
        elif len(matching_dumps) == 1:
-            dumps_to_restore.add(matching_dumps[0])
+            dumps_to_restore[matching_dumps[0]] = None
        else:
            raise ValueError(
-                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.'
+                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.',
            )
    if missing_dumps:
@@ -394,30 +491,31 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
        )
        raise ValueError(
-            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive"
+            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive",
        )
-    return dumps_to_restore
+    return tuple(dumps_to_restore.keys())
 def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
    '''
-    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    Given a tuple of requested dumps to restore and a set of dumps actually restored, raise
-    if any requested dumps to restore weren't restored, indicating that they were missing from the
+    ValueError if any requested dumps to restore weren't restored, indicating that they were missing
-    configuration.
+    from the configuration.
    '''
-    if not dumps_actually_restored:
+    if not dumps_to_restore:
        raise ValueError('No data source dumps were found to restore')
    missing_dumps = sorted(
-        dumps_to_restore - dumps_actually_restored, key=lambda dump: dump.data_source_name
+        set(dumps_to_restore) - dumps_actually_restored,
        key=lambda dump: dump.data_source_name,
    )
    if missing_dumps:
        rendered_dumps = ', '.join(f'{render_dump_metadata(dump)}' for dump in missing_dumps)
        raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from borgmatic's configuration"
+            f"Cannot restore data source{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from borgmatic's configuration",
        )
@@ -431,101 +529,96 @@ def run_restore(
    remote_path,
 ):
    '''
-    Run the "restore" action for the given repository, but only if the repository matches the
+    Run the "restore" action for the given repository.
    requested repository in restore arguments.
    Raise ValueError if a configured data source could not be found to restore or there's no
    matching dump in the archive.
    '''
    if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
        repository, restore_arguments.repository
    ):
        return
    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
    working_directory = borgmatic.config.paths.get_working_directory(config)
    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+        patterns = borgmatic.actions.pattern.process_patterns(
-            'remove_data_source_dumps',
+            borgmatic.actions.pattern.collect_patterns(config),
            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            working_directory,
            borgmatic_runtime_directory,
            global_arguments.dry_run,
        )
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        with borgmatic.actions.dump.Dump_cleanup(
-            repository['path'],
+            config, borgmatic_runtime_directory, patterns, global_arguments.dry_run
-            restore_arguments.archive,
+        ):
-            config,
+            archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            local_borg_version,
+                repository['path'],
-            global_arguments,
+                restore_arguments.archive,
            local_path,
            remote_path,
        )
        dumps_from_archive = collect_dumps_from_archive(
            repository['path'],
            archive_name,
            config,
            local_borg_version,
            global_arguments,
            local_path,
            remote_path,
            borgmatic_runtime_directory,
        )
        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
        dumps_actually_restored = set()
        connection_params = {
            'hostname': restore_arguments.hostname,
            'port': restore_arguments.port,
            'username': restore_arguments.username,
            'password': restore_arguments.password,
            'restore_path': restore_arguments.restore_path,
        }
        # Restore each dump.
        for restore_dump in dumps_to_restore:
            found_data_source = get_configured_data_source(
                config,
                restore_dump,
            )
            # For a dump that wasn't found via an exact match in the configuration, try to fallback
            # to an "all" data source.
            if not found_data_source:
                found_data_source = get_configured_data_source(
                    config,
                    Dump(restore_dump.hook_name, 'all', restore_dump.hostname, restore_dump.port),
                )
                if not found_data_source:
                    continue
                found_data_source = dict(found_data_source)
                found_data_source['name'] = restore_dump.data_source_name
            dumps_actually_restored.add(restore_dump)
            restore_single_dump(
                repository,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
            )
            dumps_from_archive = collect_dumps_from_archive(
                repository['path'],
                archive_name,
-                restore_dump.hook_name,
+                config,
-                dict(found_data_source, **{'schemas': restore_arguments.schemas}),
+                local_borg_version,
-                connection_params,
+                global_arguments,
                local_path,
                remote_path,
                borgmatic_runtime_directory,
            )
            dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            dumps_actually_restored = set()
-            'remove_data_source_dumps',
+            connection_params = {
-            config,
+                'container': restore_arguments.container,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                'hostname': restore_arguments.hostname,
-            borgmatic_runtime_directory,
+                'port': restore_arguments.port,
-            global_arguments.dry_run,
+                'username': restore_arguments.username,
-        )
+                'password': restore_arguments.password,
                'restore_path': restore_arguments.restore_path,
            }
            # Restore each dump.
            for restore_dump in dumps_to_restore:
                found_data_source = get_configured_data_source(
                    config,
                    restore_dump,
                )
                # For a dump that wasn't found via an exact match in the configuration, try to fallback
                # to an "all" data source.
                if not found_data_source:
                    found_data_source = get_configured_data_source(
                        config,
                        Dump(
                            restore_dump.hook_name,
                            'all',
                            restore_dump.hostname,
                            restore_dump.port,
                            restore_dump.label,
                            restore_dump.container,
                        ),
                    )
                    if not found_data_source:
                        continue
                    found_data_source = dict(found_data_source)
                    found_data_source['name'] = restore_dump.data_source_name
                dumps_actually_restored.add(restore_dump)
                restore_single_dump(
                    repository,
                    config,
                    local_borg_version,
                    global_arguments,
                    local_path,
                    remote_path,
                    archive_name,
                    restore_dump.hook_name,
                    dict(found_data_source, schemas=restore_arguments.schemas),
                    connection_params,
                    borgmatic_runtime_directory,
                )
    ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)
@@ -19,7 +19,7 @@ def run_transfer(
    '''
    if transfer_arguments.archive and config.get('match_archives'):
        raise ValueError(
-            'With the transfer action, only one of --archive and --match-archives/match_archives can be used.'
+            'With the transfer action, only one of --archive and --match-archives/match_archives can be used.',
        )
    logger.info('Transferring archives to repository')
@@ -39,9 +39,9 @@ def run_arbitrary_borg(
        borg_command = tuple(options[:command_options_start_index])
        command_options = tuple(options[command_options_start_index:])
-        if borg_command and borg_command[0] in borgmatic.commands.arguments.ACTION_ALIASES.keys():
+        if borg_command and borg_command[0] in borgmatic.commands.arguments.ACTION_ALIASES:
            logger.warning(
-                f"Borg's {borg_command[0]} subcommand is supported natively by borgmatic. Try this instead: borgmatic {borg_command[0]}"
+                f"Borg's {borg_command[0]} subcommand is supported natively by borgmatic. Try this instead: borgmatic {borg_command[0]}",
            )
    except IndexError:
        borg_command = ()
@@ -57,16 +57,14 @@ def run_arbitrary_borg(
        + command_options
    )
-    return execute_command(
+    return execute_command(  # noqa: S604
        tuple(shlex.quote(part) for part in full_command),
        output_file=DO_NOT_CAPTURE,
        shell=True,
        environment=dict(
            (environment.make_environment(config) or {}),
-            **{
+            BORG_REPO=repository_path,
-                'BORG_REPO': repository_path,
+            ARCHIVE=archive if archive else '',
                'ARCHIVE': archive if archive else '',
            },
        ),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 from borgmatic.borg import environment, flags
@@ -22,15 +23,17 @@ def break_lock(
    '''
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    extra_borg_options = config.get('extra_borg_options', {}).get('break_lock', '')
    full_command = (
        (local_path, 'break-lock')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.execute
@@ -25,15 +26,16 @@ def change_passphrase(
    borgmatic.logger.add_custom_log_levels()
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    extra_borg_options = config.get('extra_borg_options', {}).get('key_change_passphrase', '')
    full_command = (
        (local_path, 'key', 'change-passphrase')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
@@ -49,7 +51,7 @@ def change_passphrase(
    config_without_passphrase = {
        option_name: value
        for (option_name, value) in config.items()
-        if option_name not in ('encryption_passphrase', 'encryption_passcommand')
+        if option_name not in {'encryption_passphrase', 'encryption_passcommand'}
    }
    borgmatic.execute.execute_command(
@@ -63,5 +65,5 @@ def change_passphrase(
    )
    logger.answer(
-        f"{repository_path}: Don't forget to update your encryption_passphrase option (if needed)"
+        "Don't forget to update your encryption_passphrase option (if needed)",
    )
@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
 import shlex
 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags, repo_info
@@ -41,11 +42,12 @@ def make_archive_filter_flags(local_borg_version, config, checks, check_argument
    if check_last:
        logger.warning(
-            'Ignoring check_last option, as "archives" or "data" are not in consistency checks'
+            'Ignoring check_last option, as "archives" or "data" are not in consistency checks',
        )
    if prefix:
        logger.warning(
-            'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks'
+            'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks',
        )
    return ()
@@ -75,13 +77,18 @@ def make_check_name_flags(checks, archive_filter_flags):
        return common_flags
    return (
-        tuple(f'--{check}-only' for check in checks if check in ('repository', 'archives'))
+        tuple(f'--{check}-only' for check in checks if check in {'repository', 'archives'})
        + common_flags
    )
 def get_repository_id(
-    repository_path, config, local_borg_version, global_arguments, local_path, remote_path
+    repository_path,
    config,
    local_borg_version,
    global_arguments,
    local_path,
    remote_path,
 ):
    '''
    Given a local or remote repository path, a configuration dict, the local Borg version, global
@@ -99,7 +106,7 @@ def get_repository_id(
                global_arguments,
                local_path,
                remote_path,
-            )
+            ),
        )['repository']['id']
    except (json.JSONDecodeError, KeyError):
        raise ValueError(f'Cannot determine Borg repository ID for {repository_path}')
@@ -126,8 +133,10 @@ def check_archives(
    extra_borg_options = config.get('extra_borg_options', {}).get('check', '')
    verbosity_flags = ()
    if logger.isEnabledFor(logging.INFO):
        verbosity_flags = ('--info',)
    if logger.isEnabledFor(logging.DEBUG):
        verbosity_flags = ('--debug', '--show-rc')
@@ -140,8 +149,10 @@ def check_archives(
    max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')
    # If not configured, elevate Borg's exit code 1 (an ostensible warning) to error, because Borg
    # returns exit code 1 for repository check errors!
    borg_exit_codes = [*config.get('borg_exit_codes', []), *[{'code': 1, 'treat_as': 'error'}]]
    umask = config.get('umask')
    borg_exit_codes = config.get('borg_exit_codes')
    working_directory = borgmatic.config.paths.get_working_directory(config)
    if 'data' in checks:
@@ -167,11 +178,17 @@ def check_archives(
            + make_check_name_flags(checks_subset, archive_filter_flags)
            + (('--remote-path', remote_path) if remote_path else ())
            + (('--umask', str(umask)) if umask else ())
-            + (('--log-json',) if config.get('log_json') else ())
+            + (
                ('--log-json',)
                if (
                    config.get('log_json') or not (check_arguments.repair or config.get('progress'))
                )
                else ()
            )
            + (('--lock-wait', str(lock_wait)) if lock_wait else ())
            + verbosity_flags
            + (('--progress',) if config.get('progress') else ())
-            + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+            + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
            + flags.make_repository_flags(repository_path, local_borg_version)
        )
@@ -1,7 +1,8 @@
 import logging
 import shlex
 import borgmatic.config.paths
-from borgmatic.borg import environment, flags
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
 logger = logging.getLogger(__name__)
@@ -30,19 +31,24 @@ def compact_segments(
        (local_path, 'compact')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--progress',) if config.get('progress') else ())
        + (('--cleanup-commits',) if cleanup_commits else ())
        + (('--threshold', str(threshold)) if threshold else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (
            ('--dry-run',)
            if dry_run and feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version)
            else ()
        )
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
-    if dry_run:
+    if dry_run and not feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version):
-        logging.info('Skipping compact (dry run)')
+        logger.info('Skipping compact (dry run)')
        return
    execute_command(
@@ -1,6 +1,7 @@
 import logging
 import os
 import pathlib
 import shlex
 import stat
 import textwrap
@@ -44,35 +45,46 @@ def any_parent_directories(path, candidate_parents):
    return False
-def collect_special_file_paths(
+def validate_planned_backup_paths(
    dry_run,
    create_command,
    config,
    patterns,
    local_path,
    working_directory,
    borgmatic_runtime_directory,
    find_special_files=False,
 ):
    '''
    Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
-    a working directory, and the borgmatic runtime directory, collect the paths for any special
+    a working directory, and the borgmatic runtime directory, perform a "borg create --dry-run" to
-    files (character devices, block devices, and named pipes / FIFOs) that Borg would encounter
+    determine whether Borg's planned paths to include in a backup look good. Specifically, if the
-    during a create. These are all paths that could cause Borg to hang if its --read-special flag is
+    given runtime directory exists, validate that it will be included in a backup and hasn't been
-    used.
+    excluded.
-    Skip looking for special files in the given borgmatic runtime directory, as borgmatic creates
+    If find special files is True, then return the subset of planned backup paths that are special
-    its own special files there for database dumps and we don't want those omitted.
+    files. Otherwise, return an empty tuple.
-    Additionally, if the borgmatic runtime directory is not contained somewhere in the files Borg
+    Raise ValueError if the runtime directory has been excluded via "exclude_patterns" or similar,
-    plans to backup, that means the user must have excluded the runtime directory (e.g. via
+    because any features that rely on the runtime directory getting backed up will break. For
-    "exclude_patterns" or similar). Therefore, raise, because this means Borg won't be able to
+    instance, without the runtime directory, Borg can't consume any database dumps and borgmatic may
-    consume any database dumps and therefore borgmatic will hang when it tries to do so.
+    hang waiting for them to be consumed.
    '''
    # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
    # files including any named pipe we've created. And omit "--filter" because that can break the
-    # paths output parsing below such that path lines no longer start with th expected "- ".
+    # paths output parsing below such that path lines no longer start with the expected "- ".
-    paths_output = execute_command_and_capture_output(
+    path_lines = execute_command_and_capture_output(
-        flags.omit_flag_and_value(flags.omit_flag(create_command, '--exclude-nodump'), '--filter')
+        (
-        + ('--dry-run', '--list'),
+            *flags.omit_flag_and_value(
                flags.omit_flag(
                    flags.omit_flag(create_command, '--exclude-nodump'),
                    '--log-json',
                ),
                '--filter',
            ),
            '--dry-run',
            '--list',
        ),
        capture_stderr=True,
        working_directory=working_directory,
        environment=environment.make_environment(config),
@@ -82,38 +94,68 @@ def collect_special_file_paths(
    # These are all the individual files that Borg is planning to backup as determined by the Borg
    # create dry run above.
-    paths = tuple(
+    paths = (
        path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in path_lines
-        if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
+        if path_line and path_line.startswith(('- ', '+ '))
    )
-    # These are the subset of those files that contain the borgmatic runtime directory.
+    include_pattern_paths = {
-    paths_containing_runtime_directory = {}
+        pattern.path
-
+        for pattern in patterns
-    if os.path.exists(borgmatic_runtime_directory):
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.INCLUDE
-        paths_containing_runtime_directory = {
+    }
-            path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+    runtime_directory_root_patterns = tuple(
-        }
+        pattern
-
+        for pattern in patterns
-        # If no paths to backup contain the runtime directory, it must've been excluded.
+        if any_parent_directories(pattern.path, (borgmatic_runtime_directory,))
-        if not paths_containing_runtime_directory and not dry_run:
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
-            raise ValueError(
+        # Skip root patterns that have corresponding include patterns, because those will "punch
-                f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.'
+        # through" any subsequent excludes.
-            )
+        if pattern.path not in include_pattern_paths
    return tuple(
        path
        for path in paths
        if special_file(path, working_directory)
        if path not in paths_containing_runtime_directory
    )
    special_paths = []
    validate_runtime_directory = bool(
        not dry_run
        and os.path.exists(borgmatic_runtime_directory)
        and runtime_directory_root_patterns
    )
    runtime_directory_in_path = False
    # Do everything in this one loop because we only want to consume the paths generator once.
    for path in paths:
        # If all root patterns in the runtime directory are missing from the paths Borg is planning
        # to backup, then they must've gotten excluded, e.g. by user-configured excludes. Warn
        # accordingly (below).
        if validate_runtime_directory and any(
            any_parent_directories(path, (pattern.path,))
            for pattern in runtime_directory_root_patterns
        ):
            runtime_directory_in_path = True
        # Return the subset of output paths that are special files but *not* contained within the
        # borgmatic runtime directory. The intent is to skip runtime paths that borgmatic uses for its
        # own bookkeeping, instead focusing on user-configured paths.
        if (
            find_special_files
            and not any_parent_directories(path, (borgmatic_runtime_directory,))
            and special_file(path, working_directory)
        ):
            special_paths.append(path)
    if validate_runtime_directory and not runtime_directory_in_path:
        logger.warning(
            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes (or the snapshotted source directories are empty). Please ensure the runtime directory is not excluded.'
        )
    return tuple(special_paths)
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
-def make_base_create_command(
+def make_base_create_command(  # noqa: PLR0912
    dry_run,
    repository_path,
    config,
@@ -121,23 +163,28 @@ def make_base_create_command(
    local_borg_version,
    global_arguments,
    borgmatic_runtime_directory,
    archive_suffix='',
    local_path='borg',
    remote_path=None,
    json=False,
    comment=None,
    stream_processes=None,
 ):
    '''
    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version,
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version, global
-    global arguments as an argparse.Namespace instance, and a sequence of borgmatic source
+    arguments as an argparse.Namespace instance, the borgmatic runtime directory, a string suffix to
-    directories, return a tuple of (base Borg create command flags, Borg create command positional
+    add to the archive name, the local Borg path, the remote Borg path, whether to output JSON,
-    arguments, open pattern file handle).
+    comment text to add to the created archive, and a sequence of processes streaming data to Borg,
    return a tuple of (base Borg create command flags, Borg create command positional arguments,
    open pattern file handle).
    '''
-    if config.get('source_directories_must_exist', False):
+    if config.get('source_directories_must_exist', True):
        borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
    patterns_file = borgmatic.borg.pattern.write_patterns_file(
-        patterns, borgmatic_runtime_directory
+        patterns,
        borgmatic_runtime_directory,
    )
    checkpoint_interval = config.get('checkpoint_interval', None)
    checkpoint_volume = config.get('checkpoint_volume', None)
@@ -148,9 +195,11 @@ def make_base_create_command(
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    list_filter_flags = flags.make_list_filter_flags(local_borg_version, dry_run)
    files_changed = config.get('files_changed')
    files_cache = config.get('files_cache')
-    archive_name_format = config.get(
+    archive_name_format = (
-        'archive_name_format', flags.get_default_archive_name_format(local_borg_version)
+        config.get('archive_name_format', flags.get_default_archive_name_format(local_borg_version))
        + archive_suffix
    )
    extra_borg_options = config.get('extra_borg_options', {}).get('create', '')
@@ -183,6 +232,7 @@ def make_base_create_command(
        + ('create',)
        + (('--patterns-from', patterns_file.name) if patterns_file else ())
        + flags.make_exclude_flags(config)
        + (('--comment', comment) if comment else ())
        + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
        + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
        + (('--chunker-params', chunker_params) if chunker_params else ())
@@ -196,67 +246,80 @@ def make_base_create_command(
        + (('--nobirthtime',) if config.get('birthtime') is False else ())
        + (('--read-special',) if config.get('read_special') or stream_processes else ())
        + noflags_flags
        + (('--files-changed', files_changed) if files_changed else ())
        + (('--files-cache', files_cache) if files_cache else ())
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (
            ('--list', '--filter', list_filter_flags)
            if config.get('list_details') and not json and not config.get('progress')
            else ()
        )
        + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
    )
    create_positional_arguments = flags.make_repository_archive_flags(
-        repository_path, archive_name_format, local_borg_version
+        repository_path,
        archive_name_format,
        local_borg_version,
    )
    working_directory = borgmatic.config.paths.get_working_directory(config)
    if config.get('unsafe_skip_path_validation_before_create'):
        logger.debug(
            'Skipping pre-backup path validation due to "unsafe_skip_path_validation_before_create" option.'
        )
        return (create_flags, create_positional_arguments, patterns_file)
    logger.debug('Checking file paths Borg plans to include')
    special_file_paths = validate_planned_backup_paths(
        dry_run,
        create_flags + create_positional_arguments,
        config,
        patterns,
        local_path,
        working_directory,
        borgmatic_runtime_directory=borgmatic_runtime_directory,
        find_special_files=bool(stream_processes),
    )
    if stream_processes and config.get('read_special') is False:
        logger.warning(
            'Ignoring configured "read_special" value of false, as true is needed for database hooks.',
        )
    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
-    if stream_processes and not config.get('read_special'):
+    if special_file_paths:
        truncated_special_file_paths = textwrap.shorten(
            ', '.join(special_file_paths),
            width=MAX_SPECIAL_FILE_PATHS_LENGTH,
            placeholder=' ...',
        )
        logger.warning(
-            'Ignoring configured "read_special" value of false, as true is needed for database hooks.'
+            f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
        )
-        working_directory = borgmatic.config.paths.get_working_directory(config)
+        patterns_file = borgmatic.borg.pattern.write_patterns_file(
-
+            tuple(
-        logger.debug('Collecting special file paths')
+                borgmatic.borg.pattern.Pattern(
-        special_file_paths = collect_special_file_paths(
+                    special_file_path,
-            dry_run,
+                    borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
-            create_flags + create_positional_arguments,
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
-            config,
+                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
-            local_path,
+                )
-            working_directory,
+                for special_file_path in special_file_paths
-            borgmatic_runtime_directory=borgmatic_runtime_directory,
+            ),
            borgmatic_runtime_directory,
            patterns_file=patterns_file,
        )
-        if special_file_paths:
+        if '--patterns-from' not in create_flags:
-            truncated_special_file_paths = textwrap.shorten(
+            create_flags += ('--patterns-from', patterns_file.name)
                ', '.join(special_file_paths),
                width=MAX_SPECIAL_FILE_PATHS_LENGTH,
                placeholder=' ...',
            )
            logger.warning(
                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}'
            )
            patterns_file = borgmatic.borg.pattern.write_patterns_file(
                tuple(
                    borgmatic.borg.pattern.Pattern(
                        special_file_path,
                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
                    )
                    for special_file_path in special_file_paths
                ),
                borgmatic_runtime_directory,
                patterns_file=patterns_file,
            )
            if '--patterns-from' not in create_flags:
                create_flags += ('--patterns-from', patterns_file.name)
    return (create_flags, create_positional_arguments, patterns_file)
@@ -269,15 +332,20 @@ def create_archive(
    local_borg_version,
    global_arguments,
    borgmatic_runtime_directory,
    archive_suffix='',
    local_path='borg',
    remote_path=None,
    json=False,
    comment=None,
    stream_processes=None,
 ):
    '''
    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, and global arguments as an
+    sequence of loaded configuration paths, the local Borg version, global arguments as an
-    argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
+    argparse.Namespace instance, the borgmatic runtime directory, a string suffix to add to the
    archive name, the local Borg path, the remote Borg path, whether to output JSON, and comment
    text to add to the created archive, and a sequence of processes streaming data to Borg, create a
    Borg archive and return Borg's JSON output (if any).
    If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
    create command while also triggering the given processes to produce output.
@@ -286,7 +354,7 @@ def create_archive(
    working_directory = borgmatic.config.paths.get_working_directory(config)
-    (create_flags, create_positional_arguments, patterns_file) = make_base_create_command(
+    (create_flags, create_positional_arguments, _) = make_base_create_command(
        dry_run,
        repository_path,
        config,
@@ -294,9 +362,11 @@ def create_archive(
        local_borg_version,
        global_arguments,
        borgmatic_runtime_directory,
        archive_suffix,
        local_path,
        remote_path,
        json,
        comment,
        stream_processes,
    )
@@ -321,31 +391,40 @@ def create_archive(
    borg_exit_codes = config.get('borg_exit_codes')
    if stream_processes:
-        return execute_command_with_processes(
+        output = '\n'.join(
-            create_flags + create_positional_arguments,
+            execute_command_with_processes(
-            stream_processes,
+                create_flags + create_positional_arguments,
-            output_log_level,
+                stream_processes,
-            output_file,
+                output_log_level,
-            working_directory=working_directory,
+                output_file,
-            environment=environment.make_environment(config),
+                working_directory=working_directory,
-            borg_local_path=local_path,
+                environment=environment.make_environment(config),
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
                borg_exit_codes=borg_exit_codes,
            )
        )
-    elif output_log_level is None:
+        return output if json else None
-        return execute_command_and_capture_output(
+
-            create_flags + create_positional_arguments,
+    if output_log_level is None:
-            working_directory=working_directory,
+        output = '\n'.join(
-            environment=environment.make_environment(config),
+            execute_command_and_capture_output(
-            borg_local_path=local_path,
+                create_flags + create_positional_arguments,
-            borg_exit_codes=borg_exit_codes,
+                working_directory=working_directory,
-        )
+                environment=environment.make_environment(config),
-    else:
+                borg_local_path=local_path,
-        execute_command(
+                borg_exit_codes=borg_exit_codes,
-            create_flags + create_positional_arguments,
+            )
            output_log_level,
            output_file,
            working_directory=working_directory,
            environment=environment.make_environment(config),
            borg_local_path=local_path,
            borg_exit_codes=borg_exit_codes,
        )
        return output if json else None
    execute_command(
        create_flags + create_positional_arguments,
        output_log_level,
        output_file,
        working_directory=working_directory,
        environment=environment.make_environment(config),
        borg_local_path=local_path,
        borg_exit_codes=borg_exit_codes,
    )
    return None
@@ -1,5 +1,6 @@
 import argparse
 import logging
 import shlex
 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -11,6 +12,9 @@ import borgmatic.execute
 logger = logging.getLogger(__name__)
 FORCE_HARDER_FLAG_COUNT = 2
 def make_delete_command(
    repository,
    config,
@@ -25,6 +29,8 @@ def make_delete_command(
    arguments to the delete action as an argparse.Namespace, and global arguments, return a command
    as a tuple to delete archives from the repository.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get('delete', '')
    return (
        (local_path, 'delete')
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -32,11 +38,14 @@ def make_delete_command(
        + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
        + (
-            (('--force',) + (('--force',) if delete_arguments.force >= 2 else ()))
+            (
                ('--force',)
                + (('--force',) if delete_arguments.force >= FORCE_HARDER_FLAG_COUNT else ())
            )
            if delete_arguments.force
            else ()
        )
@@ -60,6 +69,7 @@ def make_delete_command(
                'repository',
            ),
        )
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
    )
@@ -98,10 +108,11 @@ def delete_archives(
        for argument_name in ARCHIVE_RELATED_ARGUMENT_NAMES
    ):
        if borgmatic.borg.feature.available(
-            borgmatic.borg.feature.Feature.REPO_DELETE, local_borg_version
+            borgmatic.borg.feature.Feature.REPO_DELETE,
            local_borg_version,
        ):
            logger.warning(
-                'Deleting an entire repository with the delete action is deprecated when using Borg 2.x+. Use the repo-delete action instead.'
+                'Deleting an entire repository with the delete action is deprecated when using Borg 2.x+. Use the repo-delete action instead.',
            )
        repo_delete_arguments = argparse.Namespace(
@@ -0,0 +1,90 @@
 import logging
 import shlex
 import borgmatic.borg.environment
 import borgmatic.borg.feature
 import borgmatic.config.paths
 import borgmatic.execute
 from borgmatic.borg import flags
 from borgmatic.borg.pattern import write_patterns_file
 logger = logging.getLogger(__name__)
 def diff(
    repository,
    archive,
    second_archive,
    config,
    local_borg_version,
    diff_arguments,
    global_arguments,
    local_path,
    remote_path=None,
    patterns=None,
 ):
    '''
    Given a local or remote repository path, two archive names, a configuration dict, the local Borg
    version string, an argparse.Namespace of diff arguments, an argparse.Namespace of global
    arguments, optional local and remote Borg paths, executes the diff command with the given
    arguments.
    '''
    borgmatic.logger.add_custom_log_levels()
    lock_wait = config.get('lock_wait', None)
    exclude_flags = flags.make_exclude_flags(config)
    extra_borg_options = config.get('extra_borg_options', {}).get('diff', '')
    if diff_arguments.only_patterns:
        # Write patterns to a temporary file and use that file with --patterns-from.
        patterns_file = write_patterns_file(
            patterns,
            borgmatic.config.paths.get_working_directory(config),
        )
    else:
        patterns_file = None
    if borgmatic.borg.feature.available(
        borgmatic.borg.feature.Feature.NUMERIC_IDS, local_borg_version
    ):
        numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
    else:
        numeric_ids_flags = ('--numeric-owner',) if config.get('numeric_ids') else ()
    diff_command = (
        (local_path, 'diff')
        + (('--remote-path', remote_path) if remote_path else ())
        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (
            ('--patterns-from', patterns_file.name)
            if patterns_file and diff_arguments.only_patterns
            else ()
        )
        + exclude_flags
        + numeric_ids_flags
        + (('--same-chunker-params',) if diff_arguments.same_chunker_params else ())
        + (('--sort-by', ','.join(diff_arguments.sort_keys)) if diff_arguments.sort_keys else ())
        + (('--content-only',) if diff_arguments.content_only else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            (*flags.make_repository_flags(repository, local_borg_version), archive)
            if borgmatic.borg.feature.available(
                borgmatic.borg.feature.Feature.SEPARATE_REPOSITORY_ARCHIVE,
                local_borg_version,
            )
            else flags.make_repository_archive_flags(repository, archive, local_borg_version)
        )
        + (second_archive,)
    )
    borgmatic.execute.execute_command(
        full_command=diff_command,
        output_log_level=logging.ANSWER,
        environment=borgmatic.borg.environment.make_environment(config),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
        borg_exit_codes=config.get('borg_exit_codes'),
    )
@@ -10,17 +10,22 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
    'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
    'borg_security_directory': 'BORG_SECURITY_DIR',
    'borg_keys_directory': 'BORG_KEYS_DIR',
    'borg_key_file': 'BORG_KEY_FILE',
    'ssh_command': 'BORG_RSH',
    'temporary_directory': 'TMPDIR',
 }
-DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
+DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE = {
-    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
+    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
 }
-DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
+DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
-    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
+    'debug_passphrase': 'BORG_DEBUG_PASSPHRASE',
    'display_passphrase': 'BORG_DISPLAY_PASSPHRASE',
    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
    'msgpack_version_check': 'BORG_MSGPACK_VERSION_CHECK',
 }
@@ -61,7 +66,8 @@ def make_environment(config):
        environment.pop('BORG_PASSCOMMAND', None)
    passphrase = borgmatic.hooks.credential.parse.resolve_credential(
-        config.get('encryption_passphrase'), config
+        config.get('encryption_passphrase'),
        config,
    )
    if passphrase is None:
@@ -82,15 +88,16 @@ def make_environment(config):
    for (
        option_name,
        environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
        if os.environ.get(environment_variable_name) is None:
            value = config.get(option_name)
-            environment[environment_variable_name] = 'yes' if value else 'no'
+            if value is not None:
                environment[environment_variable_name] = 'YES' if value else 'NO'
    for (
        option_name,
        environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE.items():
        value = config.get(option_name)
        if value is not None:
            environment[environment_variable_name] = 'YES' if value else 'NO'
@@ -1,5 +1,6 @@
 import logging
 import os
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -31,11 +32,12 @@ def export_key(
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    working_directory = borgmatic.config.paths.get_working_directory(config)
    extra_borg_options = config.get('extra_borg_options', {}).get('key_export', '')
    if export_arguments.path and export_arguments.path != '-':
        if os.path.exists(os.path.join(working_directory or '', export_arguments.path)):
            raise FileExistsError(
-                f'Destination path {export_arguments.path} already exists. Aborting.'
+                f'Destination path {export_arguments.path} already exists. Aborting.',
            )
        output_file = None
@@ -46,12 +48,13 @@ def export_key(
        (local_path, 'key', 'export')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if output_file is None else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('paper', export_arguments.paper)
        + flags.make_flags('qr-html', export_arguments.qr_html)
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -34,12 +35,13 @@ def export_tar_archive(
    borgmatic.logger.add_custom_log_levels()
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    extra_borg_options = config.get('extra_borg_options', {}).get('export_tar', '')
    full_command = (
        (local_path, 'export-tar')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if destination_path != '-' else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--list',) if config.get('list_details') else ())
@@ -47,6 +49,7 @@ def export_tar_archive(
        + (('--dry-run',) if dry_run else ())
        + (('--tar-filter', tar_filter) if tar_filter else ())
        + (('--strip-components', str(strip_components)) if strip_components else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
            repository_path,
            archive,
@@ -56,13 +59,10 @@ def export_tar_archive(
        + (tuple(paths) if paths else ())
    )
-    if config.get('list_details'):
+    output_log_level = logging.ANSWER if config.get('list_details') else logging.INFO
        output_log_level = logging.ANSWER
    else:
        output_log_level = logging.INFO
    if dry_run:
-        logging.info('Skipping export to tar file (dry run)')
+        logger.info('Skipping export to tar file (dry run)')
        return
    execute_command(
@@ -1,5 +1,6 @@
 import logging
 import os
 import shlex
 import subprocess
 import borgmatic.config.paths
@@ -23,6 +24,7 @@ def extract_last_archive_dry_run(
    Perform an extraction dry-run of the most recent archive. If there are no archives, skip the
    dry-run.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
    verbosity_flags = ()
    if logger.isEnabledFor(logging.DEBUG):
        verbosity_flags = ('--debug', '--show-rc')
@@ -47,22 +49,36 @@ def extract_last_archive_dry_run(
    full_extract_command = (
        (local_path, 'extract', '--dry-run')
        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not config.get('progress') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + verbosity_flags
        + (('--progress',) if config.get('progress') else ())
        + list_flag
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
-            repository_path, last_archive_name, local_borg_version
+            repository_path,
            last_archive_name,
            local_borg_version,
        )
    )
-    execute_command(
+    if config.get('progress'):
-        full_extract_command,
+        execute_command(
-        environment=environment.make_environment(config),
+            full_extract_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            output_file=DO_NOT_CAPTURE,
-        borg_local_path=local_path,
+            environment=environment.make_environment(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-    )
+            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
    else:
        execute_command(
            full_extract_command,
            environment=environment.make_environment(config),
            working_directory=borgmatic.config.paths.get_working_directory(config),
            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
 def extract_archive(
@@ -90,9 +106,7 @@ def extract_archive(
    '''
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
-
+    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
    if config.get('progress') and extract_to_stdout:
        raise ValueError('progress and extract to stdout cannot both be set')
    if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
        numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
@@ -121,14 +135,15 @@ def extract_archive(
        + (('--remote-path', remote_path) if remote_path else ())
        + numeric_ids_flags
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (('--dry-run',) if dry_run else ())
        + (('--strip-components', str(strip_components)) if strip_components else ())
-        + (('--progress',) if config.get('progress') else ())
+        + (('--progress',) if config.get('progress') and not extract_to_stdout else ())
        + (('--stdout',) if extract_to_stdout else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
            # Make the repository path absolute so the destination directory used below via changing
            # the working directory doesn't prevent Borg from finding the repo. But also apply the
@@ -145,6 +160,17 @@ def extract_archive(
        os.path.join(working_directory or '', destination_path) if destination_path else None
    )
    if extract_to_stdout:
        return execute_command(
            full_command,
            output_file=subprocess.PIPE,
            run_to_completion=False,
            environment=environment.make_environment(config),
            working_directory=full_destination_path,
            borg_local_path=local_path,
            borg_exit_codes=borg_exit_codes,
        )
    # The progress output isn't compatible with captured and logged output, as progress messes with
    # the terminal directly.
    if config.get('progress'):
@@ -158,17 +184,6 @@ def extract_archive(
        )
        return None
    if extract_to_stdout:
        return execute_command(
            full_command,
            output_file=subprocess.PIPE,
            run_to_completion=False,
            environment=environment.make_environment(config),
            working_directory=full_destination_path,
            borg_local_path=local_path,
            borg_exit_codes=borg_exit_codes,
        )
    # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
    # if the restore paths don't exist in the archive.
    execute_command(
@@ -178,3 +193,5 @@ def extract_archive(
        borg_local_path=local_path,
        borg_exit_codes=borg_exit_codes,
    )
    return None
@@ -18,6 +18,7 @@ class Feature(Enum):
    EXCLUDED_FILES_MINUS = 12
    ARCHIVE_SERIES = 13
    NO_PRUNE_STATS = 14
    DRY_RUN_COMPACT = 15
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -35,6 +36,7 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
    Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
    Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
    Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
    Feature.DRY_RUN_COMPACT: parse('1.2.9'),  # borg compact --dry-run support
 }
@@ -1,5 +1,4 @@
 import itertools
 import json
 import logging
 import re
@@ -34,7 +33,7 @@ def make_flags_from_arguments(arguments, excludes=()):
            make_flags(name, value=getattr(arguments, name))
            for name in sorted(vars(arguments))
            if name not in excludes and not name.startswith('_')
-        )
+        ),
    )
@@ -50,7 +49,7 @@ def make_repository_flags(repository_path, local_borg_version):
    ) + (repository_path,)
-ARCHIVE_HASH_PATTERN = re.compile('[0-9a-fA-F]{8,}$')
+ARCHIVE_HASH_PATTERN = re.compile(r'[0-9a-fA-F]{8,}$')
 def make_repository_archive_flags(repository_path, archive, local_borg_version):
@@ -76,8 +75,8 @@ def make_repository_archive_flags(repository_path, archive, local_borg_version):
    )
-DEFAULT_ARCHIVE_NAME_FORMAT_WITHOUT_SERIES = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'  # noqa: FS003
+DEFAULT_ARCHIVE_NAME_FORMAT_WITHOUT_SERIES = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
-DEFAULT_ARCHIVE_NAME_FORMAT_WITH_SERIES = '{hostname}'  # noqa: FS003
+DEFAULT_ARCHIVE_NAME_FORMAT_WITH_SERIES = '{hostname}'
 def get_default_archive_name_format(local_borg_version):
@@ -90,7 +89,7 @@ def get_default_archive_name_format(local_borg_version):
    return DEFAULT_ARCHIVE_NAME_FORMAT_WITHOUT_SERIES
-def make_match_archives_flags(
+def make_match_archives_flags(  # noqa: PLR0911
    match_archives,
    archive_name_format,
    local_borg_version,
@@ -115,8 +114,8 @@ def make_match_archives_flags(
                return ('--match-archives', f'aid:{match_archives}')
            return ('--match-archives', match_archives)
-        else:
+
-            return ('--glob-archives', re.sub(r'^sh:', '', match_archives))
+        return ('--glob-archives', re.sub(r'^sh:', '', match_archives))
    derived_match_archives = re.sub(
        r'\{(now|utcnow|pid)([:%\w\.-]*)\}',
@@ -131,31 +130,22 @@ def make_match_archives_flags(
    if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
        return ('--match-archives', f'sh:{derived_match_archives}')
-    else:
+
-        return ('--glob-archives', f'{derived_match_archives}')
+    return ('--glob-archives', f'{derived_match_archives}')
-def warn_for_aggressive_archive_flags(json_command, json_output):
+def warn_for_aggressive_archive_flags(command, output_lines):
    '''
-    Given a JSON archives command and the resulting JSON string output from running it, parse the
+    Given an archives command and the resulting output lines from running it, warn if the command
-    JSON and warn if the command used an archive flag but the output indicates zero archives were
+    used an archive flag but the output indicates zero archives were found.
    found.
    '''
-    archive_flags_used = {'--glob-archives', '--match-archives'}.intersection(set(json_command))
+    if {'--glob-archives', '--match-archives'}.intersection(set(command)) and len(
-
+        tuple(line for line in output_lines if not line.startswith('terminating with '))
-    if not archive_flags_used:
+    ) == 0:
-        return
+        logger.warning('An archive filter was applied, but no matching archives were found.')
-
+        logger.warning(
-    try:
+            'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.',
-        if len(json.loads(json_output)['archives']) == 0:
+        )
            logger.warning('An archive filter was applied, but no matching archives were found.')
            logger.warning(
                'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.'
            )
    except json.JSONDecodeError as error:
        logger.debug(f'Cannot parse JSON output from archive command: {error}')
    except (TypeError, KeyError):
        logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
 def omit_flag(arguments, flag):
@@ -193,8 +183,8 @@ def omit_flag_and_value(arguments, flag):
    # its value.
    return tuple(
        argument
-        for (previous_argument, argument) in zip((None,) + arguments, arguments)
+        for (previous_argument, argument) in zip((None, *arguments), arguments)
-        if flag not in (previous_argument, argument)
+        if flag not in {previous_argument, argument}
        if not argument.startswith(f'{flag}=')
    )
@@ -209,7 +199,7 @@ def make_exclude_flags(config):
        itertools.chain.from_iterable(
            ('--exclude-if-present', if_present)
            for if_present in config.get('exclude_if_present', ())
-        )
+        ),
    )
    keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
    exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
@@ -229,10 +219,10 @@ def make_list_filter_flags(local_borg_version, dry_run):
    if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
        if show_excludes or dry_run:
            return f'{base_flags}+-'
-        else:
+
-            return base_flags
+        return base_flags
    if show_excludes:
        return f'{base_flags}x-'
-    else:
+
-        return f'{base_flags}-'
+    return f'{base_flags}-'
@@ -1,10 +1,10 @@
 import logging
 import os
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, flags
-from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+from borgmatic.execute import execute_command
 logger = logging.getLogger(__name__)
@@ -30,29 +30,30 @@ def import_key(
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    working_directory = borgmatic.config.paths.get_working_directory(config)
    extra_borg_options = config.get('extra_borg_options', {}).get('key_import', '')
-    if import_arguments.path and import_arguments.path != '-':
+    if (
-        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+        import_arguments.path
-            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+        and import_arguments.path != '-'
-
+        and not os.path.exists(os.path.join(working_directory or '', import_arguments.path))
-        input_file = None
+    ):
-    else:
+        raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
        input_file = DO_NOT_CAPTURE
    full_command = (
        (local_path, 'key', 'import')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('paper', import_arguments.paper)
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
        )
-        + ((import_arguments.path,) if input_file is None else ())
+        + (import_arguments.path or '-',)
    )
    if global_arguments.dry_run:
@@ -61,7 +62,6 @@ def import_key(
    execute_command(
        full_command,
        input_file=input_file,
        output_log_level=logging.INFO,
        environment=environment.make_environment(config),
        working_directory=working_directory,
@@ -1,10 +1,11 @@
 import argparse
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
-from borgmatic.execute import execute_command, execute_command_and_capture_output
+from borgmatic.execute import execute_command_and_capture_output
 logger = logging.getLogger(__name__)
@@ -23,6 +24,8 @@ def make_info_command(
    arguments to the info action as an argparse.Namespace, and global arguments, return a command
    as a tuple to display summary information for archives in the repository.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get('info', '')
    return (
        (local_path, 'info')
        + (
@@ -37,7 +40,7 @@ def make_info_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + (
            (
@@ -55,8 +58,10 @@ def make_info_command(
            )
        )
        + flags.make_flags_from_arguments(
-            info_arguments, excludes=('repository', 'archive', 'prefix', 'match_archives')
+            info_arguments,
            excludes=('repository', 'archive', 'prefix', 'match_archives'),
        )
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
@@ -98,24 +103,28 @@ def display_archives_info(
    borg_exit_codes = config.get('borg_exit_codes')
    working_directory = borgmatic.config.paths.get_working_directory(config)
    json_info = execute_command_and_capture_output(
        json_command,
        environment=environment.make_environment(config),
        working_directory=working_directory,
        borg_local_path=local_path,
        borg_exit_codes=borg_exit_codes,
    )
    if info_arguments.json:
-        return json_info
+        return '\n'.join(
            execute_command_and_capture_output(
                json_command,
                environment=environment.make_environment(config),
                working_directory=working_directory,
                borg_local_path=local_path,
                borg_exit_codes=borg_exit_codes,
            )
        )
-    flags.warn_for_aggressive_archive_flags(json_command, json_info)
+    output_lines = tuple(
-
+        execute_command_and_capture_output(
-    execute_command(
+            main_command,
-        main_command,
+            output_log_level=logging.ANSWER,
-        output_log_level=logging.ANSWER,
+            environment=environment.make_environment(config),
-        environment=environment.make_environment(config),
+            working_directory=working_directory,
-        working_directory=working_directory,
+            borg_local_path=local_path,
-        borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
-        borg_exit_codes=borg_exit_codes,
+        )
    )
    flags.warn_for_aggressive_archive_flags(main_command, output_lines)
    return None
@@ -1,7 +1,9 @@
 import argparse
 import copy
 import json
 import logging
 import re
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -17,7 +19,10 @@ MAKE_FLAGS_EXCLUDES = (
    'archive',
    'paths',
    'find_paths',
-) + ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST
+    'format',
    'json',
    *ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST,
 )
 def make_list_command(
@@ -34,6 +39,8 @@ def make_list_command(
    and local and remote Borg paths, return a command as a tuple to list archives or paths within an
    archive.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get('list', '')
    return (
        (local_path, 'list')
        + (
@@ -48,12 +55,17 @@ def make_list_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + flags.make_flags('json-lines', list_arguments.json)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + flags.make_flags('format', list_arguments.format or config.get('file_list_format'))
        + flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            flags.make_repository_archive_flags(
-                repository_path, list_arguments.archive, local_borg_version
+                repository_path,
                list_arguments.archive,
                local_borg_version,
            )
            if list_arguments.archive
            else flags.make_repository_flags(repository_path, local_borg_version)
@@ -100,14 +112,15 @@ def capture_archive_listing(
    remote_path=None,
 ):
    '''
-    Given a local or remote repository path, an archive name, a configuration
+    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
-    dict, the local Borg version, global arguments as an argparse.Namespace,
+    version, global arguments as an argparse.Namespace, the archive paths (or Borg patterns) in
-    the archive paths (or Borg patterns) in which to list files, the Borg path
+    which to list files, the Borg path format indicating keys to include in the output, and local
-    format to use for the output, and local and remote Borg paths, capture the
+    and remote Borg paths, capture the output of listing that archive and return it as a sequence of
-    output of listing that archive and return it as a list of file paths.
+    dicts, one per path.
    '''
    return tuple(
-        execute_command_and_capture_output(
+        json.loads(entry)
        for entry in execute_command_and_capture_output(
            make_list_command(
                repository_path,
                config,
@@ -115,10 +128,10 @@ def capture_archive_listing(
                argparse.Namespace(
                    repository=repository_path,
                    archive=archive,
-                    paths=[path for path in list_paths] if list_paths else None,
+                    paths=list(list_paths) if list_paths else None,
                    find_paths=None,
-                    json=None,
+                    json=True,
-                    format=path_format or '{path}{NUL}',  # noqa: FS003
+                    format=path_format or None,
                ),
                global_arguments,
                local_path,
@@ -129,8 +142,6 @@ def capture_archive_listing(
            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
        .strip('\0')
        .split('\0')
    )
@@ -156,7 +167,7 @@ def list_archive(
    if not list_arguments.archive and not list_arguments.find_paths:
        if feature.available(feature.Feature.REPO_LIST, local_borg_version):
            logger.warning(
-                'Omitting the --archive flag on the list action is deprecated when using Borg 2.x+. Use the repo-list action instead.'
+                'Omitting the --archive flag on the list action is deprecated when using Borg 2.x+. Use the repo-list action instead.',
            )
        repo_list_arguments = argparse.Namespace(
@@ -184,14 +195,9 @@ def list_archive(
        for name in ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST:
            if getattr(list_arguments, name, None):
                logger.warning(
-                    f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag."
+                    f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag.",
                )
    if list_arguments.json:
        raise ValueError(
            'The --json flag on the list action is not supported when using the --archive/--find flags.'
        )
    borg_exit_codes = config.get('borg_exit_codes')
    # If there are any paths to find (and there's not a single archive already selected), start by
@@ -210,24 +216,20 @@ def list_archive(
        )
        # Ask Borg to list archives. Capture its output for use below.
-        archive_lines = tuple(
+        archive_lines = execute_command_and_capture_output(
-            execute_command_and_capture_output(
+            repo_list.make_repo_list_command(
-                repo_list.make_repo_list_command(
+                repository_path,
-                    repository_path,
+                config,
-                    config,
+                local_borg_version,
-                    local_borg_version,
+                repo_list_arguments,
-                    repo_list_arguments,
+                global_arguments,
-                    global_arguments,
+                local_path,
-                    local_path,
+                remote_path,
-                    remote_path,
+            ),
-                ),
+            environment=environment.make_environment(config),
-                environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-                working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
-                borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
                borg_exit_codes=borg_exit_codes,
            )
            .strip('\n')
            .splitlines()
        )
    else:
        archive_lines = (list_arguments.archive,)
@@ -262,3 +264,5 @@ def list_archive(
            borg_local_path=local_path,
            borg_exit_codes=borg_exit_codes,
        )
    return None
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags
@@ -25,12 +26,13 @@ def mount_archive(
    '''
    umask = config.get('umask', None)
    lock_wait = config.get('lock_wait', None)
    extra_borg_options = config.get('extra_borg_options', {}).get('mount', '')
    full_command = (
        (local_path, 'mount')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not mount_arguments.foreground else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -39,6 +41,7 @@ def mount_archive(
            excludes=('repository', 'archive', 'mount_point', 'paths', 'options'),
        )
        + (('-o', mount_arguments.options) if mount_arguments.options else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            (
                flags.make_repository_flags(repository_path, local_borg_version)
@@ -17,9 +17,11 @@ def run_passcommand(passcommand, working_directory):
    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
    per borgmatic invocation.
    '''
-    return borgmatic.execute.execute_command_and_capture_output(
+    return '\n'.join(
-        shlex.split(passcommand),
+        borgmatic.execute.execute_command_and_capture_output(
-        working_directory=working_directory,
+            shlex.split(passcommand),
            working_directory=working_directory,
        )
    )
@@ -4,8 +4,6 @@ import logging
 import os
 import tempfile
 import borgmatic.borg.pattern
 logger = logging.getLogger(__name__)
@@ -59,9 +57,9 @@ Pattern = collections.namedtuple(
 def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
    '''
-    Given a sequence of patterns as borgmatic.borg.pattern.Pattern instances, write them to a named
+    Given a sequence of patterns as Pattern instances, write them to a named temporary file in the
-    temporary file in the given borgmatic runtime directory and return the file object so it can
+    given borgmatic runtime directory and return the file object so it can continue to exist on disk
-    continue to exist on disk as long as the caller needs it.
+    as long as the caller needs it.
    If an optional open pattern file is given, append to it instead of making a new temporary file.
    Return None if no patterns are provided.
@@ -70,7 +68,9 @@ def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=Non
        return None
    if patterns_file is None:
-        patterns_file = tempfile.NamedTemporaryFile('w', dir=borgmatic_runtime_directory)
+        patterns_file = tempfile.NamedTemporaryFile(
            'w', dir=borgmatic_runtime_directory, encoding='utf-8'
        )
        operation_name = 'Writing'
    else:
        patterns_file.write('\n')
@@ -90,17 +90,17 @@ def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=Non
 def check_all_root_patterns_exist(patterns):
    '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances, check that all root pattern
+    Given a sequence of Pattern instances, check that all root pattern paths exist. If any don't,
-    paths exist. If any don't, raise an exception.
+    raise an exception.
    '''
    missing_paths = [
        pattern.path
        for pattern in patterns
-        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        if pattern.type == Pattern_type.ROOT
        if not os.path.exists(pattern.path)
    ]
    if missing_paths:
        raise ValueError(
-            f"Source directories or root pattern paths do not exist: {', '.join(missing_paths)}"
+            f"Source directories or root pattern paths do not exist: {', '.join(missing_paths)}",
        )
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -71,9 +72,9 @@ def prune_archives(
    full_command = (
        (local_path, 'prune')
        + make_prune_flags(config, prune_arguments, local_borg_version)
        + ('--log-json',)
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (
            ('--stats',)
@@ -90,7 +91,7 @@ def prune_archives(
        + (('--list',) if config.get('list_details') else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
@@ -32,19 +32,21 @@ def recreate_archive(
    exclude_flags = flags.make_exclude_flags(config)
    compression = config.get('compression', None)
    chunker_params = config.get('chunker_params', None)
    extra_borg_options = config.get('extra_borg_options', {}).get('recreate', '')
    # Available recompress MODES: "if-different", "always", "never" (default)
    recompress = config.get('recompress', None)
    # Write patterns to a temporary file and use that file with --patterns-from.
    patterns_file = write_patterns_file(
-        patterns, borgmatic.config.paths.get_working_directory(config)
+        patterns,
        borgmatic.config.paths.get_working_directory(config),
    )
    recreate_command = (
        (local_path, 'recreate')
        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -70,6 +72,8 @@ def recreate_archive(
        + (('--chunker-params', chunker_params) if chunker_params else ())
        + (('--recompress', recompress) if recompress else ())
        + exclude_flags
        + (('--dry-run',) if global_arguments.dry_run else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            (
                flags.make_repository_flags(repository, local_borg_version)
@@ -80,7 +84,8 @@ def recreate_archive(
                )
            )
            if borgmatic.borg.feature.available(
-                borgmatic.borg.feature.Feature.SEPARATE_REPOSITORY_ARCHIVE, local_borg_version
+                borgmatic.borg.feature.Feature.SEPARATE_REPOSITORY_ARCHIVE,
                local_borg_version,
            )
            else (
                flags.make_repository_archive_flags(repository, archive, local_borg_version)
@@ -90,10 +95,6 @@ def recreate_archive(
        )
    )
    if global_arguments.dry_run:
        logger.info('Skipping the archive recreation (dry run)')
        return
    borgmatic.execute.execute_command(
        full_command=recreate_command,
        output_log_level=logging.INFO,
@@ -0,0 +1,69 @@
 import logging
 import shlex
 import borgmatic.borg.environment
 import borgmatic.borg.flags
 logger = logging.getLogger(__name__)
 def make_rename_command(
    dry_run,
    repository_name,
    old_archive_name,
    new_archive_name,
    config,
    local_borg_version,
    local_path,
    remote_path,
 ):
    extra_borg_options = config.get('extra_borg_options', {}).get('rename', '')
    return (
        (local_path, 'rename')
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + borgmatic.borg.flags.make_flags('dry-run', dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
        + ('--log-json',)
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_archive_flags(
            repository_name,
            old_archive_name,
            local_borg_version,
        )
        + (new_archive_name,)
    )
 def rename_archive(
    repository_name,
    old_archive_name,
    new_archive_name,
    dry_run,
    config,
    local_borg_version,
    local_path,
    remote_path,
 ):
    command = make_rename_command(
        dry_run,
        repository_name,
        old_archive_name,
        new_archive_name,
        config,
        local_borg_version,
        local_path,
        remote_path,
    )
    borgmatic.execute.execute_command(
        command,
        output_log_level=logging.INFO,
        environment=borgmatic.borg.environment.make_environment(config),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
        borg_exit_codes=config.get('borg_exit_codes'),
    )
@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
 import shlex
 import subprocess
 import borgmatic.config.paths
@@ -48,13 +49,13 @@ def create_repository(
                global_arguments,
                local_path,
                remote_path,
-            )
+            ),
        )
        repository_encryption_mode = info_data.get('encryption', {}).get('mode')
        if repository_encryption_mode != encryption_mode:
            raise ValueError(
-                f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
+                f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"',
            )
        logger.info('Repository already exists. Skipping creation.')
@@ -65,8 +66,14 @@ def create_repository(
    lock_wait = config.get('lock_wait')
    umask = config.get('umask')
    extra_borg_options_from_init = config.get('extra_borg_options', {}).get('init', '')
    extra_borg_options = config.get('extra_borg_options', {}).get('repo-create', '')
    if extra_borg_options_from_init:
        logger.warning(
            'The "init" option in "extra_borg_options" is deprecated and will be removed from a future release; use "repo_create" instead.'
        )
    repo_create_command = (
        (local_path,)
        + (
@@ -82,16 +89,16 @@ def create_repository(
        + (('--make-parent-dirs',) if make_parent_directories else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (tuple(shlex.split(extra_borg_options_from_init)) if extra_borg_options_from_init else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
    if dry_run:
-        logging.info('Skipping repository creation (dry run)')
+        logger.info('Skipping repository creation (dry run)')
        return
    # Do not capture output here, so as to support interactive prompts.
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -9,6 +10,9 @@ import borgmatic.execute
 logger = logging.getLogger(__name__)
 FORCE_HARDER_FLAG_COUNT = 2
 def make_repo_delete_command(
    repository,
    config,
@@ -17,18 +21,30 @@ def make_repo_delete_command(
    global_arguments,
    local_path,
    remote_path,
    output_file,
 ):
    '''
    Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the repo_delete action as an argparse.Namespace, and global arguments, return a command
+    arguments to the repo_delete action as an argparse.Namespace, and global arguments, the Borg
-    as a tuple to repo_delete the entire repository.
+    local path, the Borg remote path, and an optional output file, return a command as a tuple to
    repo_delete the entire repository.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get(
        'repo_delete'
        if borgmatic.borg.feature.available(
            borgmatic.borg.feature.Feature.REPO_DELETE, local_borg_version
        )
        else 'delete',
        '',
    )
    return (
        (local_path,)
        + (
            ('repo-delete',)
            if borgmatic.borg.feature.available(
-                borgmatic.borg.feature.Feature.REPO_DELETE, local_borg_version
+                borgmatic.borg.feature.Feature.REPO_DELETE,
                local_borg_version,
            )
            else ('delete',)
        )
@@ -37,17 +53,22 @@ def make_repo_delete_command(
        + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if output_file is None else ())
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
        + (
-            (('--force',) + (('--force',) if repo_delete_arguments.force >= 2 else ()))
+            (
                ('--force',)
                + (('--force',) if repo_delete_arguments.force >= FORCE_HARDER_FLAG_COUNT else ())
            )
            if repo_delete_arguments.force
            else ()
        )
        + borgmatic.borg.flags.make_flags_from_arguments(
-            repo_delete_arguments, excludes=('list_details', 'force', 'repository')
+            repo_delete_arguments,
            excludes=('list_details', 'force', 'repository'),
        )
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
    )
@@ -68,6 +89,14 @@ def delete_repository(
    '''
    borgmatic.logger.add_custom_log_levels()
    # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
    # prompt won't work.
    output_file = (
        None
        if repo_delete_arguments.force or repo_delete_arguments.cache_only
        else borgmatic.execute.DO_NOT_CAPTURE
    )
    command = make_repo_delete_command(
        repository,
        config,
@@ -76,18 +105,13 @@ def delete_repository(
        global_arguments,
        local_path,
        remote_path,
        output_file,
    )
    borgmatic.execute.execute_command(
        command,
        output_log_level=logging.ANSWER,
-        # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+        output_file=output_file,
        # prompt won't work.
        output_file=(
            None
            if repo_delete_arguments.force or repo_delete_arguments.cache_only
            else borgmatic.execute.DO_NOT_CAPTURE
        ),
        environment=borgmatic.borg.environment.make_environment(config),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -24,6 +25,10 @@ def display_repository_info(
    '''
    borgmatic.logger.add_custom_log_levels()
    lock_wait = config.get('lock_wait', None)
    extra_borg_options = config.get('extra_borg_options', {}).get(
        'repo_info' if feature.available(feature.Feature.REPO_INFO, local_borg_version) else 'info',
        '',
    )
    full_command = (
        (local_path,)
@@ -32,6 +37,7 @@ def display_repository_info(
            if feature.available(feature.Feature.REPO_INFO, local_borg_version)
            else ('info',)
        )
        + (('--critical',) if repo_info_arguments.json else ())
        + (
            ('--info',)
            if logger.getEffectiveLevel() == logging.INFO and not repo_info_arguments.json
@@ -44,9 +50,10 @@ def display_repository_info(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
        + flags.make_flags('log-json', config.get('log_json'))
        + flags.make_flags('lock-wait', lock_wait)
        + ('--log-json',)
        + (('--json',) if repo_info_arguments.json else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
@@ -54,19 +61,23 @@ def display_repository_info(
    borg_exit_codes = config.get('borg_exit_codes')
    if repo_info_arguments.json:
-        return execute_command_and_capture_output(
+        return '\n'.join(
-            full_command,
+            execute_command_and_capture_output(
-            environment=environment.make_environment(config),
+                full_command,
-            working_directory=working_directory,
+                environment=environment.make_environment(config),
-            borg_local_path=local_path,
+                working_directory=working_directory,
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
-        )
+                borg_exit_codes=borg_exit_codes,
-    else:
+            )
        execute_command(
            full_command,
            output_log_level=logging.ANSWER,
            environment=environment.make_environment(config),
            working_directory=working_directory,
            borg_local_path=local_path,
            borg_exit_codes=borg_exit_codes,
        )
    execute_command(
        full_command,
        output_log_level=logging.ANSWER,
        environment=environment.make_environment(config),
        working_directory=working_directory,
        borg_local_path=local_path,
        borg_exit_codes=borg_exit_codes,
    )
    return None
@@ -1,10 +1,12 @@
 import argparse
 import json
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
-from borgmatic.execute import execute_command, execute_command_and_capture_output
+from borgmatic.execute import execute_command_and_capture_output
 logger = logging.getLogger(__name__)
@@ -22,49 +24,102 @@ def resolve_archive_name(
    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
    version, global arguments as an argparse.Namespace, a local Borg path, and a remote Borg path,
    return the archive name. But if the archive name is "latest", then instead introspect the
-    repository for the latest archive and return its name.
+    repository for the latest archive and return its name or ID, depending on whether the version of
    Borg in use supports archive series—different archives that share the same name but have unique
    IDs.
    Raise ValueError if "latest" is given but there are no archives in the repository.
    '''
    if archive != 'latest':
        return archive
-    full_command = (
+    latest_archive = get_latest_archive(
-        (
+        repository_path,
-            local_path,
+        config,
-            (
+        local_borg_version,
-                'repo-list'
+        global_arguments,
-                if feature.available(feature.Feature.REPO_LIST, local_borg_version)
+        local_path=local_path,
-                else 'list'
+        remote_path=remote_path,
            ),
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
        + flags.make_flags('log-json', config.get('log_json'))
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + flags.make_flags('last', 1)
        + ('--short',)
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
-    output = execute_command_and_capture_output(
+    return (
-        full_command,
+        latest_archive['id']
-        environment=environment.make_environment(config),
+        if feature.available(feature.Feature.ARCHIVE_SERIES, local_borg_version)
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+        else latest_archive['name']
        borg_local_path=local_path,
        borg_exit_codes=config.get('borg_exit_codes'),
    )
 def get_latest_archive(
    repository_path,
    config,
    local_borg_version,
    global_arguments,
    local_path='borg',
    remote_path=None,
    consider_checkpoints=False,
 ):
    '''
    Returns a dict with information about the latest archive of a repository.
    Raises ValueError if there are no archives in the repository.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get(
        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
        '',
    )
    full_command = (
        local_path,
        (
            'repo-list'
            if feature.available(feature.Feature.REPO_LIST, local_borg_version)
            else 'list'
        ),
        *flags.make_flags('remote-path', remote_path),
        *flags.make_flags('umask', config.get('umask')),
        *('--log-json',),
        *flags.make_flags('lock-wait', config.get('lock_wait')),
        *(
            flags.make_match_archives_flags(
                config.get('match_archives'),
                config.get('archive_name_format'),
                local_borg_version,
            )
        ),
        *(
            flags.make_flags('consider-checkpoints', consider_checkpoints)
            if not feature.available(feature.Feature.REPO_LIST, local_borg_version)
            else ()
        ),
        *flags.make_flags('last', 1),
        '--json',
        *(tuple(shlex.split(extra_borg_options)) if extra_borg_options else ()),
        *flags.make_repository_flags(repository_path, local_borg_version),
    )
    json_output = '\n'.join(
        execute_command_and_capture_output(
            full_command,
            environment=environment.make_environment(config),
            working_directory=borgmatic.config.paths.get_working_directory(config),
            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
    )
    archives = json.loads(json_output)['archives']
    try:
-        latest_archive = output.strip().splitlines()[-1]
+        latest_archive = archives[-1]
    except IndexError:
        raise ValueError('No archives found in the repository')
-    logger.debug(f'Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive["name"]} ({latest_archive["id"]})')
    return latest_archive
-MAKE_FLAGS_EXCLUDES = ('repository', 'prefix', 'match_archives')
+MAKE_FLAGS_EXCLUDES = ('repository', 'format', 'prefix', 'match_archives')
 def make_repo_list_command(
@@ -81,6 +136,11 @@ def make_repo_list_command(
    arguments to the repo_list action, global arguments as an argparse.Namespace instance, and local and
    remote Borg paths, return a command as a tuple to list archives with a repository.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get(
        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
        '',
    )
    return (
        (
            local_path,
@@ -102,7 +162,7 @@ def make_repo_list_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + (
            (
@@ -119,7 +179,11 @@ def make_repo_list_command(
                )
            )
        )
        + flags.make_flags(
            'format', repo_list_arguments.format or config.get('archive_list_format')
        )
        + flags.make_flags_from_arguments(repo_list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )
@@ -162,24 +226,28 @@ def list_repository(
    working_directory = borgmatic.config.paths.get_working_directory(config)
    borg_exit_codes = config.get('borg_exit_codes')
    json_listing = execute_command_and_capture_output(
        json_command,
        environment=environment.make_environment(config),
        working_directory=working_directory,
        borg_local_path=local_path,
        borg_exit_codes=borg_exit_codes,
    )
    if repo_list_arguments.json:
-        return json_listing
+        return '\n'.join(
            execute_command_and_capture_output(
                json_command,
                environment=environment.make_environment(config),
                working_directory=working_directory,
                borg_local_path=local_path,
                borg_exit_codes=borg_exit_codes,
            )
        )
-    flags.warn_for_aggressive_archive_flags(json_command, json_listing)
+    output_lines = tuple(
-
+        execute_command_and_capture_output(
-    execute_command(
+            main_command,
-        main_command,
+            output_log_level=logging.ANSWER,
-        output_log_level=logging.ANSWER,
+            environment=environment.make_environment(config),
-        environment=environment.make_environment(config),
+            working_directory=working_directory,
-        working_directory=working_directory,
+            borg_local_path=local_path,
-        borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
-        borg_exit_codes=borg_exit_codes,
+        )
    )
    flags.warn_for_aggressive_archive_flags(main_command, output_lines)
    return None
@@ -1 +1 @@
-DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'
+DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'  # pragma: nocover
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 import borgmatic.logger
@@ -24,6 +25,7 @@ def transfer_archives(
    instance, transfer archives to the given repository.
    '''
    borgmatic.logger.add_custom_log_levels()
    extra_borg_options = config.get('extra_borg_options', {}).get('transfer', '')
    full_command = (
        (local_path, 'transfer')
@@ -31,7 +33,7 @@ def transfer_archives(
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + flags.make_flags('progress', config.get('progress'))
        + (
@@ -53,6 +55,7 @@ def transfer_archives(
                )
            )
        )
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
        + flags.make_flags('other-repo', transfer_arguments.source_repository)
        + flags.make_flags('dry-run', dry_run)
@@ -1,4 +1,5 @@
 import logging
 import shlex
 import borgmatic.config.paths
 from borgmatic.execute import execute_command
@@ -11,10 +12,13 @@ def unmount_archive(config, mount_point, local_path='borg'):
    Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
    from the mount point.
    '''
    extra_borg_options = config.get('extra_borg_options', {}).get('umount', '')
    full_command = (
        (local_path, 'umount')
        + ('--log-json',)
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (mount_point,)
    )
@@ -16,15 +16,19 @@ def local_borg_version(config, local_path='borg'):
    '''
    full_command = (
        (local_path, '--version')
        + ('--log-json',)
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
    )
-    output = execute_command_and_capture_output(
+
-        full_command,
+    output = '\n'.join(
-        environment=environment.make_environment(config),
+        execute_command_and_capture_output(
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            full_command,
-        borg_local_path=local_path,
+            environment=environment.make_environment(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
    )
    try:
@@ -33,6 +33,7 @@ ACTION_ALIASES = {
    'key': [],
    'borg': [],
    'recreate': [],
    'diff': [],
 }
@@ -60,7 +61,7 @@ def get_subactions_for_actions(action_parsers):
        action: tuple(
            subaction_name
            for group_action in action_parser._subparsers._group_actions
-            for subaction_name in group_action.choices.keys()
+            for subaction_name in group_action.choices
        )
        for action, action_parser in action_parsers.items()
        if action_parser._subparsers
@@ -77,21 +78,25 @@ def omit_values_colliding_with_action_names(unparsed_arguments, parsed_arguments
    '''
    remaining_arguments = list(unparsed_arguments)
-    for action_name, parsed in parsed_arguments.items():
+    for parsed in parsed_arguments.values():
        for value in vars(parsed).values():
            if isinstance(value, str):
-                if value in ACTION_ALIASES.keys() and value in remaining_arguments:
+                if value in ACTION_ALIASES and value in remaining_arguments:
                    remaining_arguments.remove(value)
            elif isinstance(value, list):
                for item in value:
-                    if item in ACTION_ALIASES.keys() and item in remaining_arguments:
+                    if item in ACTION_ALIASES and item in remaining_arguments:
                        remaining_arguments.remove(item)
    return tuple(remaining_arguments)
 def parse_and_record_action_arguments(
-    unparsed_arguments, parsed_arguments, action_parser, action_name, canonical_name=None
+    unparsed_arguments,
    parsed_arguments,
    action_parser,
    action_name,
    canonical_name=None,
 ):
    '''
    Given unparsed arguments as a sequence of strings, parsed arguments as a dict from action name
@@ -102,7 +107,8 @@ def parse_and_record_action_arguments(
    given action doesn't apply to the given unparsed arguments.
    '''
    filtered_arguments = omit_values_colliding_with_action_names(
-        unparsed_arguments, parsed_arguments
+        unparsed_arguments,
        parsed_arguments,
    )
    if action_name not in filtered_arguments:
@@ -186,12 +192,12 @@ def get_unparsable_arguments(remaining_action_arguments):
        itertools.chain.from_iterable(
            argument_group
            for argument_group in dict.fromkeys(
-                itertools.chain.from_iterable(grouped_action_arguments)
+                itertools.chain.from_iterable(grouped_action_arguments),
-            ).keys()
+            )
            if all(
                argument_group in action_arguments for action_arguments in grouped_action_arguments
            )
-        )
+        ),
    )
@@ -244,7 +250,7 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
                            subaction_name,
                        )
                        if argument != action_name
-                    )
+                    ),
                )
                if subaction_name in arguments:
@@ -256,14 +262,18 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
                    sys.exit(0)
                else:
                    raise ValueError(
-                        f"Missing sub-action after {action_name} action. Expected one of: {', '.join(get_subactions_for_actions(action_parsers)[action_name])}"
+                        f"Missing sub-action after {action_name} action. Expected one of: {', '.join(get_subactions_for_actions(action_parsers)[action_name])}",
                    )
        # Otherwise, parse with the main action parser.
        else:
            remaining_action_arguments.append(
                parse_and_record_action_arguments(
-                    unparsed_arguments, arguments, action_parser, action_name, canonical_name
+                    unparsed_arguments,
-                )
+                    arguments,
                    action_parser,
                    action_name,
                    canonical_name,
                ),
            )
    # If no actions were explicitly requested, assume defaults.
@@ -272,11 +282,11 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
            default_action_parser = action_parsers[default_action_name]
            remaining_action_arguments.append(
                parse_and_record_action_arguments(
-                    tuple(unparsed_arguments) + (default_action_name,),
+                    (*unparsed_arguments, default_action_name),
                    arguments,
                    default_action_parser,
                    default_action_name,
-                )
+                ),
            )
    arguments['global'], remaining = global_parser.parse_known_args(unparsed_arguments)
@@ -288,7 +298,14 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
    )
-OMITTED_FLAG_NAMES = {'match-archives', 'progress', 'statistics', 'list-details'}
+OMITTED_FLAG_NAMES = {
    'match-archives',
    'progress',
    'statistics',
    'list-details',
    'file-list-format',
    'archive-list-format',
 }
 def make_argument_description(schema, flag_name):
@@ -304,10 +321,10 @@ def make_argument_description(schema, flag_name):
    if '[0]' in flag_name:
        pieces.append(
-            ' To specify a different list element, replace the "[0]" with another array index ("[1]", "[2]", etc.).'
+            ' To specify a different list element, replace the "[0]" with another array index ("[1]", "[2]", etc.).',
        )
-    if example and schema_type in ('array', 'object'):
+    if example and schema_type in ('array', 'object'):  # noqa: PLR6201
        example_buffer = io.StringIO()
        yaml = ruamel.yaml.YAML(typ='safe')
        yaml.default_flow_style = True
@@ -387,7 +404,7 @@ def add_array_element_arguments(arguments_group, unparsed_arguments, flag_name):
        if not pattern.match(unparsed_flag_name) or unparsed_flag_name == existing_flag_name:
            continue
-        if action_registry_name in ('store_true', 'store_false'):
+        if action_registry_name in {'store_true', 'store_false'}:
            arguments_group.add_argument(
                unparsed_flag_name,
                action=action_registry_name,
@@ -408,7 +425,7 @@ def add_array_element_arguments(arguments_group, unparsed_arguments, flag_name):
            )
-def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names=None):
+def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names=None):  # noqa: PLR0912
    '''
    Given an argparse._ArgumentGroup instance, a configuration schema dict, and a sequence of
    unparsed argument strings, convert the entire schema into corresponding command-line flags and
@@ -466,7 +483,10 @@ def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names
        if properties:
            for name, child in properties.items():
                add_arguments_from_schema(
-                    arguments_group, child, unparsed_arguments, names + (name,)
+                    arguments_group,
                    child,
                    unparsed_arguments,
                    (*names, name),
                )
            return
@@ -483,12 +503,15 @@ def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names
                    arguments_group,
                    child,
                    unparsed_arguments,
-                    names[:-1] + (f'{names[-1]}[0]',) + (name,),
+                    (*names[:-1], f'{names[-1]}[0]', name),
                )
        # If there aren't any children, then this is an array of scalars. Recurse accordingly.
        else:
            add_arguments_from_schema(
-                arguments_group, items, unparsed_arguments, names[:-1] + (f'{names[-1]}[0]',)
+                arguments_group,
                items,
                unparsed_arguments,
                (*names[:-1], f'{names[-1]}[0]'),
            )
    flag_name = '.'.join(names).replace('_', '-')
@@ -515,9 +538,9 @@ def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names
        )
        if names[-1].startswith('no_'):
-            no_flag_name = '.'.join(names[:-1] + (names[-1][len('no_') :],)).replace('_', '-')
+            no_flag_name = '.'.join((*names[:-1], names[-1][len('no_') :])).replace('_', '-')
        else:
-            no_flag_name = '.'.join(names[:-1] + ('no-' + names[-1],)).replace('_', '-')
+            no_flag_name = '.'.join((*names[:-1], 'no-' + names[-1])).replace('_', '-')
        arguments_group.add_argument(
            f'--{no_flag_name}',
@@ -545,7 +568,7 @@ def add_arguments_from_schema(arguments_group, schema, unparsed_arguments, names
    add_array_element_arguments(arguments_group, unparsed_arguments, flag_name)
-def make_parsers(schema, unparsed_arguments):
+def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
    '''
    Given a configuration schema dict and unparsed arguments as a sequence of strings, build a
    global arguments parser, individual action parsers, and a combined parser containing both.
@@ -553,8 +576,7 @@ def make_parsers(schema, unparsed_arguments):
    ignoring actions, and the combined parser is handy for displaying help that includes everything:
    global flags, a list of actions, etc.
    '''
-    config_paths = collect.get_default_config_paths(expand_home=True)
+    config_paths = collect.get_default_config_paths()
    unexpanded_config_paths = collect.get_default_config_paths(expand_home=False)
    # Using allow_abbrev=False here prevents the global parser from erroring about "ambiguous"
    # options like --encryption. Such options are intended for an action parser rather than the
@@ -567,7 +589,7 @@ def make_parsers(schema, unparsed_arguments):
        '--config',
        dest='config_paths',
        action='append',
-        help=f"Configuration filename or directory, can specify flag multiple times, defaults to: -c {' -c '.join(unexpanded_config_paths)}",
+        help='Configuration filename or directory, can specify flag multiple times, defaults to /etc/borgmatic/config.yaml, /etc/borgmatic.d, $XDG_CONFIG_HOME/borgmatic/config.yaml, and $XDG_CONFIG_HOME/borgmatic.d, where $XDG_CONFIG_HOME defaults to $HOME/.config',
    )
    global_group.add_argument(
        '-n',
@@ -626,8 +648,8 @@ def make_parsers(schema, unparsed_arguments):
    repo_create_parser = action_parsers.add_parser(
        'repo-create',
        aliases=ACTION_ALIASES['repo-create'],
-        help='Create a new, empty Borg repository',
+        help='Create a new, empty Borg repository (also known as "init")',
-        description='Create a new, empty Borg repository',
+        description='Create a new, empty Borg repository (also known as "init")',
        add_help=False,
    )
    repo_create_group = repo_create_parser.add_argument_group('repo-create arguments')
@@ -667,10 +689,13 @@ def make_parsers(schema, unparsed_arguments):
        dest='make_parent_directories',
        default=None,
        action='store_true',
-        help='Create any missing parent directories of the repository directory',
+        help='Create any missing parent directories of the repository directory [Borg 1.x only]',
    )
    repo_create_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    transfer_parser = action_parsers.add_parser(
@@ -712,7 +737,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Only transfer archives with names, hashes, or series matching this pattern',
    )
    transfer_group.add_argument(
-        '--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
+        '--sort-by',
        metavar='KEYS',
        help='Comma-separated list of sorting keys',
    )
    transfer_group.add_argument(
        '--first',
@@ -720,7 +747,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Only transfer first N archives after other filters are applied',
    )
    transfer_group.add_argument(
-        '--last', metavar='N', help='Only transfer last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='Only transfer last N archives after other filters are applied',
    )
    transfer_group.add_argument(
        '--oldest',
@@ -743,7 +772,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Transfer archives that are newer than the specified time range (e.g. 7d or 12m) from the current time [Borg 2.x+ only]',
    )
    transfer_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    prune_parser = action_parsers.add_parser(
@@ -833,7 +865,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Minimum saved space percentage threshold for compacting a segment, defaults to 10',
    )
    compact_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    create_parser = action_parsers.add_parser(
@@ -870,7 +905,16 @@ def make_parsers(schema, unparsed_arguments):
        help='Show per-file details',
    )
    create_group.add_argument(
-        '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
+        '--json',
        dest='json',
        default=False,
        action='store_true',
        help='Output results as JSON',
    )
    create_group.add_argument(
        '--comment',
        metavar='COMMENT',
        help='Add a comment text to the archive',
    )
    create_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
@@ -991,13 +1035,19 @@ def make_parsers(schema, unparsed_arguments):
        help='Only delete archives with names, hashes, or series matching this pattern',
    )
    delete_group.add_argument(
-        '--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
+        '--sort-by',
        metavar='KEYS',
        help='Comma-separated list of sorting keys',
    )
    delete_group.add_argument(
-        '--first', metavar='N', help='Delete first N archives after other filters are applied'
+        '--first',
        metavar='N',
        help='Delete first N archives after other filters are applied',
    )
    delete_group.add_argument(
-        '--last', metavar='N', help='Delete last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='Delete last N archives after other filters are applied',
    )
    delete_group.add_argument(
        '--oldest',
@@ -1034,7 +1084,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to extract, defaults to the configured repository if there is only one, quoted globs supported',
    )
    extract_group.add_argument(
-        '--archive', help='Name or hash of a single archive to extract (or "latest")', required=True
+        '--archive',
        help='Name or hash of a single archive to extract (or "latest")',
        required=True,
    )
    extract_group.add_argument(
        '--path',
@@ -1063,7 +1115,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Display progress for each file as it is extracted',
    )
    extract_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    config_parser = action_parsers.add_parser(
@@ -1088,7 +1143,7 @@ def make_parsers(schema, unparsed_arguments):
        add_help=False,
    )
    config_bootstrap_group = config_bootstrap_parser.add_argument_group(
-        'config bootstrap arguments'
+        'config bootstrap arguments',
    )
    config_bootstrap_group.add_argument(
        '--repository',
@@ -1143,7 +1198,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Command to use instead of "ssh"',
    )
    config_bootstrap_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    config_generate_parser = config_parsers.add_parser(
@@ -1162,9 +1220,9 @@ def make_parsers(schema, unparsed_arguments):
    config_generate_group.add_argument(
        '-d',
        '--destination',
-        dest='destination_filename',
+        dest='destination_path',
        default=config_paths[0],
-        help=f'Destination configuration file, default: {unexpanded_config_paths[0]}',
+        help='Destination configuration file (or directory if using --split), default: /etc/borgmatic/config.yaml',
    )
    config_generate_group.add_argument(
        '--overwrite',
@@ -1173,7 +1231,15 @@ def make_parsers(schema, unparsed_arguments):
        help='Whether to overwrite any existing destination file, defaults to false',
    )
    config_generate_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '--split',
        action='store_true',
        help='Assuming the destination is a directory instead of a file, split the configuration into separate files within it, one per option, useful for documentation',
    )
    config_generate_group.add_argument(
        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    config_validate_parser = config_parsers.add_parser(
@@ -1190,7 +1256,33 @@ def make_parsers(schema, unparsed_arguments):
        help='Show the validated configuration after all include merging has occurred',
    )
    config_validate_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    config_show_parser = config_parsers.add_parser(
        'show',
        help='Show the computed configuration for each file specified with --config (see borgmatic --help)',
        description='Show the computed configuration for each file specified with --config (see borgmatic --help)',
        add_help=False,
    )
    config_show_group = config_show_parser.add_argument_group('config show arguments')
    config_show_group.add_argument(
        '--option',
        help='Show the value of a single named configuration option instead of the entire configuration',
    )
    config_show_group.add_argument(
        '--json',
        action='store_true',
        help='Show the configuration as JSON with one array element per configuration file',
    )
    config_show_group.add_argument(
        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    export_tar_parser = action_parsers.add_parser(
@@ -1206,7 +1298,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to export from, defaults to the configured repository if there is only one, quoted globs supported',
    )
    export_tar_group.add_argument(
-        '--archive', help='Name or hash of a single archive to export (or "latest")', required=True
+        '--archive',
        help='Name or hash of a single archive to export (or "latest")',
        required=True,
    )
    export_tar_group.add_argument(
        '--path',
@@ -1223,7 +1317,8 @@ def make_parsers(schema, unparsed_arguments):
        required=True,
    )
    export_tar_group.add_argument(
-        '--tar-filter', help='Name of filter program to pipe data through'
+        '--tar-filter',
        help='Name of filter program to pipe data through',
    )
    export_tar_group.add_argument(
        '--list',
@@ -1241,7 +1336,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Number of leading path components to remove from each exported path. Skip paths with fewer elements',
    )
    export_tar_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    mount_parser = action_parsers.add_parser(
@@ -1257,7 +1355,8 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to use, defaults to the configured repository if there is only one, quoted globs supported',
    )
    mount_group.add_argument(
-        '--archive', help='Name or hash of a single archive to mount (or "latest")'
+        '--archive',
        help='Name or hash of a single archive to mount (or "latest")',
    )
    mount_group.add_argument(
        '--mount-point',
@@ -1286,7 +1385,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Mount first N archives after other filters are applied',
    )
    mount_group.add_argument(
-        '--last', metavar='N', help='Mount last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='Mount last N archives after other filters are applied',
    )
    mount_group.add_argument(
        '--oldest',
@@ -1363,7 +1464,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Do not delete the local security info when deleting a repository',
    )
    repo_delete_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    restore_parser = action_parsers.add_parser(
@@ -1406,6 +1510,10 @@ def make_parsers(schema, unparsed_arguments):
        '--port',
        help='Database port to restore to. Defaults to the "restore_port" option in borgmatic\'s configuration',
    )
    restore_group.add_argument(
        '--container',
        help='Container to restore to. Defaults to the "restore_container" option in borgmatic\'s configuration',
    )
    restore_group.add_argument(
        '--username',
        help='Username with which to connect to the database. Defaults to the "restore_username" option in borgmatic\'s configuration',
@@ -1418,10 +1526,18 @@ def make_parsers(schema, unparsed_arguments):
        '--restore-path',
        help='Path to restore SQLite database dumps to. Defaults to the "restore_path" option in borgmatic\'s configuration',
    )
    restore_group.add_argument(
        '--original-label',
        help='The label where the dump to restore came from, only necessary if you need to disambiguate dumps',
    )
    restore_group.add_argument(
        '--original-hostname',
        help='The hostname where the dump to restore came from, only necessary if you need to disambiguate dumps',
    )
    restore_group.add_argument(
        '--original-container',
        help='The container where the dump to restore came from, only necessary if you need to disambiguate dumps',
    )
    restore_group.add_argument(
        '--original-port',
        type=int,
@@ -1432,7 +1548,10 @@ def make_parsers(schema, unparsed_arguments):
        help='The name of the data source hook for the dump to restore, only necessary if you need to disambiguate dumps',
    )
    restore_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    repo_list_parser = action_parsers.add_parser(
@@ -1448,14 +1567,22 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to list, defaults to the configured repositories, quoted globs supported',
    )
    repo_list_group.add_argument(
-        '--short', default=False, action='store_true', help='Output only archive names'
+        '--short',
        default=False,
        action='store_true',
        help='Output only archive names',
    )
-    repo_list_group.add_argument('--format', help='Format for archive listing')
+    repo_list_group.add_argument('--format', help='Borg format for the archive listing')
    repo_list_group.add_argument(
-        '--json', default=False, action='store_true', help='Output results as JSON'
+        '--json',
        default=False,
        action='store_true',
        help='Output results as JSON',
    )
    repo_list_group.add_argument(
-        '-P', '--prefix', help='Deprecated. Only list archive names starting with this prefix'
+        '-P',
        '--prefix',
        help='Deprecated. Only list archive names starting with this prefix',
    )
    repo_list_group.add_argument(
        '-a',
@@ -1465,13 +1592,19 @@ def make_parsers(schema, unparsed_arguments):
        help='Only list archive names, hashes, or series matching this pattern',
    )
    repo_list_group.add_argument(
-        '--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
+        '--sort-by',
        metavar='KEYS',
        help='Comma-separated list of sorting keys',
    )
    repo_list_group.add_argument(
-        '--first', metavar='N', help='List first N archives after other filters are applied'
+        '--first',
        metavar='N',
        help='List first N archives after other filters are applied',
    )
    repo_list_group.add_argument(
-        '--last', metavar='N', help='List last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='List last N archives after other filters are applied',
    )
    repo_list_group.add_argument(
        '--oldest',
@@ -1500,7 +1633,10 @@ def make_parsers(schema, unparsed_arguments):
        help="List only deleted archives that haven't yet been compacted [Borg 2.x+ only]",
    )
    repo_list_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    list_parser = action_parsers.add_parser(
@@ -1516,7 +1652,8 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository containing archive to list, defaults to the configured repositories, quoted globs supported',
    )
    list_group.add_argument(
-        '--archive', help='Name or hash of a single archive to list (or "latest")'
+        '--archive',
        help='Name or hash of a single archive to list (or "latest")',
    )
    list_group.add_argument(
        '--path',
@@ -1533,14 +1670,22 @@ def make_parsers(schema, unparsed_arguments):
        help='Partial path or pattern to search for and list across multiple archives, can specify flag multiple times',
    )
    list_group.add_argument(
-        '--short', default=False, action='store_true', help='Output only path names'
+        '--short',
        default=False,
        action='store_true',
        help='Output only path names',
    )
-    list_group.add_argument('--format', help='Format for file listing')
+    list_group.add_argument('--format', help='Borg format for the file listing')
    list_group.add_argument(
-        '--json', default=False, action='store_true', help='Output results as JSON'
+        '--json',
        default=False,
        action='store_true',
        help='Output results as JSON',
    )
    list_group.add_argument(
-        '-P', '--prefix', help='Deprecated. Only list archive names starting with this prefix'
+        '-P',
        '--prefix',
        help='Deprecated. Only list archive names starting with this prefix',
    )
    list_group.add_argument(
        '-a',
@@ -1550,19 +1695,30 @@ def make_parsers(schema, unparsed_arguments):
        help='Only list archive names matching this pattern',
    )
    list_group.add_argument(
-        '--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
+        '--sort-by',
        metavar='KEYS',
        help='Comma-separated list of sorting keys',
    )
    list_group.add_argument(
-        '--first', metavar='N', help='List first N archives after other filters are applied'
+        '--first',
        metavar='N',
        help='List first N archives after other filters are applied',
    )
    list_group.add_argument(
-        '--last', metavar='N', help='List last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='List last N archives after other filters are applied',
    )
    list_group.add_argument(
-        '-e', '--exclude', metavar='PATTERN', help='Exclude paths matching the pattern'
+        '-e',
        '--exclude',
        metavar='PATTERN',
        help='Exclude paths matching the pattern',
    )
    list_group.add_argument(
-        '--exclude-from', metavar='FILENAME', help='Exclude paths from exclude file, one per line'
+        '--exclude-from',
        metavar='FILENAME',
        help='Exclude paths from exclude file, one per line',
    )
    list_group.add_argument('--pattern', help='Include or exclude paths matching a pattern')
    list_group.add_argument(
@@ -1585,10 +1741,17 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to show info for, defaults to the configured repository if there is only one, quoted globs supported',
    )
    repo_info_group.add_argument(
-        '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
+        '--json',
        dest='json',
        default=False,
        action='store_true',
        help='Output results as JSON',
    )
    repo_info_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    info_parser = action_parsers.add_parser(
@@ -1604,10 +1767,15 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository containing archive to show info for, defaults to the configured repository if there is only one, quoted globs supported',
    )
    info_group.add_argument(
-        '--archive', help='Archive name, hash, or series to show info for (or "latest")'
+        '--archive',
        help='Archive name, hash, or series to show info for (or "latest")',
    )
    info_group.add_argument(
-        '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
+        '--json',
        dest='json',
        default=False,
        action='store_true',
        help='Output results as JSON',
    )
    info_group.add_argument(
        '-P',
@@ -1622,7 +1790,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Only show info for archive names, hashes, or series matching this pattern',
    )
    info_group.add_argument(
-        '--sort-by', metavar='KEYS', help='Comma-separated list of sorting keys'
+        '--sort-by',
        metavar='KEYS',
        help='Comma-separated list of sorting keys',
    )
    info_group.add_argument(
        '--first',
@@ -1630,7 +1800,9 @@ def make_parsers(schema, unparsed_arguments):
        help='Show info for first N archives after other filters are applied',
    )
    info_group.add_argument(
-        '--last', metavar='N', help='Show info for last N archives after other filters are applied'
+        '--last',
        metavar='N',
        help='Show info for last N archives after other filters are applied',
    )
    info_group.add_argument(
        '--oldest',
@@ -1667,7 +1839,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to break the lock for, defaults to the configured repository if there is only one, quoted globs supported',
    )
    break_lock_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    key_parser = action_parsers.add_parser(
@@ -1712,7 +1887,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Path to export the key to, defaults to stdout (but be careful about dirtying the output with --verbosity)',
    )
    key_export_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    key_import_parser = key_parsers.add_parser(
@@ -1737,7 +1915,10 @@ def make_parsers(schema, unparsed_arguments):
        help='Path to import the key from backup, defaults to stdin',
    )
    key_import_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    key_change_passphrase_parser = key_parsers.add_parser(
@@ -1747,14 +1928,17 @@ def make_parsers(schema, unparsed_arguments):
        add_help=False,
    )
    key_change_passphrase_group = key_change_passphrase_parser.add_argument_group(
-        'key change-passphrase arguments'
+        'key change-passphrase arguments',
    )
    key_change_passphrase_group.add_argument(
        '--repository',
        help='Path of repository to change the passphrase for, defaults to the configured repository if there is only one, quoted globs supported',
    )
    key_change_passphrase_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    recreate_parser = action_parsers.add_parser(
@@ -1771,7 +1955,7 @@ def make_parsers(schema, unparsed_arguments):
    )
    recreate_group.add_argument(
        '--archive',
-        help='Archive name, hash, or series to recreate',
+        help='Archive name, hash, or series to recreate, defaults to all archives in the repository (if specified), or all archives across all repositories',
    )
    recreate_group.add_argument(
        '--list',
@@ -1804,9 +1988,53 @@ def make_parsers(schema, unparsed_arguments):
        help='Only consider archive names, hashes, or series matching this pattern [Borg 2.x+ only]',
    )
    recreate_group.add_argument(
-        '-h', '--help', action='help', help='Show this help message and exit'
+        '-h',
        '--help',
        action='help',
        help='Show this help message and exit',
    )
    diff_parser = action_parsers.add_parser(
        'diff',
        aliases=ACTION_ALIASES['diff'],
        help='This command finds differences (file contents, user/group/mode) between archives',
        description='This command finds differences (file contents, user/group/mode) between archives',
    )
    diff_group = diff_parser.add_argument_group('diff arguments')
    diff_group.add_argument(
        '--repository',
        help='Path of repository containing archive to diff, defaults to the configured repository if there is only one, quoted globs supported',
    )
    diff_group.add_argument(
        '--archive',
        help='Archive name, hash, or series to diff',
        required=True,
    )
    diff_group.add_argument(
        '--second-archive',
        help='Second archive name, hash, or series to diff',
        required=True,
    )
    diff_group.add_argument(
        '--same-chunker-params', action='store_true', help='Override check of chunker parameters'
    )
    diff_group.add_argument(
        '--sort-by',
        metavar='KEY',
        dest='sort_keys',
        action='append',
        help='Advanced sorting: specify field(s) to sort by. Prefix with > for descending or < for ascending (default)',
    )
    diff_group.add_argument(
        '--content-only',
        action='store_true',
        help='Only compare differences in content (exclude metadata differences)',
    )
    diff_group.add_argument(
        '--only-patterns',
        action='store_true',
        help='Run the diff according to borgmatic configured patterns (ie do not diff entire archives)',
    )
    borg_parser = action_parsers.add_parser(
        'borg',
        aliases=ACTION_ALIASES['borg'],
@@ -1820,7 +2048,8 @@ def make_parsers(schema, unparsed_arguments):
        help='Path of repository to pass to Borg, defaults to the configured repositories, quoted globs supported',
    )
    borg_group.add_argument(
-        '--archive', help='Archive name, hash, or series to pass to Borg (or "latest")'
+        '--archive',
        help='Archive name, hash, or series to pass to Borg (or "latest")',
    )
    borg_group.add_argument(
        '--',
@@ -1834,6 +2063,9 @@ def make_parsers(schema, unparsed_arguments):
    return global_parser, action_parsers, global_plus_action_parser
 HIGHLANDER_ACTION_ARGUMENTS_COUNT = 2  # 1 for "global" + 1 for the action
 def parse_arguments(schema, *unparsed_arguments):
    '''
    Given a configuration schema dict and the command-line arguments with which this script was
@@ -1844,21 +2076,22 @@ def parse_arguments(schema, *unparsed_arguments):
    Raise SystemExit with an error code of 0 if "--help" was requested.
    '''
    global_parser, action_parsers, global_plus_action_parser = make_parsers(
-        schema, unparsed_arguments
+        schema,
        unparsed_arguments,
    )
    arguments, remaining_action_arguments = parse_arguments_for_actions(
-        unparsed_arguments, action_parsers.choices, global_parser
+        unparsed_arguments,
        action_parsers.choices,
        global_parser,
    )
    if not arguments['global'].config_paths:
-        arguments['global'].config_paths = collect.get_default_config_paths(expand_home=True)
+        arguments['global'].config_paths = collect.get_default_config_paths()
    for action_name in ('bootstrap', 'generate', 'validate'):
-        if (
+        if action_name in arguments and len(arguments) > HIGHLANDER_ACTION_ARGUMENTS_COUNT:
            action_name in arguments.keys() and len(arguments.keys()) > 2
        ):  # 2 = 1 for 'global' + 1 for the action
            raise ValueError(
-                f'The {action_name} action cannot be combined with other actions. Please run it separately.'
+                f'The {action_name} action cannot be combined with other actions. Please run it separately.',
            )
    unknown_arguments = get_unparsable_arguments(remaining_action_arguments)
@@ -1870,11 +2103,11 @@ def parse_arguments(schema, *unparsed_arguments):
        global_plus_action_parser.print_usage()
        raise ValueError(
-            f"Unrecognized argument{'s' if len(unknown_arguments) > 1 else ''}: {' '.join(unknown_arguments)}"
+            f"Unrecognized argument{'s' if len(unknown_arguments) > 1 else ''}: {' '.join(unknown_arguments)}",
        )
    if (
-        ('list' in arguments and 'repo-info' in arguments and arguments['list'].json)
+        ('list' in arguments and 'repo-info' in arguments and arguments['list'].json)  # noqa: PLR0916
        or ('list' in arguments and 'info' in arguments and arguments['list'].json)
        or ('repo-info' in arguments and 'info' in arguments and arguments['repo-info'].json)
    ):
@@ -1882,23 +2115,23 @@ def parse_arguments(schema, *unparsed_arguments):
    if 'list' in arguments and (arguments['list'].prefix and arguments['list'].match_archives):
        raise ValueError(
-            'With the list action, only one of --prefix or --match-archives flags can be used.'
+            'With the list action, only one of --prefix or --match-archives flags can be used.',
        )
    if 'repo-list' in arguments and (
        arguments['repo-list'].prefix and arguments['repo-list'].match_archives
    ):
        raise ValueError(
-            'With the repo-list action, only one of --prefix or --match-archives flags can be used.'
+            'With the repo-list action, only one of --prefix or --match-archives flags can be used.',
        )
-    if 'info' in arguments and (
+    if 'info' in arguments and (  # noqa: PLR0916
        (arguments['info'].archive and arguments['info'].prefix)
        or (arguments['info'].archive and arguments['info'].match_archives)
        or (arguments['info'].prefix and arguments['info'].match_archives)
    ):
        raise ValueError(
-            'With the info action, only one of --archive, --prefix, or --match-archives flags can be used.'
+            'With the info action, only one of --archive, --prefix, or --match-archives flags can be used.',
        )
    if 'borg' in arguments and arguments['global'].dry_run:
@@ -17,9 +17,11 @@ import borgmatic.actions.check
 import borgmatic.actions.compact
 import borgmatic.actions.config.bootstrap
 import borgmatic.actions.config.generate
 import borgmatic.actions.config.show
 import borgmatic.actions.config.validate
 import borgmatic.actions.create
 import borgmatic.actions.delete
 import borgmatic.actions.diff
 import borgmatic.actions.export_key
 import borgmatic.actions.export_tar
 import borgmatic.actions.extract
@@ -95,7 +97,9 @@ class Monitoring_hooks:
        self.config_filename = config_filename
        self.config = config
        self.dry_run = global_arguments.dry_run
-        self.monitoring_log_level = verbosity_to_log_level(config.get('monitoring_verbosity'))
+        self.monitoring_log_level = verbosity_to_log_level(
            get_verbosity({config_filename: config}, 'monitoring_verbosity'),
        )
        self.monitoring_hooks_are_activated = (
            using_primary_action and self.monitoring_log_level != DISABLED
        )
@@ -180,7 +184,7 @@ class Monitoring_hooks:
        )
-def run_configuration(config_filename, config, config_paths, arguments):
+def run_configuration(config_filename, config, config_paths, arguments):  # noqa: PLR0912, PLR0915
    '''
    Given a config filename, the corresponding parsed config dict, a sequence of loaded
    configuration paths, and command-line arguments as a dict from subparser name to a namespace of
@@ -204,12 +208,13 @@ def run_configuration(config_filename, config, config_paths, arguments):
    if skip_actions:
        logger.debug(
-            f"Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
+            f"Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions",
        )
-    try:
+    try:  # noqa: PLR1702
-        with Monitoring_hooks(config_filename, config, arguments, global_arguments):
+        with (
-            with borgmatic.hooks.command.Before_after_hooks(
+            Monitoring_hooks(config_filename, config, arguments, global_arguments),
            borgmatic.hooks.command.Before_after_hooks(
                command_hooks=config.get('commands'),
                before_after='configuration',
                umask=config.get('umask'),
@@ -218,69 +223,80 @@ def run_configuration(config_filename, config, config_paths, arguments):
                action_names=arguments.keys(),
                configuration_filename=config_filename,
                log_file=config.get('log_file', ''),
-            ):
+            ),
-                try:
+        ):
-                    local_borg_version = borg_version.local_borg_version(config, local_path)
+            try:
-                    logger.debug(f'Borg {local_borg_version}')
+                local_borg_version = borg_version.local_borg_version(config, local_path)
-                except (OSError, CalledProcessError, ValueError) as error:
+                logger.debug(f'Borg {local_borg_version}')
-                    yield from log_error_records(
+            except (OSError, CalledProcessError, ValueError) as error:
-                        f'{config_filename}: Error getting local Borg version', error
+                yield from log_error_records(
-                    )
+                    f'{config_filename}: Error getting local Borg version',
-                    return
+                    error,
                )
                raise
-                for repo in config['repositories']:
+            for repo in config['repositories']:
-                    repo_queue.put(
+                repo_queue.put(
-                        (repo, 0),
+                    (repo, 0),
-                    )
+                )
-                while not repo_queue.empty():
+            while not repo_queue.empty():
-                    repository, retry_num = repo_queue.get()
+                repository, retry_num = repo_queue.get()
-                    with Log_prefix(repository.get('label', repository['path'])):
+                with Log_prefix(repository.get('label', repository['path'])):
-                        logger.debug('Running actions for repository')
+                    logger.debug('Running actions for repository')
-                        timeout = retry_num * retry_wait
+                    timeout = retry_num * retry_wait
-                        if timeout:
+
-                            logger.warning(f'Sleeping {timeout}s before next retry')
+                    if timeout:
-                            time.sleep(timeout)
+                        logger.warning(f'Sleeping {timeout}s before next retry')
-                        try:
+                        time.sleep(timeout)
-                            yield from run_actions(
+
-                                arguments=arguments,
+                    try:
-                                config_filename=config_filename,
+                        yield from run_actions(
-                                config=config,
+                            arguments=arguments,
-                                config_paths=config_paths,
+                            config_filename=config_filename,
-                                local_path=local_path,
+                            config=config,
-                                remote_path=remote_path,
+                            config_paths=config_paths,
-                                local_borg_version=local_borg_version,
+                            local_path=local_path,
-                                repository=repository,
+                            remote_path=remote_path,
                            local_borg_version=local_borg_version,
                            repository=repository,
                        )
                    except (OSError, CalledProcessError, ValueError) as error:
                        if retry_num < retries:
                            repo_queue.put(
                                (repository, retry_num + 1),
                            )
-                        except (OSError, CalledProcessError, ValueError) as error:
+                            tuple(  # Consume the generator so as to trigger logging.
-                            if retry_num < retries:
+                                log_error_records(
-                                repo_queue.put(
+                                    'Error running actions for repository',
-                                    (repository, retry_num + 1),
+                                    error,
-                                )
+                                    levelno=logging.WARNING,
-                                tuple(  # Consume the generator so as to trigger logging.
+                                    log_command_error_output=True,
-                                    log_error_records(
+                                ),
                                        'Error running actions for repository',
                                        error,
                                        levelno=logging.WARNING,
                                        log_command_error_output=True,
                                    )
                                )
                                logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
                                continue
                            if command.considered_soft_failure(error):
                                continue
                            yield from log_error_records(
                                'Error running actions for repository',
                                error,
                            )
-                            encountered_error = error
+                            logger.warning(f'Retrying... attempt {retry_num + 1}/{retries}')
-                            error_repository = repository
+                            continue
                        if command.considered_soft_failure(error):
                            continue
                        yield from log_error_records(
                            'Error running actions for repository',
                            error,
                        )
                        encountered_error = error
                        error_repository = repository
            # Re-raise any error, so that the Monitoring_hooks context manager wrapping this
            # code can see the error and act accordingly. Do this here rather than as soon as
            # the error is encountered so that an error with one repository doesn't prevent
            # other repositories from running.
            if encountered_error:
                raise encountered_error
    except (OSError, CalledProcessError, ValueError) as error:
-        yield from log_error_records('Error running configuration', error)
+        yield from log_error_records('Error running configuration')
        encountered_error = error
@@ -290,7 +306,10 @@ def run_configuration(config_filename, config, config_paths, arguments):
    try:
        command.execute_hooks(
            command.filter_hooks(
-                config.get('commands'), after='error', action_names=arguments.keys()
+                config.get('commands'),
                after='error',
                action_names=arguments.keys(),
                state_names=['fail'],
            ),
            config.get('umask'),
            borgmatic.config.paths.get_working_directory(config),
@@ -309,7 +328,7 @@ def run_configuration(config_filename, config, config_paths, arguments):
        yield from log_error_records(f'{config_filename}: Error running after error hook', error)
-def run_actions(
+def run_actions(  # noqa: PLR0912, PLR0915
    *,
    arguments,
    config_filename,
@@ -323,7 +342,7 @@ def run_actions(
    '''
    Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
    filename, a configuration dict, a sequence of loaded configuration paths, local and remote paths
-    to Borg, a local Borg version string, and a repository name, run all actions from the
+    to Borg, a local Borg version string, and a repository dict, run all actions from the
    command-line arguments on the given repository.
    Yield JSON output strings from executing any actions that produce JSON.
@@ -345,6 +364,22 @@ def run_actions(
        'repository': repository_path,
    }
    skip_actions = set(get_skip_actions(config, arguments))
    requested_repository = next(
        (
            repository
            for action_arguments in arguments.values()
            for repository in (getattr(action_arguments, 'repository', None),)
            if repository is not None
        ),
        None,
    )
    if requested_repository and not borgmatic.config.validate.repositories_match(
        repository,
        requested_repository,
    ):
        logger.debug('Skipping actions because the requested --repository does not match')
        return
    with borgmatic.hooks.command.Before_after_hooks(
        command_hooks=config.get('commands'),
@@ -408,6 +443,7 @@ def run_actions(
                        local_borg_version,
                        action_arguments,
                        global_arguments,
                        dry_run_label,
                        local_path,
                        remote_path,
                    )
@@ -588,6 +624,16 @@ def run_actions(
                        local_path,
                        remote_path,
                    )
                elif action_name == 'diff':
                    borgmatic.actions.diff.run_diff(
                        repository,
                        config,
                        local_borg_version,
                        action_arguments,
                        global_arguments,
                        local_path,
                        remote_path,
                    )
                elif action_name == 'borg':
                    borgmatic.actions.borg.run_borg(
                        repository,
@@ -627,9 +673,10 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                        levelno=logging.DEBUG,
                        levelname='DEBUG',
                        msg=f'{config_filename}: Loading configuration file',
-                    )
+                        name=logger.name,
                    ),
                ),
-            ]
+            ],
        )
        try:
            configs[config_filename], paths, parse_logs = validate.parse_configuration(
@@ -646,12 +693,13 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                [
                    logging.makeLogRecord(
                        dict(
-                            levelno=logging.WARNING,
+                            levelno=logging.CRITICAL,
-                            levelname='WARNING',
+                            levelname='CRITICAL',
                            msg=f'{config_filename}: Insufficient permissions to read configuration file',
-                        )
+                            name=logger.name,
                        ),
                    ),
-                ]
+                ],
            )
        except (ValueError, OSError, validate.Validation_error) as error:
            logs.extend(
@@ -661,12 +709,18 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                            levelno=logging.CRITICAL,
                            levelname='CRITICAL',
                            msg=f'{config_filename}: Error parsing configuration file',
-                        )
+                            name=logger.name,
                        ),
                    ),
                    logging.makeLogRecord(
-                        dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=str(error))
+                        dict(
                            levelno=logging.CRITICAL,
                            levelname='CRITICAL',
                            msg=str(error),
                            name=logger.name,
                        ),
                    ),
-                ]
+                ],
            )
    return (configs, sorted(config_paths), logs)
@@ -677,7 +731,7 @@ def log_record(suppress_log=False, **kwargs):
    Create a log record based on the given makeLogRecord() arguments, one of which must be
    named "levelno". Log the record (unless suppress log is set) and return it.
    '''
-    record = logging.makeLogRecord(kwargs)
+    record = logging.makeLogRecord(dict(kwargs, name=logger.name))
    if suppress_log:
        return record
@@ -689,7 +743,10 @@ BORG_REPOSITORY_ACCESS_ABORTED_EXIT_CODE = 62
 def log_error_records(
-    message, error=None, levelno=logging.CRITICAL, log_command_error_output=False
+    message,
    error=None,
    levelno=logging.CRITICAL,
    log_command_error_output=False,
 ):
    '''
    Given error message text, an optional exception object, an optional log level, and whether to
@@ -707,14 +764,14 @@ def log_error_records(
    try:
        raise error
-    except CalledProcessError as error:
+    except CalledProcessError as called_process_error:
        yield log_record(levelno=levelno, levelname=level_name, msg=str(message))
-        if error.output:
+        if called_process_error.output:
            try:
-                output = error.output.decode('utf-8')
+                output = called_process_error.output.decode('utf-8')
            except (UnicodeDecodeError, AttributeError):
-                output = error.output
+                output = called_process_error.output
            # Suppress these logs for now and save the error output for the log summary at the end.
            # Log a separate record per line, as some errors can be really verbose and overflow the
@@ -727,18 +784,18 @@ def log_error_records(
                    suppress_log=True,
                )
-        yield log_record(levelno=levelno, levelname=level_name, msg=str(error))
+        yield log_record(levelno=levelno, levelname=level_name, msg=str(called_process_error))
-        if error.returncode == BORG_REPOSITORY_ACCESS_ABORTED_EXIT_CODE:
+        if called_process_error.returncode == BORG_REPOSITORY_ACCESS_ABORTED_EXIT_CODE:
            yield log_record(
                levelno=levelno,
                levelname=level_name,
                msg='\nTo work around this, set either the "relocated_repo_access_is_ok" or "unknown_unencrypted_repo_access_is_ok" option to "true", as appropriate.',
            )
-    except (ValueError, OSError) as error:
+    except (ValueError, OSError) as other_error:
        yield log_record(levelno=levelno, levelname=level_name, msg=str(message))
-        yield log_record(levelno=levelno, levelname=level_name, msg=str(error))
+        yield log_record(levelno=levelno, levelname=level_name, msg=str(other_error))
-    except:  # noqa: E722
+    except:  # noqa: E722, S110
        # Raising above only as a means of determining the error type. Swallow the exception here
        # because we don't want the exception to propagate out of this function.
        pass
@@ -765,70 +822,63 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
    '''
    add_custom_log_levels()
-    if 'bootstrap' in arguments:
+    try:
-        try:
+        if 'bootstrap' in arguments:
-            # No configuration file is needed for bootstrap.
+            try:
-            local_borg_version = borg_version.local_borg_version(
+                # No configuration file is needed for bootstrap.
-                {}, arguments['bootstrap'].local_path
+                local_borg_version = borg_version.local_borg_version(
-            )
+                    {},
-        except (OSError, CalledProcessError, ValueError) as error:
+                    arguments['bootstrap'].local_path,
-            yield from log_error_records('Error getting local Borg version', error)
+                )
-            return
+            except (OSError, CalledProcessError, ValueError) as error:
                yield from log_error_records('Error getting local Borg version', error)
                return
        try:
            borgmatic.actions.config.bootstrap.run_bootstrap(
-                arguments['bootstrap'], arguments['global'], local_borg_version
+                arguments['bootstrap'],
                arguments['global'],
                local_borg_version,
            )
            yield logging.makeLogRecord(
                dict(
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='Bootstrap successful',
-                )
+                    name=logger.name,
                ),
            )
        except (
            CalledProcessError,
            ValueError,
            OSError,
        ) as error:
            yield from log_error_records(error)
-        return
+            return
-    if 'generate' in arguments:
+        if 'generate' in arguments:
        try:
            borgmatic.actions.config.generate.run_generate(
-                arguments['generate'], arguments['global']
+                arguments['generate'],
                arguments['global'],
            )
            yield logging.makeLogRecord(
                dict(
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='Generate successful',
-                )
+                    name=logger.name,
-            )
+                ),
        except (
            CalledProcessError,
            ValueError,
            OSError,
        ) as error:
            yield from log_error_records(error)
        return
    if 'validate' in arguments:
        if configuration_parse_errors:
            yield logging.makeLogRecord(
                dict(
                    levelno=logging.CRITICAL,
                    levelname='CRITICAL',
                    msg='Configuration validation failed',
                )
            )
            return
-        try:
+        if 'validate' in arguments:
            if configuration_parse_errors:
                yield logging.makeLogRecord(
                    dict(
                        levelno=logging.CRITICAL,
                        levelname='CRITICAL',
                        msg='Configuration validation failed',
                        name=logger.name,
                    ),
                )
                return
            borgmatic.actions.config.validate.run_validate(arguments['validate'], configs)
            yield logging.makeLogRecord(
@@ -836,19 +886,26 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='All configuration files are valid',
-                )
+                    name=logger.name,
                ),
            )
        except (
            CalledProcessError,
            ValueError,
            OSError,
        ) as error:
            yield from log_error_records(error)
-        return
+            return
        if 'show' in arguments:
            borgmatic.actions.config.show.run_show(arguments['show'], configs)
            return
    except (
        CalledProcessError,
        ValueError,
        OSError,
    ) as error:
        yield from log_error_records(error)
-def collect_configuration_run_summary_logs(configs, config_paths, arguments, log_file_path):
+def collect_configuration_run_summary_logs(configs, config_paths, arguments, log_file_path):  # noqa: PLR0912
    '''
    Given a dict of configuration filename to corresponding parsed configuration, a sequence of
    loaded configuration paths, parsed command-line arguments as a dict from subparser name to a
@@ -861,9 +918,9 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
    # Run cross-file validation checks.
    repository = None
-    for action_name, action_arguments in arguments.items():
+    for action_arguments in arguments.values():
        if hasattr(action_arguments, 'repository'):
-            repository = getattr(action_arguments, 'repository')
+            repository = action_arguments.repository
            break
    try:
@@ -879,23 +936,36 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
        return
    try:
        seen_command_hooks = []
        for config_filename, config in configs.items():
-            command.execute_hooks(
+            command_hooks = command.filter_hooks(
-                command.filter_hooks(
+                tuple(
-                    config.get('commands'), before='everything', action_names=arguments.keys()
+                    command_hook
                    for command_hook in config.get('commands', ())
                    if command_hook not in seen_command_hooks
                ),
-                config.get('umask'),
+                before='everything',
-                borgmatic.config.paths.get_working_directory(config),
+                action_names=arguments.keys(),
                arguments['global'].dry_run,
                configuration_filename=config_filename,
                log_file=log_file_path or '',
            )
            if command_hooks:
                command.execute_hooks(
                    command_hooks,
                    config.get('umask'),
                    borgmatic.config.paths.get_working_directory(config),
                    arguments['global'].dry_run,
                    configuration_filename=config_filename,
                    log_file=log_file_path or '',
                )
                seen_command_hooks += list(command_hooks)
    except (CalledProcessError, ValueError, OSError) as error:
        yield from log_error_records('Error running before everything hook', error)
        return
    # Execute the actions corresponding to each configuration file.
    json_results = []
    encountered_error = False
    for config_filename, config in configs.items():
        with Log_prefix(config_filename):
@@ -906,6 +976,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
            )
            if error_logs:
                encountered_error = True
                yield from log_error_records('An error occurred')
                yield from error_logs
            else:
@@ -914,7 +985,8 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                        levelno=logging.INFO,
                        levelname='INFO',
                        msg=f'{config_filename}: Successfully ran configuration file',
-                    )
+                        name=logger.name,
                    ),
                )
                if results:
                    json_results.extend(results)
@@ -928,23 +1000,37 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                local_path=get_local_path(configs),
            )
        except (CalledProcessError, OSError) as error:
            encountered_error = True
            yield from log_error_records('Error unmounting mount point', error)
    if json_results:
        sys.stdout.write(json.dumps(json_results))
    try:
        seen_command_hooks = []
        for config_filename, config in configs.items():
-            command.execute_hooks(
+            command_hooks = command.filter_hooks(
-                command.filter_hooks(
+                tuple(
-                    config.get('commands'), after='everything', action_names=arguments.keys()
+                    command_hook
                    for command_hook in config.get('commands', ())
                    if command_hook not in seen_command_hooks
                ),
-                config.get('umask'),
+                after='everything',
-                borgmatic.config.paths.get_working_directory(config),
+                action_names=arguments.keys(),
-                arguments['global'].dry_run,
+                state_names=['fail' if encountered_error else 'finish'],
                configuration_filename=config_filename,
                log_file=log_file_path or '',
            )
            if command_hooks:
                command.execute_hooks(
                    command_hooks,
                    config.get('umask'),
                    borgmatic.config.paths.get_working_directory(config),
                    arguments['global'].dry_run,
                    configuration_filename=config_filename,
                    log_file=log_file_path or '',
                )
                seen_command_hooks += list(command_hooks)
    except (CalledProcessError, ValueError, OSError) as error:
        yield from log_error_records('Error running after everything hook', error)
@@ -958,16 +1044,17 @@ def exit_with_help_link():  # pragma: no cover
    sys.exit(1)
-def check_and_show_help_on_no_args(configs):
+def check_and_show_help_on_no_args(configs, schema):
    '''
-    Given a dict of configuration filename to corresponding parsed configuration, check if the
+    Given a dict of configuration filename to corresponding parsed configuration and the
-    borgmatic command is run without any arguments. If the configuration option "default_actions" is
+    configuration schema as a dict, check if the borgmatic command was run without any arguments. If
-    set to False, show the help message. Otherwise, trigger the default backup behavior.
+    the configuration option "default_actions" is set to False, then show the help message an exit.
    '''
-    if len(sys.argv) == 1:  # No arguments provided
+    if len(sys.argv) == 1:  # No arguments provided.
        default_actions = any(config.get('default_actions', True) for config in configs.values())
-        if not default_actions:
+
-            parse_arguments('--help')
+        if configs and not default_actions:
            parse_arguments(schema, '--help')
            sys.exit(0)
@@ -989,17 +1076,34 @@ def get_singular_option_value(configs, option_name):
        configure_logging(logging.CRITICAL)
        joined_values = ', '.join(str(value) for value in distinct_values)
        logger.critical(
-            f'The {option_name} option has conflicting values across configuration files: {joined_values}'
+            f'The {option_name} option has conflicting values across configuration files: {joined_values}',
        )
        exit_with_help_link()
    try:
-        return tuple(distinct_values)[0]
+        return next(iter(distinct_values))
-    except IndexError:
+    except StopIteration:
        return None
-def main(extra_summary_logs=[]):  # pragma: no cover
+def display_summary(summary_logs, log_json):  # pragma: no cover
    summary_logs_max_level = max(log.levelno for log in summary_logs)
    for message in ('summary:',) if log_json else ('', 'summary:'):
        log_record(
            levelno=summary_logs_max_level,
            levelname=logging.getLevelName(summary_logs_max_level),
            msg=message,
        )
    for log in summary_logs:
        logger.handle(log)
    if summary_logs_max_level >= logging.CRITICAL:
        exit_with_help_link()
 def main(extra_summary_logs=()):  # pragma: no cover
    configure_signals()
    configure_delayed_logging()
    schema_filename = validate.schema_filename()
@@ -1019,20 +1123,24 @@ def main(extra_summary_logs=[]):  # pragma: no cover
        exit_with_help_link()
    except SystemExit as error:
        if error.code == 0:
-            raise error
+            raise
        configure_logging(logging.CRITICAL)
        logger.critical(f"Error parsing arguments: {' '.join(sys.argv)}")
        exit_with_help_link()
    global_arguments = arguments['global']
    if global_arguments.version:
-        print(importlib.metadata.version('borgmatic'))
+        print(importlib.metadata.version('borgmatic'))  # noqa: T201
        sys.exit(0)
    if global_arguments.bash_completion:
-        print(borgmatic.commands.completion.bash.bash_completion())
+        print(borgmatic.commands.completion.bash.bash_completion())  # noqa: T201
        sys.exit(0)
    if global_arguments.fish_completion:
-        print(borgmatic.commands.completion.fish.fish_completion())
+        print(borgmatic.commands.completion.fish.fish_completion())  # noqa: T201
        sys.exit(0)
    config_filenames = tuple(collect.collect_config_filenames(global_arguments.config_paths))
@@ -1043,8 +1151,7 @@ def main(extra_summary_logs=[]):  # pragma: no cover
        resolve_env=global_arguments.resolve_env and not arguments.get('validate'),
    )
-    # Use the helper function to check and show help on no arguments, passing the preloaded configs
+    check_and_show_help_on_no_args(configs, schema)
    check_and_show_help_on_no_args(configs)
    configuration_parse_errors = (
        (max(log.levelno for log in parse_logs) >= logging.CRITICAL) if parse_logs else False
@@ -1054,6 +1161,7 @@ def main(extra_summary_logs=[]):  # pragma: no cover
        getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
    )
    log_file_path = get_singular_option_value(configs, 'log_file')
    log_json = get_singular_option_value(configs, 'log_json')
    try:
        configure_logging(
@@ -1063,7 +1171,8 @@ def main(extra_summary_logs=[]):  # pragma: no cover
            verbosity_to_log_level(get_verbosity(configs, 'monitoring_verbosity')),
            log_file_path,
            get_singular_option_value(configs, 'log_file_format'),
-            color_enabled=should_do_markup(configs, any_json_flags),
+            log_json,
            color_enabled=should_do_markup(configs, any_json_flags or log_json),
        )
    except (FileNotFoundError, PermissionError) as error:
        configure_logging(logging.CRITICAL)
@@ -1071,32 +1180,29 @@ def main(extra_summary_logs=[]):  # pragma: no cover
        exit_with_help_link()
    summary_logs = (
-        extra_summary_logs
+        list(extra_summary_logs)
        + parse_logs
        + (
            list(
                collect_highlander_action_summary_logs(
-                    configs, arguments, configuration_parse_errors
+                    configs,
-                )
+                    arguments,
                    configuration_parse_errors,
                ),
            )
            or list(
                collect_configuration_run_summary_logs(
-                    configs, config_paths, arguments, log_file_path
+                    configs,
-                )
+                    config_paths,
                    arguments,
                    log_file_path,
                ),
            )
        )
    )
    summary_logs_max_level = max(log.levelno for log in summary_logs)
-    for message in ('', 'summary:'):
+    display_summary(summary_logs, log_json)
        log_record(
            levelno=summary_logs_max_level,
            levelname=logging.getLevelName(summary_logs_max_level),
            msg=message,
        )
    for log in summary_logs:
        logger.handle(log)
-    if summary_logs_max_level >= logging.CRITICAL:
+if __name__ == '__main__':  # pragma: no cover
-        exit_with_help_link()
+    main()
@@ -22,15 +22,15 @@ def available_actions(subparsers, current_action=None):
    action of "config" but not "list".
    '''
    action_to_subactions = borgmatic.commands.arguments.get_subactions_for_actions(
-        subparsers.choices
+        subparsers.choices,
    )
    current_subactions = action_to_subactions.get(current_action)
    if current_subactions:
        return current_subactions
-    all_subactions = set(
+    all_subactions = {
        subaction for subactions in action_to_subactions.values() for subaction in subactions
-    )
+    }
-    return tuple(action for action in subparsers.choices.keys() if action not in all_subactions)
+    return tuple(action for action in subparsers.choices if action not in all_subactions)
@@ -23,7 +23,7 @@ def bash_completion():
    borgmatic's command-line argument parsers.
    '''
    (
-        unused_global_parser,
+        _,
        action_parsers,
        global_plus_action_parser,
    ) = borgmatic.commands.arguments.make_parsers(
@@ -33,12 +33,13 @@ def bash_completion():
    global_flags = parser_flags(global_plus_action_parser)
    # Avert your eyes.
    # fmt: off
    return '\n'.join(
        (
            'check_version() {',
            '    local this_script="$(cat "$BASH_SOURCE" 2> /dev/null)"',
            '    local installed_script="$(borgmatic --bash-completion 2> /dev/null)"',
-            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];'
+            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];',
            f'''        then cat << EOF\n{borgmatic.commands.completion.actions.upgrade_message(
                    'bash',
                    'sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"',
@@ -47,24 +48,22 @@ def bash_completion():
            '    fi',
            '}',
            'complete_borgmatic() {',
-        )
+            *tuple(
        + tuple(
            '''    if [[ " ${COMP_WORDS[*]} " =~ " %s " ]]; then
        COMPREPLY=($(compgen -W "%s %s %s" -- "${COMP_WORDS[COMP_CWORD]}"))
        return 0
-    fi'''
+    fi'''  # noqa: UP031
            % (
                action,
                parser_flags(action_parser),
                ' '.join(
-                    borgmatic.commands.completion.actions.available_actions(action_parsers, action)
+                    borgmatic.commands.completion.actions.available_actions(action_parsers, action),
                ),
                global_flags,
            )
            for action, action_parser in reversed(action_parsers.choices.items())
-        )
+        ),
-        + (
+            '    COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'  # noqa: UP031
            '    COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'  # noqa: FS003
            % (
                ' '.join(borgmatic.commands.completion.actions.available_actions(action_parsers)),
                global_flags,
@@ -72,5 +71,5 @@ def bash_completion():
            '    (check_version &)',
            '}',
            '\ncomplete -o bashdefault -o default -F complete_borgmatic borgmatic',
-        )
+        ),
    )
@@ -11,10 +11,10 @@ def has_file_options(action: Action):
    '''
    Given an argparse.Action instance, return True if it takes a file argument.
    '''
-    return action.metavar in (
+    return action.metavar in {
        'FILENAME',
        'PATH',
-    ) or action.dest in ('config_paths',)
+    } or action.dest in {'config_paths'}
 def has_choice_options(action: Action):
@@ -36,11 +36,11 @@ def has_unknown_required_param_options(action: Action):
    return (
        action.required is True
        or action.nargs
-        in (
+        in {
            '+',
            '*',
-        )
+        }
-        or action.metavar in ('PATTERN', 'KEYS', 'N')
+        or action.metavar in {'PATTERN', 'KEYS', 'N'}
        or (action.type is not None and action.default is None)
    )
@@ -77,7 +77,7 @@ def exact_options_completion(action: Action):
        return f'''\ncomplete -c borgmatic -x -n "__borgmatic_current_arg {args}"'''
    raise ValueError(
-        f'Unexpected action: {action} passes has_exact_options but has no choices produced'
+        f'Unexpected action: {action} passes has_exact_options but has no choices produced',
    )
@@ -96,7 +96,7 @@ def fish_completion():
    borgmatic's command-line argument parsers.
    '''
    (
-        unused_global_parser,
+        _,
        action_parsers,
        global_plus_action_parser,
    ) = borgmatic.commands.arguments.make_parsers(
@@ -104,7 +104,7 @@ def fish_completion():
        unparsed_arguments=(),
    )
-    all_action_parsers = ' '.join(action for action in action_parsers.choices.keys())
+    all_action_parsers = ' '.join(action for action in action_parsers.choices)
    exact_option_args = tuple(
        ' '.join(action.option_strings)
@@ -119,8 +119,9 @@ def fish_completion():
    )
    # Avert your eyes.
-    return '\n'.join(
+    # fmt: off
-        dedent_strip_as_tuple(
+    return '\n'.join((
        *dedent_strip_as_tuple(
            f'''
            function __borgmatic_check_version
                set -fx this_filename (status current-filename)
@@ -157,27 +158,27 @@ def fish_completion():
            set --local action_parser_condition "not __fish_seen_subcommand_from {all_action_parsers}"
            set --local exact_option_condition "not __borgmatic_current_arg {' '.join(exact_option_args)}"
-            '''
+            ''',
-        )
+        ),
-        + ('\n# action_parser completions',)
+        '\n# action_parser completions',
-        + tuple(
+        *tuple(
            f'''complete -c borgmatic -f -n "$action_parser_condition" -n "$exact_option_condition" -a '{action_name}' -d {shlex.quote(action_parser.description)}'''
            for action_name, action_parser in action_parsers.choices.items()
-        )
+        ),
-        + ('\n# global flags',)
+        '\n# global flags',
-        + tuple(
+        *tuple(
            # -n is checked in order, so put faster / more likely to be true checks first
            f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)}{exact_options_completion(action)}'''
            for action in global_plus_action_parser._actions
            # ignore the noargs action, as this is an impossible completion for fish
            if len(action.option_strings) > 0
            if 'Deprecated' not in action.help
-        )
+        ),
-        + ('\n# action_parser flags',)
+        '\n# action_parser flags',
-        + tuple(
+        *tuple(
            f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)} -n "__fish_seen_subcommand_from {action_name}"{exact_options_completion(action)}'''
            for action_name, action_parser in action_parsers.choices.items()
            for action in action_parser._actions
            if 'Deprecated' not in (action.help or ())
-        )
+        ),
-    )
+    ))
@@ -5,7 +5,7 @@ def variants(flag_name):
    "--foo[9].bar".
    '''
    if '[0]' in flag_name:
-        for index in range(0, 10):
+        for index in range(10):
            yield flag_name.replace('[0]', f'[{index}]')
        return
@@ -10,8 +10,8 @@ def main():
            levelno=logging.WARNING,
            levelname='WARNING',
            msg='generate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config generate" instead.',
-        )
+        ),
    )
-    sys.argv = ['borgmatic', 'config', 'generate'] + sys.argv[1:]
+    sys.argv = ['borgmatic', 'config', 'generate', *sys.argv[1:]]
    borgmatic.commands.borgmatic.main([warning_log])
@@ -10,8 +10,8 @@ def main():
            levelno=logging.WARNING,
            levelname='WARNING',
            msg='validate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config validate" instead.',
-        )
+        ),
    )
-    sys.argv = ['borgmatic', 'config', 'validate'] + sys.argv[1:]
+    sys.argv = ['borgmatic', 'config', 'validate', *sys.argv[1:]]
    borgmatic.commands.borgmatic.main([warning_log])
@@ -146,16 +146,16 @@ def prepare_arguments_for_config(global_arguments, schema):
        keys = tuple(argument_name.split('.'))
        option_type = type_for_option(schema, keys)
-        # The argument doesn't correspond to any option in the schema, so ignore it. It's
+        # The argument doesn't correspond to any option in the schema, or it is a complex argument, so ignore it.
-        # probably a flag that borgmatic has on the command-line but not in configuration.
+        # It's probably a flag that borgmatic has on the command-line but not in configuration.
-        if option_type is None:
+        if option_type in {'object', None}:
            continue
        prepared_values.append(
            (
                keys,
                convert_value_type(value, option_type),
-            )
+            ),
        )
    return tuple(prepared_values)
@@ -1,17 +1,14 @@
 import os
-def get_default_config_paths(expand_home=True):
+def get_default_config_paths():
    '''
-    Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
+    Return a list of default configuration paths. This includes both system-wide
-    default configuration paths. This includes both system-wide configuration and configuration in
+    configuration and configuration in the current user's home directory.
    the current user's home directory.
    Don't expand the home directory ($HOME) if the expand home flag is False.
    '''
-    user_config_directory = os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
+    user_config_directory = os.path.expandvars(
-    if expand_home:
+        os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
-        user_config_directory = os.path.expandvars(user_config_directory)
+    )
    return [
        '/etc/borgmatic/config.yaml',
@@ -49,6 +46,7 @@ def collect_config_filenames(config_paths):
        for filename in sorted(os.listdir(path)):
            full_filename = os.path.join(path, filename)
-            matching_filetype = full_filename.endswith('.yaml') or full_filename.endswith('.yml')
+            matching_filetype = full_filename.endswith(('.yaml', '.yml'))
            if matching_filetype and not os.path.isdir(full_filename):
                yield os.path.abspath(full_filename)
@@ -1,3 +1,6 @@
 import contextlib
 import functools
 import re
 import shlex
@@ -6,20 +9,58 @@ def coerce_scalar(value):
    Given a configuration value, coerce it to an integer or a boolean as appropriate and return the
    result.
    '''
-    try:
+    with contextlib.suppress(TypeError, ValueError):
        return int(value)
    except (TypeError, ValueError):
        pass
-    if value == 'true' or value == 'True':
+    try:
-        return True
+        return {
-    if value == 'false' or value == 'False':
+            'true': True,
-        return False
+            'True': True,
-
+            'false': False,
-    return value
+            'False': False,
        }.get(value, value)
    except TypeError:  # e.g. for an unhashable type
        return value
-def apply_constants(value, constants, shell_escape=False):
+CONSTANT_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
 def resolve_constant(match, constants, command_hook):
    '''
    Given a re.Match instance of CONSTANT_PATTERN representing a matched constant name to be
    interpolated, a constants dict, and whether this is for a command hook, lookup the matched
    constant name within the given constants and return its value.
    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
    off the backslashing and return the literal value.
    If the variable name isn't found in the given constants, then return the literal value.
    '''
    name = match.group('name')
    # The would-be variable is escaped, so strip off the escaping and return the result without
    # resolving the name—unless this is for a command hook, in which case just return the literal
    # string. That way, subsequent variable interpolation will still see the string as escaped
    # instead of trying to interpolate it.
    if match.group('left_escape') and match.group('right_escape'):
        if command_hook:
            return match.group(0)
        return '{' + name + '}'
    value = constants.get(name)
    # The matched variable is in the constants, so return its value. And if this is for a command
    # hook, then shell escape the value so as to prevent shell injection attacks.
    if value is not None:
        return shlex.quote(str(value)) if command_hook else str(value)
    # The matched variable name isn't in the constants. Return the whole string unaltered.
    return match.group(0)
 def apply_constants(value, constants, command_hook=False):
    '''
    Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
    replace any configuration string values of the form "{constant}" (or containing it) with the
@@ -37,27 +78,25 @@ def apply_constants(value, constants, shell_escape=False):
        return value
    if isinstance(value, str):
        for constant_name, constant_value in constants.items():
            value = value.replace(
                '{' + constant_name + '}',
                shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
            )
        # Support constants within non-string scalars by coercing the value to its appropriate type.
-        value = coerce_scalar(value)
+        value = coerce_scalar(
            CONSTANT_PATTERN.sub(
                functools.partial(resolve_constant, constants=constants, command_hook=command_hook),
                value,
            )
        )
    elif isinstance(value, list):
        for index, list_value in enumerate(value):
-            value[index] = apply_constants(list_value, constants, shell_escape)
+            value[index] = apply_constants(list_value, constants, command_hook)
    elif isinstance(value, dict):
        for option_name, option_value in value.items():
            value[option_name] = apply_constants(
                option_value,
                constants,
-                shell_escape=(
+                command_hook=(
-                    shell_escape
+                    command_hook
-                    or option_name.startswith('before_')
+                    or option_name.startswith(('before_', 'after_'))
-                    or option_name.startswith('after_')
+                    or option_name in {'on_error', 'run'}
                    or option_name == 'on_error'
                ),
            )
@@ -2,7 +2,7 @@ import os
 import re
 VARIABLE_PATTERN = re.compile(
-    r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})'
+    r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})',
 )
@@ -1,5 +1,7 @@
 import collections
 import contextlib
 import io
 import json
 import os
 import re
@@ -18,7 +20,8 @@ def insert_newline_before_comment(config, field_name):
    field and its comments.
    '''
    config.ca.items[field_name][1].insert(
-        0, ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None)
+        0,
        ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None),
    )
@@ -40,18 +43,22 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
        config = ruamel.yaml.comments.CommentedSeq(
            example
            if borgmatic.config.schema.compare_types(
-                schema['items'].get('type'), SCALAR_SCHEMA_TYPES
+                schema['items'].get('type'),
                SCALAR_SCHEMA_TYPES,
            )
            else [
                schema_to_sample_configuration(
-                    schema['items'], source_config, level, parent_is_sequence=True
+                    schema['items'],
-                )
+                    source_config,
-            ]
+                    level,
                    parent_is_sequence=True,
                ),
            ],
        )
        add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
    elif borgmatic.config.schema.compare_types(schema_type, {'object'}):
        if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
-            source_config = dict(collections.ChainMap(*source_config))
+            source_config = source_config[0]
        config = (
            ruamel.yaml.comments.CommentedMap(
@@ -59,19 +66,25 @@ def schema_to_sample_configuration(schema, source_config=None, level=0, parent_i
                    (
                        field_name,
                        schema_to_sample_configuration(
-                            sub_schema, (source_config or {}).get(field_name, {}), level + 1
+                            sub_schema,
                            (source_config or {}).get(field_name, {}),
                            level + 1,
                        ),
                    )
                    for field_name, sub_schema in borgmatic.config.schema.get_properties(
-                        schema
+                        schema,
                    ).items()
-                ]
+                ],
            )
            or example
        )
        indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
        add_comments_to_configuration_object(
-            config, schema, source_config, indent=indent, skip_first=parent_is_sequence
+            config,
            schema,
            source_config,
            indent=indent,
            skip_first_field=parent_is_sequence,
        )
    elif borgmatic.config.schema.compare_types(schema_type, SCALAR_SCHEMA_TYPES, match=all):
        return example
@@ -95,7 +108,7 @@ def comment_out_line(line):
    return '# '.join((indent_spaces, line[count_indent_spaces:]))
-def comment_out_optional_configuration(rendered_config):
+def transform_optional_configuration(rendered_config, comment_out=True):
    '''
    Post-process a rendered configuration string to comment out optional key/values, as determined
    by a sentinel in the comment before each key.
@@ -105,26 +118,44 @@ def comment_out_optional_configuration(rendered_config):
    Ideally ruamel.yaml would support commenting out keys during configuration generation, but it's
    not terribly easy to accomplish that way.
    If comment_out is False, then just strip the comment sentinel without actually commenting
    anything out.
    '''
    lines = []
    optional = False
    indent_characters = None
    indent_characters_at_sentinel = None
    for line in rendered_config.split('\n'):
        indent_characters = len(line) - len(line.lstrip())
        # Upon encountering an optional configuration option, comment out lines until the next blank
        # line.
        if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
            if comment_out is False:
                continue
            optional = True
            indent_characters_at_sentinel = indent_characters
            continue
-        # Hit a blank line, so reset commenting.
+        # Hit a blank line or dedented, so reset commenting.
-        if not line.strip():
+        if not line.strip() or (
            indent_characters_at_sentinel is not None
            and indent_characters < indent_characters_at_sentinel
        ):
            optional = False
            indent_characters_at_sentinel = None
        lines.append(comment_out_line(line) if optional else line)
    return '\n'.join(lines)
 RUAMEL_YAML_END_OF_DOCUMENT_MARKER = '...\n'
 def render_configuration(config):
    '''
    Given a config data structure of nested OrderedDicts, render the config as YAML and return it.
@@ -132,7 +163,13 @@ def render_configuration(config):
    dumper = ruamel.yaml.YAML(typ='rt')
    dumper.indent(mapping=INDENT, sequence=INDENT + SEQUENCE_INDENT, offset=INDENT)
    rendered = io.StringIO()
-    dumper.dump(config, rendered)
+    dumper.dump(
        config,
        rendered,
        # Dumping certain values (integers, for instance) causes ruamel.yaml to append an
        # end-of-document "..." marker. Strip it.
        transform=lambda dumped: dumped.removesuffix(RUAMEL_YAML_END_OF_DOCUMENT_MARKER),
    )
    return rendered.getvalue()
@@ -145,15 +182,13 @@ def write_configuration(config_filename, rendered_config, mode=0o600, overwrite=
    '''
    if not overwrite and os.path.exists(config_filename):
        raise FileExistsError(
-            f'{config_filename} already exists. Aborting. Use --overwrite to replace the file.'
+            f'{config_filename} already exists. Aborting. Use --overwrite to replace the file.',
        )
-    try:
+    with contextlib.suppress(FileExistsError, FileNotFoundError):
        os.makedirs(os.path.dirname(config_filename), mode=0o700)
    except (FileExistsError, FileNotFoundError):
        pass
-    with open(config_filename, 'w') as config_file:
+    with open(config_filename, 'w', encoding='utf-8') as config_file:
        config_file.write(rendered_config)
    os.chmod(config_filename, mode)
@@ -178,7 +213,7 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
    if schema['items'].get('type') != 'object':
        return
-    for field_name in config[0].keys():
+    for field_name in config[0]:
        field_schema = borgmatic.config.schema.get_properties(schema['items']).get(field_name, {})
        description = field_schema.get('description')
@@ -198,32 +233,39 @@ COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 def add_comments_to_configuration_object(
-    config, schema, source_config=None, indent=0, skip_first=False
+    config,
    schema,
    source_config=None,
    indent=0,
    skip_first_field=False,
 ):
    '''
    Using descriptions from a schema as a source, add those descriptions as comments to the given
    configuration dict, putting them before each field. Indent the comment the given number of
    characters.
    If skip_first_field is True, omit the comment for the initial field. This is useful for
    sequences, where the comment for the first field goes before the sequence itself.
    And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
    source configuration dict. The idea is that any options used in the source configuration should
    stay active in the generated configuration.
    '''
    for index, field_name in enumerate(config.keys()):
-        if skip_first and index == 0:
+        if skip_first_field and index == 0:
            continue
        field_schema = borgmatic.config.schema.get_properties(schema).get(field_name, {})
        description = field_schema.get('description', '').strip()
-        # If this isn't a default key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment, flagging it to be commented
        # out from the sample configuration. This sentinel is consumed by downstream processing that
        # does the actual commenting out.
        if field_name not in DEFAULT_KEYS and (
            source_config is None or field_name not in source_config
        ):
            description = (
-                '\n'.join((description, COMMENTED_OUT_SENTINEL))
+                f'{description}\n{COMMENTED_OUT_SENTINEL}'
                if description
                else COMMENTED_OUT_SENTINEL
            )
@@ -251,6 +293,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
    '''
    if not source_config:
        return destination_config
    if not destination_config or not isinstance(source_config, collections.abc.Mapping):
        return source_config
@@ -258,7 +301,8 @@ def merge_source_configuration_into_destination(destination_config, source_confi
        # This is a mapping. Recurse for this key/value.
        if isinstance(source_value, collections.abc.Mapping):
            destination_config[field_name] = merge_source_configuration_into_destination(
-                destination_config[field_name], source_value
+                destination_config[field_name],
                source_value,
            )
            continue
@@ -272,42 +316,102 @@ def merge_source_configuration_into_destination(destination_config, source_confi
                        source_item,
                    )
                    for index, source_item in enumerate(source_value)
-                ]
+                ],
            )
            continue
        # This is some sort of scalar. Set it into the destination.
-        destination_config[field_name] = source_config[field_name]
+        destination_config[field_name] = source_value
    return destination_config
 def get_configuration_subset(config, option_name):  # pragma: no cover
    '''
    Given configuration as a ruamel.yaml.CommentedMap and an option name found within it at the top
    level, return a new CommentedMap containing a subset of the configuration with only the given
    option and no other top-level options.
    This is useful when generating the sample configuration for a single option instead of a whole
    configuration file.
    '''
    option_config = ruamel.yaml.CommentedMap({option_name: config[option_name]})
    # Due to a quirk of ruamel.yaml, the comment right before a top-level key is not on that key and
    # needs to get copied separately.
    option_config.ca.items[option_name] = config.ca.items[option_name]
    return option_config
 def generate_sample_configuration(
-    dry_run, source_filename, destination_filename, schema_filename, overwrite=False
+    dry_run,
    source_filename,
    destination_path,
    schema_filename,
    overwrite=False,
    split=False,
 ):
    '''
-    Given an optional source configuration filename, and a required destination configuration
+    Given an optional source configuration filename, a required destination configuration path, the
-    filename, the path to a schema filename in a YAML rendition of the JSON Schema format, and
+    path to a schema filename in a YAML rendition of the JSON Schema format, whether to overwrite a
-    whether to overwrite a destination file, write out a sample configuration file based on that
+    destination file, and whether to split the configuration into multiple files (one per option) in
-    schema. If a source filename is provided, merge the parsed contents of that configuration into
+    the assumed destination directory, write out sample configuration file(s) based on that schema.
-    the generated configuration.
+    If a source filename is provided, merge the parsed contents of that configuration into the
    generated configuration.
    '''
-    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
+    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename, encoding='utf-8'))
    source_config = None
    if source_filename:
        source_config = load.load_configuration(source_filename)
        normalize.normalize(source_filename, source_config)
        # The borgmatic.config.normalize.normalize() function tacks on an empty "bootstrap" if
        # needed, so the hook gets used by default. But we don't want it to end up in the generated
        # config unless the user has set it explicitly, as an empty "bootstrap:" won't validate.
        if source_config and source_config.get('bootstrap') == {}:
            del source_config['bootstrap']
    destination_config = merge_source_configuration_into_destination(
-        schema_to_sample_configuration(schema, source_config), source_config
+        schema_to_sample_configuration(schema, source_config),
        source_config,
    )
    if dry_run:
        return
    if split:
        if os.path.exists(destination_path) and not os.path.isdir(destination_path):
            raise ValueError('With the --split flag, the destination path must be a directory')
        os.makedirs(destination_path, exist_ok=True)
        for option_name in destination_config:
            write_configuration(
                os.path.join(destination_path, f'{option_name}.yaml'),
                transform_optional_configuration(
                    render_configuration(get_configuration_subset(destination_config, option_name)),
                    comment_out=False,
                ).strip(),
                overwrite=overwrite,
            )
        # Also dump a manifest listing all the options we've written.
        json.dump(
            {'option_names': list(destination_config.keys())},
            open(os.path.join(destination_path, 'options.json'), 'w', encoding='utf-8'),
        )
        return
    if os.path.exists(destination_path) and not os.path.isfile(destination_path):
        raise ValueError('Without the --split flag, the destination path must be a file')
    write_configuration(
-        destination_filename,
+        destination_path,
-        comment_out_optional_configuration(render_configuration(destination_config)),
+        transform_optional_configuration(
            render_configuration(destination_config), comment_out=True
        ),
        overwrite=overwrite,
    )
@@ -31,7 +31,7 @@ def probe_and_include_file(filename, include_directories, config_paths):
            return load_configuration(candidate_filename, config_paths)
    raise FileNotFoundError(
-        f'Could not find include {filename} at {" or ".join(candidate_filenames)}'
+        f'Could not find include {filename} at {" or ".join(candidate_filenames)}',
    )
@@ -69,7 +69,7 @@ def include_configuration(loader, filename_node, include_directory, config_paths
        ]
    raise ValueError(
-        'The value given for the !include tag is invalid; use a single filename or a list of filenames instead'
+        'The value given for the !include tag is invalid; use a single filename or a list of filenames instead',
    )
@@ -85,7 +85,7 @@ def raise_retain_node_error(loader, node):
    '''
    if isinstance(node, (ruamel.yaml.nodes.MappingNode, ruamel.yaml.nodes.SequenceNode)):
        raise ValueError(
-            'The !retain tag may only be used within a configuration file containing a merged !include tag.'
+            'The !retain tag may only be used within a configuration file containing a merged !include tag.',
        )
    raise ValueError('The !retain tag may only be used on a mapping or list.')
@@ -100,7 +100,7 @@ def raise_omit_node_error(loader, node):
    tags are handled by deep_merge_nodes() below.
    '''
    raise ValueError(
-        'The !omit tag may only be used on a scalar (e.g., string) or list element within a configuration file containing a merged !include tag.'
+        'The !omit tag may only be used on a scalar (e.g., string) or list element within a configuration file containing a merged !include tag.',
    )
@@ -111,9 +111,13 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
    '''
    def __init__(
-        self, preserve_quotes=None, loader=None, include_directory=None, config_paths=None
+        self,
        preserve_quotes=None,
        loader=None,
        include_directory=None,
        config_paths=None,
    ):
-        super(Include_constructor, self).__init__(preserve_quotes, loader)
+        super().__init__(preserve_quotes, loader)
        self.add_constructor(
            '!include',
            functools.partial(
@@ -147,7 +151,7 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
        representer = ruamel.yaml.representer.SafeRepresenter()
        for index, (key_node, value_node) in enumerate(node.value):
-            if key_node.tag == u'tag:yaml.org,2002:merge' and value_node.tag == '!include':
+            if key_node.tag == 'tag:yaml.org,2002:merge' and value_node.tag == '!include':
                # Replace the merge include with a sequence of included configuration nodes ready
                # for merging. The construct_object() call here triggers include_configuration()
                # among other constructors.
@@ -157,7 +161,7 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
                )
        # This super().flatten_mapping() call actually performs "<<" merges.
-        super(Include_constructor, self).flatten_mapping(node)
+        super().flatten_mapping(node)
        node.value = deep_merge_nodes(node.value)
@@ -179,7 +183,7 @@ def load_configuration(filename, config_paths=None):
    # because yaml.Constructor has to be an actual class.)
    class Include_constructor_with_extras(Include_constructor):
        def __init__(self, preserve_quotes=None, loader=None):
-            super(Include_constructor_with_extras, self).__init__(
+            super().__init__(
                preserve_quotes,
                loader,
                include_directory=os.path.dirname(filename),
@@ -190,7 +194,7 @@ def load_configuration(filename, config_paths=None):
    yaml.Constructor = Include_constructor_with_extras
    config_paths.add(filename)
-    with open(filename) as file:
+    with open(filename, encoding='utf-8') as file:
        return yaml.load(file.read())
@@ -318,17 +322,18 @@ def deep_merge_nodes(nodes):
    # Bucket the nodes by their keys. Then merge all of the values sharing the same key.
    for key_name, grouped_nodes in itertools.groupby(
-        sorted(nodes, key=get_node_key_name), get_node_key_name
+        sorted(nodes, key=get_node_key_name),
        get_node_key_name,
    ):
-        grouped_nodes = list(grouped_nodes)
+        grouped_nodes = list(grouped_nodes)  # noqa: PLW2901
        # The merged node inherits its attributes from the final node in the group.
        (last_node_key, last_node_value) = grouped_nodes[-1]
-        value_types = set(type(value) for (_, value) in grouped_nodes)
+        value_types = {type(value) for (_, value) in grouped_nodes}
        if len(value_types) > 1:
            raise ValueError(
-                f'Incompatible types found when trying to merge "{key_name}:" values across configuration files: {", ".join(value_type.id for value_type in value_types)}'
+                f'Incompatible types found when trying to merge "{key_name}:" values across configuration files: {", ".join(value_type.id for value_type in value_types)}',
            )
        # If we're dealing with MappingNodes, recurse and merge its values as well.
@@ -351,7 +356,7 @@ def deep_merge_nodes(nodes):
                            comment=last_node_value.comment,
                            anchor=last_node_value.anchor,
                        ),
-                    )
+                    ),
                )
            continue
@@ -374,7 +379,7 @@ def deep_merge_nodes(nodes):
                            comment=last_node_value.comment,
                            anchor=last_node_value.anchor,
                        ),
-                    )
+                    ),
                )
            continue
@@ -25,12 +25,12 @@ def normalize_sections(config_filename, config):
        and location.get('prefix') != consistency.get('prefix')
    ):
        raise ValueError(
-            'The retention prefix and the consistency prefix cannot have different values (unless one is not set).'
+            'The retention prefix and the consistency prefix cannot have different values (unless one is not set).',
        )
    if storage.get('umask') and hooks.get('umask') and storage.get('umask') != hooks.get('umask'):
        raise ValueError(
-            'The storage umask and the hooks umask cannot have different values (unless one is not set).'
+            'The storage umask and the hooks umask cannot have different values (unless one is not set).',
        )
    any_section_upgraded = False
@@ -51,8 +51,8 @@ def normalize_sections(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: Configuration sections (like location:, storage:, retention:, consistency:, and hooks:) are deprecated and support will be removed from a future release. To prepare for this, move your options out of sections to the global scope.',
-                )
+                ),
-            )
+            ),
        ]
    return []
@@ -68,7 +68,7 @@ def make_command_hook_deprecation_log(config_filename, option_name):  # pragma:
            levelno=logging.WARNING,
            levelname='WARNING',
            msg=f'{config_filename}: {option_name} is deprecated and support will be removed from a future release. Use commands: instead.',
-        )
+        ),
    )
@@ -90,7 +90,7 @@ def normalize_commands(config_filename, config):
                {
                    preposition: 'repository',
                    'run': commands,
-                }
+                },
            )
    # Normalize "before_backup", "before_prune", "after_backup", "after_prune", etc.
@@ -108,7 +108,7 @@ def normalize_commands(config_filename, config):
                    preposition: 'action',
                    'when': [action_name],
                    'run': commands,
-                }
+                },
            )
    # Normalize "on_error".
@@ -121,7 +121,7 @@ def normalize_commands(config_filename, config):
                'after': 'error',
                'when': ['create', 'prune', 'compact', 'check'],
                'run': commands,
-            }
+            },
        )
    # Normalize "before_everything" and "after_everything".
@@ -136,13 +136,13 @@ def normalize_commands(config_filename, config):
                    preposition: 'everything',
                    'when': ['create'],
                    'run': commands,
-                }
+                },
            )
    return logs
-def normalize(config_filename, config):
+def normalize(config_filename, config):  # noqa: PLR0912, PLR0915
    '''
    Given a configuration filename and a configuration dict of its loaded contents, apply particular
    hard-coded rules to normalize the configuration to adhere to the current schema. Return any log
@@ -159,9 +159,9 @@ def normalize(config_filename, config):
                dict(
                    levelno=logging.WARNING,
                    levelname='WARNING',
-                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use borgmatic_runtime_directory and borgmatic_state_directory instead.',
+                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use user_runtime_directory and user_state_directory instead.',
-                )
+                ),
-            )
+            ),
        )
    # Upgrade exclude_if_present from a string to a list.
@@ -173,8 +173,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The exclude_if_present option now expects a list value. String values for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['exclude_if_present'] = [exclude_if_present]
@@ -191,8 +191,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The store_config_files option has moved under the bootstrap hook. Specifying store_config_files at the global scope is deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        del config['store_config_files']
        config['bootstrap']['store_config_files'] = store_config_files
@@ -206,8 +206,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The healthchecks hook now expects a key/value pair with "ping_url" as a key. String values for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['healthchecks'] = {'ping_url': healthchecks}
@@ -219,8 +219,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['cronitor'] = {'ping_url': cronitor}
@@ -232,8 +232,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['pagerduty'] = {'integration_key': pagerduty}
@@ -245,8 +245,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['cronhub'] = {'ping_url': cronhub}
@@ -259,8 +259,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The checks option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['checks'] = [{'name': check_type} for check_type in checks]
@@ -273,8 +273,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The numeric_owner option has been renamed to numeric_ids. numeric_owner is deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['numeric_ids'] = numeric_owner
@@ -286,8 +286,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The bsd_flags option has been renamed to flags. bsd_flags is deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['flags'] = bsd_flags
@@ -299,8 +299,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The remote_rate_limit option has been renamed to upload_rate_limit. remote_rate_limit is deprecated and support will be removed from a future release.',
-                )
+                ),
-            )
+            ),
        )
        config['upload_rate_limit'] = remote_rate_limit
@@ -314,8 +314,8 @@ def normalize(config_filename, config):
                        levelno=logging.WARNING,
                        levelname='WARNING',
                        msg=f'{config_filename}: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
-                    )
+                    ),
-                )
+                ),
            )
            config['repositories'] = [
                {'path': repository} if isinstance(repository, str) else repository
@@ -338,25 +338,22 @@ def normalize(config_filename, config):
                            levelno=logging.WARNING,
                            levelname='WARNING',
                            msg=f'{config_filename}: Repository paths containing "~" are deprecated in borgmatic and support will be removed from a future release.',
-                        )
+                        ),
-                    )
+                    ),
                )
            if ':' in repository_path:
                if repository_path.startswith('file://'):
                    updated_repository_path = os.path.abspath(
-                        repository_path.partition('file://')[-1]
+                        repository_path.partition('file://')[-1],
                    )
                    config['repositories'].append(
                        dict(
                            repository_dict,
                            path=updated_repository_path,
-                        )
+                        ),
                    )
-                elif (
+                elif repository_path.startswith(('ssh://', 'sftp://', 'rclone:', 's3:', 'b2:')):
                    repository_path.startswith('ssh://')
                    or repository_path.startswith('sftp://')
                    or repository_path.startswith('rclone:')
                ):
                    config['repositories'].append(repository_dict)
                else:
                    rewritten_repository_path = f"ssh://{repository_path.replace(':~', '/~').replace(':/', '/').replace(':', '/./')}"
@@ -365,15 +362,15 @@ def normalize(config_filename, config):
                            dict(
                                levelno=logging.WARNING,
                                levelname='WARNING',
-                                msg=f'{config_filename}: Remote repository paths without ssh:// or rclone: syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
+                                msg=f'{config_filename}: Remote repository paths without ssh://, sftp://, rclone:, s3:, or b2:, syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
-                            )
+                            ),
-                        )
+                        ),
                    )
                    config['repositories'].append(
                        dict(
                            repository_dict,
                            path=rewritten_repository_path,
-                        )
+                        ),
                    )
            else:
                config['repositories'].append(repository_dict)
@@ -385,8 +382,8 @@ def normalize(config_filename, config):
                    levelno=logging.WARNING,
                    levelname='WARNING',
                    msg=f'{config_filename}: The prefix option is deprecated and support will be removed from a future release. Use archive_name_format or match_archives instead.',
-                )
+                ),
-            )
+            ),
        )
    return logs
@@ -18,7 +18,7 @@ def set_values(config, keys, value):
    if len(keys) == 1:
        if isinstance(config, list):
            raise ValueError(
-                'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)'
+                'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)',
            )
        config[first_key] = value
@@ -69,11 +69,11 @@ def type_for_option(schema, option_keys):
    '''
    option_schema = schema
-    for key in option_keys:
+    try:
-        try:
+        for key in option_keys:
            option_schema = option_schema['properties'][key]
-        except KeyError:
+    except KeyError:
-            return None
+        return None
    try:
        return option_schema['type']
@@ -103,8 +103,8 @@ def parse_overrides(raw_overrides, schema):
    parsed_overrides = []
-    for raw_override in raw_overrides:
+    try:
-        try:
+        for raw_override in raw_overrides:
            raw_keys, value = raw_override.split('=', 1)
            keys = tuple(raw_keys.split('.'))
            option_type = type_for_option(schema, keys)
@@ -113,14 +113,14 @@ def parse_overrides(raw_overrides, schema):
                (
                    keys,
                    convert_value_type(value, option_type),
-                )
+                ),
            )
-        except ValueError:
+    except ValueError:
-            raise ValueError(
+        raise ValueError(
-                f"Invalid override '{raw_override}'. Make sure you use the form: OPTION=VALUE or OPTION.SUBOPTION=VALUE"
+            f"Invalid override '{raw_override}'. Make sure you use the form: OPTION=VALUE or OPTION.SUBOPTION=VALUE",
-            )
+        )
-        except ruamel.yaml.error.YAMLError as error:
+    except ruamel.yaml.error.YAMLError as error:
-            raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
+        raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
    return tuple(parsed_overrides)
@@ -139,7 +139,7 @@ def apply_overrides(config, schema, raw_overrides):
    if overrides:
        logger.warning(
-            "The --override flag is deprecated and will be removed from a future release. Instead, use a command-line flag corresponding to the configuration option you'd like to set."
+            "The --override flag is deprecated and will be removed from a future release. Instead, use a command-line flag corresponding to the configuration option you'd like to set.",
        )
    for keys, value in overrides:
@@ -1,10 +1,20 @@
 import contextlib
 import logging
 import os
 import tempfile
 from enum import Enum
 logger = logging.getLogger(__name__)
 class Systemd_directories(Enum):
    RUNTIME_DIRECTORY = 0
    STATE_DIRECTORY = 1
    CACHE_DIRECTORY = 2
    LOGS_DIRECTORY = 3
    CONFIGURATION_DIRECTORY = 4
 def expand_user_in_path(path):
    '''
    Given a directory path, expand any tildes in it.
@@ -15,6 +25,17 @@ def expand_user_in_path(path):
        return None
 def resolve_systemd_directory(directory):
    '''
    Given a systemd directory environment variable enum, read the value if set and return the first
    configured directory.
    '''
    separator = ':'
    paths = os.environ.get(directory.name)
    return paths.split(separator)[0] if paths else None
 def get_working_directory(config):  # pragma: no cover
    '''
    Given a configuration dict, get the working directory from it, expanding any tildes.
@@ -34,7 +55,8 @@ TEMPORARY_DIRECTORY_PREFIX = 'borgmatic-'
 def replace_temporary_subdirectory_with_glob(
-    path, temporary_directory_prefix=TEMPORARY_DIRECTORY_PREFIX
+    path,
    temporary_directory_prefix=TEMPORARY_DIRECTORY_PREFIX,
 ):
    '''
    Given an absolute temporary directory path and an optional temporary directory prefix, look for
@@ -94,7 +116,9 @@ class Runtime_directory:
        runtime_directory = (
            config.get('user_runtime_directory')
            or os.environ.get('XDG_RUNTIME_DIR')  # Set by PAM on Linux.
-            or os.environ.get('RUNTIME_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
                Systemd_directories.RUNTIME_DIRECTORY
            )  # Set by systemd if configured.
        )
        if runtime_directory:
@@ -103,7 +127,9 @@ class Runtime_directory:
            self.temporary_directory = None
        else:
-            base_directory = os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'
+            base_directory = (
                os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'  # noqa: S108
            )
            if not base_directory.startswith(os.path.sep):
                raise ValueError('The temporary directory must be an absolute path')
@@ -122,7 +148,7 @@ class Runtime_directory:
                base_path if final_directory == 'borgmatic' else runtime_directory,
                '.',  # Borg 1.4+ "slashdot" hack.
                'borgmatic',
-            )
+            ),
        )
        os.makedirs(self.runtime_path, mode=0o700, exist_ok=True)
@@ -139,13 +165,11 @@ class Runtime_directory:
        Delete any temporary directory that was created as part of initialization.
        '''
        if self.temporary_directory:
            try:
                self.temporary_directory.cleanup()
            # The cleanup() call errors if, for instance, there's still a
            # mounted filesystem within the temporary directory. There's
            # nothing we can do about that here, so swallow the error.
-            except OSError:
+            with contextlib.suppress(OSError):
-                pass
+                self.temporary_directory.cleanup()
 def make_runtime_directory_glob(borgmatic_runtime_directory):
@@ -158,7 +182,7 @@ def make_runtime_directory_glob(borgmatic_runtime_directory):
        *(
            '*' if subdirectory.startswith(TEMPORARY_DIRECTORY_PREFIX) else subdirectory
            for subdirectory in os.path.normpath(borgmatic_runtime_directory).split(os.path.sep)
-        )
+        ),
    )
@@ -172,8 +196,10 @@ def get_borgmatic_state_directory(config):
        os.path.join(
            config.get('user_state_directory')
            or os.environ.get('XDG_STATE_HOME')
-            or os.environ.get('STATE_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
                Systemd_directories.STATE_DIRECTORY
            )  # Set by systemd if configured.
            or '~/.local/state',
            'borgmatic',
-        )
+        ),
    )
@@ -14,8 +14,8 @@ def get_properties(schema):
            item
            for item in itertools.chain(
                *itertools.zip_longest(
-                    *[sub_schema['properties'].items() for sub_schema in schema['oneOf']]
+                    *[sub_schema['properties'].items() for sub_schema in schema['oneOf']],
-                )
+                ),
            )
            if item is not None
        )
@@ -61,12 +61,6 @@ def compare_types(schema_type, target_types, match=any):
    list must be in the target types.
    '''
    if isinstance(schema_type, list):
-        if match(element_schema_type in target_types for element_schema_type in schema_type):
+        return match(element_schema_type in target_types for element_schema_type in schema_type)
            return True
-        return False
+    return schema_type in target_types
    if schema_type in target_types:
        return True
    return False
@@ -27,6 +27,13 @@ properties:
            - /etc
            - /var/log/syslog*
            - /home/user/path with spaces
    source_directories_must_exist:
        type: boolean
        description: |
            When true, source directories (and root pattern paths) must exist
            before a backup begins. If they don't, borgmatic errors. Defaults to
            true.
        example: false
    repositories:
        type: array
        items:
@@ -72,7 +79,8 @@ properties:
                    description: |
                        Whether any missing parent directories of the repository
                        path should be created, only used for the repo-create
-                        action. Defaults to false.
+                        action. Defaults to false. (This option is supported 
                        for Borg 1.x only.)
                    example: true
        description: |
            A required list of local or remote repositories with paths and
@@ -133,12 +141,39 @@ properties:
            false. But when a database hook is used, the setting here is ignored
            and read_special is considered true.
        example: true
    unsafe_skip_path_validation_before_create:
        type: boolean
        description: |
            Skip pre-backup safety validation: (1) runtime directory exclusion
            detection, and (2) special file exclusion for database streaming.
            Can significantly improve performance for large filesystems.
            UNSAFE because, when this option is enabled, Borg may hang on
            special files (named pipes, devices) and data from borgmatic data
            source hooks may be silently excluded if your excluded patterns
            match the runtime directory (see the "user_runtime_directory"
            option).
            Only enable if not using data source hooks and you manually verify
            that your excludes don't affect the runtime directory. Defaults
            to false.
        example: true
    flags:
        type: boolean
        description: |
            Record filesystem flags (e.g. NODUMP, IMMUTABLE) in archive.
            Defaults to true.
        example: false
    files_changed:
        type: string
        enum: ['ctime', 'mtime', 'disabled']
        description: |
            Threshold for considering a file as changed. See
            https://borgbackup.readthedocs.io/en/stable/usage/create.html for
            details. Defaults to "ctime". E.g., a file is considered changed if
            its ctime has changed since the last backup. (This option is
            supported for Borg 1.4.2+ only.)
        example: ctime
    files_cache:
        type: string
        description: |
@@ -162,9 +197,13 @@ properties:
            type: string
        description: |
            Any paths matching these patterns are included/excluded from
-            backups. Globs are expanded. (Tildes are not.) See the output of
+            backups. Recursion root patterns ("R ...") are effectively the same
-            "borg help patterns" for more details. Quote any value if it
+            as "source_directories"; they tell Borg which paths to backup
-            contains leading punctuation, so it parses correctly.
+            (modulo any excludes). Globs are expanded. (Tildes are not.) See
            the output of "borg help patterns" for more details. Quote any value
            if it contains leading punctuation, so it parses correctly. Also use
            leading slashes in absolute paths, or data source hooks may be
            unable to rewrite patterns as needed.
        example:
            - 'R /'
            - '- /home/*/.cache'
@@ -231,7 +270,8 @@ properties:
    exclude_nodump:
        type: boolean
        description: |
-            Exclude files with the NODUMP flag. Defaults to false.
+            Exclude files with the NODUMP flag. Defaults to false. (This option
            is supported for Borg 1.x only.)
        example: true
    borgmatic_source_directory:
        type: string
@@ -247,7 +287,7 @@ properties:
            Path for storing temporary runtime data like streaming database
            dumps and bootstrap metadata. borgmatic automatically creates and
            uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR
-            or or $TMPDIR or $TEMP or /run/user/$UID.
+            or $TMPDIR or $TEMP or /run/user/$UID.
        example: /run/user/1001
    user_state_directory:
        type: string
@@ -258,12 +298,6 @@ properties:
            create the check records again (and therefore re-run checks).
            Defaults to $XDG_STATE_HOME or ~/.local/state.
        example: /var/lib/borgmatic
    source_directories_must_exist:
        type: boolean
        description: |
            If true, then source directories (and root pattern paths) must
            exist. If they don't, an error is raised. Defaults to false.
        example: true
    encryption_passcommand:
        type: string
        description: |
@@ -274,7 +308,7 @@ properties:
            encryption_passphrase takes precedence. This can also be used to
            access encrypted systemd service credentials. Defaults to not set.
            For more details, see:
-            https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+            https://torsion.org/borgmatic/how-to/provide-your-passwords/
        example: "secret-tool lookup borg-repository repo-name"
    encryption_passphrase:
        type: string
@@ -389,6 +423,13 @@ properties:
            Path for Borg cache files. Defaults to
            $borg_base_directory/.cache/borg
        example: /path/to/base/cache
    use_chunks_archive:
        type: boolean
        description: |
            Enables or disables the use of chunks.archive.d for faster cache
            resyncs in Borg. If true, value is set to "yes" (default) else
            it's set to "no", reducing disk usage but slowing resyncs.
        example: true
    borg_files_cache_ttl:
        type: integer
        description: |
@@ -398,14 +439,20 @@ properties:
        type: string
        description: |
            Path for Borg security and encryption nonce files. Defaults to
-            $borg_base_directory/.config/borg/security
+            $borg_config_directory/security
        example: /path/to/base/config/security
    borg_keys_directory:
        type: string
        description: |
            Path for Borg encryption key files. Defaults to
-            $borg_base_directory/.config/borg/keys
+            $borg_config_directory/keys
        example: /path/to/base/config/keys
    borg_key_file:
        type: string
        description: |
            Path for the Borg repository key file, for use with a repository
            created with "keyfile" encryption.
        example: /path/to/base/config/keyfile
    borg_exit_codes:
        type: array
        items:
@@ -429,9 +476,9 @@ properties:
        description: |
            A list of Borg exit codes that should be elevated to errors or
            squashed to warnings as indicated. By default, Borg error exit codes
-            (2 to 99) are treated as errors while warning exit codes (1 and
+            (2 to 99) are treated as errors and most warning exit codes (1 and
-            100+) are treated as warnings. Exit codes other than 1 and 2 are
+            100+) are treated as errors as well. Exit codes other than 1 and 2
-            only present in Borg 1.4.0+.
+            are only present in Borg 1.4.0+.
        example:
            - code: 13
              treat_as: warning
@@ -471,6 +518,27 @@ properties:
            If match_archives is not specified, borgmatic defaults to deriving
            the match_archives value from archive_name_format.
        example: "sh:{hostname}-*"
    file_list_format:
        type: string
        description: |
            Borg format for the files listing of the "list" action. Corresponds
            to the "--format" flag. Defaults to "{mode} {user:6} {group:6}
            {size:8} {mtime} {path}{extra}{NL}". With "--json", the form of the
            format is ignored, but the keys used in it are added to the JSON
            output. See
            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
            details.
        example: "{path}{extra}{NL}"
    archive_list_format:
        type: string
        description: |
            Borg format for the archives listing of the "repo-list" action.
            Corresponds to the "--format" flag. Defaults to "{archive:<36}
            {time} [{id}]{NL}". With "--json", the form of the format is
            ignored, but the keys used in it are added to the JSON output. See
            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
            details.
        example: "{archive}{NL}"
    relocated_repo_access_is_ok:
        type: boolean
        description: |
@@ -483,40 +551,152 @@ properties:
            Bypass Borg error about a previously unknown unencrypted repository.
            Defaults to false.
        example: true
    debug_passphrase:
        type: boolean
        description: |
            When set true, display debugging information that includes 
            passphrases used and passphrase related environment variables set. 
            Defaults to false.
        example: true
    display_passphrase:
        type: boolean
        description: |
            When set true, always shows passphrase and its hex UTF-8 byte
            sequence. Defaults to false.
        example: true
    check_i_know_what_i_am_doing:
        type: boolean
        description: |
            Bypass Borg confirmation about check with repair option. Defaults to
            false and an interactive prompt from Borg.
        example: true
    msgpack_version_check:
        type: boolean
        description: |
            Optionally disable the msgpack version check. Default is true; use
            at your own risk. (This option is supported for Borg 1.4.2+ only.)
        example: false
    extra_borg_options:
        type: object
        additionalProperties: false
        properties:
-            init:
+            break_lock:
                type: string
                description: |
-                  Extra command-line options to pass to "borg init".
+                  Extra command-line options to pass to "borg break-lock".
                example: "--extra-option"
-            create:
+            check:
                type: string
                description: |
-                  Extra command-line options to pass to "borg create".
+                  Extra command-line options to pass to "borg check".
                example: "--extra-option"
            prune:
                type: string
                description: |
                  Extra command-line options to pass to "borg prune".
                example: "--extra-option"
            compact:
                type: string
                description: |
                  Extra command-line options to pass to "borg compact".
                example: "--extra-option"
-            check:
+            create:
                type: string
                description: |
-                  Extra command-line options to pass to "borg check".
+                  Extra command-line options to pass to "borg create".
                example: "--extra-option"
            delete:
                type: string
                description: |
                  Extra command-line options to pass to "borg delete".
                example: "--extra-option"
            export_tar:
                type: string
                description: |
                  Extra command-line options to pass to "borg export-tar".
                example: "--extra-option"
            extract:
                type: string
                description: |
                  Extra command-line options to pass to "borg extract".
                example: "--extra-option"
            key_export:
                type: string
                description: |
                  Extra command-line options to pass to "borg key export".
                example: "--extra-option"
            key_import:
                type: string
                description: |
                  Extra command-line options to pass to "borg key import".
                example: "--extra-option"
            key_change_passphrase:
                type: string
                description: |
                  Extra command-line options to pass to "borg key
                  change-passphrase".
                example: "--extra-option"
            info:
                type: string
                description: |
                  Extra command-line options to pass to "borg info".
                example: "--extra-option"
            init:
                type: string
                description: |
                  Deprecated. Use "repo_create" instead. Extra command-line
                  options to pass to "borg init" / "borg repo-create".
                example: "--extra-option"
            list:
                type: string
                description: |
                  Extra command-line options to pass to "borg list".
                example: "--extra-option"
            mount:
                type: string
                description: |
                  Extra command-line options to pass to "borg mount".
                example: "--extra-option"
            prune:
                type: string
                description: |
                  Extra command-line options to pass to "borg prune".
                example: "--extra-option"
            recreate:
                type: string
                description: |
                  Extra command-line options to pass to "borg recreate".
                example: "--extra-option"
            rename:
                type: string
                description: |
                  Extra command-line options to pass to "borg rename".
                example: "--extra-option"
            repo_create:
                type: string
                description: |
                  Extra command-line options to pass to "borg init" / "borg
                  repo-create".
                example: "--extra-option"
            repo_delete:
                type: string
                description: |
                  Extra command-line options to pass to "borg repo-delete".
                example: "--extra-option"
            repo_info:
                type: string
                description: |
                  Extra command-line options to pass to "borg repo-info".
                example: "--extra-option"
            repo_list:
                type: string
                description: |
                  Extra command-line options to pass to "borg repo-list".
                example: "--extra-option"
            transfer:
                type: string
                description: |
                  Extra command-line options to pass to "borg transfer".
                example: "--extra-option"
            umount:
                type: string
                description: |
                  Extra command-line options to pass to "borg umount".
                example: "--extra-option"
        description: |
            Additional options to pass directly to particular Borg commands,
@@ -558,6 +738,14 @@ properties:
        type: integer
        description: Number of yearly archives to keep.
        example: 1
    keep_13weekly:
        type: integer
        description: Number of quarterly archives to keep (13 week strategy).
        example: 13
    keep_3monthly:
        type: integer
        description: Number of quarterly archives to keep (3 month strategy).
        example: 3
    prefix:
        type: string
        description: |
@@ -863,6 +1051,8 @@ properties:
        type: string
        description: |
            Python format string used for log messages written to the log file.
            See https://docs.python.org/3/library/logging.html (and specifically
            the LogRecord attributes with "{}-formatting") for details.
        example: "[{asctime}] {levelname}: {prefix}{message}"
    monitoring_verbosity:
        type: integer
@@ -908,9 +1098,9 @@ properties:
    default_actions:
        type: boolean
        description: |
-            Whether to apply default actions (create, prune, compact and check)
+            Whether to run default actions (create, prune, compact, and check)
-            when no arguments are supplied to the borgmatic command. If set to
+            when no arguments are given on the command line. If set to false,
-            false, borgmatic displays the help message instead.
+            borgmatic displays the help message instead.
        example: true
    skip_actions:
        type: array
@@ -939,6 +1129,7 @@ properties:
                - break-lock
                - key
                - borg
                - diff
        description: |
            List of one or more actions to skip running for this configuration
            file, even if specified on the command-line (explicitly or
@@ -1155,6 +1346,7 @@ properties:
                                  - break-lock
                                  - key
                                  - borg
                                  - diff
                          description: |
                              List of actions for which the commands will be
                              run. Defaults to running for all actions.
@@ -1220,11 +1412,32 @@ properties:
                                  - break-lock
                                  - key
                                  - borg
                                  - diff
                          description: |
                              Only trigger the hook when borgmatic is run with
                              particular actions listed here. Defaults to
                              running for all actions.
                          example: [create, prune, compact, check]
                      states:
                          type: array
                          items:
                              type: string
                              enum:
                                  - finish
                                  - fail
                          description: |
                              Only trigger the hook if borgmatic encounters one
                              of the states (execution results) listed here,
                              where:
                               * "finish": No errors occurred.
                               * "fail": An error occurred.
                              This state is evaluated only for the scope of the
                              configured "action", "repository", etc., rather
                              than for the entire borgmatic run. Only available
                              for "after" hooks. Defaults to running the hook
                              for all states.
                          example:
                              - finish
                      run:
                          type: array
                          items:
@@ -1274,6 +1487,23 @@ properties:
                        implicitly enables read_special (see above) to support
                        dump and restore streaming.
                    example: users
                label:
                    type: string
                    description: |
                        Label to identify the database dump in the backup.
                    example: my_backup_label
                container:
                    type: string
                    description: |
                        Container name/id to connect to. When specified the
                        hostname is ignored. Requires docker/podman CLI.
                    example: debian_stable
                restore_container:
                    type: string
                    description: |
                        Container name/id to restore to. Defaults to the
                        "container" option.
                    example: restore_container
                hostname:
                    type: string
                    description: |
@@ -1475,6 +1705,32 @@ properties:
                        database hook implicitly enables read_special (see
                        above) to support dump and restore streaming.
                    example: users
                skip_names:
                    type: array
                    items:
                        type: string
                    description: |
                        Database names to skip when dumping "all" databases.
                        Ignored when the database name is not "all".
                    example:
                        - cache
                label:
                    type: string
                    description: |
                        Label to identify the database dump in the backup.
                    example: my_backup_label
                container:
                    type: string
                    description: |
                        Container name/id to connect to. When specified the
                        hostname is ignored. Requires docker/podman CLI.
                    example: debian_stable
                restore_container:
                    type: string
                    description: |
                        Container name/id to restore to. Defaults to the
                        "container" option.
                    example: restore_container
                hostname:
                    type: string
                    description: |
@@ -1496,6 +1752,20 @@ properties:
                    description: |
                        Port to restore to. Defaults to the "port" option.
                    example: 5433
                socket_path:
                    type: string
                    description: |
                        Path of a Unix socket to connect to instead of a remote
                        hostname. Ignored when "hostname" is set and not
                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
                    example: database.example.org
                restore_socket_path:
                    type: string
                    description: |
                        Path of a Unix socket to connect to instead of a remote
                        hostname. Ignored when "hostname" is set and not
                        "localhost". Defaults to the "socket_path" option.
                    example: database.example.org
                username:
                    type: string
                    description: |
@@ -1525,6 +1795,23 @@ properties:
                        Defaults to the "password" option. Supports the
                        "{credential ...}" syntax.
                    example: trustsome1
                password_transport:
                    type: string
                    enum:
                        - pipe
                        - environment
                    description: |
                        How to transmit database passwords from borgmatic to the
                        MariaDB client, one of:
                         * "pipe": Securely transmit passwords via anonymous
                           pipe. Only works if the database client is on the
                           same host as borgmatic. (The server can be
                           somewhere else.) This is the default value.
                         * "environment": Transmit passwords via environment
                           variable. Potentially less secure than a pipe, but
                           necessary when the database client is elsewhere, e.g.
                           when "mariadb_dump_command" is configured to "exec"
                           into a container and run a client there.
                tls:
                    type: boolean
                    description: |
@@ -1542,18 +1829,21 @@ properties:
                mariadb_dump_command:
                    type: string
                    description: |
-                        Command to use instead of "mariadb-dump". This can be
+                        Command to use instead of "mariadb-dump". But set any
-                        used to run a specific mariadb_dump version (e.g., one
+                        additional command-line flags in "options", not here.
-                        inside a running container). If you run it from within a
+                        This command can be used to run a specific mariadb_dump
-                        container, make sure to mount the path in the
+                        version (e.g., one inside a running container). If you
-                        "user_runtime_directory" option from the host into the
+                        run it from within a container, make sure to mount the
-                        container at the same location. Defaults to
+                        path in the "user_runtime_directory" option from the
-                        "mariadb-dump".
+                        host into the container at the same location. Defaults
                        to "mariadb-dump".
                    example: docker exec mariadb_container mariadb-dump
                mariadb_command:
                    type: string
                    description: |
-                        Command to run instead of "mariadb". This can be used to
+                        Command to run instead of "mariadb". But set any
                        additional command-line flags in "list_options" or
                        "restore_options", not here. This command can be used to
                        run a specific mariadb version (e.g., one inside a
                        running container). Defaults to "mariadb".
                    example: docker exec mariadb_container mariadb           
@@ -1623,6 +1913,32 @@ properties:
                        database hook implicitly enables read_special (see
                        above) to support dump and restore streaming.
                    example: users
                skip_names:
                    type: array
                    items:
                        type: string
                    description: |
                        Database names to skip when dumping "all" databases.
                        Ignored when the database name is not "all".
                    example:
                        - cache
                label:
                    type: string
                    description: |
                        Label to identify the database dump in the backup.
                    example: my_backup_label
                container:
                    type: string
                    description: |
                        Container name/id to connect to. When specified the
                        hostname is ignored. Requires docker/podman CLI.
                    example: debian_stable
                restore_container:
                    type: string
                    description: |
                        Container name/id to restore to. Defaults to the
                        "container" option.
                    example: restore_container
                hostname:
                    type: string
                    description: |
@@ -1644,6 +1960,20 @@ properties:
                    description: |
                        Port to restore to. Defaults to the "port" option.
                    example: 5433
                socket_path:
                    type: string
                    description: |
                        Path of a Unix socket to connect to instead of a remote
                        hostname. Ignored when "hostname" is set and not
                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
                    example: database.example.org
                restore_socket_path:
                    type: string
                    description: |
                        Path of a Unix socket to connect to instead of a remote
                        hostname. Ignored when "hostname" is set and not
                        "localhost". Defaults to the "socket_path" option.
                    example: database.example.org
                username:
                    type: string
                    description: |
@@ -1673,6 +2003,23 @@ properties:
                        Defaults to the "password" option. Supports the
                        "{credential ...}" syntax.
                    example: trustsome1
                password_transport:
                    type: string
                    enum:
                        - pipe
                        - environment
                    description: |
                        How to transmit database passwords from borgmatic to the
                        MySQL client, one of:
                         * "pipe": Securely transmit passwords via anonymous
                           pipe. Only works if the database client is on the
                           same host as borgmatic. (The server can be
                           somewhere else.) This is the default value.
                         * "environment": Transmit passwords via environment
                           variable. Potentially less secure than a pipe, but
                           necessary when the database client is elsewhere, e.g.
                           when "mysql_dump_command" is configured to "exec"
                           into a container and run a client there.
                tls:
                    type: boolean
                    description: |
@@ -1690,17 +2037,21 @@ properties:
                mysql_dump_command:
                    type: string
                    description: |
-                        Command to use instead of "mysqldump". This can be used
+                        Command to use instead of "mysqldump". But set any
-                        to run a specific mysql_dump version (e.g., one inside a
+                        additional command-line flags in "options", not here.
-                        running container). If you run it from within a
+                        This command can be used to run a specific mysql_dump
-                        container, make sure to mount the path in the
+                        version (e.g., one inside a running container). If you
-                        "user_runtime_directory" option from the host into the
+                        run it from within a container, make sure to mount the
-                        container at the same location. Defaults to "mysqldump".
+                        path in the "user_runtime_directory" option from the
                        host into the container at the same location. Defaults
                        to "mysqldump".
                    example: docker exec mysql_container mysqldump
                mysql_command:
                    type: string
                    description: |
-                        Command to run instead of "mysql". This can be used to
+                        Command to run instead of "mysql". But set any
                        additional command-line flags in "list_options" or
                        "restore_options", not here. This command can be used to
                        run a specific mysql version (e.g., one inside a running
                        container). Defaults to "mysql".
                    example: docker exec mysql_container mysql
@@ -1779,6 +2130,11 @@ properties:
                        read_special (see above) to support dump and restore
                        streaming.
                    example: /var/lib/sqlite/users.db
                label:
                    type: string
                    description: |
                        Label to identify the database dump in the backup.
                    example: my_backup_label
                restore_path:
                    type: string
                    description: |
@@ -1827,6 +2183,23 @@ properties:
                        database hook implicitly enables read_special (see
                        above) to support dump and restore streaming.
                    example: users
                label:
                    type: string
                    description: |
                        Label to identify the database dump in the backup.
                    example: my_backup_label
                container:
                    type: string
                    description: |
                        Container name/id to connect to. When specified the
                        hostname is ignored. Requires docker/podman CLI.
                    example: debian_stable
                restore_container:
                    type: string
                    description: |
                        Container name/id to restore to. Defaults to the
                        "container" option.
                    example: restore_container
                hostname:
                    type: string
                    description: |
@@ -1983,17 +2356,24 @@ properties:
                        type: string
                        description: |
                            The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have started.
                    priority:
                        type: string
                        enum:
                            - max
                            - urgent
                            - high
                            - default
                            - low
                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: min
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic
            finish:
                type: object
                additionalProperties: false
@@ -2007,17 +2387,24 @@ properties:
                        type: string
                        description: |
                            The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have finished.
                    priority:
                        type: string
                        enum:
                            - max
                            - urgent
                            - high
                            - default
                            - low
                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: min
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,+1
            fail:
                type: object
                additionalProperties: false
@@ -2034,14 +2421,21 @@ properties:
                        example: Your backups have failed.
                    priority:
                        type: string
                        enum:
                            - max
                            - urgent
                            - high
                            - default
                            - low
                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: max
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,-1,skull
            states:
                type: array
                items:
@@ -2450,8 +2844,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to Apprise services as part of the
-                    "finish", "fail", and "log" states. Defaults to true.
+                    "finish", "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
            logs_size_limit:
                type: integer
                description: |
@@ -2565,8 +2959,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to Healthchecks as part of the "finish",
-                    "fail", and "log" states. Defaults to true.
+                    "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
            ping_body_limit:
                type: integer
                description: |
@@ -2672,8 +3066,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to PagerDuty when a backup errors.
-                    Defaults to true.
+                    Defaults to false.
-                example: false
+                example: true
        description: |
            Configuration for a monitoring integration with PagerDuty. Create an
            account at https://www.pagerduty.com if you'd like to use this
@@ -2701,7 +3095,7 @@ properties:
            url:
                type: string
                description: |
-                    Grafana loki log URL to notify when a backup begins,
+                    Grafana Loki log URL to notify when a backup begins,
                    ends, or fails.
                example: "http://localhost:3100/loki/api/v1/push"
            labels:
@@ -2718,6 +3112,11 @@ properties:
                    app: "borgmatic"
                    config: "__config"
                    hostname: "__hostname"
            send_logs:
                type: boolean
                description: |
                    Send borgmatic logs to Loki. Defaults to false.
                example: true
        description: |
            Configuration for a monitoring integration with Grafana Loki. You
            can send the logs to a self-hosted instance or create an account at
@@ -2742,6 +3141,12 @@ properties:
                    project monitor. Used along with the data source name URL to
                    construct a cron URL.
                example: mymonitor
            environment:
                type: string
                description: |
                    Sentry monitor environment used in the call to Sentry. If
                    not set, the Sentry default is used.
                example: production
            states:
                type: array
                items:
@@ -2795,7 +3200,8 @@ properties:
            findmnt_command:
                type: string
                description: |
-                    Command to use instead of "findmnt".
+                    Deprecated and unused. Was the command to use instead of
                    "findmnt".
                example: /usr/local/bin/findmnt
        description: |
            Configuration for integration with the Btrfs filesystem.
@@ -2846,6 +3252,25 @@ properties:
        description: |
            Configuration for integration with Linux LVM (Logical Volume
            Manager).
    systemd:
        type: object
        additionalProperties: false
        properties:
            systemd_creds_command:
                type: string
                description: |
                    Command to use instead of "systemd-creds". Only used as a
                    fallback when borgmatic is run outside of a systemd service.
                example: /usr/local/bin/systemd-creds
            encrypted_credentials_directory:
                type: string
                description: |
                    Directory containing encrypted credentials for
                    "systemd-creds" to use instead of
                    "/etc/credstore.encrypted".
                example: /path/to/credstore.encrypted
        description: |
            Configuration for integration with systemd credentials.
    container:
        type: object
        additionalProperties: false
@@ -2866,6 +3291,14 @@ properties:
                description: |
                    Command to use instead of "keepassxc-cli".
                example: /usr/local/bin/keepassxc-cli
            ask_for_password:
                type: boolean
                description: |
                    Whether keepassxc-cli should prompt the user for a password.
                    Disabling this is only really useful if you're unlocking
                    your KeePassXC database with a key file instead of a
                    password. Defaults to true.
                example: false
            key_file:
                type: string
                description: |
@@ -17,7 +17,7 @@ def schema_filename():
    '''
    schema_path = os.path.join(os.path.dirname(borgmatic.config.__file__), 'schema.yaml')
-    with open(schema_path):
+    with open(schema_path, encoding='utf-8'):
        return schema_path
@@ -97,7 +97,11 @@ def apply_logical_validation(config_filename, parsed_configuration):
 def parse_configuration(
-    config_filename, schema_filename, arguments, overrides=None, resolve_env=True
+    config_filename,
    schema_filename,
    arguments,
    overrides=None,
    resolve_env=True,
 ):
    '''
    Given the path to a config filename in YAML format, the path to a schema filename in a YAML
@@ -147,7 +151,8 @@ def parse_configuration(
    if validation_errors:
        raise Validation_error(
-            config_filename, tuple(format_json_error(error) for error in validation_errors)
+            config_filename,
            tuple(format_json_error(error) for error in validation_errors),
        )
    apply_logical_validation(config_filename, config)
@@ -166,13 +171,14 @@ def normalize_repository_path(repository, base=None):
        return (
            os.path.abspath(os.path.join(base, repository)) if base else os.path.abspath(repository)
        )
-    elif repository.startswith('file://'):
+
    if repository.startswith('file://'):
        local_path = repository.partition('file://')[-1]
        return (
            os.path.abspath(os.path.join(base, local_path)) if base else os.path.abspath(local_path)
        )
-    else:
+
-        return repository
+    return repository
 def glob_match(first, second):
@@ -194,11 +200,13 @@ def repositories_match(first, second):
    '''
    if isinstance(first, str):
        first = {'path': first, 'label': first}
    if isinstance(second, str):
        second = {'path': second, 'label': second}
    return glob_match(first.get('label'), second.get('label')) or glob_match(
-        normalize_repository_path(first.get('path')), normalize_repository_path(second.get('path'))
+        normalize_repository_path(first.get('path')),
        normalize_repository_path(second.get('path')),
    )
@@ -219,7 +227,7 @@ def guard_configuration_contains_repository(repository, configurations):
            for config in configurations.values()
            for config_repository in config['repositories']
            if repositories_match(config_repository, repository)
-        )
+        ),
    )
    if count == 0:
@@ -1,9 +1,14 @@
 import collections
 import contextlib
 import enum
 import json
 import locale
 import logging
 import os
 import select
 import subprocess
 import textwrap
 import time
 import borgmatic.logger
@@ -14,6 +19,9 @@ ERROR_OUTPUT_MAX_LINE_COUNT = 25
 BORG_ERROR_EXIT_CODE_START = 2
 BORG_ERROR_EXIT_CODE_END = 99
 # See https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids
 BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS = {101, 102, 104, 105, 106}
 class Exit_status(enum.Enum):
    STILL_RUNNING = 1
@@ -22,7 +30,32 @@ class Exit_status(enum.Enum):
    ERROR = 4
-def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
+def command_is_borg(command, borg_local_path):
    '''
    Given a command as a sequence and the Borg local path, return whether that command is a call to
    Borg.
    '''
    parsed_command = command.split(' ', 1) if isinstance(command, str) else command
    if not parsed_command:
        return False
    return bool(borg_local_path and parsed_command[0] == borg_local_path)
 BORG_EXIT_CODE_TO_DESCRIPTION = {
    100: 'File changed while we backed it up',
    101: 'Include pattern never matched',
    102: 'General backup issue',
    103: 'File type or inode changed while we backed it up',
    104: 'Backup OS issue',
    105: 'Backup permission issue',
    106: 'Backup IO issue',
    107: 'Backup file not found',
 }
 def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):  # noqa: PLR0911
    '''
    Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
    exit code. If a Borg local path is given and matches the process' command, then interpret the
@@ -31,41 +64,51 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
    '''
    if exit_code is None:
        return Exit_status.STILL_RUNNING
    if exit_code == 0:
        return Exit_status.SUCCESS
-    if borg_local_path and command[0] == borg_local_path:
+    if not command_is_borg(command, borg_local_path):
-        # First try looking for the exit code in the borg_exit_codes configuration.
+        return Exit_status.ERROR
        for entry in borg_exit_codes or ():
            if entry.get('code') == exit_code:
                treat_as = entry.get('treat_as')
-                if treat_as == 'error':
+    description = BORG_EXIT_CODE_TO_DESCRIPTION.get(exit_code)
-                    logger.error(
+    description_parenthetical = f' ({description})' if description else ''
                        f'Treating exit code {exit_code} as an error, as per configuration'
                    )
                    return Exit_status.ERROR
                elif treat_as == 'warning':
                    logger.warning(
                        f'Treating exit code {exit_code} as a warning, as per configuration'
                    )
                    return Exit_status.WARNING
-        # If the exit code doesn't have explicit configuration, then fall back to the default Borg
+    # First try looking for the exit code in the borg_exit_codes configuration.
-        # behavior.
+    for entry in borg_exit_codes or ():
-        return (
+        if entry.get('code') == exit_code:
-            Exit_status.ERROR
+            treat_as = entry.get('treat_as')
-            if (
+
-                exit_code < 0
+            if treat_as == 'error':
-                or (
+                logger.error(
-                    exit_code >= BORG_ERROR_EXIT_CODE_START
+                    f'Treating exit code {exit_code}{description_parenthetical} as an error, as per configuration',
                    and exit_code <= BORG_ERROR_EXIT_CODE_END
                )
-            )
+                return Exit_status.ERROR
-            else Exit_status.WARNING
+
            if treat_as == 'warning':
                logger.warning(
                    f'Treating exit code {exit_code}{description_parenthetical} as a warning, as per configuration',
                )
                return Exit_status.WARNING
    # If the exit code doesn't have explicit configuration, then fall back to the default
    # behavior of treating Borg errors as errors and some Borg warnings as errors.
    if exit_code in BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS:
        logger.error(
            f'Treating exit code {exit_code}{description_parenthetical} as an error, as per borgmatic defaults',
        )
-    return Exit_status.ERROR
+        return Exit_status.ERROR
    if exit_code < 0 or (
        exit_code >= BORG_ERROR_EXIT_CODE_START and exit_code <= BORG_ERROR_EXIT_CODE_END
    ):
        return Exit_status.ERROR
    logger.warning(
        f'Treating exit code {exit_code}{description_parenthetical} as a warning, as per borgmatic defaults',
    )
    return Exit_status.WARNING
 def command_for_process(process):
@@ -76,149 +119,406 @@ def command_for_process(process):
    return process.args if isinstance(process.args, str) else ' '.join(process.args)
-def output_buffer_for_process(process, exclude_stdouts):
+def output_buffers_for_process(process, exclude_stdouts):
    '''
    Given a process as an instance of subprocess.Popen and a sequence of stdouts to exclude, return
-    either the process's stdout or stderr. The idea is that if stdout is excluded for a process, we
+    the process stdout and stderr as a tuple—but exclude the stdout if it's in the given stdouts to
-    still have stderr to log.
+    exclude.
    '''
-    return process.stderr if process.stdout in exclude_stdouts else process.stdout
+    return tuple(
        buffer for buffer in (process.stdout, process.stderr) if buffer not in exclude_stdouts
    )
-def append_last_lines(last_lines, captured_output, line, output_log_level):
+BORG_LOG_LEVEL_ELEVATION_THRESHOLD = 10
 def borg_json_log_line_to_record(line, log_level):
    '''
-    Given a rolling list of last lines, a list of captured output, a line to append, and an output
+    Given a single Borg "--log-json"-style log line and a log level, return the line converted to a
-    log level, append the line to the last lines and (if necessary) the captured output. Then log
+    logging.LogRecord instance. Return None if the line can't be parsed as JSON.
-    the line at the requested output log level.
+
    If Borg provides a log level in its JSON, prefer logging at that level. But if Borg doesn't
    provide a log level—or the log level given to this function is just a little bit higher than
    Borg's—elevate to that level. This supports use cases like elevating Borg's INFO level logs to
    borgmatic's custom ANSWER level so that requested data shows up even at the default verbosity.
    '''
-    last_lines.append(line)
+    with contextlib.suppress(json.JSONDecodeError, TypeError, KeyError, AttributeError):
        log_data = json.loads(line)
        log_type = log_data.get('type')
-    if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
+        if log_type == 'log_message':
-        last_lines.pop(0)
+            borg_log_level = logging._nameToLevel.get(log_data.get('levelname'))
            log_level_delta = 0 if log_level is None else log_level - borg_log_level
-    if output_log_level is None:
+            if log_level_delta > 0 and log_level_delta < BORG_LOG_LEVEL_ELEVATION_THRESHOLD:
-        captured_output.append(line)
+                return logging.makeLogRecord(
-    else:
+                    dict(
-        logger.log(output_log_level, line)
+                        levelno=log_level,
-
+                        created=log_data.get('time'),
-
+                        msg=log_data.get('message'),
-def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):
+                        levelname=logging.getLevelName(log_level),
-    '''
+                        name=log_data.get('name'),
    Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
    process with the requested log level. Additionally, raise a CalledProcessError if a process
    exits with an error (or a warning for exit code 1, if that process does not match the Borg local
    path).
    If output log level is None, then instead of logging, capture output for each process and return
    it as a dict from the process to its output. Use the given Borg local path and exit code
    configuration to decide what's an error and what's a warning.
    For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
    stdouts are given to exclude, then for any matching processes, log from their stderr instead.
    Note that stdout for a process can be None if output is intentionally not captured. In which
    case it won't be logged.
    '''
    # Map from output buffer to sequence of last lines.
    buffer_last_lines = collections.defaultdict(list)
    process_for_output_buffer = {
        output_buffer_for_process(process, exclude_stdouts): process
        for process in processes
        if process.stdout or process.stderr
    }
    output_buffers = list(process_for_output_buffer.keys())
    captured_outputs = collections.defaultdict(list)
    still_running = True
    # Log output for each process until they all exit.
    while True:
        if output_buffers:
            (ready_buffers, _, _) = select.select(output_buffers, [], [])
            for ready_buffer in ready_buffers:
                ready_process = process_for_output_buffer.get(ready_buffer)
                # The "ready" process has exited, but it might be a pipe destination with other
                # processes (pipe sources) waiting to be read from. So as a measure to prevent
                # hangs, vent all processes when one exits.
                if ready_process and ready_process.poll() is not None:
                    for other_process in processes:
                        if (
                            other_process.poll() is None
                            and other_process.stdout
                            and other_process.stdout not in output_buffers
                        ):
                            # Add the process's output to output_buffers to ensure it'll get read.
                            output_buffers.append(other_process.stdout)
                while True:
                    line = ready_buffer.readline().rstrip().decode()
                    if not line or not ready_process:
                        break
                    # Keep the last few lines of output in case the process errors, and we need the output for
                    # the exception below.
                    append_last_lines(
                        buffer_last_lines[ready_buffer],
                        captured_outputs[ready_process],
                        line,
                        output_log_level,
                    )
                )
-        if not still_running:
+            return logging.makeLogRecord(
-            break
+                dict(
                    levelno=borg_log_level,
                    created=log_data.get('time'),
                    msg=log_data.get('message'),
                    levelname=log_data.get('levelname'),
                    name=log_data.get('name'),
                )
            )
-        still_running = False
+        if log_type == 'file_status':
            return logging.makeLogRecord(
                dict(
                    levelno=log_level,
                    created=time.time(),
                    msg=f'{log_data.get("status")} {log_data.get("path")}',
                    levelname=logging.getLevelName(log_level),
                    name='borg.file_status',
                )
            )
-        for process in processes:
+    return None
            exit_code = process.poll() if output_buffers else process.wait()
-            if exit_code is None:
+
-                still_running = True
+def log_line_to_record(line, log_level):
-                command = process.args.split(' ') if isinstance(process.args, str) else process.args
+    '''
    Given a log data dict for a single Borg log entry and a log level, return it converted to a
    logging.LogRecord instance.
    '''
    return logging.makeLogRecord(
        dict(
            msg=line,
            levelno=log_level,
            levelname=logging.getLevelName(log_level),
        )
    )
 def parse_log_line(line, log_level, elevate_stderr, borg_local_path, command):
    '''
    Given a raw output line from an external program, whether this line came from stderr and should
    be elevated to error/warning, the Borg local path, and the command as a sequence, return a
    logging.LogRecord instance containing its parsed data.
    If the command being run is Borg, and the log line is JSON-formatted log data, then grab the log
    level from it and log the parsed JSON to be consumed later by a Python logging.Formatter.
    But for non-Borg commands, elevate stderr-sourced logs to ERROR. The one exception is if the log
    came from stderr and the string "warning:" appears at the start of the log line. In that case,
    just elevate the log level to a WARN.
    '''
    if command_is_borg(command, borg_local_path):
        log_record = borg_json_log_line_to_record(line, log_level)
        if log_record:
            return log_record
    if elevate_stderr:
        return log_line_to_record(
            line, logging.WARNING if line.lower().startswith('warning:') else logging.ERROR
        )
    return log_line_to_record(line, log_level)
 def handle_log_record(log_record, last_lines=None):
    '''
    Given a log record to be logged and a rolling list of last lines, append the record's message to
    the last lines (if given). Then (if the log level is not None), log the record.
    Return the log record.
    '''
    log_message = log_record.getMessage()
    if last_lines is not None:
        last_lines.append(log_message)
        if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
            last_lines.pop(0)
    if log_record.levelno is not None:
        logger.handle(log_record)
    return log_record
 READ_CHUNK_SIZE = 4096
 def read_lines(buffer, process, line_separator='\n'):
    '''
    Given a Python buffer (like stdout) ready for reading, its process, and a line separator,
    repeatedly yield a tuple of (decoded) lines from the buffer until the process has exited.
    It is assumed that this function's generator is used in conjunction with an external select()
    call to know when to read more lines. Otherwise, the generator will busywait if it's called in a
    tight loop.
    '''
    data = b''
    encoded_separator = line_separator.encode()
    separator_size = len(encoded_separator)
    encoding = locale.getpreferredencoding()
    while True:
        chunk = os.read(buffer.fileno(), READ_CHUNK_SIZE)
        if not chunk:  # EOF
            # The process is still running, so we keep running too.
            if process.poll() is None:  # pragma: no cover
                continue
-            command = process.args.split(' ') if isinstance(process.args, str) else process.args
+            break
            exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
-            if exit_status in (Exit_status.ERROR, Exit_status.WARNING):
+        data += chunk
-                # If an error occurs, include its output in the raised exception so that we don't
+        lines = []
                # inadvertently hide error output.
                output_buffer = output_buffer_for_process(process, exclude_stdouts)
                last_lines = buffer_last_lines[output_buffer] if output_buffer else []
-                # Collect any straggling output lines that came in since we last gathered output.
+        # Split the data into lines, holding back anything leftover that might
-                while output_buffer:  # pragma: no cover
+        # be a partial line.
-                    line = output_buffer.readline().rstrip().decode()
+        while True:
-                    if not line:
+            separator_position = data.find(encoded_separator)
                        break
-                    append_last_lines(
+            if separator_position == -1:
                        last_lines, captured_outputs[process], line, output_log_level=logging.ERROR
                    )
                if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
                    last_lines.insert(0, '...')
                # Something has gone wrong. So vent each process' output buffer to prevent it from
                # hanging. And then kill the process.
                for other_process in processes:
                    if other_process.poll() is None:
                        other_process.stdout.read(0)
                        other_process.kill()
                if exit_status == Exit_status.ERROR:
                    raise subprocess.CalledProcessError(
                        exit_code, command_for_process(process), '\n'.join(last_lines)
                    )
                still_running = False
                break
-    if captured_outputs:
+            lines.append(data[:separator_position].decode(encoding))
-        return {
+            data = data[separator_position + separator_size :]
-            process: '\n'.join(output_lines) for process, output_lines in captured_outputs.items()
+
-        }
+        yield tuple(lines)
    # Yield any leftover data from the end of the buffer.
    if data:
        yield (data.decode(encoding).rstrip(),)
 Buffer_reader = collections.namedtuple(
    'Buffer_reader',
    ('lines', 'process'),
 )
 Process_metadata = collections.namedtuple(
    'Process_metadata',
    ('last_lines', 'capture'),
 )
 def log_buffer_lines(
    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
 ):
    '''
    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
    whether to capture stderr, read and log any ready output lines from the buffers. Additionally,
    for any log records with a log level that meets or exceeds the output log level, yield those log
    messages for capture.
    This function just does one "turn of the crank" of logging buffer output. It is intended to be
    called repeatedly to continue to process buffers.
    '''
    if not buffer_readers:
        return
    (ready_buffers, _, _) = select.select(buffer_readers.keys(), [], [])
    for ready_buffer in ready_buffers:
        reader = buffer_readers[ready_buffer]
        # The "ready" process has exited, but it might be a pipe destination with other
        # processes (pipe sources) waiting to be read from. So as a measure to prevent
        # hangs, vent all processes when one exits.
        if reader.process and reader.process.poll() is not None:
            for other_process in process_metadatas:
                if (
                    other_process.poll() is None
                    and other_process.stdout
                    and other_process.stdout not in buffer_readers
                ):
                    # Add the process's output to buffer_readers to ensure it'll get read.
                    buffer_readers[other_process.stdout] = Buffer_reader(
                        read_lines(other_process.stdout, other_process), other_process
                    )
        try:
            lines = next(reader.lines)
        except StopIteration:
            continue
        for line in lines:
            if not line or not reader.process:
                continue
            # Keep the last few lines of output in case the process errors and we need the
            # output for the exception below.
            log_record = handle_log_record(
                parse_log_line(
                    line=line,
                    log_level=output_log_level,
                    elevate_stderr=(ready_buffer == reader.process.stderr and not capture_stderr),
                    borg_local_path=borg_local_path,
                    command=reader.process.args,
                ),
                last_lines=process_metadatas[reader.process].last_lines,
            )
            if (
                log_record.levelno is None
                or output_log_level is None
                or log_record.levelno >= output_log_level
            ) and process_metadatas[reader.process].capture:
                yield log_record.getMessage()
 def raise_for_process_errors(buffer_readers, process_metadatas, borg_local_path, borg_exit_codes):
    '''
    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
    Process_metadata instance, Borg's local path, a sequence of exit code configuration dicts, check
    the given processes for error or warning exit codes. If found, vent or kill any running
    processes. In the case of an error exit code, raise. In the case of warning, return
    Exit_status.WARNING. Otherwise, return None.
    '''
    result_status = None
    for process in process_metadatas:
        exit_code = process.poll() if buffer_readers else process.wait()
        if exit_code is None:
            continue
        exit_status = interpret_exit_code(process.args, exit_code, borg_local_path, borg_exit_codes)
        if exit_status not in {Exit_status.ERROR, Exit_status.WARNING}:
            continue
        # Something has gone wrong. So vent each process' output buffer to prevent it from
        # hanging. And then kill the process.
        for other_process in process_metadatas:
            if other_process.poll() is None:
                other_process.stdout.read(0)
                other_process.kill()
        if exit_status == Exit_status.WARNING:
            result_status = Exit_status.WARNING
            continue
        last_lines = process_metadatas[process].last_lines
        # If an error occurs, include its output in the raised exception so that we don't
        # inadvertently hide error output.
        if len(last_lines) >= ERROR_OUTPUT_MAX_LINE_COUNT:
            last_lines.insert(0, '...')
        raise subprocess.CalledProcessError(
            exit_code,
            command_for_process(process),
            '\n'.join(last_lines),
        )
    return result_status
 def log_remaining_buffer_lines(
    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
 ):
    '''
    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
    whether to capture stderr, drain and log any remaining output lines from the buffers until
    they're empty. Additionally, for any log records with a log level that meets or exceeds the
    output log level, yield those log messages for capture.
    '''
    for output_buffer, reader in buffer_readers.items():
        if not reader.process:
            continue
        for lines in reader.lines:
            for line in lines:
                log_record = handle_log_record(
                    parse_log_line(
                        line=line.rstrip(),
                        log_level=output_log_level,
                        elevate_stderr=(
                            output_buffer == reader.process.stderr and not capture_stderr
                        ),
                        borg_local_path=borg_local_path,
                        command=reader.process.args,
                    ),
                )
                if (
                    log_record.levelno is None
                    or output_log_level is None
                    or log_record.levelno >= output_log_level
                ) and process_metadatas[reader.process].capture:
                    yield log_record.getMessage()
 def log_outputs(
    processes,
    exclude_stdouts,
    output_log_level,
    borg_local_path,
    borg_exit_codes,
    capture_stderr=False,
 ):
    '''
    Given a sequence of subprocess.Popen() instances for multiple processes, log the outputs (stderr
    and stdout). Use the requested output log level for stdout, but always log stderr to the ERROR
    log level. Additionally, raise a CalledProcessError if a process exits with an error (or a
    warning for exit code 1, if that process does not match the Borg local path).
    If the output log level is None, then instead of logging, capture the output for the last
    process given and yield it one line at a time. This includes stderr if capture stderr is set.
    But if the output log level is not None, don't yield anything.
    This yielding means that this function is a generator, and must be consumed in order to execute.
    Use the given Borg local path and exit code configuration to decide what's an error and what's a
    warning. If any stdouts are given to exclude, then for any matching processes, ignore those
    buffers. Also note that stdout for a process can be None if output is intentionally not
    captured, in which case it won't be logged.
    '''
    # Map from output buffer to Process_metadata instance. By convention, the last process is the
    # process to capture.
    process_metadatas = {
        process: Process_metadata(last_lines=[], capture=bool(process == processes[-1]))
        for process in processes
    }
    # Map from buffer to Buffer_reader instance.
    buffer_readers = {
        buffer: Buffer_reader(read_lines(buffer, process), process)
        for process in processes
        if process.stdout or process.stderr
        for buffer in output_buffers_for_process(process, exclude_stdouts)
    }
    # Log output lines for each process until they all exit.
    while True:
        yield from log_buffer_lines(
            buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
        )
        if (
            raise_for_process_errors(
                buffer_readers, process_metadatas, borg_local_path, borg_exit_codes
            )
            == Exit_status.WARNING
        ):
            break
        if all(process.poll() is not None for process in processes):
            break
    # Now that all processes have exited, drain and consume any last output.
    yield from log_remaining_buffer_lines(
        buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
    )
 SECRET_COMMAND_FLAG_NAMES = {'--password'}
@@ -255,19 +555,19 @@ def log_command(full_command, input_file=None, output_file=None, environment=Non
            ' '.join(
                tuple(
                    f'{key}=***'
-                    for key in (environment or {}).keys()
+                    for key in (environment or {})
                    if any(
                        key.startswith(prefix)
                        for prefix in PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG
                    )
                )
-                + mask_command_secrets(full_command)
+                + mask_command_secrets(full_command),
            ),
            width=MAX_LOGGED_COMMAND_LENGTH,
            placeholder=' ...',
        )
        + (f" < {getattr(input_file, 'name', input_file)}" if input_file else '')
-        + (f" > {getattr(output_file, 'name', output_file)}" if output_file else '')
+        + (f" > {getattr(output_file, 'name', output_file)}" if output_file else ''),
    )
@@ -288,18 +588,19 @@ def execute_command(
    borg_local_path=None,
    borg_exit_codes=None,
    run_to_completion=True,
    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
    '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. If an open output file object is given, then write stdout to the file and only
+    the given log level. If an open output file object is given, then write stdout to the file and
-    log stderr. If an open input file object is given, then read stdin from the file. If shell is
+    only log stderr. If an open input file object is given, then read stdin from the file. If shell
-    True, execute the command within a shell. If an environment variables dict is given, then pass
+    is True, execute the command within a shell. If an environment variables dict is given, then
-    it into the command. If a working directory is given, use that as the present working directory
+    pass it into the command. If a working directory is given, use that as the present working
-    when running the command. If a Borg local path is given, and the command matches it (regardless
+    directory when running the command. If a Borg local path is given, and the command matches it
-    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
-    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    codes are given as a sequence of exit code configuration dicts, then use that configuration to
-    what's an error and what's a warning. If run to completion is False, then return the process for
+    decide what's an error and what's a warning. If run to completion is False, then return the
-    the command without executing it to completion.
+    process for the command without executing it to completion.
    Raise subprocesses.CalledProcessError if an error occurs while running the command.
    '''
@@ -307,32 +608,36 @@ def execute_command(
    do_not_capture = bool(output_file is DO_NOT_CAPTURE)
    command = ' '.join(full_command) if shell else full_command
-    process = subprocess.Popen(
+    process = subprocess.Popen(  # noqa: S603
        command,
        stdin=input_file,
        stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-        stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
+        stderr=None if do_not_capture else subprocess.PIPE,
        shell=shell,
        env=environment,
        cwd=working_directory,
-        # Necessary for passing credentials via anonymous pipe.
+        close_fds=close_fds,
        close_fds=False,
    )
    if not run_to_completion:
        return process
    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        log_outputs(
+        tuple(
-            (process,),
+            log_outputs(
-            (input_file, output_file),
+                (process,),
-            output_log_level,
+                (input_file, output_file),
-            borg_local_path,
+                output_log_level,
-            borg_exit_codes,
+                borg_local_path,
                borg_exit_codes,
            )
        )
    return None
 def execute_command_and_capture_output(
    full_command,
    output_log_level=None,
    input_file=None,
    capture_stderr=False,
    shell=False,
@@ -340,17 +645,22 @@ def execute_command_and_capture_output(
    working_directory=None,
    borg_local_path=None,
    borg_exit_codes=None,
    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
    '''
    Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    output (stdout) as a generator that yields one line at a time. The generator must be consumed in
-    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    order for the called command to execute.
-    execute the command within a shell. If an environment variables dict is given, then pass it into
+
-    the command. If a working directory is given, use that as the present working directory when
+    If an output log level is given, then instead of suppressing log output, also output the
-    running the command. If a Borg local path is given, and the command matches it (regardless of
+    captured lines at the given log level.  If an input file descriptor is given, then pipe it to
-    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    the command's stdin. If capture stderr is True, then capture stderr in addition to stdout. If
-    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    shell is True, execute the command within a shell. If an environment variables dict is given,
-    error and what's a warning.
+    then pass it into the command. If a working directory is given, use that as the present working
    directory when running the command. If a Borg local path is given, and the command matches it
    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
    codes are given as a sequence of exit code configuration dicts, then use that configuration to
    decide what's an error and what's a warning.
    Raise subprocesses.CalledProcessError if an error occurs while running the command.
    '''
@@ -358,15 +668,17 @@ def execute_command_and_capture_output(
    command = ' '.join(full_command) if shell else full_command
    try:
-        output = subprocess.check_output(
+        process = subprocess.Popen(  # noqa: S603
            command,
            stdin=input_file,
-            stderr=subprocess.STDOUT if capture_stderr else None,
+            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE
            if capture_stderr or command_is_borg(command, borg_local_path)
            else None,
            shell=shell,
            env=environment,
            cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
+            close_fds=close_fds,
            close_fds=False,
        )
    except subprocess.CalledProcessError as error:
        if (
@@ -374,9 +686,21 @@ def execute_command_and_capture_output(
            == Exit_status.ERROR
        ):
            raise
        output = error.output
-    return output.decode() if output is not None else None
+        if error.output is not None:
            yield from iter(error.output.decode(locale.getpreferredencoding()).splitlines())
        return
    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
        yield from log_outputs(
            (process,),
            (input_file,),
            output_log_level,
            borg_local_path,
            borg_exit_codes,
            capture_stderr=capture_stderr,
        )
 def execute_command_with_processes(
@@ -390,22 +714,26 @@ def execute_command_with_processes(
    working_directory=None,
    borg_local_path=None,
    borg_exit_codes=None,
    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
    '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. Simultaneously, continue to poll one or more active processes so that they
+    the given log level. Simultaneously, continue to poll one or more active processes so that they
    run as well. This is useful, for instance, for processes that are streaming output to a named
    pipe that the given command is consuming from.
    If an open output file object is given, then write stdout to the file and only log stderr. But
    if output log level is None, instead suppress logging and return the captured output for (only)
-    the given command. If an open input file object is given, then read stdin from the file. If
+    the given command as a generator that yields one line at a time. The generator must be consumed
-    shell is True, execute the command within a shell. If an environment variables dict is given,
+    in order for the called command to execute—regardless of the output log level.
-    then pass it into the command. If a working directory is given, use that as the present working
+
-    directory when running the command. If a Borg local path is given, then for any matching command
+    If an open input file object is given, then read stdin from the file. If shell is True, execute
-    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    the command within a shell. If an environment variables dict is given, then pass it into the
-    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    command. If a working directory is given, use that as the present working directory when running
-    configuration to decide what's an error and what's a warning.
+    the command. If a Borg local path is given, then for any matching command or process (regardless
    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
    given as a sequence of exit code configuration dicts, then use that configuration to decide
    what's an error and what's a warning.
    Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
    upstream process.
@@ -415,18 +743,15 @@ def execute_command_with_processes(
    command = ' '.join(full_command) if shell else full_command
    try:
-        command_process = subprocess.Popen(
+        command_process = subprocess.Popen(  # noqa: S603
            command,
            stdin=input_file,
            stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-            stderr=(
+            stderr=None if do_not_capture else subprocess.PIPE,
                None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
            ),
            shell=shell,
            env=environment,
            cwd=working_directory,
-            # Necessary for passing credentials via anonymous pipe.
+            close_fds=close_fds,
            close_fds=False,
        )
    except (subprocess.CalledProcessError, OSError):
        # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.
@@ -435,16 +760,14 @@ def execute_command_with_processes(
            if process.poll() is None:
                process.stdout.read(0)
                process.kill()
        raise
    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        captured_outputs = log_outputs(
+        yield from log_outputs(
-            tuple(processes) + (command_process,),
+            (*processes, command_process),
            (input_file, output_file),
            output_log_level,
            borg_local_path,
            borg_exit_codes,
        )
    if output_log_level is None:
        return captured_outputs.get(command_process)
@@ -13,6 +13,60 @@ logger = logging.getLogger(__name__)
 SOFT_FAIL_EXIT_CODE = 75
 BORG_PLACEHOLDER_NAMES = {
    'hostname',
    'fqdn',
    'reverse-fqdn',
    'now',
    'utcnow',
    'unixtime',
    'user',
    'pid',
    'borgversion',
    'borgmajor',
    'borgminor',
    'borgpatch',
 }
 VARIABLE_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
 def resolve_variable(match, context, hook_description):
    '''
    Given a re.Match instance of VARIABLE_PATTERN representing a matched variable name to be
    interpolated, a context dict, and a description of the current command hook, lookup the matched
    variable name within the given context and return its value.
    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
    off the backslashing and return the literal value.
    If the variable name isn't found in the given context (and isn't a Borg placeholder), then
    warn and return the literal value.
    '''
    name = match.group('name')
    # The would-be variable is escaped, so strip off the escaping and return the result without
    # resolving the name.
    if match.group('left_escape') and match.group('right_escape'):
        return '{' + name + '}'
    value = context.get(name)
    # The matched variable is in the context, so return its value.
    if value is not None:
        return shlex.quote(str(value))
    # The matched variable name isn't in the context. Warn about variables unknown to borgmatic, but
    # don't warn if the variable name happens to be a Borg placeholder, as Borg should hopefully
    # consume it.
    if name not in BORG_PLACEHOLDER_NAMES:
        logger.warning(
            f'Variable "{name}" is not supported in the {hook_description} hook',
        )
    # Return the whole string unaltered.
    return match.group(0)
 def interpolate_context(hook_description, command, context):
@@ -20,15 +74,10 @@ def interpolate_context(hook_description, command, context):
    Given a config filename, a hook description, a single hook command, and a dict of context
    names/values, interpolate the values by "{name}" into the command and return the result.
    '''
-    for name, value in context.items():
+    return VARIABLE_PATTERN.sub(
-        command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
+        functools.partial(resolve_variable, context=context, hook_description=hook_description),
-
+        command,
-    for unsupported_variable in re.findall(r'{\w+}', command):
+    )
        logger.warning(
            f"Variable '{unsupported_variable}' is not supported in {hook_description} hook"
        )
    return command
 def make_environment(current_environment, sys_module=sys):
@@ -47,25 +96,29 @@ def make_environment(current_environment, sys_module=sys):
    return environment
-def filter_hooks(command_hooks, before=None, after=None, hook_name=None, action_names=None):
+def filter_hooks(command_hooks, before=None, after=None, action_names=None, state_names=None):
    '''
    Given a sequence of command hook dicts from configuration and one or more filters (before name,
-    after name, calling hook name, or a sequence of action names), filter down the command hooks to
+    after name, a sequence of action names, and/or a sequence of execution result state names),
-    just the ones that match the given filters.
+    filter down the command hooks to just the ones that match the given filters.
    '''
    return tuple(
        hook_config
        for hook_config in command_hooks or ()
        for config_action_names in (hook_config.get('when'),)
        for config_state_names in (hook_config.get('states'),)
        if before is None or hook_config.get('before') == before
        if after is None or hook_config.get('after') == after
        if action_names is None
        or config_action_names is None
        or set(config_action_names or ()).intersection(set(action_names))
        if state_names is None
        or config_state_names is None
        or set(config_state_names or ()).intersection(set(state_names))
    )
-def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
+def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):  # noqa: PLR0912
    '''
    Given a sequence of command hook dicts from configuration, a umask to execute with (or None), a
    working directory to execute with, and whether this is a dry run, run the commands for each
@@ -118,7 +171,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                if dry_run:
                    continue
-                borgmatic.execute.execute_command(
+                borgmatic.execute.execute_command(  # noqa: S604
                    [command],
                    output_log_level=(
                        logging.ERROR if hook_config.get('after') == 'error' else logging.ANSWER
@@ -126,6 +179,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                    shell=True,
                    environment=make_environment(os.environ),
                    working_directory=working_directory,
                    close_fds=True,
                )
        finally:
            if original_umask:
@@ -143,7 +197,7 @@ class Before_after_hooks:
           before_after='do_stuff',
           umask=config.get('umask'),
           dry_run=dry_run,
-           hook_name='myhook',
+           action_names=['create'],
       ):
            do()
            some()
@@ -160,22 +214,20 @@ class Before_after_hooks:
        umask,
        working_directory,
        dry_run,
        hook_name=None,
        action_names=None,
        **context,
    ):
        '''
        Given a sequence of command hook configuration dicts, the before/after name, a umask to run
-        commands with, a working directory to run commands with, a dry run flag, the name of the
+        commands with, a working directory to run commands with, a dry run flag, a sequence of
-        calling hook, a sequence of action names, and any context for the executed commands, save
+        action names, and any context for the executed commands, save those data points for use
-        those data points for use below.
+        below.
        '''
        self.command_hooks = command_hooks
        self.before_after = before_after
        self.umask = umask
        self.working_directory = working_directory
        self.dry_run = dry_run
        self.hook_name = hook_name
        self.action_names = action_names
        self.context = context
@@ -188,7 +240,6 @@ class Before_after_hooks:
                borgmatic.hooks.command.filter_hooks(
                    self.command_hooks,
                    before=self.before_after,
                    hook_name=self.hook_name,
                    action_names=self.action_names,
                ),
                self.umask,
@@ -202,7 +253,7 @@ class Before_after_hooks:
            # Trigger the after hook manually, since raising here will prevent it from being run
            # otherwise.
-            self.__exit__(None, None, None)
+            self.__exit__(exception_type=type(error), exception=error, traceback=None)
            raise ValueError(f'Error running before {self.before_after} hook: {error}')
@@ -215,8 +266,8 @@ class Before_after_hooks:
                borgmatic.hooks.command.filter_hooks(
                    self.command_hooks,
                    after=self.before_after,
                    hook_name=self.hook_name,
                    action_names=self.action_names,
                    state_names=['fail' if exception_type else 'finish'],
                ),
                self.umask,
                self.working_directory,
@@ -34,7 +34,8 @@ def load_credential(hook_config, config, credential_parameters):
                config.get('working_directory', ''),
                (hook_config or {}).get('secrets_directory', DEFAULT_SECRETS_DIRECTORY),
                secret_name,
-            )
+            ),
            encoding='utf-8',
        ) as secret_file:
            return secret_file.read().rstrip(os.linesep)
    except (FileNotFoundError, OSError) as error:
@@ -23,7 +23,8 @@ def load_credential(hook_config, config, credential_parameters):
    try:
        with open(
-            os.path.join(config.get('working_directory', ''), expanded_credential_path)
+            os.path.join(config.get('working_directory', ''), expanded_credential_path),
            encoding='utf-8',
        ) as credential_file:
            return credential_file.read().rstrip(os.linesep)
    except (FileNotFoundError, OSError) as error:
@@ -29,6 +29,7 @@ def load_credential(hook_config, config, credential_parameters):
    command = (
        tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
        + ('show', '--show-protected', '--attributes', 'Password')
        + (('--no-password',) if not (hook_config or {}).get('ask_for_password', True) else ())
        + (
            ('--key-file', hook_config['key_file'])
            if hook_config and hook_config.get('key_file')
@@ -42,4 +43,6 @@ def load_credential(hook_config, config, credential_parameters):
        + (expanded_database_path, attribute_name)  # Ensure database and entry are last.
    )
-    return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+    return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
        os.linesep
    )
@@ -120,5 +120,8 @@ def resolve_credential(value, config):
        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
    return borgmatic.hooks.dispatch.call_hook(
-        'load_credential', config, hook_name, tuple(credential_parameters)
+        'load_credential',
        config,
        hook_name,
        tuple(credential_parameters),
    )
@@ -1,6 +1,9 @@
 import logging
 import os
 import re
 import shlex
 import borgmatic.execute
 logger = logging.getLogger(__name__)
@@ -24,18 +27,35 @@ def load_credential(hook_config, config, credential_parameters):
        raise ValueError(f'Cannot load invalid credential name: "{name}"')
    credentials_directory = os.environ.get('CREDENTIALS_DIRECTORY')
    if not credentials_directory:
        raise ValueError(
            f'Cannot load credential "{credential_name}" because the systemd CREDENTIALS_DIRECTORY environment variable is not set'
        )
    if not CREDENTIAL_NAME_PATTERN.match(credential_name):
        raise ValueError(f'Cannot load invalid credential name "{credential_name}"')
    credentials_directory = os.environ.get('CREDENTIALS_DIRECTORY')
    if not credentials_directory:
        logger.debug(
            f'Falling back to loading credential "{credential_name}" via systemd-creds because the systemd CREDENTIALS_DIRECTORY environment variable is not set'
        )
        command = (
            *shlex.split((hook_config or {}).get('systemd_creds_command', 'systemd-creds')),
            'decrypt',
            os.path.join(
                (hook_config or {}).get(
                    'encrypted_credentials_directory', '/etc/credstore.encrypted'
                ),
                credential_name,
            ),
        )
        return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
            os.linesep
        )
    try:
-        with open(os.path.join(credentials_directory, credential_name)) as credential_file:
+        with open(
            os.path.join(credentials_directory, credential_name), encoding='utf-8'
        ) as credential_file:
            return credential_file.read().rstrip(os.linesep)
    except (FileNotFoundError, OSError) as error:
        logger.warning(error)
@@ -1,11 +1,14 @@
 import contextlib
 import glob
 import importlib
 import itertools
 import json
 import logging
 import os
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.data_source.config
 logger = logging.getLogger(__name__)
@@ -17,6 +20,29 @@ def use_streaming(hook_config, config):  # pragma: no cover
    return False
 MAXIMUM_CONFIG_SYMLINKS_TO_FOLLOW = 10
 def resolve_config_path_symlinks(path):
    '''
    Given a path, resolve and yield each successive symlink until the final non-symlink target. If
    the given path isn't a symlink, then just yield it.
    Raise ValueError if we have to follow too many symlinks without getting to the final target.
    '''
    original_path = path
    for _ in range(MAXIMUM_CONFIG_SYMLINKS_TO_FOLLOW):
        yield os.path.abspath(path)
        if not os.path.islink(path):
            return
        path = os.readlink(path)
    raise ValueError(f'Too many symlinks to follow for configuration path: {original_path}')
 def dump_data_sources(
    hook_config,
    config,
@@ -32,13 +58,22 @@ def dump_data_sources(
    the archive. But skip this if the bootstrap store_config_files option is False or if this is a
    dry run.
    If any configuration paths are symlinks, then store each symlink along with any destination
    paths as well.
    Return an empty sequence, since there are no ongoing dump processes from this hook.
    '''
    if hook_config and hook_config.get('store_config_files') is False:
        return []
    borgmatic_manifest_path = os.path.join(
-        borgmatic_runtime_directory, 'bootstrap', 'manifest.json'
+        borgmatic_runtime_directory,
        'bootstrap',
        'manifest.json',
    )
    resolved_config_paths = tuple(
        itertools.chain.from_iterable(resolve_config_path_symlinks(path) for path in config_paths)
    )
    if dry_run:
@@ -46,36 +81,40 @@ def dump_data_sources(
    os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
-    with open(borgmatic_manifest_path, 'w') as manifest_file:
+    with open(borgmatic_manifest_path, 'w', encoding='utf-8') as manifest_file:
        json.dump(
            {
                'borgmatic_version': importlib.metadata.version('borgmatic'),
-                'config_paths': config_paths,
+                'config_paths': resolved_config_paths,
            },
            manifest_file,
        )
-    patterns.extend(
+    borgmatic.hooks.data_source.config.inject_pattern(
-        borgmatic.borg.pattern.Pattern(
+        patterns,
            config_path, source=borgmatic.borg.pattern.Pattern_source.HOOK
        )
        for config_path in config_paths
    )
    patterns.append(
        borgmatic.borg.pattern.Pattern(
            os.path.join(borgmatic_runtime_directory, 'bootstrap'),
            source=borgmatic.borg.pattern.Pattern_source.HOOK,
-        )
+        ),
    )
    for config_path in resolved_config_paths:
        borgmatic.hooks.data_source.config.inject_pattern(
            patterns,
            borgmatic.borg.pattern.Pattern(
                config_path,
                source=borgmatic.borg.pattern.Pattern_source.HOOK,
            ),
        )
    return []
-def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):
    '''
-    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, and
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, the
-    whether this is a dry run, then remove the manifest file created above. If this is a dry run,
+    configured patterns, and whether this is a dry run, then remove the manifest file created above.
-    then don't actually remove anything.
+    If this is a dry run, then don't actually remove anything.
    '''
    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
@@ -86,7 +125,7 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
        'bootstrap',
    )
    logger.debug(
-        f'Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}'
+        f'Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}',
    )
    for manifest_directory in glob.glob(manifest_glob):
@@ -96,19 +135,22 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
        if dry_run:
            continue
-        try:
+        with contextlib.suppress(FileNotFoundError):
            os.remove(manifest_file_path)
        except FileNotFoundError:
            pass
-        try:
+        with contextlib.suppress(FileNotFoundError):
            os.rmdir(manifest_directory)
        except FileNotFoundError:
            pass
 def make_data_source_dump_patterns(
-    hook_config, config, borgmatic_runtime_directory, name=None
+    hook_config,
    config,
    borgmatic_runtime_directory,
    name=None,
    hostname=None,
    port=None,
    container=None,
    label=None,
 ):  # pragma: no cover
    '''
    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
--- a/Show More
+++ b/Show More
`@@ -1 +1 @@`
	`DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'`	`DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic' # pragma: nocover`