Fix schema example for "archive_username" (#1317 ).

Fix the (unreleased) "archive_hostname" option / "--archive-hostname" flag to pass an environment variable to Borg instead of a flag. Also add an "archive_username" option and corresponding "--archive-username" flag to override the "{user}" plaecholder (#1317 ).
Fix an error from the "diff" action when exclude options are configured (#1327 ).
2026-07-22 18:13:02 +02:00 · 2026-07-20 23:04:51 -07:00 · 2026-07-20 18:57:11 -07:00 · 2026-07-19 18:59:03 -07:00 · 2026-07-16 12:28:31 -07:00 · 2026-07-16 11:58:10 -07:00
359 changed files with 22579 additions and 7785 deletions
@@ -21,7 +21,7 @@ module.exports = function(eleventyConfig) {
            if (process.env.NODE_ENV == "production") {
                return link;
            }
-            return link.replace('https://torsion.org/borgmatic/', 'http://localhost:8080/');
+            return link.replace('https://torsion.org/', 'http://localhost:8080/');
        }
    };
    let markdownItAnchorOptions = {
@@ -44,7 +44,7 @@ module.exports = function(eleventyConfig) {
        templateFormats: [
          "md",
          "txt"
-        ]
+        ],
    }
 };

@@ -1,5 +1,5 @@
 name: build
-run-name: ${{ gitea.actor }} is building
+run-name: ${{ forgejo.actor }} is building
 on:
  push:
    branches: [main]
@@ -17,7 +17,7 @@ jobs:
  docs:
    needs: [test]
    runs-on: host
-    if: gitea.event_name == 'push'
+    if: forgejo.event_name == 'push'
    env:
      IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs

@@ -29,5 +29,3 @@ jobs:
          PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
      - run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
      - run: podman push "$IMAGE_NAME"
-      - run: scripts/export-docs-from-image
-      - run: curl --user "${{ secrets.REGISTRY_USERNAME }}:${{ secrets.REGISTRY_PASSWORD }}" --upload-file borgmatic-docs.tar.gz https://projects.torsion.org/api/packages/borgmatic-collective/generic/borgmatic-docs/$(head --lines=1 NEWS)/borgmatic-docs.tar.gz
@@ -1,10 +0,0 @@
-## Hold up, GitHub users
-
-Thanks for your contribution! 
-
-Unfortunately, we don't use GitHub pull requests to manage code contributions to this repository (and GitHub doesn't have any way to disable pull requests entirely). Instead, please see:
-
-https://torsion.org/borgmatic/#contributing
-
-... which provides full instructions on how to submit pull requests. You can even use your GitHub account to login.
-
@@ -0,0 +1,28 @@
+# AGENTS.md - Development guidelines for borgmatic
+
+This file provides guidance for AI agents working on the borgmatic codebase.
+
+## Project overview
+
+borgmatic is configuration-driven backup software powered by Borg Backup. It's a
+Python project using setuptools.
+
+Please do not use AI agents to modify this codebase. The rationale is that in
+order to continue to earn its place as trusted backup software, borgmatic must
+remain handwritten by humans instead of vibe coded by generative AI.
+
+Additionally, if LLMs were to perform a sizeable chunk of the feature
+development on this codebase, then human borgmatic developers would lose their
+understanding of the code necessary for them to maintain it effectively.
+
+Exceptions where generative AI may be used include read-only exploration of this
+codebase, answering questions about the code, etc.
+
+## Architecture notes
+
+- **main entry point**: `borgmatic.commands.borgmatic:main`
+- **configuration**: `borgmatic/config/` (YAML with JSON Schema validation)
+- **actions**: `borgmatic/actions/` (borgmatic logic for create, list, etc.)
+- **Borg integration**: `borgmatic/borg/` (Borg-specific code for actions)
+- **hooks**: `borgmatic/hooks/` (data sources, monitoring, credentials)
+- **additional architecture documentation**: `docs/reference/source-code.md`
@@ -1,4 +1,5 @@
 # This file only applies to the source dist tarball, not the built wheel.
+include NEWS
 include borgmatic/config/schema.yaml
 graft docs
 graft sample
@@ -1,3 +1,273 @@
+2.1.7.dev0
+ * #1309: Add support for the "--quick-stats" flag and the "quick_statistics" option to the "prune"
+   action. Borg >= 1.4.5 and < 2 only.
+ * #1317: Add an "archive_hostname" option and a corresponding "--archive-hostname" flag for
+   overriding the hostname used for the "{hostname}" placeholder in the "archive_name_format"
+   option. Also add an "archive_username" option and corresponding "--archive-username" flag to
+   override the "{user}" plaecholder. Both options/flags are Borg 1.4.5+ only.
+ * #1319: Fix the ZFS hook's overzealous unmounting of snapshot paths when a source dataset is at
+   "/".
+ * #1322: Fix for the "restore" action sometimes failing to find a database dump that was dumped
+   with a default port.
+ * #1324: For the MariaDB and MySQL hooks, add "events", "routines", and "tablespaces" options for
+   disabling dumping of scheduled events, stored routines, and tablespaces, respectively.
+ * #1327: Fix an error from the "diff" action when exclude options are configured.
+ * #1331: Fix the "repo-create" action to more surgically suppress Borg "Repository does not exist"
+   logs and avoid inadvertently suppressing other error logs.
+ * #1333: Fix the "--progress" flag on the "compact" action to actually update the progress of
+   segment compaction.
+ * #1334: Add the "CAP_FOWNER" capability to "CapabilityBoundingSet" in the sample systemd service,
+   so that Borg can open source files without changing file access times.
+ * Fix a bug in which the "compact" action does not pass a compact threshold of zero to Borg.
+
+2.1.6
+ * #1256: Fix a race condition in which borgmatic sometimes swallows Borg error output without
+   logging it.
+ * #1300: Fix the "source_directories_must_exist" option to support source directories relative to a
+   "working_directory".
+ * #1301: Expand the "patterns_from" and "exclude_from" options to support paths containing tildes
+   and globs.
+ * #1303: For the MariaDB and MySQL database hooks, include events, routines, and tablespaces when
+   dumping a database.
+ * #1303: For the MariaDB hook, include only a subset of system data when dumping the "mysql" system
+   database (or "all" databases), so the dump is actually restorable. See the documentation for more
+   information: https://torsion.org/borgmatic/reference/configuration/data-sources/mariadb/
+ * #1308: Update the Apprise monitoring hook's "url" option to support loading credentials with the
+   "{credential ...}" syntax. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/configuration/credentials/
+ * #1309: Add a "--quick-stats" flag and corresponding "quick_statistics" option for showing only
+   abbreviated statistics for the "create" action. Borg 1.4.5+ only.
+ * #1314: Add a minimal stand-alone borgmatic binary in addition to the standard one. The minimal
+   binary omits support for the Apprise monitoring hook.
+ * Add an experimental "browse" action providing a console UI for browsing your backups. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/how-to/inspect-your-backups/#browsing-backups
+ * Update the KeePassXC credential hook to support KeePassXC's secret service integration. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/reference/configuration/credentials/keepassxc/
+ * Add the NEWS changelog file to release tarball (#1298).
+ * Enable reply by email on projects.torsion.org, so replies to notification emails get posted as
+   comments on tickets.
+ * For the MariaDB and MySQL database hooks, escape quotes in passwords when the
+   "password_transport" option is "pipe".
+ * Add a development script for upgrading pinned dependencies.
+ * Fix the PostgreSQL database hook to properly parse "*options" values containing quoted spaces.
+ * Update the documentation to use Pagefind's component-based search UI.
+
+2.1.5
+ * #1229: Document the permissions needed for the PostgreSQL database hook:
+   https://torsion.org/borgmatic/reference/configuration/data-sources/postgresql/
+ * #1289: Add mutual TLS support for the Loki monitoring hook. See the documentation for more
+   information: https://torsion.org/borgmatic/reference/configuration/monitoring/loki/
+ * #1292: Fix a "source directories do not exist" regression when configuration paths are relative
+   symlinks and the bootstrap data source hook is enabled.
+ * #1294: Fix a regression in which SSH warnings from remote repositories broke the "spot" check
+   and other actions as well.
+ * #1295: Fix the ZFS hook to properly unmount snapshots for empty datasets.
+
+2.1.4
+ * #1266: Add a stand-alone borgmatic Linux binary to the release downloads to serve as another way
+   to install borgmatic. Consider this binary a beta feature.
+ * #1286: Fix a regression in which running borgmatic with no arguments and no configuration files
+   doesn't error as expected.
+ * #1257: Fix for the Loki monitoring hook not respecting the monitoring verbosity.
+ * #1264: Improve performance of the "info" and "repo-list" actions by eliminating a second "borg
+   info" call that supports a "no matching archives" warning. The warning still occurs; it's just
+   done now without the extra "borg info" call.
+ * #1268: Fix the "spot" check, "extract" check, and all uses of the archive name "latest" to
+   respect the "match_archives" and "archives_name_format" options. This means that borgmatic now
+   uses the "latest" archive that also matches those options instead of the latest archive overall.
+ * When Borg exits with a warning exit code, show a description of it, so you don't have to look up
+   the code.
+ * Split out borgmatic installation documentation to its own page, so it's easier to find.
+ * Switch the default borgmatic installation method from pipx to uv, as uv is faster and used for
+   borgmatic development. If you'd like to switch, see the documentation for more information:
+   https://torsion.org/borgmatic/how-to/upgrade/
+ * Move the project tracker from Gitea to Forgejo.
+ * Fix a regression in which borgmatic didn't show an error message when run with no configuration.
+ * Fix a traceback in the "spot" check with Borg 2.
+
+2.1.3
+ * #1175: Add a "files_changed" option for customizing Borg's file modification detection.
+ * #1175: Add a "msgpack_version_check" option to prevent Borg from validating msgpack's version.
+ * #1218: Add a "config show" action to display computed borgmatic configuration as YAML or JSON,
+   handy for fetching borgmatic configuration from external scripts. See the documentation for more
+   information:
+   https://torsion.org/borgmatic/reference/command-line/actions/config-show/
+ * #1228: Adjust the "spot" check so error output includes more information about what failed.
+ * #1236: Fix the "spot" check to skip hard links, as Borg doesn't produces hashes for them.
+ * #1243: Add a "diff" action for viewing the difference between the contents of two archives.
+ * #1248: Go back to treating Borg "file not found" warnings (exit code 107) as
+   warnings instead of errors. Otherwise, borgmatic can error on files that a user intentionally
+   deletes while a backup is running. You can still override this behavior with the
+   "borg_exit_codes" option. See the documentation for more information:
+   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
+ * #1248: Un-deprecate the "source_directories_must_exist" option and default it to true, to
+   compensate for Borg "file not found" warnings no longer being treated as errors.
+ * #1269: Fix the ZFS hook to support datasets with a "canmount" property of "noauto".
+ * #1270: Follow symlinks when backing up borgmatic configuration files to support the "bootstrap"
+   action.
+ * #1274: Add an optional override for the documentation development listen port and use Podman
+   Compose if present.
+ * #1281: Fix a unicode error when backing up a non-UTF-8 source filename with a
+   corresponding system locale.
+ * Add a policy about the use of generative AI in the borgmatic codebase:
+   https://torsion.org/borgmatic/how-to/develop-on-borgmatic/#use-of-generative-ai
+
+2.1.2
+ * #1231: If a source file is deleted during a "spot" check, consider the file as non-matching
+   and move on instead of immediately failing the entire check.
+ * #1250: Fix a regression in which the "--stats" flag hides statistics at default verbosity.
+ * #1251: Fix a regression in the ntfy monitoring hook in which borgmatic sends tags incorrectly,
+   resulting in "400 Bad Request" from ntfy.
+ * #1258: Fix a "codec can't decode byte" error when running commands that output multi-byte unicode
+   characters.
+ * #1260: Fix for SSH warnings from Borg showing up as JSON logs even without the "--log-json" flag.
+ * #1252: Work around Borg returning a warning exit code when a repository/archive check fails. Now,
+   borgmatic interprets such failures as errors.
+ * Deduplicate overlapping source directories and patterns so they don't throw off "spot" check file
+   counts and cause spurious check failures.
+
+2.1.1
+ * #1241: For the "recreate" action, actually pass the "--dry-run" flag through to Borg instead of
+   just skipping the Borg call.
+ * #1242: Fix a regression in which the "spot" check hung while collecting archive contents.
+ * #1244: When the "unsafe_skip_path_validation_before_create" option is enabled, don't log a
+   warning about it.
+ * #1245: Fix a regression in which the KeePassXC credential hook password prompt was invisible.
+ * #1246: Fix a regression in which the ntfy monitoring hook failed to send a ping when the
+   "priority" option was set.
+
+2.1.0
+ * TL;DR: Many logging, memory, and performance improvements. Mind those breaking changes!
+ * #485: When running commands (database clients, command hooks, etc.), elevate stderr output to
+   show up as borgmatic error logs.
+ * #858: With the "--log-json" flag, log borgmatic's own logs as JSON, not just Borg's.
+ * #1092: BREAKING: Treat most Borg warnings as errors by default, so for instance backups now fail
+   when source directories are missing. You can still override this behavior with the
+   "borg_exit_codes" option. See the documentation for more information:
+   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
+ * #1092: Deprecate the "source_directories_must_exist" option, as borgmatic now treats "backup file
+   not found" warnings from Borg as errors, which accomplishes the same thing.
+ * #1132: BREAKING/SECURITY: For the Healthchecks, Apprise, Pagerduty, and Loki monitoring hooks,
+   disable log sending when not explicitly enabled. This avoids revealing private log information to
+   third-party services. To send logs anyway, set the monitoring hook's "send_logs" option to
+   "true".
+ * #1204: When verbosity levels differ between console/monitoring/syslog/file, log Borg's output to
+   each one at a different level. Previously, it was logged at the maximum level of all the
+   verbosities.
+ * #1208: Fix for the "restore" action incorrectly extracting more database dumps than the
+   "--database" flag specifies.
+ * #1210: Fix an error when running the "spot" check or "extract" action with the "progress" option
+   or "--progress" flag.
+ * #1211: Fix an error about the runtime directory getting excluded by tweaking its logic and
+   lowering the error to a warning.
+ * #1212: Fix an error when restoring multiple directory-format database dumps at once.
+ * #1213: BREAKING: Support disabling both constant and variable interpolation by escaping with
+   backslashes. For instance, interpret "\{name\}" as literally "{name}" instead of trying to
+   resolve it as a constant/variable.
+ * #1220: Cleanup snapshots immediately after ZFS, LVM, or Btrfs hooks error—rather than waiting
+   until the next time borgmatic runs.
+ * #1221: Add an "unsafe_skip_path_validation_before_create" option to skip pre-backup safety
+   validation so as to reduce backup times on large filesystems.
+ * #1224: When running an "extract" check with the "--progress" flag, show file extraction progress.
+ * #1225: Improve performance and greatly reduce memory usage during pre-backup safety validation on
+   large filesystems.
+ * #1230: Fix another warning from LVM about leaked file descriptors, this time when calling
+   "lvcreate" from a command hook.
+ * #1234: Fix for the ntfy monitoring hook erroring on emojis in the "title" option.
+ * When syslog verbosity is enabled, log to systemd's journal (if present) with
+   structured data. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/command-line/logging/#systemd-journal
+ * SECURITY: Prevent shell injection attacks via constant interpolation in command hooks. (This was
+   already implemented for deprecated "before_*"/"after_*" command hooks.)
+ * Fix for an error in the "key import" action when importing a key from stdin.
+ * Fix the "recreate" action to include borgmatic-specific paths (database dumps, etc.) in recreated
+   archives.
+ * Update the "list" action to support the "--json" flag when the "--archive" flag is also used.
+ * Promote the ZFS, LVM, and Btrfs hooks from beta features to stable.
+
+2.0.13
+ * #1054: Allow the Btrfs hook to create and delete snapshots even when running
+   as a non-root user. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#non-root-user
+ * #1179: Add a "file_list_format" option for setting the "list" action's output format and an
+   "archive_list_format" option for setting the "repo-list" action's format.
+ * #1192: Fix for over-aggressive deduplication of source directories that contain the borgmatic
+   runtime directory, potentially resulting in data loss (data not getting backed up) when
+   snapshotting these source directories.
+ * #1192, #1163: Document potential interactions between security settings in borgmatic's sample
+   systemd service file and the ZFS, LVM, and Btrfs hooks.
+ * #1193: In the documentation for the MariaDB/MySQL database hooks, clarify how to set custom
+   command-line flags for database commands.
+ * #1193: For the MariaDB and MySQL database hooks, add a "socket_path" option for Unix socket
+   database connections.
+ * #1193: For the MariaDB and MySQL database hooks, creates a consistent snapshot by dumping all
+   tables in a single transaction.
+ * #1194: Fix for an incorrect diff command shown when running the "generate config" action with a
+   source configuration file. 
+ * #1195: Fix a regression in the ZFS, LVM, and Btrfs hooks in which snapshotted paths ignored
+   global excludes.
+ * #1201: Document a problematic interaction between borgmatic and systemd-tmpfiles:
+   https://torsion.org/borgmatic/reference/configuration/runtime-directory/#systemd-tmpfiles
+ * #1203: Fix that errors and exits when the borgmatic runtime directory is partially excluded by
+   configured excludes. Previously, borgmatic only errored when the runtime directory was completely
+   excluded.
+ * #1206: Adjust Btrfs snapshot paths so that Borg 1.x gets file cache hits when backing them up,
+   improving performance.
+ * Update the sample systemd timer with a shorter random delay when catching up on a missed run.
+
+2.0.12
+ * #1127: Fix for the database hooks not respecting the "working_directory" option.
+ * #1181: Add an "ask_for_password" option to the KeePassXC credential hook for disabling
+   KeePassXC's password prompt, e.g. if you're only using a key file to decrypt your database.
+ * #1184: Fix the fish shell completion's detection of version mismatches.
+ * #1186: Fix a regression in the Btrfs hook in which subvolume snapshots didn't get cleaned up
+   until the start of the next borgmatic run.
+ * In the SQLite database hook, run SQLite such that it exits upon encountering an error instead of,
+   you know, not doing that.
+ * Add documentation on repositories, including SSH, Rclone, S3, and B2:
+   https://torsion.org/borgmatic/reference/configuration/repositories/
+ * Improve documentation search results for individual configuration options.
+ * Add borgmatic release artifacts (wheel and tarball) to each release on the releases page:
+   https://projects.torsion.org/borgmatic-collective/borgmatic/releases
+ * Move the tarball of borgmatic's HTML documentation from the packages page to the releases page.
+
+2.0.11
+ * #957: Document borgmatic's limitations around parallelism—both its own and Borg's. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/how-to/make-per-application-backups/#limitations
+ * #1165: Fix for when the systemd service directories (RuntimeDirectory and StateDirectory) each
+   contain multiple paths.
+ * #1168: Fix for the "list", "info", and "delete" options in "extra_borg_options" being ignored
+   when "--archive" is omitted with Borg 1.x.
+ * #1169: Fix for a regression in the ZFS, LVM, and Btrfs hooks in which partial excludes of
+   snapshot paths were ignored.
+ * #1170: Fix for an inconsistent log level for Borg's last output line before exiting.
+ * #1172: Add an "environment" option to the Sentry monitoring hook.
+ * #1176: Fix the "--repository" flag not applying to command hooks.
+ * Add a "rename" option to "extra_borg_options" to support passing arbitrary flags to "borg
+   rename".
+ * Add documentation on patterns and excludes:
+   https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/
+ * Drop support for Python 3.9, which has been end-of-lifed.
+
+2.0.10
+ * #427: Expand the "extra_borg_options" option to support passing arbitrary Borg flags to every
+   Borg sub-command that borgmatic uses. As part of this, deprecate the "init" option under
+   "borg_extra_options" in favor of "repo_create".
+ * #942: Factor reference material out of the documentation how-to guides. This means there's now a
+   whole reference section in the docs! Check it out: https://torsion.org/borgmatic/
+ * #973: For the MariaDB and MySQL database hooks, add a "skip_names" option to ignore particular
+   databases when dumping "all".
+ * #1150: Fix for a runtime directory error when the "create" action is used with the "--log-json"
+   flag.
+ * #1150: Fix for a runtime directory error when the configured patterns contain a global exclude.
+ * #1161: Fix a traceback (TypeError) in the "check" action with Python 3.14.
+ * #1166: Add a "borg_key_file" option for setting the Borg repository key file path.
+ * Add documentation search.
+ * Change the URL of the local documentation development server to be more like the production URL.
+
 2.0.9
 * #1105: More accurately collect Btrfs subvolumes to snapshot. As part of this, the Btrfs hook no
   longer uses "findmnt" and the "findmnt_command" option is deprecated.
@@ -2,10 +2,7 @@
 title: borgmatic
 permalink: index.html
 ---
-
-## It's your data. Keep it that way.
-
-<img src="docs/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
+<img src="https://torsion.org/borgmatic/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">

 borgmatic is simple, configuration-driven backup software for servers and
 workstations. Protect your files with client-side encryption. Backup your
@@ -60,47 +57,48 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).

 ### Data

-<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.postgresql.org/"><img src="https://torsion.org/borgmatic/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mysql.com/"><img src="https://torsion.org/borgmatic/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://mariadb.com/"><img src="https://torsion.org/borgmatic/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mongodb.com/"><img src="https://torsion.org/borgmatic/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sqlite.org/"><img src="https://torsion.org/borgmatic/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://openzfs.org/"><img src="https://torsion.org/borgmatic/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="https://torsion.org/borgmatic/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="https://torsion.org/borgmatic/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://rclone.org"><img src="https://torsion.org/borgmatic/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="https://torsion.org/borgmatic/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>


 ### Monitoring

-<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pushover.net/"><img src="docs/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://healthchecks.io/"><img src="https://torsion.org/borgmatic/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://uptime.kuma.pet/"><img src="https://torsion.org/borgmatic/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronitor.io/"><img src="https://torsion.org/borgmatic/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronhub.io/"><img src="https://torsion.org/borgmatic/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pagerduty.com/"><img src="https://torsion.org/borgmatic/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pushover.net/"><img src="https://torsion.org/borgmatic/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://ntfy.sh/"><img src="https://torsion.org/borgmatic/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://grafana.com/oss/loki/"><img src="https://torsion.org/borgmatic/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://github.com/caronc/apprise/wiki"><img src="https://torsion.org/borgmatic/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.zabbix.com/"><img src="https://torsion.org/borgmatic/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="https://torsion.org/borgmatic/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>


 ### Credentials

-<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://systemd.io/"><img src="https://torsion.org/borgmatic/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="https://torsion.org/borgmatic/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="https://torsion.org/borgmatic/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="https://torsion.org/borgmatic/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>


 ## Getting started

-Your first step is to [install and configure
-borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/).
+Your first steps are to
+[install](https://torsion.org/borgmatic/how-to/install-borgmatic/) and
+[configure borgmatic](https://torsion.org/borgmatic/how-to/set-up-backups/).

-For additional documentation, check out the links above (left panel on wide screens)
+For additional documentation, check out the links on the top/left
 for <a href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
 reference guides</a>.

@@ -108,16 +106,16 @@ reference guides</a>.
 ## Hosting providers

 Need somewhere to store your encrypted off-site backups? The following hosting
-providers include specific support for Borg/borgmatic—and fund borgmatic
-development and hosting when you use these referral links to sign up:
+provider includes specific support for Borg/borgmatic—and funds borgmatic
+development and hosting when you use this referral links to sign up:

 <ul>
 <li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic">BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li>
- <li class="referral"><a href="https://hetzner.cloud/?ref=v9dOJ98Ic9I8">Hetzner</a>: A "storage box" that includes support for Borg</li>
 </ul>

-Additionally, rsync.net has a compatible storage offering, but does not fund
-borgmatic development or hosting.
+Additionally, Hetzner and rsync\.net have compatible storage offerings, but do
+not fund borgmatic development or hosting.
+

 ## Support and contributing

@@ -132,7 +130,7 @@ first. If you prefer to use an existing GitHub account, you can skip account
 creation and [login directly](https://projects.torsion.org/user/login).

 Also see the [security
-policy](https://torsion.org/borgmatic/docs/security-policy/) for any security
+policy](https://torsion.org/borgmatic/security-policy/) for any security
 issues.


@@ -177,7 +175,7 @@ discuss your idea. Note that you'll need to
 first. In general, contributions are very welcome. We don't bite!

 Also, please check out the [borgmatic development
-how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
+how-to](https://torsion.org/borgmatic/how-to/develop-on-borgmatic/) for
 info on cloning source code, running tests, etc.

 ### Recent contributors
@@ -0,0 +1,19 @@
+.
+apprise
+attrs
+certifi
+charset-normalizer
+click
+idna
+jsonschema
+jsonschema-specifications
+markdown
+oauthlib
+packaging
+pyyaml
+referencing
+requests
+requests-oauthlib
+rpds-py
+ruamel-yaml
+urllib3
@@ -0,0 +1,21 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --annotation-style line binary_requirements.in -o binary_requirements.txt
+apprise==1.10.0           # via -r binary_requirements.in
+attrs==26.1.0             # via jsonschema, referencing, -r binary_requirements.in
+.                         # via -r binary_requirements.in
+certifi==2026.5.20        # via apprise, requests, -r binary_requirements.in
+charset-normalizer==3.4.7  # via requests, -r binary_requirements.in
+click==8.4.1              # via apprise, -r binary_requirements.in
+idna==3.16                # via requests, -r binary_requirements.in
+jsonschema==4.26.0        # via borgmatic, -r binary_requirements.in
+jsonschema-specifications==2025.9.1  # via jsonschema, -r binary_requirements.in
+markdown==3.10.2          # via apprise, -r binary_requirements.in
+oauthlib==3.3.1           # via requests-oauthlib, -r binary_requirements.in
+packaging==26.2           # via borgmatic, -r binary_requirements.in
+pyyaml==6.0.3             # via apprise, -r binary_requirements.in
+referencing==0.37.0       # via jsonschema, jsonschema-specifications, -r binary_requirements.in
+requests==2.34.2          # via apprise, borgmatic, requests-oauthlib, -r binary_requirements.in
+requests-oauthlib==2.0.0  # via apprise, -r binary_requirements.in
+rpds-py==0.30.0           # via jsonschema, referencing, -r binary_requirements.in
+ruamel-yaml==0.19.1       # via borgmatic, -r binary_requirements.in
+urllib3==2.7.0            # via requests, -r binary_requirements.in
@@ -2,7 +2,6 @@ import logging

 import borgmatic.borg.borg
 import borgmatic.borg.repo_list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -19,26 +18,22 @@ def run_borg(
    '''
    Run the "borg" action for the given repository.
    '''
-    if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        borg_arguments.repository,
-    ):
-        logger.info('Running arbitrary Borg command')
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
-            borg_arguments.archive,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        borgmatic.borg.borg.run_arbitrary_borg(
-            repository['path'],
-            config,
-            local_borg_version,
-            options=borg_arguments.options,
-            archive=archive_name,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+    logger.info('Running arbitrary Borg command')
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        borg_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    borgmatic.borg.borg.run_arbitrary_borg(
+        repository['path'],
+        config,
+        local_borg_version,
+        options=borg_arguments.options,
+        archive=archive_name,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -1,7 +1,6 @@
 import logging

 import borgmatic.borg.break_lock
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -18,16 +17,12 @@ def run_break_lock(
    '''
    Run the "break-lock" action for the given repository.
    '''
-    if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        break_lock_arguments.repository,
-    ):
-        logger.info('Breaking repository and cache locks')
-        borgmatic.borg.break_lock.break_lock(
-            repository['path'],
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+    logger.info('Breaking repository and cache locks')
+    borgmatic.borg.break_lock.break_lock(
+        repository['path'],
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -0,0 +1,108 @@
+import signal
+
+import textual.app
+import textual.binding
+import textual.widgets
+
+import borgmatic.actions.browse.carousel
+import borgmatic.actions.browse.configuration_files_list
+import borgmatic.actions.browse.logs
+import borgmatic.actions.browse.repositories_list
+
+
+class Browse_app(textual.app.App):
+    '''
+    The main app / entry point for the browse action UI.
+    '''
+
+    BINDINGS = (
+        textual.binding.Binding(key='q', action='quit', description='quit'),
+        textual.binding.Binding(key='v', action='toggle_logs', description='view logs'),
+        textual.binding.Binding(
+            key='c', action='command_palette', description='commands', show=False
+        ),
+    )
+    COMMAND_PALETTE_BINDING = 'c'
+    CSS = '''
+        .panel {
+            border: round $primary;
+            border-title-color: $text-primary;
+            width: 100%;
+            height: 100%;
+        }
+
+        #logs {
+            width: 100%;
+            height: 50%;
+            display: none;
+        }
+    '''
+
+    def __init__(self, configs):
+        self.configs = configs
+
+        super().__init__()
+
+    def get_system_commands(self, screen):  # pragma: no cover
+        '''
+        Remove the screenshot system command because it produces broken screenshots. (Emoji
+        weirdness, etc.) Also remove minimize and maximize because they don't accomplish much with
+        out particular layout.
+        '''
+        yield from (
+            command
+            for command in super().get_system_commands(screen)
+            if command.title not in {'Screenshot', 'Minimize', 'Maximize'}
+        )
+
+    def compose(self):
+        '''
+        Compose a UI consisting of:
+
+          * a header with the application name
+          * a carousel container that contains the main UI panels
+          * a logs panel where Python logs show up (panel hidden by default)
+          * a footer with available keys listed
+        '''
+        yield textual.widgets.Header()
+        yield borgmatic.actions.browse.carousel.Carousel(
+            [
+                borgmatic.actions.browse.configuration_files_list.Configuration_files_list(
+                    self.configs
+                )
+            ]
+            if len(self.configs) > 1
+            else [
+                borgmatic.actions.browse.repositories_list.Repositories_list(
+                    next(iter(self.configs.values()))
+                )
+            ]
+        )
+
+        logs_panel = borgmatic.actions.browse.logs.Logs()
+        yield logs_panel
+        yield textual.widgets.Footer()
+
+        borgmatic.actions.browse.logs.log_to_widget(logs_panel)
+
+    def on_mount(self):
+        '''
+        Set the application title, which ends up in the header.
+        '''
+        self.title = 'borgmatic browse'
+
+    def action_toggle_logs(self):
+        '''
+        Toggle the show/hide status of the logs panel.
+        '''
+        logs_panel = self.query_one('#logs')
+        logs_panel.styles.display = 'none' if logs_panel.styles.display == 'block' else 'block'
+
+    def exit(self):  # pragma: no cover
+        '''
+        Exit the application. But first raise a SIGTERM (handled in borgmatic/signals.py) to
+        encourage a fast exit by killing any ongoing Borg subprocesses.
+        '''
+        signal.raise_signal(signal.SIGTERM)
+
+        super().exit()
@@ -0,0 +1,143 @@
+import argparse
+import collections
+import enum
+import json
+import logging
+
+import binaryornot.helpers
+
+import borgmatic.borg.extract
+import borgmatic.borg.list
+import borgmatic.borg.repo_list
+import borgmatic.borg.version
+
+logger = logging.getLogger(__name__)
+
+
+class Path_type(enum.Enum):
+    DIRECTORY = 'd'
+    LINK = 'l'
+    PIPE = 'p'
+    FILE = '-'
+
+
+# A data structure capturing a path stored in a Borg archive.
+Archive_path = collections.namedtuple(
+    'Archive_path',
+    ('path_type', 'file_path', 'link_target'),
+)
+
+
+def get_repository_archives(config, repository):
+    '''
+    Given a configuration dict and a repository dict, return a list of the repository's archives,
+    one dict per archive.
+    '''
+    with borgmatic.logger.Log_prefix(repository.get('label', repository['path'])):
+        logger.info('Listing repository')
+        repo_list_arguments = argparse.Namespace(
+            repository=repository['path'],
+            short=None,
+            format=None,
+            json=True,
+            prefix=None,
+            match_archives=None,
+            sort_by=None,
+            first=None,
+            last=None,
+        )
+        global_arguments = argparse.Namespace()
+        local_path = config.get('local_path', 'borg')
+        remote_path = config.get('remote_path')
+        local_borg_version = borgmatic.borg.version.local_borg_version(config, local_path)
+
+        return json.loads(
+            borgmatic.borg.repo_list.list_repository(
+                repository['path'],
+                config,
+                local_borg_version,
+                repo_list_arguments,
+                global_arguments,
+                local_path,
+                remote_path,
+            )
+        )
+
+
+def get_archive_paths(config, repository, archive_name):
+    '''
+    Given a configuration dict, a repository dict, and an archive name in that repository, get a
+    generator of files, directories, symlinks, etc. found in the archive, each as an Archive_path
+    instance.
+    '''
+    with borgmatic.logger.Log_prefix(repository.get('label', repository['path'])):
+        logger.info(f'Listing archive {archive_name}')
+
+        global_arguments = argparse.Namespace()
+        local_path = config.get('local_path', 'borg')
+        remote_path = config.get('remote_path')
+        local_borg_version = borgmatic.borg.version.local_borg_version(config, local_path)
+
+        return (
+            Archive_path(
+                path_data['type'],
+                path_data['path'],
+                path_data.get('linktarget'),
+            )
+            for path_data in borgmatic.borg.list.capture_archive_listing(
+                repository['path'],
+                archive_name,
+                config,
+                local_borg_version,
+                global_arguments,
+                local_path=local_path,
+                remote_path=remote_path,
+            )
+        )
+
+
+READLINES_HINT_BYTES = 100000
+TRUNCATION_MESSAGE = '[... truncated for display ...]'
+
+
+def get_archive_file_content(config, repository, archive_name, file_path):
+    '''
+    Given a configuration dict, a repository dict, an archive name in that repository, and a file
+    path in that archive, return the file's contents or None if the file can't be loaded, e.g.
+    because it's binary or can't be decoded.
+
+    If the file is too large, then truncate the returned content.
+    '''
+    with borgmatic.logger.Log_prefix(repository.get('label', repository['path'])):
+        logger.info(f'Getting archive content of file {file_path}')
+        local_path = config.get('local_path', 'borg')
+        remote_path = config.get('remote_path')
+
+        lines = borgmatic.borg.extract.extract_archive(
+            dry_run=False,
+            repository=repository['path'],
+            archive=archive_name,
+            paths=(file_path,),
+            config=config,
+            local_borg_version=borgmatic.borg.version.local_borg_version(config, local_path),
+            global_arguments=argparse.Namespace(),
+            local_path=local_path,
+            remote_path=remote_path,
+            destination_path=None,
+            strip_components=None,
+            extract_to_stdout=True,
+        ).stdout.readlines(READLINES_HINT_BYTES)
+
+        content = b''.join(lines)
+
+        if binaryornot.helpers.is_binary_string(content):
+            return None
+
+        try:
+            return (
+                content.decode()
+                if len(content) < READLINES_HINT_BYTES
+                else f'{content.decode()}\n{TRUNCATION_MESSAGE}'
+            )
+        except UnicodeDecodeError:
+            return None
@@ -0,0 +1,89 @@
+import textual.widgets
+
+import borgmatic.actions.browse.bindings
+import borgmatic.actions.browse.loading
+import borgmatic.actions.browse.workers
+
+
+class Archives_list(textual.widgets.OptionList):
+    '''
+    A widget for selecting a single Borg archive from among the archives in a repository. The item
+    selection event is handled in a Carousel instance, the parent widget of an Archives_list.
+    '''
+
+    BINDINGS = borgmatic.actions.browse.bindings.OPTION_LIST_BINDINGS
+
+    def __init__(self, config, repository):
+        '''
+        Given a configuration dict and a repository dict, prepare to load the archives from the
+        repository for eventual display in this widget. Actual loading kicks off in on_mount()
+        below.
+        '''
+        self.config = config
+        self.repository = repository
+
+        super().__init__(classes='panel')
+        self.border_title = '📚 archives'
+        self.highlighted_option_changed = False
+        self.archive_loaded = borgmatic.actions.browse.workers.Archive_loaded(
+            self, 'archive loaded'
+        )
+
+        self.loading_timer = borgmatic.actions.browse.loading.add_inline_loading_indicator(self)
+
+    def on_mount(self):
+        '''
+        When this widget gets mounted in the DOM, subscribe to archive loaded events so that we can
+        find out about archives as they load. Also start loading archives from the repository.
+
+        Loading is started *after* subscribing to the archive loaded signal so that there's not a
+        gap where we might miss out on signal publishes.
+        '''
+        self.archive_loaded.subscribe(self, self.on_archive_loaded)
+
+        borgmatic.actions.browse.workers.add_repository_archives(
+            self.app,
+            archive_loaded=self.archive_loaded,
+            config=self.config,
+            repository=self.repository,
+        )
+
+    def on_archive_loaded(self, archive_name):
+        '''
+        When an archive loads, add it as an option to this archives list. But if we get a
+        signal that all path loading is complete, stop and remove our loading indicator.
+        '''
+        if archive_name is borgmatic.actions.browse.workers.LOADING_DONE:
+            self.loading_timer.stop()
+            self.remove_option('loading-indicator')
+            return
+
+        label_pieces = (
+            (archive_name, '[dim](latest)[/dim]') if len(self.options) == 1 else (archive_name,)
+        )
+        highlighted_option = self.highlighted_option
+
+        loading_indicator = self.get_option('loading-indicator')
+        self.remove_option('loading-indicator')
+        self.add_options(
+            (
+                textual.widgets.option_list.Option(' '.join(label_pieces), id=archive_name),
+                loading_indicator,
+            ),
+        )
+
+        # Retain the highlighted option position even as other options load around it.
+        self.highlighted = (
+            self.get_option_index(highlighted_option.id)
+            if highlighted_option and self.highlighted_option_changed
+            else 0
+        )
+
+    def on_option_list_option_highlighted(self, event):
+        '''
+        When the highlighted option changes, record that fact. This flag is consumed in
+        borgmatic.actions.browse.workers.add_repository_archives() in order to retain the
+        highlighted option even as other options load around it.
+        '''
+        if self.highlighted not in {None, 0}:
+            self.highlighted_option_changed = True
@@ -0,0 +1,22 @@
+import textual.binding
+import textual.widgets
+
+OPTION_LIST_BINDINGS = (
+    *textual.widgets.OptionList.BINDINGS,
+    textual.binding.Binding(
+        key='up,k', action='cursor_up', description='scroll up', show=True, priority=True
+    ),
+    textual.binding.Binding(
+        key='down,j', action='cursor_down', description='scroll down', show=True, priority=True
+    ),
+    textual.binding.Binding(
+        key='pageup', action='page_up', description='page up', show=True, priority=True
+    ),
+    textual.binding.Binding(
+        key='pagedown', action='page_down', description='page down', show=True, priority=True
+    ),
+    textual.binding.Binding(
+        key='enter', action='select', description='select', show=True, priority=True
+    ),
+    textual.binding.Binding(key='right,l', action='select', description='select', show=False),
+)
@@ -0,0 +1,153 @@
+import os
+
+import textual.binding
+import textual.containers
+
+import borgmatic.actions.browse.archive
+import borgmatic.actions.browse.archives_list
+import borgmatic.actions.browse.configuration_files_list
+import borgmatic.actions.browse.directory_list
+import borgmatic.actions.browse.file_preview
+import borgmatic.actions.browse.icons
+import borgmatic.actions.browse.repositories_list
+
+
+def make_next_panel(focused_panel, option_id):
+    '''
+    Given a focused panel widget and the selected option ID, return the next panel corresponding to
+    that selection. This is the mechanism by which the user can successively drill down from
+    configuration file to repository to archive to root directory to non-root directory or file.
+
+    If the particular option ID on the focused panel doesn't have a supported next panel, then
+    return None.
+    '''
+    if isinstance(
+        focused_panel, borgmatic.actions.browse.configuration_files_list.Configuration_files_list
+    ):
+        return borgmatic.actions.browse.repositories_list.Repositories_list(
+            config=focused_panel.configs[option_id]
+        )
+
+    if isinstance(focused_panel, borgmatic.actions.browse.repositories_list.Repositories_list):
+        return borgmatic.actions.browse.archives_list.Archives_list(
+            config=focused_panel.config, repository=focused_panel.repositories[option_id]
+        )
+
+    if isinstance(focused_panel, borgmatic.actions.browse.archives_list.Archives_list):
+        return borgmatic.actions.browse.directory_list.Directory_list(
+            config=focused_panel.config, repository=focused_panel.repository, archive_name=option_id
+        )
+
+    if isinstance(focused_panel, borgmatic.actions.browse.directory_list.Directory_list):
+        option = focused_panel.get_option(option_id)
+
+        if option.prompt.startswith(
+            borgmatic.actions.browse.icons.PATH_TYPE_ICONS[
+                borgmatic.actions.browse.archive.Path_type.DIRECTORY.value
+            ]
+        ):
+            return borgmatic.actions.browse.directory_list.Directory_list(
+                focused_panel.config,
+                focused_panel.repository,
+                focused_panel.archive_name,
+                path_loaded=focused_panel.path_loaded,
+                path_components=(*focused_panel.path_components, option_id),
+            )
+
+        if option.prompt.startswith(
+            borgmatic.actions.browse.icons.PATH_TYPE_ICONS[
+                borgmatic.actions.browse.archive.Path_type.FILE.value
+            ]
+        ):
+            return borgmatic.actions.browse.file_preview.File_preview(
+                focused_panel.config,
+                focused_panel.repository,
+                focused_panel.archive_name,
+                file_path=os.path.sep.join((*focused_panel.path_components, option_id)),
+            )
+
+    return None
+
+
+class Carousel(textual.containers.Horizontal):
+    BINDINGS = (
+        textual.binding.Binding(
+            key='left,h', action='previous', description='previous', priority=True
+        ),
+    )
+
+    def __init__(self, panels):
+        self.panels = panels
+        self.focused_panel = panels[0]
+
+        super().__init__()
+
+    def compose(self):
+        '''
+        Compose with each of the contained panels and focus the first one.
+        '''
+        yield from self.panels
+
+        self.focused_panel.focus()
+
+    def action_previous(self):
+        '''
+        Make the previous panel into the focused panel.
+        '''
+        previous_panel_index = self.panels.index(self.focused_panel) - 1
+
+        if previous_panel_index < 0:
+            return
+
+        self.focused_panel.styles.display = 'none'
+
+        self.focused_panel = self.panels[previous_panel_index]
+        self.focused_panel.styles.display = 'block'
+        self.focused_panel.focus()
+
+    def action_next(self, option_id):
+        '''
+        Hide the current focused panel and create the next one.
+        '''
+        next_panel_index = self.panels.index(self.focused_panel) + 1
+
+        if next_panel_index < len(self.panels):
+            next_panel = self.panels[next_panel_index]
+            next_panel.styles.display = 'block'
+        else:
+            next_panel = make_next_panel(self.focused_panel, option_id)
+
+            if next_panel is None:
+                self.notify('Cannot display this content', severity='warning')
+                return
+
+            self.panels.append(next_panel)
+            next_panel.highlighted = 0
+
+        self.focused_panel.styles.display = 'none'
+        self.focused_panel = next_panel
+        self.focused_panel.focus()
+        self.mount(self.focused_panel)
+
+    def on_option_list_option_highlighted(self, event):
+        '''
+        The highlighted option has changed, so truncate any next panels.
+        '''
+        next_panel_index = self.panels.index(self.focused_panel) + 1
+
+        del self.panels[next_panel_index:]
+
+    def on_option_list_option_selected(self, event):
+        '''
+        An option has been selected, so advance to the next panel—unless the option selected is
+        "..", in which case go to the previous panel.
+        '''
+        if (
+            event.option_list != self.focused_panel or event.option_id == 'loading-indicator'
+        ):  # pragma: no cover
+            return
+
+        if event.option_id == '..':
+            self.action_previous()
+        else:
+            self.action_next(event.option_id)
@@ -0,0 +1,33 @@
+import os
+
+import textual.widgets
+
+import borgmatic.actions.browse.bindings
+
+
+class Configuration_files_list(textual.widgets.OptionList):
+    '''
+    A widget for selecting a single borgmatic configuration file from among available configuration
+    files. The item selection event is handled in a Carousel instance, the parent widget of an
+    Configuration_files_list.
+    '''
+
+    BINDINGS = borgmatic.actions.browse.bindings.OPTION_LIST_BINDINGS
+
+    def __init__(self, configs):
+        '''
+        Given a dict mapping from configuration path to corresponding configuration dict, add each
+        configuration path as an option to this widget.
+        '''
+        self.configs = configs
+        home_directory = os.path.expanduser('~')
+
+        super().__init__(
+            *(
+                textual.widgets.option_list.Option(unexpanded_path, id=config_path)
+                for config_path in configs
+                for unexpanded_path in (config_path.replace(home_directory, '~'),)
+            ),
+            classes='panel',
+        )
+        self.border_title = '📄 configuration files'
@@ -0,0 +1,211 @@
+import os
+
+import textual.widgets
+
+import borgmatic.actions.browse.bindings
+import borgmatic.actions.browse.icons
+import borgmatic.actions.browse.loading
+import borgmatic.actions.browse.workers
+
+
+def get_relative_archive_path_components(archive_path, current_directory_path_components):
+    '''
+    Given an Archive_path instance and a tuple of path components for the currently browsed
+    directory, get the path components as a tuple for the archive path relative to that directory.
+
+    For instance, given an archive path with a path of 'foo/bar/baz/quux.txt' and current
+    directory path components of ('foo', 'bar'), return ('baz', 'quux.txt').
+
+    If the archive path is not actually relative to the current directory, return None.
+    '''
+    archive_path_components = tuple(archive_path.file_path.split(os.path.sep))
+
+    if not current_directory_path_components:
+        return archive_path_components
+
+    # If the loaded path doesn't match this directory list's own path, then we don't care about
+    # it for purposes of displaying this particular directory.
+    if (
+        tuple(archive_path_components[: len(current_directory_path_components)])
+        != current_directory_path_components
+    ):
+        return None
+
+    # Strip off the portion of the archive path that matches the directory list's own path.
+    return archive_path_components[len(current_directory_path_components) :]
+
+
+def make_directory_list_option(archive_path, relative_path_components):
+    '''
+    Given an Archive_path instance and a tuple of relative path components for it, make a
+    textual.widgets.option_list.Option for the path. Use an the icon based on whether this looks
+    like a terminal filename or a directory.
+    '''
+    pieces = (
+        borgmatic.actions.browse.icons.PATH_TYPE_ICONS.get(
+            archive_path.path_type if len(relative_path_components) == 1 else 'd', '❓'
+        ),
+        relative_path_components[0],
+    ) + (('→', archive_path.link_target) if archive_path.link_target else ())
+
+    return textual.widgets.option_list.Option(
+        prompt=' '.join(pieces), id=relative_path_components[0]
+    )
+
+
+def add_archive_paths(
+    directory_list,
+    config,
+    repository,
+    archive_name,
+    archive_paths,
+):
+    '''
+    Given a DirectoryList instance, a configuration dict, a repository dict, an archive name, and a
+    sequence of ArchivePath instances, add the paths to the directory list as options, sorting and
+    deduplicating the resulting directory list's options.
+
+    After all of this reshuffling, make sure the original highlighted option remains highlighted.
+    '''
+    highlighted_option = directory_list.highlighted_option
+    original_options_count = len(directory_list.options)
+
+    sorted_options = sorted(
+        (
+            *directory_list.options,
+            *(
+                make_directory_list_option(archive_path, relative_path_components)
+                for archive_path in archive_paths
+                for relative_path_components in (
+                    get_relative_archive_path_components(
+                        archive_path,
+                        directory_list.path_components,
+                    ),
+                )
+                if relative_path_components
+                if relative_path_components[0] not in directory_list._id_to_option
+            ),
+        ),
+        # The loading indicator "option" always goes to the bottom.
+        key=lambda option: ((option.id == 'loading-indicator'), option.prompt),
+    )
+
+    # If there aren't actually any options to add (due to deduplication), bail.
+    if len(sorted_options) == original_options_count:
+        return
+
+    # Retain the highlighted option position even as other options load around it.
+    directory_list.set_options(sorted_options)
+    directory_list.highlighted = (
+        directory_list.get_option_index(highlighted_option.id)
+        if highlighted_option and directory_list.highlighted_option_changed
+        else 0
+    )
+
+
+class Directory_list(textual.widgets.OptionList):
+    '''
+    A widget for selecting a path from among the contents of a particular directory in a Borg
+    archive. The item selection event is handled in a Carousel instance, the parent widget of a
+    Directory_list.
+    '''
+
+    BINDINGS = borgmatic.actions.browse.bindings.OPTION_LIST_BINDINGS
+
+    def __init__(self, config, repository, archive_name, path_loaded=None, path_components=None):
+        '''
+        Given a configuration dict, a repository dict, an archive name, an optional
+        Archive_path_loaded instance for signalling new paths as they load, and an optional tuple of
+        path components indicating this directory's position in the backed up filesystem, prepare to
+        load paths from the archive for eventual display in this widget. Actual loading kicks off in
+        on_mount() below.
+        '''
+        self.config = config
+        self.repository = repository
+        self.archive_name = archive_name
+        self.path_components = path_components or ()
+        self.highlighted_option_changed = False
+
+        super().__init__(classes='panel')
+
+        self.border_title = ' '.join(
+            (
+                '📁',
+                os.path.sep.join(self.path_components)
+                if self.path_components
+                else f'{archive_name}',
+            )
+        )
+
+        if self.path_components:
+            self.add_option(
+                textual.widgets.option_list.Option(
+                    '📁 ..',
+                    id='..',
+                ),
+            )
+
+        self.path_loaded = path_loaded or borgmatic.actions.browse.workers.Archive_path_loaded(
+            self, 'archive path loaded'
+        )
+
+        if not self.path_loaded.complete:
+            self.timer = borgmatic.actions.browse.loading.add_inline_loading_indicator(self)
+
+    def on_mount(self):
+        '''
+        When this widget gets mounted in the DOM, subscribe to path loaded events so that we can
+        find out about relevant archive paths as they load. And if this is a root directory list,
+        start loading paths from the archive. If this is a non-root directory list, add any already
+        loaded archive paths to this widget as options.
+
+        Loading is started *after* subscribing to path loaded signals so that there's not a gap
+        where we might miss out on any paths.
+        '''
+        self.path_loaded.subscribe(self, self.on_archive_path_loaded)
+
+        if self.path_components:
+            add_archive_paths(
+                directory_list=self,
+                config=self.config,
+                repository=self.repository,
+                archive_name=self.archive_name,
+                archive_paths=borgmatic.actions.browse.workers.get_paths(
+                    self.path_loaded.path_hierarchy, self.path_components
+                ),
+            )
+        else:
+            borgmatic.actions.browse.workers.load_archive_paths(
+                self.app,
+                path_loaded=self.path_loaded,
+                config=self.config,
+                repository=self.repository,
+                archive_name=self.archive_name,
+            )
+
+    def on_archive_path_loaded(self, data):
+        '''
+        When an archive path loads, add it as an option to this directory list. But if we get a
+        signal that all path loading is complete, stop and remove our loading indicator.
+        '''
+        if data is borgmatic.actions.browse.workers.LOADING_DONE:
+            self.timer.stop()
+            self.remove_option('loading-indicator')
+            return
+
+        add_archive_paths(
+            directory_list=self,
+            config=self.config,
+            repository=self.repository,
+            archive_name=self.archive_name,
+            archive_paths=(data,),
+        )
+
+    def on_option_list_option_highlighted(self, event):
+        '''
+        When the highlighted option changes, record that fact. This flag is consumed in
+        add_archive_paths() in order to retain the highlighted option even as other options load
+        around it.
+        '''
+        if self.highlighted not in {None, 0}:
+            self.highlighted_option_changed = True
@@ -0,0 +1,87 @@
+import logging
+
+import rich.syntax
+import textual.binding
+import textual.widgets
+
+import borgmatic.actions.browse.loading
+import borgmatic.actions.browse.workers
+
+logger = logging.getLogger('__name__')
+
+
+class File_preview(textual.widgets.RichLog):
+    '''
+    A widget for extracting and previewing the contents of a file stored in a Borg archive.
+    '''
+
+    BINDINGS = (
+        *textual.widgets.RichLog.BINDINGS,
+        textual.binding.Binding(
+            key='up,k', action='scroll_up', description='scroll up', show=True, priority=True
+        ),
+        textual.binding.Binding(
+            key='down,j', action='scroll_down', description='scroll down', show=True, priority=True
+        ),
+        textual.binding.Binding(
+            key='pageup', action='page_up', description='page up', show=True, priority=True
+        ),
+        textual.binding.Binding(
+            key='pagedown', action='page_down', description='page down', show=True, priority=True
+        ),
+    )
+
+    def __init__(self, config, repository, archive_name, file_path):
+        '''
+        Given a configuration dict, a repository dict, an archive name, and the path of a file in
+        the archive, prepare to load the file's contents for eventual display in this widget. Actual
+        loading kicks off in on_mount() below.
+        '''
+        self.config = config
+        self.repository = repository
+        self.archive_name = archive_name
+        self.file_path = file_path
+
+        super().__init__(classes='panel')
+        self.border_title = f'📄 {self.file_path} preview'
+        self.auto_scroll = False
+        self.file_preview_loaded = borgmatic.actions.browse.workers.File_preview_loaded(
+            self, 'file preview loaded'
+        )
+
+        self.loading_timer = borgmatic.actions.browse.loading.add_inline_loading_indicator(self)
+
+    def on_mount(self):
+        '''
+        When this widget gets mounted in the DOM, subscribe to archive loaded events so that we can
+        find out about archives as they load. Also start loading file contents from the archive.
+
+        Loading is started *after* subscribing to the file preview loaded signal so that there's not
+        a gap where we might miss out on signal publishes.
+        '''
+        self.file_preview_loaded.subscribe(self, self.on_file_preview_loaded)
+
+        borgmatic.actions.browse.workers.load_file_preview(
+            self.app,
+            file_preview_loaded=self.file_preview_loaded,
+            config=self.config,
+            repository=self.repository,
+            archive_name=self.archive_name,
+            file_path=self.file_path,
+        )
+
+    def on_file_preview_loaded(self, file_contents):
+        '''
+        When a file loads, write its contents (syntax highlighted) to this file preview widget.
+        '''
+        self.loading_timer.stop()
+        self.clear()
+
+        if file_contents is None:
+            self.write('Cannot display a preview for this file')
+        else:
+            # Only pass the file path and not its contents to guess_lexer(). Passing the contents is
+            # more accurate, but also much slower.
+            self.write(
+                rich.syntax.Syntax(file_contents, rich.syntax.Syntax.guess_lexer(self.file_path))
+            )
@@ -0,0 +1,8 @@
+import borgmatic.actions.browse.archive
+
+PATH_TYPE_ICONS = {
+    borgmatic.actions.browse.archive.Path_type.DIRECTORY.value: '📁',
+    borgmatic.actions.browse.archive.Path_type.LINK.value: '🔗',
+    borgmatic.actions.browse.archive.Path_type.PIPE.value: '🚰',
+    borgmatic.actions.browse.archive.Path_type.FILE.value: '📄',
+}
@@ -0,0 +1,54 @@
+import contextlib
+import functools
+import logging
+
+import textual.widgets
+import textual.widgets.option_list
+
+LOADING_DOT_INTERVAL_SECONDS = 0.3
+
+
+logger = logging.getLogger('__name__')
+
+
+def update_inline_loading_indicator(widget):
+    '''
+    Given a textual.widgets.OptionList or a textual.widgets.RichLog instance, animate the existing
+    loading indicator inside it.
+    '''
+    if isinstance(widget, textual.widgets.OptionList):
+        with contextlib.suppress(textual.widgets.option_list.OptionDoesNotExist):
+            widget.replace_option_prompt(
+                'loading-indicator',
+                (str(widget.get_option('loading-indicator').prompt) + '.').replace('....', ''),
+            )
+    elif isinstance(widget, textual.widgets.RichLog):
+        with contextlib.suppress(IndexError):
+            loading_message = str(widget.lines[0].text)
+            widget.clear()
+            widget.write((loading_message + '.').replace('....', ''))
+    else:
+        raise ValueError(f'Unsupported widget type: {type(widget)}')
+
+
+LOADING_MESSAGE = '⏳ loading...'
+
+
+def add_inline_loading_indicator(widget):
+    '''
+    Given a textual.widgets.OptionList or a textual.widgets.RichLog instance, add a loading
+    indicator to it.
+    '''
+    if isinstance(widget, textual.widgets.OptionList):
+        loading_option = textual.widgets.option_list.Option(LOADING_MESSAGE, id='loading-indicator')
+        widget.add_option(loading_option)
+        widget.highlighted = None
+    elif isinstance(widget, textual.widgets.RichLog):
+        widget.write(LOADING_MESSAGE)
+    else:
+        raise ValueError(f'Unsupported widget type: {type(widget)}')
+
+    return widget.set_interval(
+        LOADING_DOT_INTERVAL_SECONDS,
+        functools.partial(update_inline_loading_indicator, widget),
+    )
@@ -0,0 +1,102 @@
+import contextlib
+import logging
+
+import textual._context
+import textual.widgets
+import textual.worker
+
+import borgmatic.logger
+
+
+class Rich_color_formatter(logging.Formatter):
+    '''
+    A Python logging formatter that formats log records with Rich-compatible color markup according
+    to their levels.
+    '''
+
+    def __init__(self, *args, **kwargs):
+        self.prefix = None
+        super().__init__(
+            '{prefix}{message}',
+            *args,
+            style='{',
+            **kwargs,
+        )
+
+    def format(self, record):
+        '''
+        Given a log record, format it with Rich-compatibe color markup corresponding to its log
+        level.
+        '''
+        borgmatic.logger.add_custom_log_levels()
+
+        color = {
+            logging.CRITICAL: 'bright_red',
+            logging.ERROR: 'bright_red',
+            logging.WARNING: 'bright_yellow',
+            logging.ANSWER: 'bright_magenta',
+            logging.INFO: 'bright_green',
+            logging.DEBUG: 'bright_cyan',
+        }.get(record.levelno)
+        record.prefix = f'{self.prefix}: ' if self.prefix else ''
+
+        return f'[{color}]{super().format(record)}[/{color}]'
+
+
+class Browse_log_handler(logging.Handler):
+    '''
+    A Python log handler that writes any log records to a logging widget.
+    '''
+
+    def __init__(self, logs_widget):
+        '''
+        Given a logs widget, save it for use below.
+        '''
+        self.logs_widget = logs_widget
+
+        super().__init__()
+
+    def emit(self, record):
+        '''
+        Given a log record, format it and log it to the logs widgets. This works whether or not the
+        logging is happening in the main thread.
+        '''
+        message = self.format(record)
+
+        try:
+            textual.worker.get_current_worker()
+            self.logs_widget.app.call_from_thread(self.logs_widget.write, message)
+        except (RuntimeError, textual.worker.NoActiveWorker):
+            with contextlib.suppress(textual._context.NoActiveAppError):
+                self.logs_widget.write(message)
+
+
+def log_to_widget(logs_widget):
+    '''
+    Given a Textual RichLog logs widget, add a log handler and formatter that logs to it. Also
+    remove the default borgmatic console log handler so it doesn't try to log all over our UI.
+    '''
+    handler = Browse_log_handler(logs_widget)
+    handler.setFormatter(Rich_color_formatter())
+    logger = logging.getLogger()
+    logger.setLevel(min(handler.level for handler in logger.handlers))
+    logger.addHandler(handler)
+
+    with contextlib.suppress(StopIteration):
+        console_handler = next(
+            handler
+            for handler in logging.getLogger().handlers
+            if isinstance(handler, borgmatic.logger.Multi_stream_handler)
+        )
+        logger.removeHandler(console_handler)
+
+
+class Logs(textual.widgets.RichLog):
+    '''
+    A widget for viewing borgmatic logs in realtime. The log level is determined by borgmatic's
+    current verbosity level.
+    '''
+
+    def __init__(self):
+        super().__init__(markup=True, id='logs', classes='panel')
+        self.border_title = '🪵 logs'
@@ -0,0 +1,30 @@
+import textual.widgets
+
+import borgmatic.actions.browse.bindings
+
+
+class Repositories_list(textual.widgets.OptionList):
+    '''
+    A widget for selecting a single Borg repository from among the repositories in a borgmatic
+    configuration file. The item selection event is handled in a Carousel instance, the parent
+    widget of a Repositories_list.
+    '''
+
+    BINDINGS = borgmatic.actions.browse.bindings.OPTION_LIST_BINDINGS
+
+    def __init__(self, config):
+        '''
+        Given a configuration dict, populate the repositories in this widget.
+        '''
+        self.config = config
+        self.repositories = config['repositories']
+
+        super().__init__(
+            *(
+                textual.widgets.option_list.Option(label, id=index)
+                for index, repository in enumerate(self.repositories)
+                for label in (repository.get('label', repository.get('path')),)
+            ),
+            classes='panel',
+        )
+        self.border_title = '📦 repositories'
@@ -0,0 +1,29 @@
+import logging
+
+
+def run_browse(
+    diff_arguments,
+    global_arguments,
+    configs,
+):
+    '''
+    Run the "browse" action for the given borgmatic configurations. This launches a console UI.
+
+    Raise ValueError if the Textual library (a prerequisite for this action) can't be imported.
+    '''
+    if not configs:
+        return
+
+    logging.getLogger('asyncio').setLevel(logging.WARNING)
+
+    try:
+        import textual  # noqa: F401, PLC0415
+    except ImportError:  # pragma: no cover
+        raise ValueError(
+            'Unable to import the Textual library for the browse action; try installing "borgmatic[browse]"'
+        )
+
+    import borgmatic.actions.browse.app  # noqa: PLC0415
+
+    app = borgmatic.actions.browse.app.Browse_app(configs)
+    app.run()
@@ -0,0 +1,192 @@
+import logging
+import os
+
+import textual
+import textual.signal
+
+import borgmatic.actions.browse.archive
+
+logger = logging.getLogger('__name__')
+
+
+LOADING_DONE = object()
+
+
+class Archive_loaded(textual.signal.Signal):
+    '''
+    A signal that publishes when each subsequent archive is loaded from a repository, intended for
+    consumption in widgets that display archives as they are loaded. This signal also publishes
+    when loading is complete.
+
+    Each subscribed callback call includes the archive as an archive name string. Given the lack of
+    other identifying information (configuration file, repository), there should be a separate
+    Archive_loaded instance per repository.
+    '''
+
+
+@textual.work(thread=True)
+def add_repository_archives(browse_app, archive_loaded, config, repository):
+    '''
+    Given a running Browse_app instance, an Archive_loaded instance, a configuration dict, and a
+    repository dict, load a list of the archives from the repository and add them as options in the
+    archives list. Reverse the order so the most recent archive is first.
+
+    This function runs in a separate thread from the main UI. When loading is complete, publish a
+    loading done signal.
+    '''
+    archives_data = borgmatic.actions.browse.archive.get_repository_archives(config, repository)
+
+    # Reverse the archives, so the common case of accessing the latest archive is easy because it's
+    # at the top.
+    for archive in reversed(archives_data['archives']):
+        archive_loaded.publish(archive['archive'])
+
+    archive_loaded.publish(LOADING_DONE)
+
+
+def record_path(archive_path, hierarchy, path_components):
+    '''
+    Given an Archive_path instance, a dict capturing a filesystem hierarchy of paths, and a tuple of
+    path components for the archive path, set the archive path into the hierarchy data structure.
+
+    For instance, if given an archive path and path components representing "foo/bar/baz.txt",
+    produce a hierarchy that looks like:
+
+        {'foo': {'bar': {'baz.txt': Archive_path('-', 'foo/bar/baz.txt', '')}}}
+
+    Note that the hierarchy is modified in place, so any existing paths there are retained.
+    '''
+    if len(path_components) == 1:
+        hierarchy[path_components[0]] = {} if archive_path.path_type == 'd' else archive_path
+        return
+
+    record_path(archive_path, hierarchy.setdefault(path_components[0], {}), path_components[1:])
+
+
+def get_paths(hierarchy, path_components, full_path_components=None):
+    '''
+    Given a dict capturing a filesystem hierarchy of paths (or a subset thereof), a tuple of path
+    components for a directory path relative to the hierarchy root, and an optional tuple of
+    *absolute* path components for the same path (if different), return a generator of the
+    contained file and directory Archive_path instances from the hierarchy.
+
+    For instance, given the following hierarchy:
+
+        {'foo': {'bar': {'baz.txt': Archive_path('-', 'foo/bar/baz.txt', ''), 'quux': {}}}}
+
+    ... and path components of ('foo', 'bar'), return a generator with the following:
+
+        * Archive_path('-', 'foo/bar/baz.txt', '')
+        * Archive_path('d', 'foo/bar/quux', '')
+
+    The given absolute path components are use to construct directory paths like that last archive
+    path.
+    '''
+    if full_path_components is None:
+        full_path_components = path_components
+
+    if len(path_components) == 1:
+        try:
+            return (
+                archive_path
+                if isinstance(archive_path, borgmatic.actions.browse.archive.Archive_path)
+                else borgmatic.actions.browse.archive.Archive_path(
+                    'd', os.path.join(*full_path_components, component), ''
+                )
+                for component, archive_path in hierarchy[path_components[0]].items()
+            )
+        except KeyError:
+            raise ValueError(f'Unknown file or directory: {path_components[0]}')
+
+    try:
+        return get_paths(hierarchy[path_components[0]], path_components[1:], full_path_components)
+    except KeyError:
+        raise ValueError(f'Unknown directory: {path_components[0]}')
+
+
+class Archive_path_loaded(textual.signal.Signal):
+    '''
+    A signal that publishes when each subsequent path is loaded from an archive, intended for
+    consumption in widgets that display paths as they are loaded. This signal also tracks the
+    complete filesystem hierarchy seen thus far, so new widgets that get created after loading has
+    started can "catch up" with existing known paths. Lastly, this signal publishes and tracks when
+    loading is complete.
+
+    Each subscrided callback call includes the loaded path as an Archive_path instance. There is
+    intended to be a separate Archive_path_loaded instance per archive, but that instance should be
+    shared among several different widgets for the same archive for performance reasons.
+    '''
+
+    def __init__(self, owner, name):
+        self.path_hierarchy = {}
+        self.complete = False
+
+        super().__init__(owner, name)
+
+    def publish(self, archive_path):
+        '''
+        Publish the given archive path to subscribers and record its path locally. But if the
+        archive path is actually LOADING_DONE, then record loading as complete.
+        '''
+        super().publish(archive_path)
+
+        if archive_path is LOADING_DONE:
+            self.complete = True
+        else:
+            record_path(
+                archive_path, self.path_hierarchy, archive_path.file_path.split(os.path.sep)
+            )
+
+
+@textual.work(thread=True)
+def load_archive_paths(browse_app, path_loaded, config, repository, archive_name):
+    '''
+    Given a running Browse_app instance, an Archive_path_loaded instance, a configuration dict, a
+    repository dict, and an archive name, load the paths in this archive and publish each one via
+    the Archive_path_loaded signal, so interested widgets can subscribe. Also send a "loading done"
+    signal when loading completes.
+
+    This function runs in a separate thread from the main UI. When loading is complete, publish a
+    loading done signal.
+    '''
+    for archive_path in borgmatic.actions.browse.archive.get_archive_paths(
+        config, repository, archive_name
+    ):
+        path_loaded.publish(archive_path)
+
+    path_loaded.publish(LOADING_DONE)
+
+
+class File_preview_loaded(textual.signal.Signal):
+    '''
+    A signal that publishes when file contents are loaded from an archive, intended for consumption
+    in widgets that display loaded files. This signal also publishes when loading is complete.
+
+    Each published callback includes a the file's contents as a string. Given the lack of other
+    identifying information (configuration file, repository, archive), there should be a separate
+    Archive_loaded instance per previewed file.
+    '''
+
+
+@textual.work(thread=True)
+def load_file_preview(
+    browse_app,
+    file_preview_loaded,
+    config,
+    repository,
+    archive_name,
+    file_path,
+):
+    '''
+    Given a running Browse_app instance, a File_preview_loaded instance, a configuration dict, a
+    repository dict, an archive name, and the path of a file in that archive, load the contents of
+    the file and write it into the given file preview widget.
+
+    This function runs in a separate thread from the main UI. When loading is complete, publish a
+    loading done signal.
+    '''
+    file_contents = borgmatic.actions.browse.archive.get_archive_file_content(
+        config, repository, archive_name, file_path
+    )
+
+    file_preview_loaded.publish(file_contents)
@@ -1,7 +1,6 @@
 import logging

 import borgmatic.borg.change_passphrase
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -18,20 +17,13 @@ def run_change_passphrase(
    '''
    Run the "key change-passphrase" action for the given repository.
    '''
-    if (
-        change_passphrase_arguments.repository is None
-        or borgmatic.config.validate.repositories_match(
-            repository,
-            change_passphrase_arguments.repository,
-        )
-    ):
-        logger.info('Changing repository passphrase')
-        borgmatic.borg.change_passphrase.change_passphrase(
-            repository['path'],
-            config,
-            local_borg_version,
-            change_passphrase_arguments,
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+    logger.info('Changing repository passphrase')
+    borgmatic.borg.change_passphrase.change_passphrase(
+        repository['path'],
+        config,
+        local_borg_version,
+        change_passphrase_arguments,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -9,6 +9,7 @@ import pathlib
 import random
 import shlex
 import shutil
+import subprocess
 import textwrap

 import borgmatic.actions.config.bootstrap
@@ -20,11 +21,8 @@ import borgmatic.borg.extract
 import borgmatic.borg.list
 import borgmatic.borg.pattern
 import borgmatic.borg.repo_list
-import borgmatic.borg.state
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.execute
-import borgmatic.hooks.command

 DEFAULT_CHECKS = (
    {'name': 'repository', 'frequency': '1 month'},
@@ -246,7 +244,7 @@ def write_check_time(path):  # pragma: no cover
    logger.debug(f'Writing check time at {path}')

    os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
-    pathlib.Path(path, mode=0o600).touch()
+    pathlib.Path(path).touch(mode=0o600)


 def read_check_time(path):
@@ -383,12 +381,9 @@ def collect_spot_check_source_paths(
        dry_run=True,
        repository_path=repository['path'],
        # Omit "progress" because it interferes with "list_details".
-        config=dict(
-            {option: value for option, value in config.items() if option != 'progress'},
-            list_details=True,
-        ),
+        config=dict(config, progress=False, list_details=True),
        patterns=borgmatic.actions.pattern.process_patterns(
-            borgmatic.actions.pattern.collect_patterns(config)
+            borgmatic.actions.pattern.collect_patterns(config, working_directory)
            + tuple(
                borgmatic.borg.pattern.Pattern(
                    config_path,
@@ -408,7 +403,7 @@ def collect_spot_check_source_paths(
    )
    working_directory = borgmatic.config.paths.get_working_directory(config)

-    paths_output = borgmatic.execute.execute_command_and_capture_output(
+    path_lines = borgmatic.execute.execute_command_and_capture_output(
        create_flags + create_positional_arguments,
        capture_stderr=True,
        environment=borgmatic.borg.environment.make_environment(config),
@@ -417,14 +412,20 @@ def collect_spot_check_source_paths(
        borg_exit_codes=config.get('borg_exit_codes'),
    )

-    paths = tuple(
+    paths = (
        path_line.split(' ', 1)[1]
-        for path_line in paths_output.splitlines()
+        for path_line in path_lines
        if path_line and path_line.startswith(('- ', '+ '))
    )

    return tuple(
-        path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+        # Use dict.fromkeys() to deduplicate file paths, which are present in Borg's dry run output
+        # when there are overlapping source patterns. For instance, if both "/foo" and
+        # "/foo/file.txt" are in configured patterns, then "/foo/file.txt" will show up in Borg's
+        # dry run output twice.
+        dict.fromkeys(
+            path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+        )
    )


@@ -454,24 +455,22 @@ def collect_spot_check_archive_paths(
    borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)

    return tuple(
-        path
-        for line in borgmatic.borg.list.capture_archive_listing(
+        entry['path']
+        for entry in borgmatic.borg.list.capture_archive_listing(
            repository['path'],
            archive,
            config,
            local_borg_version,
            global_arguments,
-            path_format='{type} {path}{NUL}',
            local_path=local_path,
            remote_path=remote_path,
        )
-        for (file_type, path) in (line.split(' ', 1),)
-        if file_type not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
-        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
+        if entry['type'] not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
+        if pathlib.Path('borgmatic') not in pathlib.Path(entry['path']).parents
        if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
        if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
    )


@@ -526,52 +525,100 @@ def compare_spot_check_hashes(
        source_sample_paths_subset = tuple(
            itertools.islice(source_sample_paths_iterator, SAMPLE_PATHS_SUBSET_COUNT),
        )
+
        if not source_sample_paths_subset:
            break

-        hash_output = borgmatic.execute.execute_command_and_capture_output(
-            tuple(
-                shlex.quote(part)
-                for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
-            )
-            + tuple(
-                path for path in source_sample_paths_subset if path in hashable_source_sample_path
-            ),
-            working_directory=working_directory,
+        hash_paths = tuple(
+            path for path in source_sample_paths_subset if path in hashable_source_sample_path
        )

-        source_hashes.update(
-            **dict(
-                (reversed(line.split('  ', 1)) for line in hash_output.splitlines()),
-                # Represent non-existent files as having empty hashes so the comparison below still
-                # works. Same thing for filesystem links, since Borg produces empty archive hashes
-                # for them.
-                **{
-                    path: ''
-                    for path in source_sample_paths_subset
-                    if path not in hashable_source_sample_path
-                },
-            ),
-        )
+        try:
+            hash_lines = borgmatic.execute.execute_command_and_capture_output(
+                tuple(
+                    shlex.quote(part)
+                    for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
+                )
+                + hash_paths,
+                working_directory=working_directory,
+            )
+            source_hashes.update(
+                **dict(
+                    zip(
+                        # xxh64sum rewrites/escapes the paths that it returns alongside its hashes, for
+                        # instance if they contain special characters. When that happens, they don't
+                        # match the original source paths and therefore hash lookups fail. So when
+                        # building this lookup dict, use the original unaltered paths we provided as
+                        # input to xxh64sum.
+                        hash_paths,
+                        (
+                            # For some reason, xxh64sum prefixes the hash with a backslash if the path
+                            # contains a newline. Work around that.
+                            line.split('  ', 1)[0].lstrip('\\')
+                            for line in hash_lines
+                        ),
+                    ),
+                    # Represent non-existent files as having empty hashes so the comparison below still
+                    # works. Same thing for filesystem links, since Borg produces empty archive hashes
+                    # for them.
+                    **{
+                        path: ''
+                        for path in source_sample_paths_subset
+                        if path not in hashable_source_sample_path
+                    },
+                ),
+            )
+        except subprocess.CalledProcessError:
+            # This can happen if a file we planned to hash gets deleted right before we try to hash
+            # it. Falling back to individual file hashing allows us to find and mark just the
+            # file(s) with problems instead of failing the whole batch.
+            logger.warning(
+                'Bulk source path hashing failed for this batch; falling back to individual file hashing'
+            )
+
+            for hash_path in hash_paths:
+                try:
+                    hash_lines = borgmatic.execute.execute_command_and_capture_output(
+                        (
+                            *(
+                                shlex.quote(part)
+                                for part in shlex.split(
+                                    spot_check_config.get('xxh64sum_command', 'xxh64sum')
+                                )
+                            ),
+                            hash_path,
+                        ),
+                        working_directory=working_directory,
+                    )
+                    source_hashes[hash_path] = next(hash_lines).split('  ', 1)[0].lstrip('\\')
+                except (subprocess.CalledProcessError, StopIteration):  # noqa: PERF203
+                    logger.warning(
+                        f'Source path hashing failed for {hash_path}; treating as missing'
+                    )
+                    source_hashes[hash_path] = ''

        # Get the hash for each file in the archive.
-        archive_hashes.update(
-            **dict(
-                reversed(line.split(' ', 1))
-                for line in borgmatic.borg.list.capture_archive_listing(
-                    repository['path'],
-                    archive,
-                    config,
-                    local_borg_version,
-                    global_arguments,
-                    list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} {path}{NUL}',
-                    local_path=local_path,
-                    remote_path=remote_path,
-                )
-                if line
-            ),
-        )
+        for entry in borgmatic.borg.list.capture_archive_listing(
+            repository['path'],
+            archive,
+            config,
+            local_borg_version,
+            global_arguments,
+            list_paths=source_sample_paths_subset,
+            path_format='{xxh64}{path}{linktarget}{target}',
+            local_path=local_path,
+            remote_path=remote_path,
+        ):
+            if not entry:
+                continue
+
+            # Borg can't get hashes of stored hard links. So if this is a hard link path (and not
+            # deemed as the "original" by Borg), then skip hashing of it.
+            if entry.get('linktarget') or entry.get('target'):
+                source_hashes.pop(os.path.join('/', entry['path']), None)
+                continue
+
+            archive_hashes[entry['path']] = entry['xxh64']

    # Compare the source hashes with the archive hashes to see how many match.
    failing_paths = []
@@ -608,8 +655,6 @@ def spot_check(
    disk to those stored in the latest archive. If any differences are beyond configured tolerances,
    then the check fails.
    '''
-    logger.debug('Running spot check')
-
    try:
        spot_check_config = next(
            check for check in config.get('checks', ()) if check.get('name') == 'spot'
@@ -672,7 +717,7 @@ def spot_check(
        )
        logger.debug(f'Paths in latest archive but not source paths: {truncated_archive_paths}')
        raise ValueError(
-            'Spot check failed: There are no source paths to compare against the archive',
+            'Spot check failed; there are no source paths to compare against the archive',
        )

    # Calculate the percentage delta between the source paths count and the archive paths count, and
@@ -686,19 +731,13 @@ def spot_check(
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
-        logger.debug(
-            f'Paths in source paths but not latest archive: {truncated_exclusive_source_paths}',
-        )
        truncated_exclusive_archive_paths = textwrap.shorten(
            ', '.join(set(archive_paths) - rootless_source_paths) or 'none',
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
-        logger.debug(
-            f'Paths in latest archive but not source paths: {truncated_exclusive_archive_paths}',
-        )
        raise ValueError(
-            f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)',
+            f'Spot check failed\n{count_delta_percentage:.2f}% file count delta between source paths ({len(source_paths)} total) and latest archive ({len(archive_paths)} total); tolerance is {spot_check_config["count_tolerance_percentage"]}%\nOnly in source paths: {truncated_exclusive_source_paths}\nOnly in latest archive: {truncated_exclusive_archive_paths}',
        )

    failing_paths = compare_spot_check_hashes(
@@ -723,11 +762,8 @@ def spot_check(
            width=MAX_SPOT_CHECK_PATHS_LENGTH,
            placeholder=' ...',
        )
-        logger.debug(
-            f'Source paths with data not matching the latest archive: {truncated_failing_paths}',
-        )
        raise ValueError(
-            f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)',
+            f'Spot check failed\n{failing_percentage:.2f}% of source paths ({len(failing_paths)} out of {len(source_paths)} checked) with data not matching the latest archive; tolerance is {data_tolerance_percentage}%\nSource paths with non-matching data: {truncated_failing_paths}',
        )

    logger.info(
@@ -750,12 +786,6 @@ def run_check(

    Raise ValueError if the Borg repository ID cannot be determined.
    '''
-    if check_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        check_arguments.repository,
-    ):
-        return
-
    logger.info('Running consistency checks')

    repository_id = borgmatic.borg.check.get_repository_id(
@@ -800,6 +830,7 @@ def run_check(
            write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))

    if 'extract' in checks:
+        logger.info('Running extract check')
        borgmatic.borg.extract.extract_last_archive_dry_run(
            config,
            local_borg_version,
@@ -812,6 +843,7 @@ def run_check(
        write_check_time(make_check_time_path(config, repository_id, 'extract'))

    if 'spot' in checks:
+        logger.info('Running spot check')
        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
            spot_check(
                repository,
@@ -2,8 +2,6 @@ import logging

 import borgmatic.borg.compact
 import borgmatic.borg.feature
-import borgmatic.config.validate
-import borgmatic.hooks.command

 logger = logging.getLogger(__name__)

@@ -22,12 +20,6 @@ def run_compact(
    '''
    Run the "compact" action for the given repository.
    '''
-    if compact_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        compact_arguments.repository,
-    ):
-        return
-
    if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
        logger.info(f'Compacting segments{dry_run_label}')
        borgmatic.borg.compact.compact_segments(
@@ -5,8 +5,6 @@ import os
 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
-import borgmatic.hooks.command

 logger = logging.getLogger(__name__)

@@ -122,15 +120,13 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
            borgmatic_runtime_directory,
        )

-    logger.info(f"Bootstrapping config paths: {', '.join(manifest_config_paths)}")
+    logger.info(f"Bootstrapping configuration paths: {', '.join(manifest_config_paths)}")

    borgmatic.borg.extract.extract_archive(
        global_arguments.dry_run,
        bootstrap_arguments.repository,
        archive_name,
        [config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
-        # Only add progress here and not the extract_archive() call above, because progress
-        # conflicts with extract_to_stdout.
        dict(config, progress=bootstrap_arguments.progress or False),
        local_borg_version,
        global_arguments,
@@ -19,26 +19,33 @@ def run_generate(generate_arguments, global_arguments):
    dry_run_label = ' (dry run; not actually writing anything)' if global_arguments.dry_run else ''

    logger.answer(
-        f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}',
+        f'Generating configuration files within: {generate_arguments.destination_path}{dry_run_label}'
+        if generate_arguments.split
+        else f'Generating a configuration file at: {generate_arguments.destination_path}{dry_run_label}'
    )

    borgmatic.config.generate.generate_sample_configuration(
        global_arguments.dry_run,
        generate_arguments.source_filename,
-        generate_arguments.destination_filename,
+        generate_arguments.destination_path,
        borgmatic.config.validate.schema_filename(),
        overwrite=generate_arguments.overwrite,
+        split=generate_arguments.split,
    )

    if generate_arguments.source_filename:
        logger.answer(
            f'''
-Merged in the contents of configuration file at: {generate_arguments.source_filename}
-To review the changes made, run:
-
-    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}''',
+Merged the contents of the source configuration file: {generate_arguments.source_filename}'''
        )

+        if not generate_arguments.split:
+            logger.answer(
+                f'''To review the changes made, run:
+
+    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_path}''',
+            )
+
    logger.answer(
        '''
 This includes all available configuration options with example values, the few
@@ -0,0 +1,44 @@
+import json
+import logging
+import sys
+
+import borgmatic.config.generate
+import borgmatic.logger
+
+logger = logging.getLogger(__name__)
+
+
+def run_show(show_arguments, configs):
+    '''
+    Given the show arguments as an argparse.Namespace instance and a dict of configuration filename
+    to corresponding parsed configuration, run the "show" action. That consists of rendering and
+    logging the computed configuration as YAML, separating the configuration for each file with
+    "---".
+
+    If show_arguments.option is set, limit the results to the value of that single option. If
+    show_arguments.json is True, render the results as JSON with one array element per configuration
+    file.
+    '''
+    borgmatic.logger.add_custom_log_levels()
+
+    if show_arguments.json:
+        sys.stdout.write(
+            json.dumps(
+                [
+                    config.get(show_arguments.option) if show_arguments.option else config
+                    for config in configs.values()
+                ]
+            )
+        )
+
+        return
+
+    for config in configs.values():
+        if len(configs) > 1:
+            logger.answer('---')
+
+        logger.answer(
+            borgmatic.config.generate.render_configuration(
+                config.get(show_arguments.option) if show_arguments.option else config
+            ).rstrip()
+        )
@@ -1,12 +1,12 @@
 import logging

+import borgmatic.actions.dump
 import borgmatic.actions.json
 import borgmatic.borg.create
 import borgmatic.borg.feature
 import borgmatic.borg.rename
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.hooks.dispatch
 from borgmatic.actions import pattern

@@ -30,12 +30,6 @@ def run_create(

    If create_arguments.json is True, yield the JSON output from creating the archive.
    '''
-    if create_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        create_arguments.repository,
-    ):
-        return
-
    if config.get('list_details') and config.get('progress'):
        raise ValueError(
            'With the create action, only one of --list/--files/list_details and --progress/progress can be used.',
@@ -51,99 +45,92 @@ def run_create(

    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
        patterns = pattern.process_patterns(
-            pattern.collect_patterns(config),
+            pattern.collect_patterns(config, working_directory),
            config,
            working_directory,
            borgmatic_runtime_directory,
        )
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )
-        active_dumps = borgmatic.hooks.dispatch.call_hooks(
-            'dump_data_sources',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            config_paths,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )

-        # Process the patterns again in case any data source hooks updated them. Without this step,
-        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
-        # the same named pipe twice.
-        patterns = pattern.process_patterns(
-            patterns,
-            config,
-            working_directory,
-            borgmatic_runtime_directory,
-            skip_expand_paths=config_paths,
-        )
-        stream_processes = [process for processes in active_dumps.values() for process in processes]
+        original_patterns = list(patterns)

-        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
-        # to make sure we only create a real archive if all the streaming processes completed
-        # successfully (create_archive will fail if a streaming process fails, but the archive might
-        # have already been created at this point).
-        use_checkpoint = bool(stream_processes)
-
-        json_output = borgmatic.borg.create.create_archive(
-            global_arguments.dry_run,
-            repository['path'],
-            config,
-            patterns,
-            local_borg_version,
-            global_arguments,
-            borgmatic_runtime_directory,
-            archive_suffix='.checkpoint' if use_checkpoint else '',
-            local_path=local_path,
-            remote_path=remote_path,
-            json=create_arguments.json,
-            comment=create_arguments.comment,
-            stream_processes=stream_processes,
-        )
-
-        if use_checkpoint:
-            rename_checkpoint_archive(
-                repository['path'],
-                global_arguments,
+        # Use the original patterns so as to disregard any modifications made by any data source
+        # hooks, e.g. via dump_data_sources() below.
+        with borgmatic.actions.dump.Dump_cleanup(
+            config, borgmatic_runtime_directory, original_patterns, global_arguments.dry_run
+        ):
+            active_dumps = borgmatic.hooks.dispatch.call_hooks(
+                'dump_data_sources',
                config,
+                borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                config_paths,
+                borgmatic_runtime_directory,
+                patterns,
+                global_arguments.dry_run,
+            )
+
+            # Process the patterns again in case any data source hooks updated them. Without this step,
+            # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+            # the same named pipe twice.
+            patterns = pattern.process_patterns(
+                patterns,
+                config,
+                working_directory,
+                borgmatic_runtime_directory,
+                skip_expand_paths=config_paths,
+            )
+            stream_processes = [
+                process for processes in active_dumps.values() for process in processes
+            ]
+
+            # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+            # to make sure we only create a real archive if all the streaming processes completed
+            # successfully (create_archive will fail if a streaming process fails, but the archive might
+            # have already been created at this point).
+            use_checkpoint = bool(stream_processes)
+
+            json_output = borgmatic.borg.create.create_archive(
+                global_arguments.dry_run,
+                repository['path'],
+                config,
+                patterns,
                local_borg_version,
-                local_path,
-                remote_path,
+                global_arguments,
+                borgmatic_runtime_directory,
+                archive_suffix='.checkpoint' if use_checkpoint else '',
+                local_path=local_path,
+                remote_path=remote_path,
+                json=create_arguments.json,
+                comment=create_arguments.comment,
+                stream_processes=stream_processes,
            )

-        if json_output:
-            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
            if use_checkpoint:
-                # Patch archive name and ID
-                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                rename_checkpoint_archive(
                    repository['path'],
+                    global_arguments,
                    config,
                    local_borg_version,
-                    global_arguments,
                    local_path,
                    remote_path,
                )

-                output['archive']['name'] = renamed_archive['name']
-                output['archive']['id'] = renamed_archive['id']
+            if json_output:
+                output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+                if use_checkpoint:
+                    # Patch archive name and ID
+                    renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                        repository['path'],
+                        config,
+                        local_borg_version,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )

-            yield output
+                    output['archive']['name'] = renamed_archive['name']
+                    output['archive']['id'] = renamed_archive['id']

-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )
+                yield output


 def rename_checkpoint_archive(
@@ -2,7 +2,6 @@ import logging

 import borgmatic.actions.arguments
 import borgmatic.borg.delete
-import borgmatic.borg.repo_delete
 import borgmatic.borg.repo_list

 logger = logging.getLogger(__name__)
@@ -20,32 +19,28 @@ def run_delete(
    '''
    Run the "delete" action for the given repository and archive(s).
    '''
-    if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        delete_arguments.repository,
-    ):
-        logger.answer('Deleting archives')
+    logger.answer('Deleting archives')

-        archive_name = (
-            borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
-                delete_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-            if delete_arguments.archive
-            else None
-        )
-
-        borgmatic.borg.delete.delete_archives(
-            repository,
+    archive_name = (
+        borgmatic.borg.repo_list.resolve_archive_name(
+            repository['path'],
+            delete_arguments.archive,
            config,
            local_borg_version,
-            borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
            global_arguments,
            local_path,
            remote_path,
        )
+        if delete_arguments.archive
+        else None
+    )
+
+    borgmatic.borg.delete.delete_archives(
+        repository,
+        config,
+        local_borg_version,
+        borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
+        global_arguments,
+        local_path,
+        remote_path,
+    )
@@ -0,0 +1,63 @@
+import logging
+
+import borgmatic.actions.pattern
+import borgmatic.borg.diff
+
+logger = logging.getLogger(__name__)
+
+
+def run_diff(
+    repository,
+    config,
+    local_borg_version,
+    diff_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "diff" action for the given repository.
+    '''
+
+    # Only process patterns if only_patterns flag is set
+    if diff_arguments.only_patterns:
+        working_directory = borgmatic.config.paths.get_working_directory(config)
+        processed_patterns = borgmatic.actions.pattern.process_patterns(
+            (*borgmatic.actions.pattern.collect_patterns(config, working_directory),),
+            config,
+            working_directory,
+        )
+    else:
+        processed_patterns = None
+
+    archive = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        diff_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    second_archive = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        diff_arguments.second_archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+
+    borgmatic.borg.diff.diff(
+        repository['path'],
+        archive,
+        second_archive,
+        config,
+        local_borg_version,
+        diff_arguments,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+        patterns=processed_patterns,
+    )
@@ -0,0 +1,53 @@
+import borgmatic.hooks.dispatch
+
+
+class Dump_cleanup:
+    '''
+    A Python context manager for removing data source dumps before and after wrapped code. Besides
+    doing this for general filesystem cleanliness reasons, leaving old dumps lying around can
+    prevent future dumping from working (in the case of filesystem snapshots) or cause Borg hangs
+    (in the case of database dump named pipes).
+
+    Example use as a context manager:
+
+        with borgmatic.actions.dump.Dump_cleanup(
+            config, borgmatic_runtime_directory, patterns, dry_run,
+        ):
+            do_something_like_perform_a_dump_or_restore()
+    '''
+
+    def __init__(self, config, borgmatic_runtime_directory, patterns, dry_run):
+        '''
+        Given a configuration dict, the borgmatic runtime directory, the configured patterns, and
+        whether this is a dry-run, store these values for use below.
+        '''
+        self.config = config
+        self.borgmatic_runtime_directory = borgmatic_runtime_directory
+        self.patterns = patterns
+        self.dry_run = dry_run
+
+    def __enter__(self):
+        '''
+        Remove all data source dumps that exist prior to the wrapped code running.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Remove all data source dumps, including any created by the wrapped code.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )
@@ -1,7 +1,6 @@
 import logging

 import borgmatic.borg.export_key
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -18,17 +17,13 @@ def run_export_key(
    '''
    Run the "key export" action for the given repository.
    '''
-    if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        export_arguments.repository,
-    ):
-        logger.info('Exporting repository key')
-        borgmatic.borg.export_key.export_key(
-            repository['path'],
-            config,
-            local_borg_version,
-            export_arguments,
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+    logger.info('Exporting repository key')
+    borgmatic.borg.export_key.export_key(
+        repository['path'],
+        config,
+        local_borg_version,
+        export_arguments,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -2,7 +2,6 @@ import logging

 import borgmatic.borg.export_tar
 import borgmatic.borg.repo_list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -19,30 +18,26 @@ def run_export_tar(
    '''
    Run the "export-tar" action for the given repository.
    '''
-    if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        export_tar_arguments.repository,
-    ):
-        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
-        borgmatic.borg.export_tar.export_tar_archive(
-            global_arguments.dry_run,
+    logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
+    borgmatic.borg.export_tar.export_tar_archive(
+        global_arguments.dry_run,
+        repository['path'],
+        borgmatic.borg.repo_list.resolve_archive_name(
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
-                export_tar_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            export_tar_arguments.paths,
-            export_tar_arguments.destination,
+            export_tar_arguments.archive,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-            tar_filter=export_tar_arguments.tar_filter,
-            strip_components=export_tar_arguments.strip_components,
-        )
+            local_path,
+            remote_path,
+        ),
+        export_tar_arguments.paths,
+        export_tar_arguments.destination,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+        tar_filter=export_tar_arguments.tar_filter,
+        strip_components=export_tar_arguments.strip_components,
+    )
@@ -2,8 +2,6 @@ import logging

 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
-import borgmatic.hooks.command

 logger = logging.getLogger(__name__)

@@ -21,29 +19,25 @@ def run_extract(
    '''
    Run the "extract" action for the given repository.
    '''
-    if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        extract_arguments.repository,
-    ):
-        logger.info(f'Extracting archive {extract_arguments.archive}')
-        borgmatic.borg.extract.extract_archive(
-            global_arguments.dry_run,
+    logger.info(f'Extracting archive {extract_arguments.archive}')
+    borgmatic.borg.extract.extract_archive(
+        global_arguments.dry_run,
+        repository['path'],
+        borgmatic.borg.repo_list.resolve_archive_name(
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
-                extract_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            extract_arguments.paths,
+            extract_arguments.archive,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-            destination_path=extract_arguments.destination,
-            strip_components=extract_arguments.strip_components,
-        )
+            local_path,
+            remote_path,
+        ),
+        extract_arguments.paths,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+        destination_path=extract_arguments.destination,
+        strip_components=extract_arguments.strip_components,
+    )
@@ -1,7 +1,6 @@
 import logging

 import borgmatic.borg.import_key
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -18,17 +17,13 @@ def run_import_key(
    '''
    Run the "key import" action for the given repository.
    '''
-    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        import_arguments.repository,
-    ):
-        logger.info('Importing repository key')
-        borgmatic.borg.import_key.import_key(
-            repository['path'],
-            config,
-            local_borg_version,
-            import_arguments,
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+    logger.info('Importing repository key')
+    borgmatic.borg.import_key.import_key(
+        repository['path'],
+        config,
+        local_borg_version,
+        import_arguments,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -4,7 +4,6 @@ import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.info
 import borgmatic.borg.repo_list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -23,30 +22,26 @@ def run_info(

    If info_arguments.json is True, yield the JSON output from the info for the archive.
    '''
-    if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        info_arguments.repository,
-    ):
-        if not info_arguments.json:
-            logger.answer('Displaying archive summary information')
+    if not info_arguments.json:
+        logger.answer('Displaying archive summary information')

-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
-            info_arguments.archive,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        json_output = borgmatic.borg.info.display_archives_info(
-            repository['path'],
-            config,
-            local_borg_version,
-            borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        info_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    json_output = borgmatic.borg.info.display_archives_info(
+        repository['path'],
+        config,
+        local_borg_version,
+        borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    if json_output:
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -3,7 +3,6 @@ import logging
 import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -22,33 +21,29 @@ def run_list(

    If list_arguments.json is True, yield the JSON output from listing the archive.
    '''
-    if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        list_arguments.repository,
-    ):
-        if not list_arguments.json:
-            if list_arguments.find_paths:  # pragma: no cover
-                logger.answer('Searching archives')
-            elif not list_arguments.archive:  # pragma: no cover
-                logger.answer('Listing archives')
+    if not list_arguments.json:
+        if list_arguments.find_paths:  # pragma: no cover
+            logger.answer('Searching archives')
+        elif not list_arguments.archive:  # pragma: no cover
+            logger.answer('Listing archives')

-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
-            list_arguments.archive,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        json_output = borgmatic.borg.list.list_archive(
-            repository['path'],
-            config,
-            local_borg_version,
-            borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        list_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    json_output = borgmatic.borg.list.list_archive(
+        repository['path'],
+        config,
+        local_borg_version,
+        borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+    if json_output:
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -2,7 +2,6 @@ import logging

 import borgmatic.borg.mount
 import borgmatic.borg.repo_list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -19,30 +18,26 @@ def run_mount(
    '''
    Run the "mount" action for the given repository.
    '''
-    if mount_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        mount_arguments.repository,
-    ):
-        if mount_arguments.archive:
-            logger.info(f'Mounting archive {mount_arguments.archive}')
-        else:  # pragma: nocover
-            logger.info('Mounting repository')
+    if mount_arguments.archive:
+        logger.info(f'Mounting archive {mount_arguments.archive}')
+    else:  # pragma: nocover
+        logger.info('Mounting repository')

-        borgmatic.borg.mount.mount_archive(
+    borgmatic.borg.mount.mount_archive(
+        repository['path'],
+        borgmatic.borg.repo_list.resolve_archive_name(
            repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
-                mount_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            mount_arguments,
+            mount_arguments.archive,
            config,
            local_borg_version,
            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
+            local_path,
+            remote_path,
+        ),
+        mount_arguments,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
@@ -34,10 +34,11 @@ def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_sty
    )


-def collect_patterns(config):
+def collect_patterns(config, working_directory):
    '''
-    Given a configuration dict, produce a single sequence of patterns comprised of the configured
-    source directories, patterns, excludes, pattern files, and exclude files.
+    Given a configuration dict and the working directory, produce a single sequence of patterns
+    comprised of the configured source directories, patterns, excludes, pattern files, and exclude
+    files.

    The idea is that Borg has all these different ways of specifying includes, excludes, source
    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
@@ -68,7 +69,8 @@ def collect_patterns(config):
            + tuple(
                parse_pattern(pattern_line.strip())
                for filename in config.get('patterns_from', ())
-                for pattern_line in open(filename, encoding='utf-8').readlines()
+                for expanded_path in expand_directory(filename, working_directory)
+                for pattern_line in open(expanded_path, encoding='utf-8')
                if not pattern_line.lstrip().startswith('#')
                if pattern_line.strip()
            )
@@ -78,7 +80,8 @@ def collect_patterns(config):
                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
                )
                for filename in config.get('exclude_from', ())
-                for exclude_line in open(filename, encoding='utf-8').readlines()
+                for expanded_path in expand_directory(filename, working_directory)
+                for exclude_line in open(expanded_path, encoding='utf-8')
                if not exclude_line.lstrip().startswith('#')
                if exclude_line.strip()
            )
@@ -259,31 +262,37 @@ def deduplicate_runtime_directory_patterns(patterns, config, borgmatic_runtime_d
        return patterns

    deduplicated = {}  # Use just the keys as an ordered set.
+    runtime_directory_parents = set(pathlib.PurePath(borgmatic_runtime_directory).parents).union(
+        {pathlib.PurePath(borgmatic_runtime_directory)}
+    )

    for pattern in patterns:
        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
            deduplicated[pattern] = True
            continue

-        parents = pathlib.PurePath(pattern.path).parents
+        pattern_parents = pathlib.PurePath(pattern.path).parents

-        # If another directory in the given list is a parent of current directory (even n levels up)
-        # and both are on the same filesystem (or one_file_system is not set), then the current
-        # directory is a duplicate.
+        # If:
+        #
+        #   1. another pattern is a parent of the current pattern (even n levels up),
+        #   2. both patterns are parents of the runtime directory (even n levels up),
+        #   3. and both patterns are on the same filesystem (or one_file_system is not set)
+        #
+        # ... then consider the current pattern as a duplicate.
        for other_pattern in patterns:
            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
                continue

-            if any(
-                pathlib.PurePath(other_pattern.path) == parent
-                and pathlib.PurePosixPath(other_pattern.path)
-                in pathlib.PurePath(borgmatic_runtime_directory).parents
-                and pattern.device is not None
-                and (
-                    other_pattern.device == pattern.device
-                    or config.get('one_file_system') is not True
-                )
-                for parent in parents
+            device_matches = pattern.device is not None and (
+                other_pattern.device == pattern.device or config.get('one_file_system') is not True
+            )
+
+            if (
+                pathlib.PurePath(other_pattern.path) in pattern_parents
+                and pathlib.PurePosixPath(other_pattern.path) in runtime_directory_parents
+                and pathlib.PurePosixPath(pattern.path) in runtime_directory_parents
+                and device_matches
            ):
                break
        else:
@@ -1,8 +1,6 @@
 import logging

 import borgmatic.borg.prune
-import borgmatic.config.validate
-import borgmatic.hooks.command

 logger = logging.getLogger(__name__)

@@ -21,12 +19,6 @@ def run_prune(
    '''
    Run the "prune" action for the given repository.
    '''
-    if prune_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        prune_arguments.repository,
-    ):
-        return
-
    logger.info(f'Pruning archives{dry_run_label}')
    borgmatic.borg.prune.prune_archives(
        global_arguments.dry_run,
@@ -1,11 +1,10 @@
 import logging
 import subprocess

-import borgmatic.borg.info
+import borgmatic.actions.pattern
+import borgmatic.borg.pattern
 import borgmatic.borg.recreate
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
-from borgmatic.actions.pattern import collect_patterns, process_patterns

 logger = logging.getLogger(__name__)

@@ -19,70 +18,77 @@ def run_recreate(
    local_borg_version,
    recreate_arguments,
    global_arguments,
+    dry_run_label,
    local_path,
    remote_path,
 ):
    '''
    Run the "recreate" action for the given repository.
    '''
-    if recreate_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        recreate_arguments.repository,
-    ):
-        if recreate_arguments.archive:
-            logger.answer(f'Recreating archive {recreate_arguments.archive}')
-        else:
-            logger.answer('Recreating repository')
+    if recreate_arguments.archive:
+        logger.answer(f'Recreating archive {recreate_arguments.archive}{dry_run_label}')
+    else:
+        logger.answer(f'Recreating repository{dry_run_label}')

-        # Collect and process patterns.
-        processed_patterns = process_patterns(
-            collect_patterns(config),
-            config,
-            borgmatic.config.paths.get_working_directory(config),
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    # Collect and process patterns.
+    processed_patterns = borgmatic.actions.pattern.process_patterns(
+        (
+            *borgmatic.actions.pattern.collect_patterns(config, working_directory),
+            # Also add borgmatic-specific paths, so they don't get excluded from the recreated
+            # archive. Note that this doesn't currently work for archives created with Borg 1.2 or
+            # below.
+            borgmatic.borg.pattern.Pattern(
+                '/borgmatic', source=borgmatic.borg.pattern.Pattern_source.INTERNAL
+            ),
+        ),
+        config,
+        working_directory,
+    )
+
+    archive = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        recreate_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+
+    if archive and archive.endswith('.recreate'):
+        if recreate_arguments.archive == 'latest':
+            raise ValueError(
+                f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
+            )
+
+        raise ValueError(
+            f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
        )

-        archive = borgmatic.borg.repo_list.resolve_archive_name(
+    try:
+        borgmatic.borg.recreate.recreate_archive(
            repository['path'],
-            recreate_arguments.archive,
+            archive,
            config,
            local_borg_version,
+            recreate_arguments,
            global_arguments,
-            local_path,
-            remote_path,
+            local_path=local_path,
+            remote_path=remote_path,
+            patterns=processed_patterns,
        )
-
-        if archive and archive.endswith('.recreate'):
-            if recreate_arguments.archive == 'latest':
+    except subprocess.CalledProcessError as error:
+        if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
+            if recreate_arguments.target:
                raise ValueError(
-                    f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
+                    f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
                )

-            raise ValueError(
-                f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
-            )
+            if archive:
+                raise ValueError(
+                    f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
+                )

-        try:
-            borgmatic.borg.recreate.recreate_archive(
-                repository['path'],
-                archive,
-                config,
-                local_borg_version,
-                recreate_arguments,
-                global_arguments,
-                local_path=local_path,
-                remote_path=remote_path,
-                patterns=processed_patterns,
-            )
-        except subprocess.CalledProcessError as error:
-            if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
-                if recreate_arguments.target:
-                    raise ValueError(
-                        f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
-                    )
-
-                if archive:
-                    raise ValueError(
-                        f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
-                    )
-
-            raise
+        raise
@@ -1,7 +1,6 @@
 import logging

 import borgmatic.borg.repo_create
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -18,12 +17,6 @@ def run_repo_create(
    '''
    Run the "repo-create" action for the given repository.
    '''
-    if repo_create_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        repo_create_arguments.repository,
-    ):
-        return
-
    logger.info('Creating repository')

    encryption_mode = repo_create_arguments.encryption_mode or repository.get('encryption')
@@ -17,20 +17,16 @@ def run_repo_delete(
    '''
    Run the "repo-delete" action for the given repository.
    '''
-    if repo_delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        repo_delete_arguments.repository,
-    ):
-        logger.answer(
-            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
-        )
+    logger.answer(
+        'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
+    )

-        borgmatic.borg.repo_delete.delete_repository(
-            repository,
-            config,
-            local_borg_version,
-            repo_delete_arguments,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
+    borgmatic.borg.repo_delete.delete_repository(
+        repository,
+        config,
+        local_borg_version,
+        repo_delete_arguments,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
@@ -2,7 +2,6 @@ import logging

 import borgmatic.actions.json
 import borgmatic.borg.repo_info
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -21,21 +20,17 @@ def run_repo_info(

    If repo_info_arguments.json is True, yield the JSON output from the info for the repository.
    '''
-    if repo_info_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        repo_info_arguments.repository,
-    ):
-        if not repo_info_arguments.json:
-            logger.answer('Displaying repository summary information')
+    if not repo_info_arguments.json:
+        logger.answer('Displaying repository summary information')

-        json_output = borgmatic.borg.repo_info.display_repository_info(
-            repository['path'],
-            config,
-            local_borg_version,
-            repo_info_arguments=repo_info_arguments,
-            global_arguments=global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+    json_output = borgmatic.borg.repo_info.display_repository_info(
+        repository['path'],
+        config,
+        local_borg_version,
+        repo_info_arguments=repo_info_arguments,
+        global_arguments=global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
+    if json_output:
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -2,7 +2,6 @@ import logging

 import borgmatic.actions.json
 import borgmatic.borg.repo_list
-import borgmatic.config.validate

 logger = logging.getLogger(__name__)

@@ -21,21 +20,17 @@ def run_repo_list(

    If repo_list_arguments.json is True, yield the JSON output from listing the repository.
    '''
-    if repo_list_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository,
-        repo_list_arguments.repository,
-    ):
-        if not repo_list_arguments.json:
-            logger.answer('Listing repository')
+    if not repo_list_arguments.json:
+        logger.answer('Listing repository')

-        json_output = borgmatic.borg.repo_list.list_repository(
-            repository['path'],
-            config,
-            local_borg_version,
-            repo_list_arguments=repo_list_arguments,
-            global_arguments=global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+    json_output = borgmatic.borg.repo_list.list_repository(
+        repository['path'],
+        config,
+        local_borg_version,
+        repo_list_arguments=repo_list_arguments,
+        global_arguments=global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )
+    if json_output:
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
@@ -1,17 +1,17 @@
 import collections
+import locale
 import logging
 import os
 import pathlib
 import shutil
 import tempfile

+import borgmatic.actions.dump
 import borgmatic.actions.pattern
 import borgmatic.borg.extract
 import borgmatic.borg.list
-import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch

@@ -131,14 +131,14 @@ def get_configured_data_source(config, restore_dump):


 def strip_path_prefix_from_extracted_dump_destination(
-    destination_path,
+    extract_path,
    borgmatic_runtime_directory,
 ):
    '''
    Directory-format dump files get extracted into a temporary directory containing a path prefix
-    that depends how the files were stored in the archive. So, given the destination path where the
-    dump was extracted and the borgmatic runtime directory, move the dump files such that the
-    restore doesn't have to deal with that varying path prefix.
+    that depends how the files were stored in the archive. So, given the path where the dump was
+    extracted and the borgmatic runtime directory, move the dump files such that the restore doesn't
+    have to deal with that varying path prefix.

    For instance, if the dump was extracted to:

@@ -152,16 +152,16 @@ def strip_path_prefix_from_extracted_dump_destination(

      /run/user/0/borgmatic/postgresql_databases/test/...
    '''
-    for subdirectory_path, _, _ in os.walk(destination_path):
+    for subdirectory_path, _, _ in os.walk(extract_path):
        databases_directory = os.path.basename(subdirectory_path)

        if not databases_directory.endswith('_databases'):
            continue

-        shutil.move(
-            subdirectory_path,
-            os.path.join(borgmatic_runtime_directory, databases_directory),
-        )
+        destination_path = os.path.join(borgmatic_runtime_directory, databases_directory)
+        shutil.rmtree(destination_path, ignore_errors=True)
+        shutil.move(subdirectory_path, destination_path)
+
        break


@@ -202,6 +202,10 @@ def restore_single_dump(
        borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
        borgmatic_runtime_directory,
        data_source['name'],
+        data_source.get('hostname'),
+        data_source.get('port'),
+        data_source.get('container'),
+        data_source.get('label'),
    )[hook_name.split('_databases', 1)[0]]

    destination_path = (
@@ -269,12 +273,12 @@ def collect_dumps_from_archive(
    Given a local or remote repository path, a resolved archive name, a configuration dict, the
    local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
    borgmatic runtime directory, query the archive for the names of data sources dumps it contains
-    and return them as a set of Dump instances.
+    and return them as a tuple of Dump instances.
    '''
-    dumps_from_archive = set()
+    dumps_from_archive = {}  # Use a dict as an ordered set.

    # There is (at most) one dump metadata file per data source hook. Load each.
-    for dumps_metadata_path in borgmatic.borg.list.capture_archive_listing(
+    for dumps_metadata_entry in borgmatic.borg.list.capture_archive_listing(
        repository,
        archive,
        config,
@@ -297,36 +301,33 @@ def collect_dumps_from_archive(
        local_path=local_path,
        remote_path=remote_path,
    ):
-        if not dumps_metadata_path:
+        if not dumps_metadata_entry.get('path'):
            continue

-        dumps_from_archive.update(
-            set(
-                borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
-                    borgmatic.borg.extract.extract_archive(
-                        global_arguments.dry_run,
-                        repository,
-                        archive,
-                        [dumps_metadata_path],
-                        config,
-                        local_borg_version,
-                        global_arguments,
-                        local_path=local_path,
-                        remote_path=remote_path,
-                        extract_to_stdout=True,
-                    )
-                    .stdout.read()
-                    .decode(),
-                    dumps_metadata_path,
-                )
+        for dump in borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
+            borgmatic.borg.extract.extract_archive(
+                global_arguments.dry_run,
+                repository,
+                archive,
+                [dumps_metadata_entry['path']],
+                config,
+                local_borg_version,
+                global_arguments,
+                local_path=local_path,
+                remote_path=remote_path,
+                extract_to_stdout=True,
            )
-        )
+            .stdout.read()
+            .decode(locale.getpreferredencoding()),
+            dumps_metadata_entry['path'],
+        ):
+            dumps_from_archive[dump] = None

    # If we've successfully loaded any dumps metadata, we're done.
    if dumps_from_archive:
        logger.debug('Collecting database dumps from archive data source dumps metadata files')

-        return dumps_from_archive
+        return tuple(dumps_from_archive.keys())

    # No dumps metadata files were found, so for backwards compatibility, fall back to parsing the
    # paths of dumps found in the archive to get their respective dump metadata.
@@ -338,7 +339,7 @@ def collect_dumps_from_archive(
    # Probe for the data source dumps in multiple locations, as the default location has moved to
    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
    # we still want to support reading dumps from previously created archives as well.
-    dump_paths = borgmatic.borg.list.capture_archive_listing(
+    dump_entries = borgmatic.borg.list.capture_archive_listing(
        repository,
        archive,
        config,
@@ -360,7 +361,9 @@ def collect_dumps_from_archive(
        remote_path=remote_path,
    )

-    for dump_path in dump_paths:
+    for dump_entry in dump_entries:
+        dump_path = dump_entry.get('path')
+
        if not dump_path:
            continue

@@ -390,11 +393,11 @@ def collect_dumps_from_archive(
            except (ValueError, TypeError):
                port = None

-            dumps_from_archive.add(
+            dumps_from_archive[
                Dump(
                    hook_name, data_source_name, None if hostname == 'localhost' else hostname, port
                )
-            )
+            ] = None

            # We've successfully parsed the dump path, so need to probe any further.
            break
@@ -403,20 +406,21 @@ def collect_dumps_from_archive(
                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}',
            )

-    return dumps_from_archive
+    return tuple(dumps_from_archive.keys())


 def get_dumps_to_restore(restore_arguments, dumps_from_archive):
    '''
    Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
-    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    a tuple of Dump instances representing the dumps found in an archive, return a tuple of specific
    Dump instances from the archive to restore. As part of this, replace any Dump having a data
    source name of "all" with multiple named Dump instances as appropriate.

    Raise ValueError if any of the requested data source names cannot be found in the archive or if
    there are multiple archive dump matches for a given requested dump.
    '''
-    requested_dumps = (
+    requested_dumps = tuple(
+        # Use a dict comprehension as an ordered set.
        {
            Dump(
                hook_name=(
@@ -433,16 +437,16 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                port=restore_arguments.original_port,
                label=restore_arguments.original_label or UNSPECIFIED,
                container=restore_arguments.original_container or UNSPECIFIED,
-            )
+            ): None
            for name in restore_arguments.data_sources or (UNSPECIFIED,)
-        }
+        }.keys()
        if restore_arguments.hook
        or restore_arguments.data_sources
        or restore_arguments.original_hostname
        or restore_arguments.original_port
        or restore_arguments.original_label
        or restore_arguments.original_container
-        else {
+        else (
            Dump(
                hook_name=UNSPECIFIED,
                data_source_name='all',
@@ -451,14 +455,15 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                label=UNSPECIFIED,
                container=UNSPECIFIED,
            ),
-        }
+        )
    )
    missing_dumps = set()
-    dumps_to_restore = set()
+    dumps_to_restore = {}  # Use a dict as an ordered set.

    # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
    if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
-        dumps_to_restore.update(dumps_from_archive)
+        for dump in dumps_from_archive:
+            dumps_to_restore[dump] = None

    # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
    for requested_dump in requested_dumps:
@@ -474,7 +479,7 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
        if len(matching_dumps) == 0:
            missing_dumps.add(requested_dump)
        elif len(matching_dumps) == 1:
-            dumps_to_restore.add(matching_dumps[0])
+            dumps_to_restore[matching_dumps[0]] = None
        else:
            raise ValueError(
                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.',
@@ -489,20 +494,20 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive",
        )

-    return dumps_to_restore
+    return tuple(dumps_to_restore.keys())


 def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
    '''
-    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
-    if any requested dumps to restore weren't restored, indicating that they were missing from the
-    configuration.
+    Given a tuple of requested dumps to restore and a set of dumps actually restored, raise
+    ValueError if any requested dumps to restore weren't restored, indicating that they were missing
+    from the configuration.
    '''
-    if not dumps_actually_restored:
+    if not dumps_to_restore:
        raise ValueError('No data source dumps were found to restore')

    missing_dumps = sorted(
-        dumps_to_restore - dumps_actually_restored,
+        set(dumps_to_restore) - dumps_actually_restored,
        key=lambda dump: dump.data_source_name,
    )

@@ -524,117 +529,96 @@ def run_restore(
    remote_path,
 ):
    '''
-    Run the "restore" action for the given repository, but only if the repository matches the
-    requested repository in restore arguments.
+    Run the "restore" action for the given repository.

    Raise ValueError if a configured data source could not be found to restore or there's no
    matching dump in the archive.
    '''
-    if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        restore_arguments.repository,
-    ):
-        return
-
    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
    working_directory = borgmatic.config.paths.get_working_directory(config)

    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
        patterns = borgmatic.actions.pattern.process_patterns(
-            borgmatic.actions.pattern.collect_patterns(config),
+            borgmatic.actions.pattern.collect_patterns(config, working_directory),
            config,
            working_directory,
        )
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )

-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
-            restore_arguments.archive,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        dumps_from_archive = collect_dumps_from_archive(
-            repository['path'],
-            archive_name,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-            borgmatic_runtime_directory,
-        )
-        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
-
-        dumps_actually_restored = set()
-        connection_params = {
-            'container': restore_arguments.container,
-            'hostname': restore_arguments.hostname,
-            'port': restore_arguments.port,
-            'username': restore_arguments.username,
-            'password': restore_arguments.password,
-            'restore_path': restore_arguments.restore_path,
-        }
-
-        # Restore each dump.
-        for restore_dump in dumps_to_restore:
-            found_data_source = get_configured_data_source(
-                config,
-                restore_dump,
-            )
-            # For a dump that wasn't found via an exact match in the configuration, try to fallback
-            # to an "all" data source.
-            if not found_data_source:
-                found_data_source = get_configured_data_source(
-                    config,
-                    Dump(
-                        restore_dump.hook_name,
-                        'all',
-                        restore_dump.hostname,
-                        restore_dump.port,
-                        restore_dump.label,
-                        restore_dump.container,
-                    ),
-                )
-
-                if not found_data_source:
-                    continue
-
-                found_data_source = dict(found_data_source)
-                found_data_source['name'] = restore_dump.data_source_name
-
-            dumps_actually_restored.add(restore_dump)
-
-            restore_single_dump(
-                repository,
+        with borgmatic.actions.dump.Dump_cleanup(
+            config, borgmatic_runtime_directory, patterns, global_arguments.dry_run
+        ):
+            archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+                repository['path'],
+                restore_arguments.archive,
                config,
                local_borg_version,
                global_arguments,
                local_path,
                remote_path,
+            )
+            dumps_from_archive = collect_dumps_from_archive(
+                repository['path'],
                archive_name,
-                restore_dump.hook_name,
-                dict(found_data_source, schemas=restore_arguments.schemas),
-                connection_params,
+                config,
+                local_borg_version,
+                global_arguments,
+                local_path,
+                remote_path,
                borgmatic_runtime_directory,
            )
+            dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)

-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )
+            dumps_actually_restored = set()
+            connection_params = {
+                'container': restore_arguments.container,
+                'hostname': restore_arguments.hostname,
+                'port': restore_arguments.port,
+                'username': restore_arguments.username,
+                'password': restore_arguments.password,
+                'restore_path': restore_arguments.restore_path,
+            }
+
+            # Restore each dump.
+            for restore_dump in dumps_to_restore:
+                found_data_source = get_configured_data_source(
+                    config,
+                    restore_dump,
+                )
+                # For a dump that wasn't found via an exact match in the configuration, try to fallback
+                # to an "all" data source.
+                if not found_data_source:
+                    found_data_source = get_configured_data_source(
+                        config,
+                        Dump(
+                            restore_dump.hook_name,
+                            'all',
+                            restore_dump.hostname,
+                            restore_dump.port,
+                            restore_dump.label,
+                            restore_dump.container,
+                        ),
+                    )
+
+                    if not found_data_source:
+                        continue
+
+                    found_data_source = dict(found_data_source)
+                    found_data_source['name'] = restore_dump.data_source_name
+
+                dumps_actually_restored.add(restore_dump)
+
+                restore_single_dump(
+                    repository,
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                    archive_name,
+                    restore_dump.hook_name,
+                    dict(found_data_source, schemas=restore_arguments.schemas),
+                    connection_params,
+                    borgmatic_runtime_directory,
+                )

    ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)
@@ -29,7 +29,6 @@ def run_arbitrary_borg(
    command.
    '''
    borgmatic.logger.add_custom_log_levels()
-    lock_wait = config.get('lock_wait', None)

    try:
        options = options[1:] if options[0] == '--' else options
@@ -53,7 +52,7 @@ def run_arbitrary_borg(
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('lock-wait', lock_wait)
+        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + command_options
    )

@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 from borgmatic.borg import environment, flags
@@ -20,17 +21,19 @@ def break_lock(
    argparse.Namespace of global arguments, and optional local and remote Borg paths, break any
    repository and cache locks leftover from Borg aborting.
    '''
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('break_lock', '')

    full_command = (
        (local_path, 'break-lock')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.execute
@@ -23,17 +24,18 @@ def change_passphrase(
    based on an interactive prompt.
    '''
    borgmatic.logger.add_custom_log_levels()
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_change_passphrase', '')

    full_command = (
        (local_path, 'key', 'change-passphrase')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
@@ -63,5 +65,5 @@ def change_passphrase(
    )

    logger.answer(
-        f"{repository_path}: Don't forget to update your encryption_passphrase option (if needed)",
+        "Don't forget to update your encryption_passphrase option (if needed)",
    )
@@ -20,7 +20,7 @@ def make_archive_filter_flags(local_borg_version, config, checks, check_argument
    flag. And if "prefix" is set in configuration and "archives" is in checks, then include a
    "--match-archives" flag.
    '''
-    check_last = config.get('check_last', None)
+    check_last = config.get('check_last')
    prefix = config.get('prefix')

    if 'archives' in checks or 'data' in checks:
@@ -149,8 +149,10 @@ def check_archives(

    max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')

+    # If not configured, elevate Borg's exit code 1 (an ostensible warning) to error, because Borg
+    # returns exit code 1 for repository check errors!
+    borg_exit_codes = [*config.get('borg_exit_codes', []), *[{'code': 1, 'treat_as': 'error'}]]
    umask = config.get('umask')
-    borg_exit_codes = config.get('borg_exit_codes')
    working_directory = borgmatic.config.paths.get_working_directory(config)

    if 'data' in checks:
@@ -176,7 +178,13 @@ def check_archives(
            + make_check_name_flags(checks_subset, archive_filter_flags)
            + (('--remote-path', remote_path) if remote_path else ())
            + (('--umask', str(umask)) if umask else ())
-            + (('--log-json',) if config.get('log_json') else ())
+            + (
+                ('--log-json',)
+                if (
+                    config.get('log_json') or not (check_arguments.repair or config.get('progress'))
+                )
+                else ()
+            )
            + (('--lock-wait', str(lock_wait)) if lock_wait else ())
            + verbosity_flags
            + (('--progress',) if config.get('progress') else ())
@@ -3,7 +3,7 @@ import shlex

 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags
-from borgmatic.execute import execute_command
+from borgmatic.execute import DO_NOT_CAPTURE, execute_command

 logger = logging.getLogger(__name__)

@@ -22,8 +22,8 @@ def compact_segments(
    Given dry-run flag, a local or remote repository path, a configuration dict, and the local Borg
    version, compact the segments in a repository.
    '''
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
    extra_borg_options = config.get('extra_borg_options', {}).get('compact', '')
    threshold = config.get('compact_threshold')

@@ -31,11 +31,11 @@ def compact_segments(
        (local_path, 'compact')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--progress',) if config.get('progress') else ())
        + (('--cleanup-commits',) if cleanup_commits else ())
-        + (('--threshold', str(threshold)) if threshold else ())
+        + (('--threshold', str(threshold)) if threshold is not None else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (
@@ -54,6 +54,7 @@ def compact_segments(
    execute_command(
        full_command,
        output_log_level=logging.INFO,
+        output_file=DO_NOT_CAPTURE if config.get('progress') else None,
        environment=environment.make_environment(config),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
@@ -53,6 +53,7 @@ def validate_planned_backup_paths(
    local_path,
    working_directory,
    borgmatic_runtime_directory,
+    find_special_files=False,
 ):
    '''
    Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
@@ -61,6 +62,9 @@ def validate_planned_backup_paths(
    given runtime directory exists, validate that it will be included in a backup and hasn't been
    excluded.

+    If find special files is True, then return the subset of planned backup paths that are special
+    files. Otherwise, return an empty tuple.
+
    Raise ValueError if the runtime directory has been excluded via "exclude_patterns" or similar,
    because any features that rely on the runtime directory getting backed up will break. For
    instance, without the runtime directory, Borg can't consume any database dumps and borgmatic may
@@ -69,10 +73,14 @@ def validate_planned_backup_paths(
    # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
    # files including any named pipe we've created. And omit "--filter" because that can break the
    # paths output parsing below such that path lines no longer start with the expected "- ".
-    paths_output = execute_command_and_capture_output(
+    path_lines = execute_command_and_capture_output(
        (
            *flags.omit_flag_and_value(
-                flags.omit_flag(create_command, '--exclude-nodump'), '--filter'
+                flags.omit_flag(
+                    flags.omit_flag(create_command, '--exclude-nodump'),
+                    '--log-json',
+                ),
+                '--filter',
            ),
            '--dry-run',
            '--list',
@@ -86,44 +94,68 @@ def validate_planned_backup_paths(

    # These are all the individual files that Borg is planning to backup as determined by the Borg
    # create dry run above.
-    paths = tuple(
+    paths = (
        path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in path_lines
        if path_line and path_line.startswith(('- ', '+ '))
    )

-    # These are the subset of output paths contained within the borgmatic runtime directory.
-    paths_inside_runtime_directory = {
-        path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+    include_pattern_paths = {
+        pattern.path
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.INCLUDE
    }
-
-    # If the runtime directory isn't present in the source patterns, then we shouldn't expect it to
-    # be in the paths output from the Borg dry run.
-    runtime_directory_present_in_patterns = any(
+    runtime_directory_root_patterns = tuple(
        pattern
        for pattern in patterns
        if any_parent_directories(pattern.path, (borgmatic_runtime_directory,))
        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        # Skip root patterns that have corresponding include patterns, because those will "punch
+        # through" any subsequent excludes.
+        if pattern.path not in include_pattern_paths
    )

-    # If no paths to backup are inside the runtime directory, it must've been excluded.
-    if (
-        not paths_inside_runtime_directory
-        and runtime_directory_present_in_patterns
-        and not dry_run
+    special_paths = []
+    validate_runtime_directory = bool(
+        not dry_run
        and os.path.exists(borgmatic_runtime_directory)
-    ):
-        raise ValueError(
-            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.',
+        and runtime_directory_root_patterns
+    )
+    runtime_directory_in_path = False
+
+    # Do everything in this one loop because we only want to consume the paths generator once.
+    for path in paths:
+        # If all root patterns in the runtime directory are missing from the paths Borg is planning
+        # to backup, then they must've gotten excluded, e.g. by user-configured excludes. Warn
+        # accordingly (below).
+        if validate_runtime_directory and any(
+            any_parent_directories(path, (pattern.path,))
+            for pattern in runtime_directory_root_patterns
+        ):
+            runtime_directory_in_path = True
+
+        # Return the subset of output paths that are special files but *not* contained within the
+        # borgmatic runtime directory. The intent is to skip runtime paths that borgmatic uses for its
+        # own bookkeeping, instead focusing on user-configured paths.
+        if (
+            find_special_files
+            and not any_parent_directories(path, (borgmatic_runtime_directory,))
+            and special_file(path, working_directory)
+        ):
+            special_paths.append(path)
+
+    if validate_runtime_directory and not runtime_directory_in_path:
+        logger.warning(
+            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes (or the snapshotted source directories are empty). Please ensure the runtime directory is not excluded.'
        )

-    return tuple(path for path in paths if path not in paths_inside_runtime_directory)
+    return tuple(special_paths)


 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000


-def make_base_create_command(
+def make_base_create_command(  # noqa: PLR0912
    dry_run,
    repository_path,
    config,
@@ -147,22 +179,25 @@ def make_base_create_command(
    return a tuple of (base Borg create command flags, Borg create command positional arguments,
    open pattern file handle).
    '''
-    if config.get('source_directories_must_exist', False):
-        borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
+    if config.get('source_directories_must_exist', True):
+        borgmatic.borg.pattern.check_all_root_patterns_exist(
+            patterns, borgmatic.config.paths.get_working_directory(config)
+        )

    patterns_file = borgmatic.borg.pattern.write_patterns_file(
        patterns,
        borgmatic_runtime_directory,
    )
-    checkpoint_interval = config.get('checkpoint_interval', None)
-    checkpoint_volume = config.get('checkpoint_volume', None)
-    chunker_params = config.get('chunker_params', None)
-    compression = config.get('compression', None)
-    upload_rate_limit = config.get('upload_rate_limit', None)
-    upload_buffer_size = config.get('upload_buffer_size', None)
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    checkpoint_interval = config.get('checkpoint_interval')
+    checkpoint_volume = config.get('checkpoint_volume')
+    chunker_params = config.get('chunker_params')
+    compression = config.get('compression')
+    upload_rate_limit = config.get('upload_rate_limit')
+    upload_buffer_size = config.get('upload_buffer_size')
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
    list_filter_flags = flags.make_list_filter_flags(local_borg_version, dry_run)
+    files_changed = config.get('files_changed')
    files_cache = config.get('files_cache')
    archive_name_format = (
        config.get('archive_name_format', flags.get_default_archive_name_format(local_borg_version))
@@ -213,11 +248,12 @@ def make_base_create_command(
        + (('--nobirthtime',) if config.get('birthtime') is False else ())
        + (('--read-special',) if config.get('read_special') or stream_processes else ())
        + noflags_flags
+        + (('--files-changed', files_changed) if files_changed else ())
        + (('--files-cache', files_cache) if files_cache else ())
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (
            ('--list', '--filter', list_filter_flags)
            if config.get('list_details') and not json and not config.get('progress')
@@ -234,8 +270,16 @@ def make_base_create_command(
    )
    working_directory = borgmatic.config.paths.get_working_directory(config)

+    if config.get('unsafe_skip_path_validation_before_create'):
+        logger.debug(
+            'Skipping pre-backup path validation due to "unsafe_skip_path_validation_before_create" option.'
+        )
+
+        return (create_flags, create_positional_arguments, patterns_file)
+
    logger.debug('Checking file paths Borg plans to include')
-    planned_backup_paths = validate_planned_backup_paths(
+
+    special_file_paths = validate_planned_backup_paths(
        dry_run,
        create_flags + create_positional_arguments,
        config,
@@ -243,44 +287,41 @@ def make_base_create_command(
        local_path,
        working_directory,
        borgmatic_runtime_directory=borgmatic_runtime_directory,
+        find_special_files=bool(stream_processes),
    )

-    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
-    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
-    if stream_processes and not config.get('read_special'):
+    if stream_processes and config.get('read_special') is False:
        logger.warning(
            'Ignoring configured "read_special" value of false, as true is needed for database hooks.',
        )

-        special_file_paths = tuple(
-            path for path in planned_backup_paths if special_file(path, working_directory)
+    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
+    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
+    if special_file_paths:
+        truncated_special_file_paths = textwrap.shorten(
+            ', '.join(special_file_paths),
+            width=MAX_SPECIAL_FILE_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.warning(
+            f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
+        )
+        patterns_file = borgmatic.borg.pattern.write_patterns_file(
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    special_file_path,
+                    borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                )
+                for special_file_path in special_file_paths
+            ),
+            borgmatic_runtime_directory,
+            patterns_file=patterns_file,
        )

-        if special_file_paths:
-            truncated_special_file_paths = textwrap.shorten(
-                ', '.join(special_file_paths),
-                width=MAX_SPECIAL_FILE_PATHS_LENGTH,
-                placeholder=' ...',
-            )
-            logger.warning(
-                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
-            )
-            patterns_file = borgmatic.borg.pattern.write_patterns_file(
-                tuple(
-                    borgmatic.borg.pattern.Pattern(
-                        special_file_path,
-                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
-                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
-                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
-                    )
-                    for special_file_path in special_file_paths
-                ),
-                borgmatic_runtime_directory,
-                patterns_file=patterns_file,
-            )
-
-            if '--patterns-from' not in create_flags:
-                create_flags += ('--patterns-from', patterns_file.name)
+        if '--patterns-from' not in create_flags:
+            create_flags += ('--patterns-from', patterns_file.name)

    return (create_flags, create_positional_arguments, patterns_file)

@@ -333,7 +374,9 @@ def create_archive(

    if json:
        output_log_level = None
-    elif config.get('list_details') or (config.get('statistics') and not dry_run):
+    elif config.get('list_details') or (
+        (config.get('statistics') or config.get('quick_statistics')) and not dry_run
+    ):
        output_log_level = logging.ANSWER
    else:
        output_log_level = logging.INFO
@@ -345,6 +388,11 @@ def create_archive(
    create_flags += (
        (('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
        + (('--stats',) if config.get('statistics') and not json and not dry_run else ())
+        + (
+            ('--quick-stats',)
+            if config.get('quick_statistics') and not json and not dry_run
+            else ()
+        )
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) and not json else ())
        + (('--progress',) if config.get('progress') else ())
        + (('--json',) if json else ())
@@ -352,25 +400,31 @@ def create_archive(
    borg_exit_codes = config.get('borg_exit_codes')

    if stream_processes:
-        return execute_command_with_processes(
-            create_flags + create_positional_arguments,
-            stream_processes,
-            output_log_level,
-            output_file,
-            working_directory=working_directory,
-            environment=environment.make_environment(config),
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
+        output = '\n'.join(
+            execute_command_with_processes(
+                create_flags + create_positional_arguments,
+                stream_processes,
+                output_log_level,
+                output_file,
+                working_directory=working_directory,
+                environment=environment.make_environment(config),
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
        )
+        return output if json else None

    if output_log_level is None:
-        return execute_command_and_capture_output(
-            create_flags + create_positional_arguments,
-            working_directory=working_directory,
-            environment=environment.make_environment(config),
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
+        output = '\n'.join(
+            execute_command_and_capture_output(
+                create_flags + create_positional_arguments,
+                working_directory=working_directory,
+                environment=environment.make_environment(config),
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
        )
+        return output if json else None

    execute_command(
        create_flags + create_positional_arguments,
@@ -1,5 +1,6 @@
 import argparse
 import logging
+import shlex

 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -28,6 +29,8 @@ def make_delete_command(
    arguments to the delete action as an argparse.Namespace, and global arguments, return a command
    as a tuple to delete archives from the repository.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('delete', '')
+
    return (
        (local_path, 'delete')
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -35,7 +38,7 @@ def make_delete_command(
        + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
        + (
@@ -66,6 +69,7 @@ def make_delete_command(
                'repository',
            ),
        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
    )

@@ -0,0 +1,88 @@
+import logging
+import shlex
+
+import borgmatic.borg.environment
+import borgmatic.borg.feature
+import borgmatic.config.paths
+import borgmatic.execute
+from borgmatic.borg import flags
+from borgmatic.borg.pattern import write_patterns_file
+
+logger = logging.getLogger(__name__)
+
+
+def diff(
+    repository,
+    archive,
+    second_archive,
+    config,
+    local_borg_version,
+    diff_arguments,
+    global_arguments,
+    local_path,
+    remote_path=None,
+    patterns=None,
+):
+    '''
+    Given a local or remote repository path, two archive names, a configuration dict, the local Borg
+    version string, an argparse.Namespace of diff arguments, an argparse.Namespace of global
+    arguments, optional local and remote Borg paths, executes the diff command with the given
+    arguments.
+    '''
+    borgmatic.logger.add_custom_log_levels()
+
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('diff', '')
+
+    if diff_arguments.only_patterns:
+        # Write patterns to a temporary file and use that file with --patterns-from.
+        patterns_file = write_patterns_file(
+            patterns,
+            borgmatic.config.paths.get_working_directory(config),
+        )
+    else:
+        patterns_file = None
+
+    if borgmatic.borg.feature.available(
+        borgmatic.borg.feature.Feature.NUMERIC_IDS, local_borg_version
+    ):
+        numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
+    else:
+        numeric_ids_flags = ('--numeric-owner',) if config.get('numeric_ids') else ()
+
+    diff_command = (
+        (local_path, 'diff')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + ('--log-json',)
+        + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (
+            ('--patterns-from', patterns_file.name)
+            if patterns_file and diff_arguments.only_patterns
+            else ()
+        )
+        + numeric_ids_flags
+        + (('--same-chunker-params',) if diff_arguments.same_chunker_params else ())
+        + (('--sort-by', ','.join(diff_arguments.sort_keys)) if diff_arguments.sort_keys else ())
+        + (('--content-only',) if diff_arguments.content_only else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
+        + (
+            (*flags.make_repository_flags(repository, local_borg_version), archive)
+            if borgmatic.borg.feature.available(
+                borgmatic.borg.feature.Feature.SEPARATE_REPOSITORY_ARCHIVE,
+                local_borg_version,
+            )
+            else flags.make_repository_archive_flags(repository, archive, local_borg_version)
+        )
+        + (second_archive,)
+    )
+
+    borgmatic.execute.execute_command(
+        full_command=diff_command,
+        output_log_level=logging.ANSWER,
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )
@@ -4,12 +4,15 @@ import borgmatic.borg.passcommand
 import borgmatic.hooks.credential.parse

 OPTION_TO_ENVIRONMENT_VARIABLE = {
+    'archive_hostname': 'BORG_HOSTNAME',
+    'archive_username': 'BORG_USERNAME',
    'borg_base_directory': 'BORG_BASE_DIR',
    'borg_config_directory': 'BORG_CONFIG_DIR',
    'borg_cache_directory': 'BORG_CACHE_DIR',
    'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
    'borg_security_directory': 'BORG_SECURITY_DIR',
    'borg_keys_directory': 'BORG_KEYS_DIR',
+    'borg_key_file': 'BORG_KEY_FILE',
    'ssh_command': 'BORG_RSH',
    'temporary_directory': 'TMPDIR',
 }
@@ -24,6 +27,7 @@ DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
+    'msgpack_version_check': 'BORG_MSGPACK_VERSION_CHECK',
 }


@@ -89,7 +93,8 @@ def make_environment(config):
    ) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
        if os.environ.get(environment_variable_name) is None:
            value = config.get(option_name)
-            environment[environment_variable_name] = 'YES' if value else 'NO'
+            if value is not None:
+                environment[environment_variable_name] = 'YES' if value else 'NO'

    for (
        option_name,
@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
@@ -28,9 +29,10 @@ def export_key(
    Raise FileExistsError if a path is given but it already exists on disk.
    '''
    borgmatic.logger.add_custom_log_levels()
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
    working_directory = borgmatic.config.paths.get_working_directory(config)
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_export', '')

    if export_arguments.path and export_arguments.path != '-':
        if os.path.exists(os.path.join(working_directory or '', export_arguments.path)):
@@ -46,12 +48,13 @@ def export_key(
        (local_path, 'key', 'export')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if output_file is None else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('paper', export_arguments.paper)
        + flags.make_flags('qr-html', export_arguments.qr_html)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
@@ -32,14 +33,15 @@ def export_tar_archive(
    If the destination path is "-", then stream the output to stdout instead of to a file.
    '''
    borgmatic.logger.add_custom_log_levels()
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('export_tar', '')

    full_command = (
        (local_path, 'export-tar')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if destination_path != '-' else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--list',) if config.get('list_details') else ())
@@ -47,6 +49,7 @@ def export_tar_archive(
        + (('--dry-run',) if dry_run else ())
        + (('--tar-filter', tar_filter) if tar_filter else ())
        + (('--strip-components', str(strip_components)) if strip_components else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
            repository_path,
            archive,
@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex
 import subprocess

 import borgmatic.config.paths
@@ -23,6 +24,7 @@ def extract_last_archive_dry_run(
    Perform an extraction dry-run of the most recent archive. If there are no archives, skip the
    dry-run.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
    verbosity_flags = ()
    if logger.isEnabledFor(logging.DEBUG):
        verbosity_flags = ('--debug', '--show-rc')
@@ -47,10 +49,12 @@ def extract_last_archive_dry_run(
    full_extract_command = (
        (local_path, 'extract', '--dry-run')
        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not config.get('progress') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + verbosity_flags
+        + (('--progress',) if config.get('progress') else ())
        + list_flag
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
            repository_path,
            last_archive_name,
@@ -58,13 +62,23 @@ def extract_last_archive_dry_run(
        )
    )

-    execute_command(
-        full_extract_command,
-        environment=environment.make_environment(config),
-        working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_local_path=local_path,
-        borg_exit_codes=config.get('borg_exit_codes'),
-    )
+    if config.get('progress'):
+        execute_command(
+            full_extract_command,
+            output_file=DO_NOT_CAPTURE,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
+    else:
+        execute_command(
+            full_extract_command,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )


 def extract_archive(
@@ -90,11 +104,9 @@ def extract_archive(
    If extract to stdout is True, then start the extraction streaming to stdout, and return that
    extract process as an instance of subprocess.Popen.
    '''
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
-
-    if config.get('progress') and extract_to_stdout:
-        raise ValueError('progress and extract to stdout cannot both be set')
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')

    if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
        numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
@@ -123,14 +135,15 @@ def extract_archive(
        + (('--remote-path', remote_path) if remote_path else ())
        + numeric_ids_flags
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + (('--dry-run',) if dry_run else ())
        + (('--strip-components', str(strip_components)) if strip_components else ())
-        + (('--progress',) if config.get('progress') else ())
+        + (('--progress',) if config.get('progress') and not extract_to_stdout else ())
        + (('--stdout',) if extract_to_stdout else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_archive_flags(
            # Make the repository path absolute so the destination directory used below via changing
            # the working directory doesn't prevent Borg from finding the repo. But also apply the
@@ -147,6 +160,17 @@ def extract_archive(
        os.path.join(working_directory or '', destination_path) if destination_path else None
    )

+    if extract_to_stdout:
+        return execute_command(
+            full_command,
+            output_file=subprocess.PIPE,
+            run_to_completion=False,
+            environment=environment.make_environment(config),
+            working_directory=full_destination_path,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
+
    # The progress output isn't compatible with captured and logged output, as progress messes with
    # the terminal directly.
    if config.get('progress'):
@@ -160,17 +184,6 @@ def extract_archive(
        )
        return None

-    if extract_to_stdout:
-        return execute_command(
-            full_command,
-            output_file=subprocess.PIPE,
-            run_to_completion=False,
-            environment=environment.make_environment(config),
-            working_directory=full_destination_path,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-
    # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
    # if the restore paths don't exist in the archive.
    execute_command(
@@ -1,5 +1,4 @@
 import itertools
-import json
 import logging
 import re

@@ -135,27 +134,18 @@ def make_match_archives_flags(  # noqa: PLR0911
    return ('--glob-archives', f'{derived_match_archives}')


-def warn_for_aggressive_archive_flags(json_command, json_output):
+def warn_for_aggressive_archive_flags(command, output_lines):
    '''
-    Given a JSON archives command and the resulting JSON string output from running it, parse the
-    JSON and warn if the command used an archive flag but the output indicates zero archives were
-    found.
+    Given an archives command and the resulting output lines from running it, warn if the command
+    used an archive flag but the output indicates zero archives were found.
    '''
-    archive_flags_used = {'--glob-archives', '--match-archives'}.intersection(set(json_command))
-
-    if not archive_flags_used:
-        return
-
-    try:
-        if len(json.loads(json_output)['archives']) == 0:
-            logger.warning('An archive filter was applied, but no matching archives were found.')
-            logger.warning(
-                'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.',
-            )
-    except json.JSONDecodeError as error:
-        logger.debug(f'Cannot parse JSON output from archive command: {error}')
-    except (TypeError, KeyError):
-        logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
+    if {'--glob-archives', '--match-archives'}.intersection(set(command)) and len(
+        tuple(line for line in output_lines if not line.startswith('terminating with '))
+    ) == 0:
+        logger.warning('An archive filter was applied, but no matching archives were found.')
+        logger.warning(
+            'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.',
+        )


 def omit_flag(arguments, flag):
@@ -1,10 +1,10 @@
 import logging
 import os
+import shlex

 import borgmatic.config.paths
-import borgmatic.logger
 from borgmatic.borg import environment, flags
-from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+from borgmatic.execute import execute_command

 logger = logging.getLogger(__name__)

@@ -27,32 +27,33 @@ def import_key(

    Raise ValueError if the path is given and it does not exist.
    '''
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
    working_directory = borgmatic.config.paths.get_working_directory(config)
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_import', '')

-    if import_arguments.path and import_arguments.path != '-':
-        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
-            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
-
-        input_file = None
-    else:
-        input_file = DO_NOT_CAPTURE
+    if (
+        import_arguments.path
+        and import_arguments.path != '-'
+        and not os.path.exists(os.path.join(working_directory or '', import_arguments.path))
+    ):
+        raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')

    full_command = (
        (local_path, 'key', 'import')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('paper', import_arguments.paper)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(
            repository_path,
            local_borg_version,
        )
-        + ((import_arguments.path,) if input_file is None else ())
+        + (import_arguments.path or '-',)
    )

    if global_arguments.dry_run:
@@ -61,7 +62,6 @@ def import_key(

    execute_command(
        full_command,
-        input_file=input_file,
        output_log_level=logging.INFO,
        environment=environment.make_environment(config),
        working_directory=working_directory,
@@ -1,10 +1,11 @@
 import argparse
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
-from borgmatic.execute import execute_command, execute_command_and_capture_output
+from borgmatic.execute import execute_command_and_capture_output

 logger = logging.getLogger(__name__)

@@ -23,6 +24,8 @@ def make_info_command(
    arguments to the info action as an argparse.Namespace, and global arguments, return a command
    as a tuple to display summary information for archives in the repository.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('info', '')
+
    return (
        (local_path, 'info')
        + (
@@ -37,7 +40,7 @@ def make_info_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + (
            (
@@ -58,6 +61,7 @@ def make_info_command(
            info_arguments,
            excludes=('repository', 'archive', 'prefix', 'match_archives'),
        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

@@ -99,26 +103,28 @@ def display_archives_info(
    borg_exit_codes = config.get('borg_exit_codes')
    working_directory = borgmatic.config.paths.get_working_directory(config)

-    json_info = execute_command_and_capture_output(
-        json_command,
-        environment=environment.make_environment(config),
-        working_directory=working_directory,
-        borg_local_path=local_path,
-        borg_exit_codes=borg_exit_codes,
-    )
-
    if info_arguments.json:
-        return json_info
+        return '\n'.join(
+            execute_command_and_capture_output(
+                json_command,
+                environment=environment.make_environment(config),
+                working_directory=working_directory,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
+        )

-    flags.warn_for_aggressive_archive_flags(json_command, json_info)
-
-    execute_command(
-        main_command,
-        output_log_level=logging.ANSWER,
-        environment=environment.make_environment(config),
-        working_directory=working_directory,
-        borg_local_path=local_path,
-        borg_exit_codes=borg_exit_codes,
+    output_lines = tuple(
+        execute_command_and_capture_output(
+            main_command,
+            output_log_level=logging.ANSWER,
+            environment=environment.make_environment(config),
+            working_directory=working_directory,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
    )

+    flags.warn_for_aggressive_archive_flags(main_command, output_lines)
+
    return None
@@ -1,7 +1,9 @@
 import argparse
 import copy
+import json
 import logging
 import re
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
@@ -17,6 +19,8 @@ MAKE_FLAGS_EXCLUDES = (
    'archive',
    'paths',
    'find_paths',
+    'format',
+    'json',
    *ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST,
 )

@@ -35,6 +39,8 @@ def make_list_command(
    and local and remote Borg paths, return a command as a tuple to list archives or paths within an
    archive.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('list', '')
+
    return (
        (local_path, 'list')
        + (
@@ -49,9 +55,12 @@ def make_list_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
+        + flags.make_flags('json-lines', list_arguments.json)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
+        + flags.make_flags('format', list_arguments.format or config.get('file_list_format'))
        + flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            flags.make_repository_archive_flags(
                repository_path,
@@ -103,14 +112,15 @@ def capture_archive_listing(
    remote_path=None,
 ):
    '''
-    Given a local or remote repository path, an archive name, a configuration
-    dict, the local Borg version, global arguments as an argparse.Namespace,
-    the archive paths (or Borg patterns) in which to list files, the Borg path
-    format to use for the output, and local and remote Borg paths, capture the
-    output of listing that archive and return it as a list of file paths.
+    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
+    version, global arguments as an argparse.Namespace, the archive paths (or Borg patterns) in
+    which to list files, the Borg path format indicating keys to include in the output, and local
+    and remote Borg paths, capture the output of listing that archive and return it as a generator
+    of dicts, one per path.
    '''
-    return tuple(
-        execute_command_and_capture_output(
+    return (
+        json.loads(entry)
+        for entry in execute_command_and_capture_output(
            make_list_command(
                repository_path,
                config,
@@ -120,8 +130,8 @@ def capture_archive_listing(
                    archive=archive,
                    paths=list(list_paths) if list_paths else None,
                    find_paths=None,
-                    json=None,
-                    format=path_format or '{path}{NUL}',
+                    json=True,
+                    format=path_format or None,
                ),
                global_arguments,
                local_path,
@@ -132,8 +142,6 @@ def capture_archive_listing(
            borg_local_path=local_path,
            borg_exit_codes=config.get('borg_exit_codes'),
        )
-        .strip('\0')
-        .split('\0'),
    )


@@ -190,11 +198,6 @@ def list_archive(
                    f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag.",
                )

-    if list_arguments.json:
-        raise ValueError(
-            'The --json flag on the list action is not supported when using the --archive/--find flags.',
-        )
-
    borg_exit_codes = config.get('borg_exit_codes')

    # If there are any paths to find (and there's not a single archive already selected), start by
@@ -213,24 +216,20 @@ def list_archive(
        )

        # Ask Borg to list archives. Capture its output for use below.
-        archive_lines = tuple(
-            execute_command_and_capture_output(
-                repo_list.make_repo_list_command(
-                    repository_path,
-                    config,
-                    local_borg_version,
-                    repo_list_arguments,
-                    global_arguments,
-                    local_path,
-                    remote_path,
-                ),
-                environment=environment.make_environment(config),
-                working_directory=borgmatic.config.paths.get_working_directory(config),
-                borg_local_path=local_path,
-                borg_exit_codes=borg_exit_codes,
-            )
-            .strip('\n')
-            .splitlines(),
+        archive_lines = execute_command_and_capture_output(
+            repo_list.make_repo_list_command(
+                repository_path,
+                config,
+                local_borg_version,
+                repo_list_arguments,
+                global_arguments,
+                local_path,
+                remote_path,
+            ),
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
        )
    else:
        archive_lines = (list_arguments.archive,)
@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags
@@ -23,14 +24,15 @@ def mount_archive(
    dict, the local Borg version, global arguments as an argparse.Namespace instance, and optional
    local and remote Borg paths, mount the archive onto the mount point.
    '''
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get('mount', '')

    full_command = (
        (local_path, 'mount')
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not mount_arguments.foreground else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -39,6 +41,7 @@ def mount_archive(
            excludes=('repository', 'archive', 'mount_point', 'paths', 'options'),
        )
        + (('-o', mount_arguments.options) if mount_arguments.options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            (
                flags.make_repository_flags(repository_path, local_borg_version)
@@ -17,9 +17,11 @@ def run_passcommand(passcommand, working_directory):
    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
    per borgmatic invocation.
    '''
-    return borgmatic.execute.execute_command_and_capture_output(
-        shlex.split(passcommand),
-        working_directory=working_directory,
+    return '\n'.join(
+        borgmatic.execute.execute_command_and_capture_output(
+            shlex.split(passcommand),
+            working_directory=working_directory,
+        )
    )


@@ -88,16 +88,16 @@ def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=Non
    return patterns_file


-def check_all_root_patterns_exist(patterns):
+def check_all_root_patterns_exist(patterns, working_directory):
    '''
-    Given a sequence of Pattern instances, check that all root pattern paths exist. If any don't,
-    raise an exception.
+    Given a sequence of Pattern instances and the current working directory, check that all root
+    pattern paths exist. If any don't, raise an exception.
    '''
    missing_paths = [
        pattern.path
        for pattern in patterns
        if pattern.type == Pattern_type.ROOT
-        if not os.path.exists(pattern.path)
+        if not os.path.exists(os.path.join(working_directory or '', pattern.path))
    ]

    if missing_paths:
@@ -65,16 +65,16 @@ def prune_archives(
    archives according to the retention policy specified in that configuration.
    '''
    borgmatic.logger.add_custom_log_levels()
-    umask = config.get('umask', None)
-    lock_wait = config.get('lock_wait', None)
+    umask = config.get('umask')
+    lock_wait = config.get('lock_wait')
    extra_borg_options = config.get('extra_borg_options', {}).get('prune', '')

    full_command = (
        (local_path, 'prune')
        + make_prune_flags(config, prune_arguments, local_borg_version)
+        + ('--log-json',)
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (
            ('--stats',)
@@ -83,10 +83,23 @@ def prune_archives(
            and not feature.available(feature.Feature.NO_PRUNE_STATS, local_borg_version)
            else ()
        )
+        + (
+            ('--quick-stats',)
+            if config.get('quick_statistics')
+            and not dry_run
+            and not feature.available(feature.Feature.NO_PRUNE_STATS, local_borg_version)
+            else ()
+        )
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + flags.make_flags_from_arguments(
            prune_arguments,
-            excludes=('repository', 'match_archives', 'statistics', 'list_details'),
+            excludes=(
+                'repository',
+                'match_archives',
+                'statistics',
+                'quick_statistics',
+                'list_details',
+            ),
        )
        + (('--list',) if config.get('list_details') else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -95,7 +108,7 @@ def prune_archives(
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

-    if config.get('statistics') or config.get('list_details'):
+    if config.get('statistics') or config.get('quick_statistics') or config.get('list_details'):
        output_log_level = logging.ANSWER
    else:
        output_log_level = logging.INFO
@@ -28,13 +28,14 @@ def recreate_archive(
    arguments, optional local and remote Borg paths, executes the recreate command with the given
    arguments.
    '''
-    lock_wait = config.get('lock_wait', None)
+    lock_wait = config.get('lock_wait')
    exclude_flags = flags.make_exclude_flags(config)
-    compression = config.get('compression', None)
-    chunker_params = config.get('chunker_params', None)
+    compression = config.get('compression')
+    chunker_params = config.get('chunker_params')
+    extra_borg_options = config.get('extra_borg_options', {}).get('recreate', '')

    # Available recompress MODES: "if-different", "always", "never" (default)
-    recompress = config.get('recompress', None)
+    recompress = config.get('recompress')

    # Write patterns to a temporary file and use that file with --patterns-from.
    patterns_file = write_patterns_file(
@@ -45,7 +46,7 @@ def recreate_archive(
    recreate_command = (
        (local_path, 'recreate')
        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
        + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -71,6 +72,8 @@ def recreate_archive(
        + (('--chunker-params', chunker_params) if chunker_params else ())
        + (('--recompress', recompress) if recompress else ())
        + exclude_flags
+        + (('--dry-run',) if global_arguments.dry_run else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (
            (
                flags.make_repository_flags(repository, local_borg_version)
@@ -92,10 +95,6 @@ def recreate_archive(
        )
    )

-    if global_arguments.dry_run:
-        logger.info('Skipping the archive recreation (dry run)')
-        return
-
    borgmatic.execute.execute_command(
        full_command=recreate_command,
        output_log_level=logging.INFO,
@@ -1,5 +1,7 @@
 import logging
+import shlex

+import borgmatic.borg.environment
 import borgmatic.borg.flags

 logger = logging.getLogger(__name__)
@@ -15,6 +17,8 @@ def make_rename_command(
    local_path,
    remote_path,
 ):
+    extra_borg_options = config.get('extra_borg_options', {}).get('rename', '')
+
    return (
        (local_path, 'rename')
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -22,8 +26,9 @@ def make_rename_command(
        + borgmatic.borg.flags.make_flags('dry-run', dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_archive_flags(
            repository_name,
            old_archive_name,
@@ -5,6 +5,7 @@ import shlex
 import subprocess

 import borgmatic.config.paths
+import borgmatic.logger
 from borgmatic.borg import environment, feature, flags, repo_info
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command

@@ -40,17 +41,21 @@ def create_repository(
    Raise subprocess.CalledProcessError if "borg info" returns an error exit code.
    '''
    try:
-        info_data = json.loads(
-            repo_info.display_repository_info(
-                repository_path,
-                config,
-                local_borg_version,
-                argparse.Namespace(json=True),
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-        )
+        # Suppress Borg's "repository does not exist" error log, so the user isn't confused by
+        # seeing an error during successful repository creation.
+        with borgmatic.logger.Logs_suppressed(msgid='Repository.DoesNotExist'):
+            info_data = json.loads(
+                repo_info.display_repository_info(
+                    repository_path,
+                    config,
+                    local_borg_version,
+                    argparse.Namespace(json=True),
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                ),
+            )
+
        repository_encryption_mode = info_data.get('encryption', {}).get('mode')

        if repository_encryption_mode != encryption_mode:
@@ -66,8 +71,14 @@ def create_repository(

    lock_wait = config.get('lock_wait')
    umask = config.get('umask')
+    extra_borg_options_from_init = config.get('extra_borg_options', {}).get('init', '')
    extra_borg_options = config.get('extra_borg_options', {}).get('repo-create', '')

+    if extra_borg_options_from_init:
+        logger.warning(
+            'The "init" option in "extra_borg_options" is deprecated and will be removed from a future release; use "repo_create" instead.'
+        )
+
    repo_create_command = (
        (local_path,)
        + (
@@ -83,11 +94,11 @@ def create_repository(
        + (('--make-parent-dirs',) if make_parent_directories else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
-        + (('--log-json',) if config.get('log_json') else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--umask', str(umask)) if umask else ())
        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options_from_init)) if extra_borg_options_from_init else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -20,12 +21,23 @@ def make_repo_delete_command(
    global_arguments,
    local_path,
    remote_path,
+    output_file,
 ):
    '''
    Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the repo_delete action as an argparse.Namespace, and global arguments, return a command
-    as a tuple to repo_delete the entire repository.
+    arguments to the repo_delete action as an argparse.Namespace, and global arguments, the Borg
+    local path, the Borg remote path, and an optional output file, return a command as a tuple to
+    repo_delete the entire repository.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_delete'
+        if borgmatic.borg.feature.available(
+            borgmatic.borg.feature.Feature.REPO_DELETE, local_borg_version
+        )
+        else 'delete',
+        '',
+    )
+
    return (
        (local_path,)
        + (
@@ -41,7 +53,7 @@ def make_repo_delete_command(
        + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if output_file is None else ())
        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
        + (
@@ -56,6 +68,7 @@ def make_repo_delete_command(
            repo_delete_arguments,
            excludes=('list_details', 'force', 'repository'),
        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
    )

@@ -76,6 +89,14 @@ def delete_repository(
    '''
    borgmatic.logger.add_custom_log_levels()

+    # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+    # prompt won't work.
+    output_file = (
+        None
+        if repo_delete_arguments.force or repo_delete_arguments.cache_only
+        else borgmatic.execute.DO_NOT_CAPTURE
+    )
+
    command = make_repo_delete_command(
        repository,
        config,
@@ -84,18 +105,13 @@ def delete_repository(
        global_arguments,
        local_path,
        remote_path,
+        output_file,
    )

    borgmatic.execute.execute_command(
        command,
        output_log_level=logging.ANSWER,
-        # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
-        # prompt won't work.
-        output_file=(
-            None
-            if repo_delete_arguments.force or repo_delete_arguments.cache_only
-            else borgmatic.execute.DO_NOT_CAPTURE
-        ),
+        output_file=output_file,
        environment=borgmatic.borg.environment.make_environment(config),
        working_directory=borgmatic.config.paths.get_working_directory(config),
        borg_local_path=local_path,
@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
@@ -23,7 +24,11 @@ def display_repository_info(
    information for the Borg repository or return JSON summary information.
    '''
    borgmatic.logger.add_custom_log_levels()
-    lock_wait = config.get('lock_wait', None)
+    lock_wait = config.get('lock_wait')
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_info' if feature.available(feature.Feature.REPO_INFO, local_borg_version) else 'info',
+        '',
+    )

    full_command = (
        (local_path,)
@@ -44,9 +49,10 @@ def display_repository_info(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
        + flags.make_flags('lock-wait', lock_wait)
+        + ('--log-json',)
        + (('--json',) if repo_info_arguments.json else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

@@ -54,12 +60,14 @@ def display_repository_info(
    borg_exit_codes = config.get('borg_exit_codes')

    if repo_info_arguments.json:
-        return execute_command_and_capture_output(
-            full_command,
-            environment=environment.make_environment(config),
-            working_directory=working_directory,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
+        return '\n'.join(
+            execute_command_and_capture_output(
+                full_command,
+                environment=environment.make_environment(config),
+                working_directory=working_directory,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
        )

    execute_command(
@@ -1,11 +1,12 @@
 import argparse
 import json
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
-from borgmatic.execute import execute_command, execute_command_and_capture_output
+from borgmatic.execute import execute_command_and_capture_output

 logger = logging.getLogger(__name__)

@@ -62,6 +63,10 @@ def get_latest_archive(

    Raises ValueError if there are no archives in the repository.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
+        '',
+    )

    full_command = (
        local_path,
@@ -72,8 +77,15 @@ def get_latest_archive(
        ),
        *flags.make_flags('remote-path', remote_path),
        *flags.make_flags('umask', config.get('umask')),
-        *flags.make_flags('log-json', config.get('log_json')),
+        *('--log-json',),
        *flags.make_flags('lock-wait', config.get('lock_wait')),
+        *(
+            flags.make_match_archives_flags(
+                config.get('match_archives'),
+                config.get('archive_name_format'),
+                local_borg_version,
+            )
+        ),
        *(
            flags.make_flags('consider-checkpoints', consider_checkpoints)
            if not feature.available(feature.Feature.REPO_LIST, local_borg_version)
@@ -81,15 +93,18 @@ def get_latest_archive(
        ),
        *flags.make_flags('last', 1),
        '--json',
+        *(tuple(shlex.split(extra_borg_options)) if extra_borg_options else ()),
        *flags.make_repository_flags(repository_path, local_borg_version),
    )

-    json_output = execute_command_and_capture_output(
-        full_command,
-        environment=environment.make_environment(config),
-        working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_local_path=local_path,
-        borg_exit_codes=config.get('borg_exit_codes'),
+    json_output = '\n'.join(
+        execute_command_and_capture_output(
+            full_command,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
    )

    archives = json.loads(json_output)['archives']
@@ -104,7 +119,7 @@ def get_latest_archive(
    return latest_archive


-MAKE_FLAGS_EXCLUDES = ('repository', 'prefix', 'match_archives')
+MAKE_FLAGS_EXCLUDES = ('repository', 'format', 'prefix', 'match_archives')


 def make_repo_list_command(
@@ -121,6 +136,11 @@ def make_repo_list_command(
    arguments to the repo_list action, global arguments as an argparse.Namespace instance, and local and
    remote Borg paths, return a command as a tuple to list archives with a repository.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
+        '',
+    )
+
    return (
        (
            local_path,
@@ -142,7 +162,7 @@ def make_repo_list_command(
        )
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + (
            (
@@ -159,7 +179,11 @@ def make_repo_list_command(
                )
            )
        )
+        + flags.make_flags(
+            'format', repo_list_arguments.format or config.get('archive_list_format')
+        )
        + flags.make_flags_from_arguments(repo_list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
    )

@@ -202,26 +226,28 @@ def list_repository(
    working_directory = borgmatic.config.paths.get_working_directory(config)
    borg_exit_codes = config.get('borg_exit_codes')

-    json_listing = execute_command_and_capture_output(
-        json_command,
-        environment=environment.make_environment(config),
-        working_directory=working_directory,
-        borg_local_path=local_path,
-        borg_exit_codes=borg_exit_codes,
-    )
-
    if repo_list_arguments.json:
-        return json_listing
+        return '\n'.join(
+            execute_command_and_capture_output(
+                json_command,
+                environment=environment.make_environment(config),
+                working_directory=working_directory,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
+        )

-    flags.warn_for_aggressive_archive_flags(json_command, json_listing)
-
-    execute_command(
-        main_command,
-        output_log_level=logging.ANSWER,
-        environment=environment.make_environment(config),
-        working_directory=working_directory,
-        borg_local_path=local_path,
-        borg_exit_codes=borg_exit_codes,
+    output_lines = tuple(
+        execute_command_and_capture_output(
+            main_command,
+            output_log_level=logging.ANSWER,
+            environment=environment.make_environment(config),
+            working_directory=working_directory,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
    )

+    flags.warn_for_aggressive_archive_flags(main_command, output_lines)
+
    return None
@@ -1 +1 @@
-DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'
+DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'  # pragma: nocover
@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 import borgmatic.logger
@@ -24,6 +25,7 @@ def transfer_archives(
    instance, transfer archives to the given repository.
    '''
    borgmatic.logger.add_custom_log_levels()
+    extra_borg_options = config.get('extra_borg_options', {}).get('transfer', '')

    full_command = (
        (local_path, 'transfer')
@@ -31,7 +33,7 @@ def transfer_archives(
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
        + flags.make_flags('remote-path', remote_path)
        + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
        + flags.make_flags('lock-wait', config.get('lock_wait'))
        + flags.make_flags('progress', config.get('progress'))
        + (
@@ -53,6 +55,7 @@ def transfer_archives(
                )
            )
        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + flags.make_repository_flags(repository_path, local_borg_version)
        + flags.make_flags('other-repo', transfer_arguments.source_repository)
        + flags.make_flags('dry-run', dry_run)
@@ -1,4 +1,5 @@
 import logging
+import shlex

 import borgmatic.config.paths
 from borgmatic.execute import execute_command
@@ -11,10 +12,13 @@ def unmount_archive(config, mount_point, local_path='borg'):
    Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
    from the mount point.
    '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('umount', '')
    full_command = (
        (local_path, 'umount')
+        + ('--log-json',)
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
        + (mount_point,)
    )

@@ -16,15 +16,19 @@ def local_borg_version(config, local_path='borg'):
    '''
    full_command = (
        (local_path, '--version')
+        + ('--log-json',)
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
    )
-    output = execute_command_and_capture_output(
-        full_command,
-        environment=environment.make_environment(config),
-        working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_local_path=local_path,
-        borg_exit_codes=config.get('borg_exit_codes'),
+
+    output = '\n'.join(
+        execute_command_and_capture_output(
+            full_command,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
    )

    try:
@@ -31,8 +31,10 @@ ACTION_ALIASES = {
    'transfer': [],
    'break-lock': [],
    'key': [],
-    'borg': [],
    'recreate': [],
+    'diff': [],
+    'browse': [],
+    'borg': [],
 }


@@ -297,7 +299,14 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
    )


-OMITTED_FLAG_NAMES = {'match-archives', 'progress', 'statistics', 'list-details'}
+OMITTED_FLAG_NAMES = {
+    'match-archives',
+    'progress',
+    'statistics',
+    'list-details',
+    'file-list-format',
+    'archive-list-format',
+}


 def make_argument_description(schema, flag_name):
@@ -316,7 +325,7 @@ def make_argument_description(schema, flag_name):
            ' To specify a different list element, replace the "[0]" with another array index ("[1]", "[2]", etc.).',
        )

-    if example and schema_type in ('array', 'object'):  # noqa: PLR6201
+    if example and schema_type in ('array', 'object'):
        example_buffer = io.StringIO()
        yaml = ruamel.yaml.YAML(typ='safe')
        yaml.default_flow_style = True
@@ -568,8 +577,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
    ignoring actions, and the combined parser is handy for displaying help that includes everything:
    global flags, a list of actions, etc.
    '''
-    config_paths = collect.get_default_config_paths(expand_home=True)
-    unexpanded_config_paths = collect.get_default_config_paths(expand_home=False)
+    config_paths = collect.get_default_config_paths()

    # Using allow_abbrev=False here prevents the global parser from erroring about "ambiguous"
    # options like --encryption. Such options are intended for an action parser rather than the
@@ -582,7 +590,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        '--config',
        dest='config_paths',
        action='append',
-        help=f"Configuration filename or directory, can specify flag multiple times, defaults to: -c {' -c '.join(unexpanded_config_paths)}",
+        help='Configuration filename or directory, can specify flag multiple times, defaults to /etc/borgmatic/config.yaml, /etc/borgmatic.d, $XDG_CONFIG_HOME/borgmatic/config.yaml, and $XDG_CONFIG_HOME/borgmatic.d, where $XDG_CONFIG_HOME defaults to $HOME/.config',
    )
    global_group.add_argument(
        '-n',
@@ -641,8 +649,8 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
    repo_create_parser = action_parsers.add_parser(
        'repo-create',
        aliases=ACTION_ALIASES['repo-create'],
-        help='Create a new, empty Borg repository',
-        description='Create a new, empty Borg repository',
+        help='Create a new, empty Borg repository (also known as "init")',
+        description='Create a new, empty Borg repository (also known as "init")',
        add_help=False,
    )
    repo_create_group = repo_create_parser.add_argument_group('repo-create arguments')
@@ -797,6 +805,13 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        action='store_true',
        help='Display statistics of the pruned archive [Borg 1 only]',
    )
+    prune_group.add_argument(
+        '--quick-stats',
+        dest='quick_statistics',
+        default=None,
+        action='store_true',
+        help='Display statistics of the pruned archive, skipping repository-wide "All archives" and chunk index statistics [Borg >= 1.4.5 and < 2 only]',
+    )
    prune_group.add_argument(
        '--list',
        dest='list_details',
@@ -889,6 +904,13 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        action='store_true',
        help='Display statistics of archive',
    )
+    create_group.add_argument(
+        '--quick-stats',
+        dest='quick_statistics',
+        default=None,
+        action='store_true',
+        help='Display statistics of archive, skipping repository-wide "All archives" and chunk index statistics [Borg 1.4.5+ only]',
+    )
    create_group.add_argument(
        '--list',
        '--files',
@@ -1213,9 +1235,9 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
    config_generate_group.add_argument(
        '-d',
        '--destination',
-        dest='destination_filename',
+        dest='destination_path',
        default=config_paths[0],
-        help=f'Destination configuration file, default: {unexpanded_config_paths[0]}',
+        help='Destination configuration file (or directory if using --split), default: /etc/borgmatic/config.yaml',
    )
    config_generate_group.add_argument(
        '--overwrite',
@@ -1223,6 +1245,11 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        action='store_true',
        help='Whether to overwrite any existing destination file, defaults to false',
    )
+    config_generate_group.add_argument(
+        '--split',
+        action='store_true',
+        help='Assuming the destination is a directory instead of a file, split the configuration into separate files within it, one per option, useful for documentation',
+    )
    config_generate_group.add_argument(
        '-h',
        '--help',
@@ -1250,6 +1277,29 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        help='Show this help message and exit',
    )

+    config_show_parser = config_parsers.add_parser(
+        'show',
+        help='Show the computed configuration for each file specified with --config (see borgmatic --help)',
+        description='Show the computed configuration for each file specified with --config (see borgmatic --help)',
+        add_help=False,
+    )
+    config_show_group = config_show_parser.add_argument_group('config show arguments')
+    config_show_group.add_argument(
+        '--option',
+        help='Show the value of a single named configuration option instead of the entire configuration',
+    )
+    config_show_group.add_argument(
+        '--json',
+        action='store_true',
+        help='Show the configuration as JSON with one array element per configuration file',
+    )
+    config_show_group.add_argument(
+        '-h',
+        '--help',
+        action='help',
+        help='Show this help message and exit',
+    )
+
    export_tar_parser = action_parsers.add_parser(
        'export-tar',
        aliases=ACTION_ALIASES['export-tar'],
@@ -1537,7 +1587,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        action='store_true',
        help='Output only archive names',
    )
-    repo_list_group.add_argument('--format', help='Format for archive listing')
+    repo_list_group.add_argument('--format', help='Borg format for the archive listing')
    repo_list_group.add_argument(
        '--json',
        default=False,
@@ -1640,7 +1690,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        action='store_true',
        help='Output only path names',
    )
-    list_group.add_argument('--format', help='Format for file listing')
+    list_group.add_argument('--format', help='Borg format for the file listing')
    list_group.add_argument(
        '--json',
        default=False,
@@ -1920,7 +1970,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
    )
    recreate_group.add_argument(
        '--archive',
-        help='Archive name, hash, or series to recreate',
+        help='Archive name, hash, or series to recreate, defaults to all archives in the repository (if specified), or all archives across all repositories',
    )
    recreate_group.add_argument(
        '--list',
@@ -1959,6 +2009,60 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
        help='Show this help message and exit',
    )

+    diff_parser = action_parsers.add_parser(
+        'diff',
+        aliases=ACTION_ALIASES['diff'],
+        help='Find differences (file contents, user/group/mode) between archives',
+        description='Find differences (file contents, user/group/mode) between archives',
+        add_help=False,
+    )
+    diff_group = diff_parser.add_argument_group('diff arguments')
+    diff_group.add_argument(
+        '--repository',
+        help='Path of repository containing archive to diff, defaults to the configured repository if there is only one, quoted globs supported',
+    )
+    diff_group.add_argument(
+        '--archive',
+        help='Archive name, hash, or series to diff',
+        required=True,
+    )
+    diff_group.add_argument(
+        '--second-archive',
+        help='Second archive name, hash, or series to diff',
+        required=True,
+    )
+    diff_group.add_argument(
+        '--same-chunker-params', action='store_true', help='Override check of chunker parameters'
+    )
+    diff_group.add_argument(
+        '--sort-by',
+        metavar='KEY',
+        dest='sort_keys',
+        action='append',
+        help='Advanced sorting: specify field(s) to sort by. Prefix with > for descending or < for ascending (default)',
+    )
+    diff_group.add_argument(
+        '--content-only',
+        action='store_true',
+        help='Only compare differences in content (exclude metadata differences)',
+    )
+    diff_group.add_argument(
+        '--only-patterns',
+        action='store_true',
+        help='Run the diff according to borgmatic configured patterns (ie do not diff entire archives)',
+    )
+    diff_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
+
+    browse_parser = action_parsers.add_parser(
+        'browse',
+        aliases=ACTION_ALIASES['browse'],
+        help='Browse repositories, archives, and files in a console UI',
+        description='Browse repositories, archives, and files in a console UI',
+        add_help=False,
+    )
+    browse_group = browse_parser.add_argument_group('browse arguments')
+    browse_group.add_argument('-h', '--help', action='help', help='Show this help message and exit')
+
    borg_parser = action_parsers.add_parser(
        'borg',
        aliases=ACTION_ALIASES['borg'],
@@ -2010,7 +2114,7 @@ def parse_arguments(schema, *unparsed_arguments):
    )

    if not arguments['global'].config_paths:
-        arguments['global'].config_paths = collect.get_default_config_paths(expand_home=True)
+        arguments['global'].config_paths = collect.get_default_config_paths()

    for action_name in ('bootstrap', 'generate', 'validate'):
        if action_name in arguments and len(arguments) > HIGHLANDER_ACTION_ARGUMENTS_COUNT:
@@ -2031,7 +2135,7 @@ def parse_arguments(schema, *unparsed_arguments):
        )

    if (
-        ('list' in arguments and 'repo-info' in arguments and arguments['list'].json)  # noqa: PLR0916
+        ('list' in arguments and 'repo-info' in arguments and arguments['list'].json)
        or ('list' in arguments and 'info' in arguments and arguments['list'].json)
        or ('repo-info' in arguments and 'info' in arguments and arguments['repo-info'].json)
    ):
@@ -2049,7 +2153,7 @@ def parse_arguments(schema, *unparsed_arguments):
            'With the repo-list action, only one of --prefix or --match-archives flags can be used.',
        )

-    if 'info' in arguments and (  # noqa: PLR0916
+    if 'info' in arguments and (
        (arguments['info'].archive and arguments['info'].prefix)
        or (arguments['info'].archive and arguments['info'].match_archives)
        or (arguments['info'].prefix and arguments['info'].match_archives)
@@ -12,14 +12,17 @@ import ruamel.yaml

 import borgmatic.actions.borg
 import borgmatic.actions.break_lock
+import borgmatic.actions.browse.run
 import borgmatic.actions.change_passphrase
 import borgmatic.actions.check
 import borgmatic.actions.compact
 import borgmatic.actions.config.bootstrap
 import borgmatic.actions.config.generate
+import borgmatic.actions.config.show
 import borgmatic.actions.config.validate
 import borgmatic.actions.create
 import borgmatic.actions.delete
+import borgmatic.actions.diff
 import borgmatic.actions.export_key
 import borgmatic.actions.export_tar
 import borgmatic.actions.extract
@@ -209,7 +212,7 @@ def run_configuration(config_filename, config, config_paths, arguments):  # noqa
            f"Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions",
        )

-    try:  # noqa: PLR1702
+    try:
        with (
            Monitoring_hooks(config_filename, config, arguments, global_arguments),
            borgmatic.hooks.command.Before_after_hooks(
@@ -340,7 +343,7 @@ def run_actions(  # noqa: PLR0912, PLR0915
    '''
    Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
    filename, a configuration dict, a sequence of loaded configuration paths, local and remote paths
-    to Borg, a local Borg version string, and a repository name, run all actions from the
+    to Borg, a local Borg version string, and a repository dict, run all actions from the
    command-line arguments on the given repository.

    Yield JSON output strings from executing any actions that produce JSON.
@@ -362,6 +365,22 @@ def run_actions(  # noqa: PLR0912, PLR0915
        'repository': repository_path,
    }
    skip_actions = set(get_skip_actions(config, arguments))
+    requested_repository = next(
+        (
+            repository
+            for action_arguments in arguments.values()
+            for repository in (getattr(action_arguments, 'repository', None),)
+            if repository is not None
+        ),
+        None,
+    )
+
+    if requested_repository and not borgmatic.config.validate.repositories_match(
+        repository,
+        requested_repository,
+    ):
+        logger.debug('Skipping actions because the requested --repository does not match')
+        return

    with borgmatic.hooks.command.Before_after_hooks(
        command_hooks=config.get('commands'),
@@ -425,6 +444,7 @@ def run_actions(  # noqa: PLR0912, PLR0915
                        local_borg_version,
                        action_arguments,
                        global_arguments,
+                        dry_run_label,
                        local_path,
                        remote_path,
                    )
@@ -605,6 +625,16 @@ def run_actions(  # noqa: PLR0912, PLR0915
                        local_path,
                        remote_path,
                    )
+                elif action_name == 'diff':
+                    borgmatic.actions.diff.run_diff(
+                        repository,
+                        config,
+                        local_borg_version,
+                        action_arguments,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
                elif action_name == 'borg':
                    borgmatic.actions.borg.run_borg(
                        repository,
@@ -644,6 +674,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                        levelno=logging.DEBUG,
                        levelname='DEBUG',
                        msg=f'{config_filename}: Loading configuration file',
+                        name=logger.name,
                    ),
                ),
            ],
@@ -666,6 +697,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                            levelno=logging.CRITICAL,
                            levelname='CRITICAL',
                            msg=f'{config_filename}: Insufficient permissions to read configuration file',
+                            name=logger.name,
                        ),
                    ),
                ],
@@ -678,10 +710,16 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                            levelno=logging.CRITICAL,
                            levelname='CRITICAL',
                            msg=f'{config_filename}: Error parsing configuration file',
+                            name=logger.name,
                        ),
                    ),
                    logging.makeLogRecord(
-                        dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=str(error)),
+                        dict(
+                            levelno=logging.CRITICAL,
+                            levelname='CRITICAL',
+                            msg=str(error),
+                            name=logger.name,
+                        ),
                    ),
                ],
            )
@@ -694,7 +732,7 @@ def log_record(suppress_log=False, **kwargs):
    Create a log record based on the given makeLogRecord() arguments, one of which must be
    named "levelno". Log the record (unless suppress log is set) and return it.
    '''
-    record = logging.makeLogRecord(kwargs)
+    record = logging.makeLogRecord(dict(kwargs, name=logger.name))
    if suppress_log:
        return record

@@ -785,18 +823,18 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
    '''
    add_custom_log_levels()

-    if 'bootstrap' in arguments:
-        try:
-            # No configuration file is needed for bootstrap.
-            local_borg_version = borg_version.local_borg_version(
-                {},
-                arguments['bootstrap'].local_path,
-            )
-        except (OSError, CalledProcessError, ValueError) as error:
-            yield from log_error_records('Error getting local Borg version', error)
-            return
+    try:
+        if 'bootstrap' in arguments:
+            try:
+                local_borg_version = borg_version.local_borg_version(
+                    # No configuration file is needed for bootstrap.
+                    {},
+                    arguments['bootstrap'].local_path,
+                )
+            except (OSError, CalledProcessError, ValueError) as error:
+                yield from log_error_records('Error getting local Borg version', error)
+                return

-        try:
            borgmatic.actions.config.bootstrap.run_bootstrap(
                arguments['bootstrap'],
                arguments['global'],
@@ -807,19 +845,13 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='Bootstrap successful',
+                    name=logger.name,
                ),
            )
-        except (
-            CalledProcessError,
-            ValueError,
-            OSError,
-        ) as error:
-            yield from log_error_records(error)

-        return
+            return

-    if 'generate' in arguments:
-        try:
+        if 'generate' in arguments:
            borgmatic.actions.config.generate.run_generate(
                arguments['generate'],
                arguments['global'],
@@ -829,30 +861,25 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='Generate successful',
-                ),
-            )
-        except (
-            CalledProcessError,
-            ValueError,
-            OSError,
-        ) as error:
-            yield from log_error_records(error)
-
-        return
-
-    if 'validate' in arguments:
-        if configuration_parse_errors:
-            yield logging.makeLogRecord(
-                dict(
-                    levelno=logging.CRITICAL,
-                    levelname='CRITICAL',
-                    msg='Configuration validation failed',
+                    name=logger.name,
                ),
            )

            return

-        try:
+        if 'validate' in arguments:
+            if configuration_parse_errors:
+                yield logging.makeLogRecord(
+                    dict(
+                        levelno=logging.CRITICAL,
+                        levelname='CRITICAL',
+                        msg='Configuration validation failed',
+                        name=logger.name,
+                    ),
+                )
+
+                return
+
            borgmatic.actions.config.validate.run_validate(arguments['validate'], configs)

            yield logging.makeLogRecord(
@@ -860,16 +887,30 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                    levelno=logging.ANSWER,
                    levelname='ANSWER',
                    msg='All configuration files are valid',
+                    name=logger.name,
                ),
            )
-        except (
-            CalledProcessError,
-            ValueError,
-            OSError,
-        ) as error:
-            yield from log_error_records(error)

-        return
+            return
+
+        if 'show' in arguments:
+            borgmatic.actions.config.show.run_show(arguments['show'], configs)
+
+            return
+
+        if 'browse' in arguments:
+            borgmatic.actions.browse.run.run_browse(
+                arguments['browse'],
+                arguments['global'],
+                configs,
+            )
+
+    except (
+        CalledProcessError,
+        ValueError,
+        OSError,
+    ) as error:
+        yield from log_error_records(error)


 def collect_configuration_run_summary_logs(configs, config_paths, arguments, log_file_path):  # noqa: PLR0912
@@ -952,6 +993,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                        levelno=logging.INFO,
                        levelname='INFO',
                        msg=f'{config_filename}: Successfully ran configuration file',
+                        name=logger.name,
                    ),
                )
                if results:
@@ -1010,16 +1052,17 @@ def exit_with_help_link():  # pragma: no cover
    sys.exit(1)


-def check_and_show_help_on_no_args(configs):
+def check_and_show_help_on_no_args(configs, schema):
    '''
-    Given a dict of configuration filename to corresponding parsed configuration, check if the
-    borgmatic command is run without any arguments. If the configuration option "default_actions" is
-    set to False, show the help message. Otherwise, trigger the default backup behavior.
+    Given a dict of configuration filename to corresponding parsed configuration and the
+    configuration schema as a dict, check if the borgmatic command was run without any arguments. If
+    the configuration option "default_actions" is set to False, then show the help message an exit.
    '''
-    if len(sys.argv) == 1:  # No arguments provided
+    if len(sys.argv) == 1:  # No arguments provided.
        default_actions = any(config.get('default_actions', True) for config in configs.values())
-        if not default_actions:
-            parse_arguments('--help')
+
+        if configs and not default_actions:
+            parse_arguments(schema, '--help')
            sys.exit(0)


@@ -1051,6 +1094,23 @@ def get_singular_option_value(configs, option_name):
        return None


+def display_summary(summary_logs, log_json):  # pragma: no cover
+    summary_logs_max_level = max(log.levelno for log in summary_logs)
+
+    for message in ('summary:',) if log_json else ('', 'summary:'):
+        log_record(
+            levelno=summary_logs_max_level,
+            levelname=logging.getLevelName(summary_logs_max_level),
+            msg=message,
+        )
+
+    for log in summary_logs:
+        logger.handle(log)
+
+    if summary_logs_max_level >= logging.CRITICAL:
+        exit_with_help_link()
+
+
 def main(extra_summary_logs=()):  # pragma: no cover
    configure_signals()
    configure_delayed_logging()
@@ -1071,7 +1131,7 @@ def main(extra_summary_logs=()):  # pragma: no cover
        exit_with_help_link()
    except SystemExit as error:
        if error.code == 0:
-            raise error
+            raise

        configure_logging(logging.CRITICAL)
        logger.critical(f"Error parsing arguments: {' '.join(sys.argv)}")
@@ -1099,8 +1159,7 @@ def main(extra_summary_logs=()):  # pragma: no cover
        resolve_env=global_arguments.resolve_env and not arguments.get('validate'),
    )

-    # Use the helper function to check and show help on no arguments, passing the preloaded configs
-    check_and_show_help_on_no_args(configs)
+    check_and_show_help_on_no_args(configs, schema)

    configuration_parse_errors = (
        (max(log.levelno for log in parse_logs) >= logging.CRITICAL) if parse_logs else False
@@ -1110,6 +1169,7 @@ def main(extra_summary_logs=()):  # pragma: no cover
        getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
    )
    log_file_path = get_singular_option_value(configs, 'log_file')
+    log_json = get_singular_option_value(configs, 'log_json')

    try:
        configure_logging(
@@ -1119,7 +1179,8 @@ def main(extra_summary_logs=()):  # pragma: no cover
            verbosity_to_log_level(get_verbosity(configs, 'monitoring_verbosity')),
            log_file_path,
            get_singular_option_value(configs, 'log_file_format'),
-            color_enabled=should_do_markup(configs, any_json_flags),
+            log_json,
+            color_enabled=should_do_markup(configs, any_json_flags or log_json),
        )
    except (FileNotFoundError, PermissionError) as error:
        configure_logging(logging.CRITICAL)
@@ -1147,17 +1208,9 @@ def main(extra_summary_logs=()):  # pragma: no cover
            )
        )
    )
-    summary_logs_max_level = max(log.levelno for log in summary_logs)

-    for message in ('', 'summary:'):
-        log_record(
-            levelno=summary_logs_max_level,
-            levelname=logging.getLevelName(summary_logs_max_level),
-            msg=message,
-        )
+    display_summary(summary_logs, log_json)

-    for log in summary_logs:
-        logger.handle(log)

-    if summary_logs_max_level >= logging.CRITICAL:
-        exit_with_help_link()
+if __name__ == '__main__':  # pragma: no cover
+    main()
@@ -39,7 +39,7 @@ def bash_completion():
            'check_version() {',
            '    local this_script="$(cat "$BASH_SOURCE" 2> /dev/null)"',
            '    local installed_script="$(borgmatic --bash-completion 2> /dev/null)"',
-            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];'
+            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];',
            f'''        then cat << EOF\n{borgmatic.commands.completion.actions.upgrade_message(
                    'bash',
                    'sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"',
@@ -1,17 +1,14 @@
 import os


-def get_default_config_paths(expand_home=True):
+def get_default_config_paths():
    '''
-    Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
-    default configuration paths. This includes both system-wide configuration and configuration in
-    the current user's home directory.
-
-    Don't expand the home directory ($HOME) if the expand home flag is False.
+    Return a list of default configuration paths. This includes both system-wide
+    configuration and configuration in the current user's home directory.
    '''
-    user_config_directory = os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
-    if expand_home:
-        user_config_directory = os.path.expandvars(user_config_directory)
+    user_config_directory = os.path.expandvars(
+        os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
+    )

    return [
        '/etc/borgmatic/config.yaml',
@@ -50,5 +47,6 @@ def collect_config_filenames(config_paths):
        for filename in sorted(os.listdir(path)):
            full_filename = os.path.join(path, filename)
            matching_filetype = full_filename.endswith(('.yaml', '.yml'))
+
            if matching_filetype and not os.path.isdir(full_filename):
                yield os.path.abspath(full_filename)
@@ -1,4 +1,6 @@
 import contextlib
+import functools
+import re
 import shlex


@@ -21,7 +23,44 @@ def coerce_scalar(value):
        return value


-def apply_constants(value, constants, shell_escape=False):
+CONSTANT_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_constant(match, constants, command_hook):
+    '''
+    Given a re.Match instance of CONSTANT_PATTERN representing a matched constant name to be
+    interpolated, a constants dict, and whether this is for a command hook, lookup the matched
+    constant name within the given constants and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given constants, then return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name—unless this is for a command hook, in which case just return the literal
+    # string. That way, subsequent variable interpolation will still see the string as escaped
+    # instead of trying to interpolate it.
+    if match.group('left_escape') and match.group('right_escape'):
+        if command_hook:
+            return match.group(0)
+
+        return '{' + name + '}'
+
+    value = constants.get(name)
+
+    # The matched variable is in the constants, so return its value. And if this is for a command
+    # hook, then shell escape the value so as to prevent shell injection attacks.
+    if value is not None:
+        return shlex.quote(str(value)) if command_hook else str(value)
+
+    # The matched variable name isn't in the constants. Return the whole string unaltered.
+    return match.group(0)
+
+
+def apply_constants(value, constants, command_hook=False):
    '''
    Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
    replace any configuration string values of the form "{constant}" (or containing it) with the
@@ -39,26 +78,25 @@ def apply_constants(value, constants, shell_escape=False):
        return value

    if isinstance(value, str):
-        for constant_name, constant_value in constants.items():
-            value = value.replace(
-                '{' + constant_name + '}',
-                shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
-            )
-
        # Support constants within non-string scalars by coercing the value to its appropriate type.
-        value = coerce_scalar(value)
+        value = coerce_scalar(
+            CONSTANT_PATTERN.sub(
+                functools.partial(resolve_constant, constants=constants, command_hook=command_hook),
+                value,
+            )
+        )
    elif isinstance(value, list):
        for index, list_value in enumerate(value):
-            value[index] = apply_constants(list_value, constants, shell_escape)
+            value[index] = apply_constants(list_value, constants, command_hook)
    elif isinstance(value, dict):
        for option_name, option_value in value.items():
            value[option_name] = apply_constants(
                option_value,
                constants,
-                shell_escape=(
-                    shell_escape
+                command_hook=(
+                    command_hook
                    or option_name.startswith(('before_', 'after_'))
-                    or option_name == 'on_error'
+                    or option_name in {'on_error', 'run'}
                ),
            )

@@ -1,6 +1,7 @@
 import collections
 import contextlib
 import io
+import json
 import os
 import re

@@ -107,7 +108,7 @@ def comment_out_line(line):
    return '# '.join((indent_spaces, line[count_indent_spaces:]))


-def comment_out_optional_configuration(rendered_config):
+def transform_optional_configuration(rendered_config, comment_out=True):
    '''
    Post-process a rendered configuration string to comment out optional key/values, as determined
    by a sentinel in the comment before each key.
@@ -117,6 +118,9 @@ def comment_out_optional_configuration(rendered_config):

    Ideally ruamel.yaml would support commenting out keys during configuration generation, but it's
    not terribly easy to accomplish that way.
+
+    If comment_out is False, then just strip the comment sentinel without actually commenting
+    anything out.
    '''
    lines = []
    optional = False
@@ -129,6 +133,9 @@ def comment_out_optional_configuration(rendered_config):
        # Upon encountering an optional configuration option, comment out lines until the next blank
        # line.
        if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
+            if comment_out is False:
+                continue
+
            optional = True
            indent_characters_at_sentinel = indent_characters
            continue
@@ -146,6 +153,9 @@ def comment_out_optional_configuration(rendered_config):
    return '\n'.join(lines)


+RUAMEL_YAML_END_OF_DOCUMENT_MARKER = '...\n'
+
+
 def render_configuration(config):
    '''
    Given a config data structure of nested OrderedDicts, render the config as YAML and return it.
@@ -153,7 +163,13 @@ def render_configuration(config):
    dumper = ruamel.yaml.YAML(typ='rt')
    dumper.indent(mapping=INDENT, sequence=INDENT + SEQUENCE_INDENT, offset=INDENT)
    rendered = io.StringIO()
-    dumper.dump(config, rendered)
+    dumper.dump(
+        config,
+        rendered,
+        # Dumping certain values (integers, for instance) causes ruamel.yaml to append an
+        # end-of-document "..." marker. Strip it.
+        transform=lambda dumped: dumped.removesuffix(RUAMEL_YAML_END_OF_DOCUMENT_MARKER),
+    )

    return rendered.getvalue()

@@ -310,19 +326,39 @@ def merge_source_configuration_into_destination(destination_config, source_confi
    return destination_config


+def get_configuration_subset(config, option_name):  # pragma: no cover
+    '''
+    Given configuration as a ruamel.yaml.CommentedMap and an option name found within it at the top
+    level, return a new CommentedMap containing a subset of the configuration with only the given
+    option and no other top-level options.
+
+    This is useful when generating the sample configuration for a single option instead of a whole
+    configuration file.
+    '''
+    option_config = ruamel.yaml.CommentedMap({option_name: config[option_name]})
+
+    # Due to a quirk of ruamel.yaml, the comment right before a top-level key is not on that key and
+    # needs to get copied separately.
+    option_config.ca.items[option_name] = config.ca.items[option_name]
+
+    return option_config
+
+
 def generate_sample_configuration(
    dry_run,
    source_filename,
-    destination_filename,
+    destination_path,
    schema_filename,
    overwrite=False,
+    split=False,
 ):
    '''
-    Given an optional source configuration filename, and a required destination configuration
-    filename, the path to a schema filename in a YAML rendition of the JSON Schema format, and
-    whether to overwrite a destination file, write out a sample configuration file based on that
-    schema. If a source filename is provided, merge the parsed contents of that configuration into
-    the generated configuration.
+    Given an optional source configuration filename, a required destination configuration path, the
+    path to a schema filename in a YAML rendition of the JSON Schema format, whether to overwrite a
+    destination file, and whether to split the configuration into multiple files (one per option) in
+    the assumed destination directory, write out sample configuration file(s) based on that schema.
+    If a source filename is provided, merge the parsed contents of that configuration into the
+    generated configuration.
    '''
    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename, encoding='utf-8'))
    source_config = None
@@ -345,8 +381,37 @@ def generate_sample_configuration(
    if dry_run:
        return

+    if split:
+        if os.path.exists(destination_path) and not os.path.isdir(destination_path):
+            raise ValueError('With the --split flag, the destination path must be a directory')
+
+        os.makedirs(destination_path, exist_ok=True)
+
+        for option_name in destination_config:
+            write_configuration(
+                os.path.join(destination_path, f'{option_name}.yaml'),
+                transform_optional_configuration(
+                    render_configuration(get_configuration_subset(destination_config, option_name)),
+                    comment_out=False,
+                ).strip(),
+                overwrite=overwrite,
+            )
+
+        # Also dump a manifest listing all the options we've written.
+        json.dump(
+            {'option_names': list(destination_config.keys())},
+            open(os.path.join(destination_path, 'options.json'), 'w', encoding='utf-8'),
+        )
+
+        return
+
+    if os.path.exists(destination_path) and not os.path.isfile(destination_path):
+        raise ValueError('Without the --split flag, the destination path must be a file')
+
    write_configuration(
-        destination_filename,
-        comment_out_optional_configuration(render_configuration(destination_config)),
+        destination_path,
+        transform_optional_configuration(
+            render_configuration(destination_config), comment_out=True
+        ),
        overwrite=overwrite,
    )
@@ -2,10 +2,19 @@ import contextlib
 import logging
 import os
 import tempfile
+from enum import Enum

 logger = logging.getLogger(__name__)


+class Systemd_directories(Enum):
+    RUNTIME_DIRECTORY = 0
+    STATE_DIRECTORY = 1
+    CACHE_DIRECTORY = 2
+    LOGS_DIRECTORY = 3
+    CONFIGURATION_DIRECTORY = 4
+
+
 def expand_user_in_path(path):
    '''
    Given a directory path, expand any tildes in it.
@@ -16,6 +25,17 @@ def expand_user_in_path(path):
        return None


+def resolve_systemd_directory(directory):
+    '''
+    Given a systemd directory environment variable enum, read the value if set and return the first
+    configured directory.
+    '''
+    separator = ':'
+
+    paths = os.environ.get(directory.name)
+    return paths.split(separator)[0] if paths else None
+
+
 def get_working_directory(config):  # pragma: no cover
    '''
    Given a configuration dict, get the working directory from it, expanding any tildes.
@@ -96,7 +116,9 @@ class Runtime_directory:
        runtime_directory = (
            config.get('user_runtime_directory')
            or os.environ.get('XDG_RUNTIME_DIR')  # Set by PAM on Linux.
-            or os.environ.get('RUNTIME_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
+                Systemd_directories.RUNTIME_DIRECTORY
+            )  # Set by systemd if configured.
        )

        if runtime_directory:
@@ -174,7 +196,9 @@ def get_borgmatic_state_directory(config):
        os.path.join(
            config.get('user_state_directory')
            or os.environ.get('XDG_STATE_HOME')
-            or os.environ.get('STATE_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
+                Systemd_directories.STATE_DIRECTORY
+            )  # Set by systemd if configured.
            or '~/.local/state',
            'borgmatic',
        ),
@@ -27,6 +27,13 @@ properties:
            - /etc
            - /var/log/syslog*
            - /home/user/path with spaces
+    source_directories_must_exist:
+        type: boolean
+        description: |
+            When true, source directories (and root pattern paths) must exist
+            before a backup begins. If they don't, borgmatic errors. Defaults to
+            true.
+        example: false
    repositories:
        type: array
        items:
@@ -73,7 +80,7 @@ properties:
                        Whether any missing parent directories of the repository
                        path should be created, only used for the repo-create
                        action. Defaults to false. (This option is supported 
-                        for Borg 1.x only)
+                        for Borg 1.x only.)
                    example: true
        description: |
            A required list of local or remote repositories with paths and
@@ -134,12 +141,39 @@ properties:
            false. But when a database hook is used, the setting here is ignored
            and read_special is considered true.
        example: true
+    unsafe_skip_path_validation_before_create:
+        type: boolean
+        description: |
+            Skip pre-backup safety validation: (1) runtime directory exclusion
+            detection, and (2) special file exclusion for database streaming.
+            Can significantly improve performance for large filesystems.
+
+            UNSAFE because, when this option is enabled, Borg may hang on
+            special files (named pipes, devices) and data from borgmatic data
+            source hooks may be silently excluded if your excluded patterns
+            match the runtime directory (see the "user_runtime_directory"
+            option).
+
+            Only enable if not using data source hooks and you manually verify
+            that your excludes don't affect the runtime directory. Defaults
+            to false.
+        example: true
    flags:
        type: boolean
        description: |
            Record filesystem flags (e.g. NODUMP, IMMUTABLE) in archive.
            Defaults to true.
        example: false
+    files_changed:
+        type: string
+        enum: ['ctime', 'mtime', 'disabled']
+        description: |
+            Threshold for considering a file as changed. See
+            https://borgbackup.readthedocs.io/en/stable/usage/create.html for
+            details. Defaults to "ctime". E.g., a file is considered changed if
+            its ctime has changed since the last backup. (This option is
+            supported for Borg 1.4.2+ only.)
+        example: ctime
    files_cache:
        type: string
        description: |
@@ -163,9 +197,13 @@ properties:
            type: string
        description: |
            Any paths matching these patterns are included/excluded from
-            backups. Globs are expanded. (Tildes are not.) See the output of
-            "borg help patterns" for more details. Quote any value if it
-            contains leading punctuation, so it parses correctly.
+            backups. Recursion root patterns ("R ...") are effectively the same
+            as "source_directories"; they tell Borg which paths to backup
+            (modulo any excludes). Globs are expanded. (Tildes are not.) See
+            the output of "borg help patterns" for more details. Quote any value
+            if it contains leading punctuation, so it parses correctly. Also use
+            leading slashes in absolute paths, or data source hooks may be
+            unable to rewrite patterns as needed.
        example:
            - 'R /'
            - '- /home/*/.cache'
@@ -232,7 +270,8 @@ properties:
    exclude_nodump:
        type: boolean
        description: |
-            Exclude files with the NODUMP flag. Defaults to false.
+            Exclude files with the NODUMP flag. Defaults to false. (This option
+            is supported for Borg 1.x only.)
        example: true
    borgmatic_source_directory:
        type: string
@@ -248,7 +287,7 @@ properties:
            Path for storing temporary runtime data like streaming database
            dumps and bootstrap metadata. borgmatic automatically creates and
            uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR
-            or or $TMPDIR or $TEMP or /run/user/$UID.
+            or $TMPDIR or $TEMP or /run/user/$UID.
        example: /run/user/1001
    user_state_directory:
        type: string
@@ -259,12 +298,6 @@ properties:
            create the check records again (and therefore re-run checks).
            Defaults to $XDG_STATE_HOME or ~/.local/state.
        example: /var/lib/borgmatic
-    source_directories_must_exist:
-        type: boolean
-        description: |
-            If true, then source directories (and root pattern paths) must
-            exist. If they don't, an error is raised. Defaults to false.
-        example: true
    encryption_passcommand:
        type: string
        description: |
@@ -275,7 +308,7 @@ properties:
            encryption_passphrase takes precedence. This can also be used to
            access encrypted systemd service credentials. Defaults to not set.
            For more details, see:
-            https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+            https://torsion.org/borgmatic/how-to/provide-your-passwords/
        example: "secret-tool lookup borg-repository repo-name"
    encryption_passphrase:
        type: string
@@ -406,14 +439,20 @@ properties:
        type: string
        description: |
            Path for Borg security and encryption nonce files. Defaults to
-            $borg_base_directory/.config/borg/security
+            $borg_config_directory/security
        example: /path/to/base/config/security
    borg_keys_directory:
        type: string
        description: |
            Path for Borg encryption key files. Defaults to
-            $borg_base_directory/.config/borg/keys
+            $borg_config_directory/keys
        example: /path/to/base/config/keys
+    borg_key_file:
+        type: string
+        description: |
+            Path for the Borg repository key file, for use with a repository
+            created with "keyfile" encryption.
+        example: /path/to/base/config/keyfile
    borg_exit_codes:
        type: array
        items:
@@ -437,9 +476,9 @@ properties:
        description: |
            A list of Borg exit codes that should be elevated to errors or
            squashed to warnings as indicated. By default, Borg error exit codes
-            (2 to 99) are treated as errors while warning exit codes (1 and
-            100+) are treated as warnings. Exit codes other than 1 and 2 are
-            only present in Borg 1.4.0+.
+            (2 to 99) are treated as errors and most warning exit codes (1 and
+            100+) are treated as errors as well. Exit codes other than 1 and 2
+            are only present in Borg 1.4.0+.
        example:
            - code: 13
              treat_as: warning
@@ -450,6 +489,9 @@ properties:
        description: |
            Umask used for when executing Borg or calling hooks. Defaults to
            0077 for Borg or the umask that borgmatic is run with for hooks.
+            Even though this value is a YAML integer, borgmatic interprets it as
+            octal. YAML's "0o"-prefixed octal notation is not currently
+            supported.
        example: 0077
    lock_wait:
        type: integer
@@ -479,6 +521,42 @@ properties:
            If match_archives is not specified, borgmatic defaults to deriving
            the match_archives value from archive_name_format.
        example: "sh:{hostname}-*"
+    archive_hostname:
+        type: string
+        description: |
+            Hostname to use for the "{hostname}" placeholder in
+            "archive_name_format", "match_archives", etc. Defaults to the system
+            hostname. (This option is supported for Borg 1.4.5+ only.)
+        example: example.org
+    archive_username:
+        type: string
+        description: |
+            Username to use for the "{user}" placeholder in
+            "archive_name_format", "match_archives", etc. Defaults to the
+            username of the user running borgmatic. (This option is supported
+            for Borg 1.4.5+ only.)
+        example: backup_user
+    file_list_format:
+        type: string
+        description: |
+            Borg format for the files listing of the "list" action. Corresponds
+            to the "--format" flag. Defaults to "{mode} {user:6} {group:6}
+            {size:8} {mtime} {path}{extra}{NL}". With "--json", the form of the
+            format is ignored, but the keys used in it are added to the JSON
+            output. See
+            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
+            details.
+        example: "{path}{extra}{NL}"
+    archive_list_format:
+        type: string
+        description: |
+            Borg format for the archives listing of the "repo-list" action.
+            Corresponds to the "--format" flag. Defaults to "{archive:<36}
+            {time} [{id}]{NL}". With "--json", the form of the format is
+            ignored, but the keys used in it are added to the JSON output. See
+            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
+            details.
+        example: "{archive}{NL}"
    relocated_repo_access_is_ok:
        type: boolean
        description: |
@@ -510,34 +588,133 @@ properties:
            Bypass Borg confirmation about check with repair option. Defaults to
            false and an interactive prompt from Borg.
        example: true
+    msgpack_version_check:
+        type: boolean
+        description: |
+            Optionally disable the msgpack version check. Default is true; use
+            at your own risk. (This option is supported for Borg 1.4.2+ only.)
+        example: false
    extra_borg_options:
        type: object
        additionalProperties: false
        properties:
-            init:
+            break_lock:
                type: string
                description: |
-                  Extra command-line options to pass to "borg init".
+                  Extra command-line options to pass to "borg break-lock".
                example: "--extra-option"
-            create:
+            check:
                type: string
                description: |
-                  Extra command-line options to pass to "borg create".
-                example: "--extra-option"
-            prune:
-                type: string
-                description: |
-                  Extra command-line options to pass to "borg prune".
+                  Extra command-line options to pass to "borg check".
                example: "--extra-option"
            compact:
                type: string
                description: |
                  Extra command-line options to pass to "borg compact".
                example: "--extra-option"
-            check:
+            create:
                type: string
                description: |
-                  Extra command-line options to pass to "borg check".
+                  Extra command-line options to pass to "borg create".
+                example: "--extra-option"
+            delete:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg delete".
+                example: "--extra-option"
+            export_tar:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg export-tar".
+                example: "--extra-option"
+            extract:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg extract".
+                example: "--extra-option"
+            key_export:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key export".
+                example: "--extra-option"
+            key_import:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key import".
+                example: "--extra-option"
+            key_change_passphrase:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key
+                  change-passphrase".
+                example: "--extra-option"
+            info:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg info".
+                example: "--extra-option"
+            init:
+                type: string
+                description: |
+                  Deprecated. Use "repo_create" instead. Extra command-line
+                  options to pass to "borg init" / "borg repo-create".
+                example: "--extra-option"
+            list:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg list".
+                example: "--extra-option"
+            mount:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg mount".
+                example: "--extra-option"
+            prune:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg prune".
+                example: "--extra-option"
+            recreate:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg recreate".
+                example: "--extra-option"
+            rename:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg rename".
+                example: "--extra-option"
+            repo_create:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg init" / "borg
+                  repo-create".
+                example: "--extra-option"
+            repo_delete:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-delete".
+                example: "--extra-option"
+            repo_info:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-info".
+                example: "--extra-option"
+            repo_list:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-list".
+                example: "--extra-option"
+            transfer:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg transfer".
+                example: "--extra-option"
+            umount:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg umount".
                example: "--extra-option"
        description: |
            Additional options to pass directly to particular Borg commands,
@@ -892,6 +1069,8 @@ properties:
        type: string
        description: |
            Python format string used for log messages written to the log file.
+            See https://docs.python.org/3/library/logging.html (and specifically
+            the LogRecord attributes with "{}-formatting") for details.
        example: "[{asctime}] {levelname}: {prefix}{message}"
    monitoring_verbosity:
        type: integer
@@ -927,6 +1106,16 @@ properties:
            Corresponds to the "--stats" flag on those actions. Defaults to
            false.
        example: true
+    quick_statistics:
+        type: boolean
+        description: |
+            Display statistics for an archive when running supported actions,
+            skipping the repository-wide "All archives" and chunk index
+            statistics to save some time. Corresponds to the "--quick-stats"
+            flag on those actions. Defaults to false. (This option is supported
+            for Borg 1.4.5+ only for the "create" action and for Borg >= 1.4.5
+            and < Borg 2 for the "prune" action.)
+        example: true
    list_details:
        type: boolean
        description: |
@@ -937,9 +1126,9 @@ properties:
    default_actions:
        type: boolean
        description: |
-            Whether to apply default actions (create, prune, compact and check)
-            when no arguments are supplied to the borgmatic command. If set to
-            false, borgmatic displays the help message instead.
+            Whether to run default actions (create, prune, compact, and check)
+            when no arguments are given on the command line. If set to false,
+            borgmatic displays the help message instead.
        example: true
    skip_actions:
        type: array
@@ -967,6 +1156,8 @@ properties:
                - info
                - break-lock
                - key
+                - diff
+                - browse
                - borg
        description: |
            List of one or more actions to skip running for this configuration
@@ -1183,6 +1374,8 @@ properties:
                                  - info
                                  - break-lock
                                  - key
+                                  - diff
+                                  - browse
                                  - borg
                          description: |
                              List of actions for which the commands will be
@@ -1248,6 +1441,8 @@ properties:
                                  - info
                                  - break-lock
                                  - key
+                                  - diff
+                                  - browse
                                  - borg
                          description: |
                              Only trigger the hook when borgmatic is run with
@@ -1541,6 +1736,15 @@ properties:
                        database hook implicitly enables read_special (see
                        above) to support dump and restore streaming.
                    example: users
+                skip_names:
+                    type: array
+                    items:
+                        type: string
+                    description: |
+                        Database names to skip when dumping "all" databases.
+                        Ignored when the database name is not "all".
+                    example:
+                        - cache
                label:
                    type: string
                    description: |
@@ -1579,6 +1783,20 @@ properties:
                    description: |
                        Port to restore to. Defaults to the "port" option.
                    example: 5433
+                socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
+                    example: database.example.org
+                restore_socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to the "socket_path" option.
+                    example: database.example.org
                username:
                    type: string
                    description: |
@@ -1639,21 +1857,45 @@ properties:
                        client and restore server. The default varies based on
                        the MariaDB version.
                    example: false
+                events:
+                    type: boolean
+                    description: |
+                        Whether to include scheduled events within the dump.
+                        Disable if your database user doesn't have the
+                        permissions to dump events. Defaults to true.
+                    example: false
+                routines:
+                    type: boolean
+                    description: |
+                        Whether to include stored routines within the dump.
+                        Disable if your user database doesn't have the
+                        permissions to dump routines. Defaults to true.
+                    example: false
+                tablespaces:
+                    type: boolean
+                    description: |
+                        Whether to include tablespaces within the dump. Disable
+                        if your database user doesn't have the permissions to
+                        dump tablespaces. Defaults to true.
+                    example: false
                mariadb_dump_command:
                    type: string
                    description: |
-                        Command to use instead of "mariadb-dump". This can be
-                        used to run a specific mariadb_dump version (e.g., one
-                        inside a running container). If you run it from within a
-                        container, make sure to mount the path in the
-                        "user_runtime_directory" option from the host into the
-                        container at the same location. Defaults to
-                        "mariadb-dump".
+                        Command to use instead of "mariadb-dump". But set any
+                        additional command-line flags in "options", not here.
+                        This command can be used to run a specific mariadb_dump
+                        version (e.g., one inside a running container). If you
+                        run it from within a container, make sure to mount the
+                        path in the "user_runtime_directory" option from the
+                        host into the container at the same location. Defaults
+                        to "mariadb-dump".
                    example: docker exec mariadb_container mariadb-dump
                mariadb_command:
                    type: string
                    description: |
-                        Command to run instead of "mariadb". This can be used to
+                        Command to run instead of "mariadb". But set any
+                        additional command-line flags in "list_options" or
+                        "restore_options", not here. This command can be used to
                        run a specific mariadb version (e.g., one inside a
                        running container). Defaults to "mariadb".
                    example: docker exec mariadb_container mariadb           
@@ -1723,6 +1965,15 @@ properties:
                        database hook implicitly enables read_special (see
                        above) to support dump and restore streaming.
                    example: users
+                skip_names:
+                    type: array
+                    items:
+                        type: string
+                    description: |
+                        Database names to skip when dumping "all" databases.
+                        Ignored when the database name is not "all".
+                    example:
+                        - cache
                label:
                    type: string
                    description: |
@@ -1761,6 +2012,20 @@ properties:
                    description: |
                        Port to restore to. Defaults to the "port" option.
                    example: 5433
+                socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
+                    example: database.example.org
+                restore_socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to the "socket_path" option.
+                    example: database.example.org
                username:
                    type: string
                    description: |
@@ -1821,20 +2086,45 @@ properties:
                        client and restore server. The default varies based on
                        the MySQL installation.
                    example: false
+                events:
+                    type: boolean
+                    description: |
+                        Whether to include scheduled events within the dump.
+                        Disable if your database user doesn't have the
+                        permissions to dump events. Defaults to true.
+                    example: false
+                routines:
+                    type: boolean
+                    description: |
+                        Whether to include stored routines within the dump.
+                        Disable if your database user doesn't have the
+                        permissions to dump routines. Defaults to true.
+                    example: false
+                tablespaces:
+                    type: boolean
+                    description: |
+                        Whether to include tablespaces within the dump. Disable
+                        if your database user doesn't have the permissions to
+                        dump tablespaces. Defaults to true.
+                    example: false
                mysql_dump_command:
                    type: string
                    description: |
-                        Command to use instead of "mysqldump". This can be used
-                        to run a specific mysql_dump version (e.g., one inside a
-                        running container). If you run it from within a
-                        container, make sure to mount the path in the
-                        "user_runtime_directory" option from the host into the
-                        container at the same location. Defaults to "mysqldump".
+                        Command to use instead of "mysqldump". But set any
+                        additional command-line flags in "options", not here.
+                        This command can be used to run a specific mysql_dump
+                        version (e.g., one inside a running container). If you
+                        run it from within a container, make sure to mount the
+                        path in the "user_runtime_directory" option from the
+                        host into the container at the same location. Defaults
+                        to "mysqldump".
                    example: docker exec mysql_container mysqldump
                mysql_command:
                    type: string
                    description: |
-                        Command to run instead of "mysql". This can be used to
+                        Command to run instead of "mysql". But set any
+                        additional command-line flags in "list_options" or
+                        "restore_options", not here. This command can be used to
                        run a specific mysql version (e.g., one inside a running
                        container). Defaults to "mysql".
                    example: docker exec mysql_container mysql
@@ -2139,17 +2429,24 @@ properties:
                        type: string
                        description: |
                            The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have started.
                    priority:
                        type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: min
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic
            finish:
                type: object
                additionalProperties: false
@@ -2163,17 +2460,24 @@ properties:
                        type: string
                        description: |
                            The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have finished.
                    priority:
                        type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: min
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,+1
            fail:
                type: object
                additionalProperties: false
@@ -2190,14 +2494,21 @@ properties:
                        example: Your backups have failed.
                    priority:
                        type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                        description: |
                            The priority to set.
-                        example: urgent
+                        example: max
                    tags:
                        type: string
                        description: |
                            Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,-1,skull
            states:
                type: array
                items:
@@ -2584,7 +2895,9 @@ properties:
                    properties:
                        url:
                            type: string
-                            description: URL of this Apprise service.
+                            description: |
+                                URL of this Apprise service. Supports the
+                                "{credential ...}" syntax.
                            example: "gotify://hostname/token"
                        label:
                            type: string
@@ -2606,8 +2919,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to Apprise services as part of the
-                    "finish", "fail", and "log" states. Defaults to true.
-                example: false
+                    "finish", "fail", and "log" states. Defaults to false.
+                example: true
            logs_size_limit:
                type: integer
                description: |
@@ -2721,8 +3034,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to Healthchecks as part of the "finish",
-                    "fail", and "log" states. Defaults to true.
-                example: false
+                    "fail", and "log" states. Defaults to false.
+                example: true
            ping_body_limit:
                type: integer
                description: |
@@ -2828,8 +3141,8 @@ properties:
                type: boolean
                description: |
                    Send borgmatic logs to PagerDuty when a backup errors.
-                    Defaults to true.
-                example: false
+                    Defaults to false.
+                example: true
        description: |
            Configuration for a monitoring integration with PagerDuty. Create an
            account at https://www.pagerduty.com if you'd like to use this
@@ -2857,9 +3170,27 @@ properties:
            url:
                type: string
                description: |
-                    Grafana loki log URL to notify when a backup begins,
+                    Grafana Loki log URL to notify when a backup begins,
                    ends, or fails.
                example: "http://localhost:3100/loki/api/v1/push"
+            tls:
+                type: object
+                additionalProperties: false
+                properties:
+                    cert_path:
+                        type: string
+                        description: |
+                            Path to a PEM client certificate file for mutual
+                            TLS authentication.
+                        example: /etc/borgmatic/loki-client.crt
+                    key_path:
+                        type: string
+                        description: |
+                            Path to a PEM private key file for the client
+                            certificate.
+                        example: /etc/borgmatic/loki-client.key
+                description: |
+                    TLS options for mutual TLS (mTLS) authentication with Loki.
            labels:
                type: object
                additionalProperties:
@@ -2874,6 +3205,11 @@ properties:
                    app: "borgmatic"
                    config: "__config"
                    hostname: "__hostname"
+            send_logs:
+                type: boolean
+                description: |
+                    Send borgmatic logs to Loki. Defaults to false.
+                example: true
        description: |
            Configuration for a monitoring integration with Grafana Loki. You
            can send the logs to a self-hosted instance or create an account at
@@ -2898,6 +3234,12 @@ properties:
                    project monitor. Used along with the data source name URL to
                    construct a cron URL.
                example: mymonitor
+            environment:
+                type: string
+                description: |
+                    Sentry monitor environment used in the call to Sentry. If
+                    not set, the Sentry default is used.
+                example: production
            states:
                type: array
                items:
@@ -3042,16 +3384,32 @@ properties:
                description: |
                    Command to use instead of "keepassxc-cli".
                example: /usr/local/bin/keepassxc-cli
+            secret_tool_command:
+                type: string
+                description: |
+                    Command to use instead of "secret-tool".
+                example: /usr/local/bin/secret-tool
+            ask_for_password:
+                type: boolean
+                description: |
+                    Whether keepassxc-cli should prompt the user for a password.
+                    Disabling this is only really useful if you're unlocking
+                    your KeePassXC database with a key file instead of a
+                    password. Ignored when using KeePassXC's secret service
+                    integration. Defaults to true.
+                example: false
            key_file:
                type: string
                description: |
                    Path to a key file for unlocking the KeePassXC database.
+                    Ignored when using KeePassXC's secret service integration.
                example: /path/to/keyfile
            yubikey:
                type: string
                description: |
                    YubiKey slot and optional serial number used to access the
-                    KeePassXC database. The format is "<slot[:serial]>", where:
+                    KeePassXC database. Ignored when using KeePassXC's secret
+                    service integration. The format is "<slot[:serial]>", where:
                     * <slot> is the YubiKey slot number (e.g., `1` or `2`).
                     * <serial> (optional) is the YubiKey's serial number (e.g.,
                       `7370001`).
@@ -1,9 +1,14 @@
 import collections
+import contextlib
 import enum
+import json
+import locale
 import logging
+import os
 import select
 import subprocess
 import textwrap
+import time

 import borgmatic.logger

@@ -14,6 +19,9 @@ ERROR_OUTPUT_MAX_LINE_COUNT = 25
 BORG_ERROR_EXIT_CODE_START = 2
 BORG_ERROR_EXIT_CODE_END = 99

+# See https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids
+BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS = {101, 102, 104, 105, 106}
+

 class Exit_status(enum.Enum):
    STILL_RUNNING = 1
@@ -22,7 +30,32 @@ class Exit_status(enum.Enum):
    ERROR = 4


-def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
+def command_is_borg(command, borg_local_path):
+    '''
+    Given a command as a sequence and the Borg local path, return whether that command is a call to
+    Borg.
+    '''
+    parsed_command = command.split(' ', 1) if isinstance(command, str) else command
+
+    if not parsed_command:
+        return False
+
+    return bool(borg_local_path and parsed_command[0] == borg_local_path)
+
+
+BORG_EXIT_CODE_TO_DESCRIPTION = {
+    100: 'File changed while we backed it up',
+    101: 'Include pattern never matched',
+    102: 'General backup issue',
+    103: 'File type or inode changed while we backed it up',
+    104: 'Backup OS issue',
+    105: 'Backup permission issue',
+    106: 'Backup IO issue',
+    107: 'Backup file not found',
+}
+
+
+def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):  # noqa: PLR0911
    '''
    Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
    exit code. If a Borg local path is given and matches the process' command, then interpret the
@@ -35,39 +68,47 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
    if exit_code == 0:
        return Exit_status.SUCCESS

-    if borg_local_path and command[0] == borg_local_path:
-        # First try looking for the exit code in the borg_exit_codes configuration.
-        for entry in borg_exit_codes or ():
-            if entry.get('code') == exit_code:
-                treat_as = entry.get('treat_as')
+    if not command_is_borg(command, borg_local_path):
+        return Exit_status.ERROR

-                if treat_as == 'error':
-                    logger.error(
-                        f'Treating exit code {exit_code} as an error, as per configuration',
-                    )
-                    return Exit_status.ERROR
+    description = BORG_EXIT_CODE_TO_DESCRIPTION.get(exit_code)
+    description_parenthetical = f' ({description})' if description else ''

-                if treat_as == 'warning':
-                    logger.warning(
-                        f'Treating exit code {exit_code} as a warning, as per configuration',
-                    )
-                    return Exit_status.WARNING
+    # First try looking for the exit code in the borg_exit_codes configuration.
+    for entry in borg_exit_codes or ():
+        if entry.get('code') == exit_code:
+            treat_as = entry.get('treat_as')

-        # If the exit code doesn't have explicit configuration, then fall back to the default Borg
-        # behavior.
-        return (
-            Exit_status.ERROR
-            if (
-                exit_code < 0
-                or (
-                    exit_code >= BORG_ERROR_EXIT_CODE_START
-                    and exit_code <= BORG_ERROR_EXIT_CODE_END
+            if treat_as == 'error':
+                logger.error(
+                    f'Treating exit code {exit_code}{description_parenthetical} as an error, as per configuration',
                )
-            )
-            else Exit_status.WARNING
+                return Exit_status.ERROR
+
+            if treat_as == 'warning':
+                logger.warning(
+                    f'Treating exit code {exit_code}{description_parenthetical} as a warning, as per configuration',
+                )
+                return Exit_status.WARNING
+
+    # If the exit code doesn't have explicit configuration, then fall back to the default
+    # behavior of treating Borg errors as errors and some Borg warnings as errors.
+    if exit_code in BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS:
+        logger.error(
+            f'Treating exit code {exit_code}{description_parenthetical} as an error, as per borgmatic defaults',
        )

-    return Exit_status.ERROR
+        return Exit_status.ERROR
+
+    if exit_code < 0 or (
+        exit_code >= BORG_ERROR_EXIT_CODE_START and exit_code <= BORG_ERROR_EXIT_CODE_END
+    ):
+        return Exit_status.ERROR
+
+    logger.warning(
+        f'Treating exit code {exit_code}{description_parenthetical} as a warning, as per borgmatic defaults',
+    )
+    return Exit_status.WARNING


 def command_for_process(process):
@@ -78,158 +119,431 @@ def command_for_process(process):
    return process.args if isinstance(process.args, str) else ' '.join(process.args)


-def output_buffer_for_process(process, exclude_stdouts):
+def output_buffers_for_process(process, exclude_stdouts):
    '''
    Given a process as an instance of subprocess.Popen and a sequence of stdouts to exclude, return
-    either the process's stdout or stderr. The idea is that if stdout is excluded for a process, we
-    still have stderr to log.
+    the process stdout and stderr as a tuple—but exclude the stdout if it's in the given stdouts to
+    exclude.
    '''
-    return process.stderr if process.stdout in exclude_stdouts else process.stdout
+    return tuple(
+        buffer for buffer in (process.stdout, process.stderr) if buffer not in exclude_stdouts
+    )


-def append_last_lines(last_lines, captured_output, line, output_log_level):
+BORG_LOG_LEVEL_ELEVATION_THRESHOLD = 10
+
+
+def borg_json_log_line_to_record(line, log_level):
    '''
-    Given a rolling list of last lines, a list of captured output, a line to append, and an output
-    log level, append the line to the last lines and (if necessary) the captured output. Then log
-    the line at the requested output log level.
+    Given a single Borg "--log-json"-style log line and a log level, return the line converted to a
+    logging.LogRecord instance. Return None if the line can't be parsed as JSON.
+
+    If Borg provides a log level in its JSON, prefer logging at that level. But if Borg doesn't
+    provide a log level—or the log level given to this function is just a little bit higher than
+    Borg's—elevate to that level. This supports use cases like elevating Borg's INFO level logs to
+    borgmatic's custom ANSWER level so that requested data shows up even at the default verbosity.
    '''
-    last_lines.append(line)
+    with contextlib.suppress(json.JSONDecodeError, TypeError, KeyError, AttributeError):
+        log_data = json.loads(line)
+        log_type = log_data.get('type')

-    if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
-        last_lines.pop(0)
+        if log_type == 'log_message':
+            borg_log_level = logging._nameToLevel.get(log_data.get('levelname'))
+            log_level_delta = 0 if log_level is None else log_level - borg_log_level

-    if output_log_level is None:
-        captured_output.append(line)
-    else:
-        logger.log(output_log_level, line)
-
-
-def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):  # noqa: PLR0912
-    '''
-    Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
-    process with the requested log level. Additionally, raise a CalledProcessError if a process
-    exits with an error (or a warning for exit code 1, if that process does not match the Borg local
-    path).
-
-    If output log level is None, then instead of logging, capture output for each process and return
-    it as a dict from the process to its output. Use the given Borg local path and exit code
-    configuration to decide what's an error and what's a warning.
-
-    For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
-    stdouts are given to exclude, then for any matching processes, log from their stderr instead.
-
-    Note that stdout for a process can be None if output is intentionally not captured. In which
-    case it won't be logged.
-    '''
-    # Map from output buffer to sequence of last lines.
-    buffer_last_lines = collections.defaultdict(list)
-    process_for_output_buffer = {
-        output_buffer_for_process(process, exclude_stdouts): process
-        for process in processes
-        if process.stdout or process.stderr
-    }
-    output_buffers = list(process_for_output_buffer.keys())
-    captured_outputs = collections.defaultdict(list)
-    still_running = True
-
-    # Log output for each process until they all exit.
-    while True:  # noqa: PLR1702
-        if output_buffers:
-            (ready_buffers, _, _) = select.select(output_buffers, [], [])
-
-            for ready_buffer in ready_buffers:
-                ready_process = process_for_output_buffer.get(ready_buffer)
-
-                # The "ready" process has exited, but it might be a pipe destination with other
-                # processes (pipe sources) waiting to be read from. So as a measure to prevent
-                # hangs, vent all processes when one exits.
-                if ready_process and ready_process.poll() is not None:
-                    for other_process in processes:
-                        if (
-                            other_process.poll() is None
-                            and other_process.stdout
-                            and other_process.stdout not in output_buffers
-                        ):
-                            # Add the process's output to output_buffers to ensure it'll get read.
-                            output_buffers.append(other_process.stdout)
-
-                while True:
-                    line = ready_buffer.readline().rstrip().decode()
-                    if not line or not ready_process:
-                        break
-
-                    # Keep the last few lines of output in case the process errors, and we need the output for
-                    # the exception below.
-                    append_last_lines(
-                        buffer_last_lines[ready_buffer],
-                        captured_outputs[ready_process],
-                        line,
-                        output_log_level,
+            if log_level_delta > 0 and log_level_delta < BORG_LOG_LEVEL_ELEVATION_THRESHOLD:
+                return logging.makeLogRecord(
+                    dict(
+                        levelno=log_level,
+                        created=log_data.get('time'),
+                        msg=log_data.get('message'),
+                        msgid=log_data.get('msgid'),
+                        levelname=logging.getLevelName(log_level),
+                        name=log_data.get('name'),
                    )
+                )

-        if not still_running:
-            break
+            return logging.makeLogRecord(
+                dict(
+                    levelno=borg_log_level,
+                    created=log_data.get('time'),
+                    msg=log_data.get('message'),
+                    msgid=log_data.get('msgid'),
+                    levelname=log_data.get('levelname'),
+                    name=log_data.get('name'),
+                )
+            )

-        still_running = False
-
-        for process in processes:
-            exit_code = process.poll() if output_buffers else process.wait()
-
-            if exit_code is None:
-                still_running = True
-                command = process.args.split(' ') if isinstance(process.args, str) else process.args
-                continue
-
-            command = process.args.split(' ') if isinstance(process.args, str) else process.args
-            exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
-
-            if exit_status in {Exit_status.ERROR, Exit_status.WARNING}:
-                # If an error occurs, include its output in the raised exception so that we don't
-                # inadvertently hide error output.
-                output_buffer = output_buffer_for_process(process, exclude_stdouts)
-                last_lines = buffer_last_lines[output_buffer] if output_buffer else []
-
-                # Collect any straggling output lines that came in since we last gathered output.
-                while output_buffer:  # pragma: no cover
-                    line = output_buffer.readline().rstrip().decode()
-                    if not line:
-                        break
-
-                    append_last_lines(
-                        last_lines,
-                        captured_outputs[process],
-                        line,
-                        output_log_level=logging.ERROR,
-                    )
-
-                if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
-                    last_lines.insert(0, '...')
-
-                # Something has gone wrong. So vent each process' output buffer to prevent it from
-                # hanging. And then kill the process.
-                for other_process in processes:
-                    if other_process.poll() is None:
-                        other_process.stdout.read(0)
-                        other_process.kill()
-
-                if exit_status == Exit_status.ERROR:
-                    raise subprocess.CalledProcessError(
-                        exit_code,
-                        command_for_process(process),
-                        '\n'.join(last_lines),
-                    )
-
-                still_running = False
-                break
-
-    if captured_outputs:
-        return {
-            process: '\n'.join(output_lines) for process, output_lines in captured_outputs.items()
-        }
+        if log_type == 'file_status':
+            return logging.makeLogRecord(
+                dict(
+                    levelno=log_level,
+                    created=time.time(),
+                    msg=f'{log_data.get("status")} {log_data.get("path")}',
+                    msgid=log_data.get('msgid'),
+                    levelname=logging.getLevelName(log_level),
+                    name='borg.file_status',
+                )
+            )

    return None


+def log_line_to_record(line, log_level):
+    '''
+    Given a log data dict for a single Borg log entry and a log level, return it converted to a
+    logging.LogRecord instance.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            msg=line,
+            levelno=log_level,
+            levelname=logging.getLevelName(log_level),
+        )
+    )
+
+
+def parse_log_line(line, log_level, elevate_stderr, borg_local_path, command):
+    '''
+    Given a raw output line from an external program, whether this line came from stderr and should
+    be elevated to error/warning, the Borg local path, and the command as a sequence, return a
+    logging.LogRecord instance containing its parsed data.
+
+    If the command being run is Borg, and the log line is JSON-formatted log data, then grab the log
+    level from it and log the parsed JSON to be consumed later by a Python logging.Formatter.
+
+    But for non-Borg commands, elevate stderr-sourced logs to ERROR. The one exception is if the log
+    came from stderr and the string "warning:" appears at the start of the log line. In that case,
+    just elevate the log level to a WARN.
+    '''
+    if command_is_borg(command, borg_local_path):
+        log_record = borg_json_log_line_to_record(line, log_level)
+
+        if log_record:
+            return log_record
+
+    if elevate_stderr:
+        return log_line_to_record(
+            line, logging.WARNING if line.lower().startswith('warning:') else logging.ERROR
+        )
+
+    return log_line_to_record(line, log_level)
+
+
+def handle_log_record(log_record, last_lines=None):
+    '''
+    Given a log record to be logged and a rolling list of last lines, append the record's message to
+    the last lines (if given). Then (if the log level is not None), log the record.
+
+    Return the log record.
+    '''
+    log_message = log_record.getMessage()
+
+    if last_lines is not None:
+        last_lines.append(log_message)
+
+        if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
+            last_lines.pop(0)
+
+    if log_record.levelno is not None:
+        logger.handle(log_record)
+
+    return log_record
+
+
+READ_CHUNK_SIZE = 4096
+
+
+def read_lines(buffer, process, line_separator='\n'):
+    '''
+    Given a Python buffer (like stdout) ready for reading, its process, and a line separator,
+    repeatedly yield a tuple of (decoded) lines from the buffer until the process has exited.
+
+    It is assumed that this function's generator is used in conjunction with an external select()
+    call to know when to read more lines. Otherwise, the generator will busywait if it's called in a
+    tight loop.
+    '''
+    data = b''
+    encoded_separator = line_separator.encode()
+    separator_size = len(encoded_separator)
+    encoding = locale.getpreferredencoding()
+
+    while True:
+        chunk = os.read(buffer.fileno(), READ_CHUNK_SIZE)
+
+        if not chunk:  # EOF
+            # The process is still running, so we keep running too.
+            if process.poll() is None:  # pragma: no cover
+                continue
+
+            break
+
+        data += chunk
+        lines = []
+
+        # Split the data into lines, holding back anything leftover that might
+        # be a partial line.
+        while True:
+            separator_position = data.find(encoded_separator)
+
+            if separator_position == -1:
+                break
+
+            lines.append(data[:separator_position].decode(encoding))
+            data = data[separator_position + separator_size :]
+
+        yield tuple(lines)
+
+    # Yield any leftover data from the end of the buffer.
+    if data:
+        yield (data.decode(encoding).rstrip(),)
+
+
+Buffer_reader = collections.namedtuple(
+    'Buffer_reader',
+    ('lines', 'process'),
+)
+
+
+Process_metadata = collections.namedtuple(
+    'Process_metadata',
+    ('last_lines', 'capture'),
+)
+
+
+def log_buffer_lines(
+    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
+):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
+    whether to capture stderr, read and log any ready output lines from the buffers. Additionally,
+    for any log records with a log level the same as the output log level, yield those log messages
+    for capture.
+
+    This function just does one "turn of the crank" of logging buffer output. It is intended to be
+    called repeatedly to continue to process buffers.
+    '''
+    if not buffer_readers:
+        return
+
+    (ready_buffers, _, _) = select.select(buffer_readers.keys(), [], [])
+
+    for ready_buffer in ready_buffers:
+        reader = buffer_readers[ready_buffer]
+
+        # The "ready" process has exited, but it might be a pipe destination with other
+        # processes (pipe sources) waiting to be read from. So as a measure to prevent
+        # hangs, vent all processes when one exits.
+        if reader.process and reader.process.poll() is not None:
+            for other_process in process_metadatas:
+                if (
+                    other_process.poll() is None
+                    and other_process.stdout
+                    and other_process.stdout not in buffer_readers
+                ):
+                    # Add the process's output to buffer_readers to ensure it'll get read.
+                    buffer_readers[other_process.stdout] = Buffer_reader(
+                        read_lines(other_process.stdout, other_process), other_process
+                    )
+
+        try:
+            lines = next(reader.lines)
+        except StopIteration:
+            continue
+
+        for line in lines:
+            if not line or not reader.process:
+                continue
+
+            # Keep the last few lines of output in case the process errors and we need the
+            # output for the exception below.
+            log_record = handle_log_record(
+                parse_log_line(
+                    line=line,
+                    log_level=output_log_level,
+                    elevate_stderr=(ready_buffer == reader.process.stderr and not capture_stderr),
+                    borg_local_path=borg_local_path,
+                    command=reader.process.args,
+                ),
+                last_lines=process_metadatas[reader.process].last_lines,
+            )
+
+            if (
+                log_record.levelno is None or log_record.levelno == output_log_level
+            ) and process_metadatas[reader.process].capture:
+                yield log_record.getMessage()
+
+
+def raise_for_process_errors(
+    buffer_readers,
+    process_metadatas,
+    output_log_level,
+    borg_local_path,
+    borg_exit_codes,
+    capture_stderr=False,
+):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, a requested output log level for stdout, Borg's local path, a
+    sequence of exit code configuration dicts, and whether to capture stderr, check the given
+    processes for error or warning exit codes. If found, vent or kill any running processes and
+    drain any remaining buffer lines. In the case of an error exit code, raise.  In the case of
+    warning, return Exit_status.WARNING. Otherwise, return None.
+    '''
+    result_status = None
+
+    for process in process_metadatas:
+        exit_code = process.poll() if buffer_readers else process.wait()
+
+        if exit_code is None:
+            continue
+
+        exit_status = interpret_exit_code(process.args, exit_code, borg_local_path, borg_exit_codes)
+
+        if exit_status not in {Exit_status.ERROR, Exit_status.WARNING}:
+            continue
+
+        # Something has gone wrong. So vent each process' output buffer to prevent it from
+        # hanging. And then kill the process.
+        for other_process in process_metadatas:
+            if other_process.poll() is None:
+                other_process.stdout.read(0)
+                other_process.kill()
+
+        if exit_status == Exit_status.WARNING:
+            result_status = Exit_status.WARNING
+            continue
+
+        # Drain and log remaining buffer lines, so no output gets lost. But swallow captured output,
+        # since we're raising below instead of yielding captured lines.
+        tuple(
+            log_remaining_buffer_lines(
+                buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
+            )
+        )
+        last_lines = process_metadatas[process].last_lines
+
+        # If an error occurs, include its output in the raised exception so that we don't
+        # inadvertently hide error output.
+        if len(last_lines) >= ERROR_OUTPUT_MAX_LINE_COUNT:
+            last_lines.insert(0, '...')
+
+        raise subprocess.CalledProcessError(
+            exit_code,
+            command_for_process(process),
+            '\n'.join(last_lines),
+        )
+
+    return result_status
+
+
+def log_remaining_buffer_lines(
+    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
+):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
+    whether to capture stderr, drain and log any remaining output lines from the buffers until
+    they're empty. Additionally, for any log records with a log level the same as the output log
+    level, yield those log messages for capture.
+
+    The main difference between this function and log_buffer_lines() is that this one doesn't just
+    do one "turn of the crank" of logging buffer output; it completely drains any remaining output.
+    '''
+    for output_buffer, reader in buffer_readers.items():
+        if not reader.process:
+            continue
+
+        for lines in reader.lines:
+            for line in lines:
+                log_record = handle_log_record(
+                    parse_log_line(
+                        line=line.rstrip(),
+                        log_level=output_log_level,
+                        elevate_stderr=(
+                            output_buffer == reader.process.stderr and not capture_stderr
+                        ),
+                        borg_local_path=borg_local_path,
+                        command=reader.process.args,
+                    ),
+                    last_lines=process_metadatas[reader.process].last_lines,
+                )
+
+                if (
+                    log_record.levelno is None or log_record.levelno == output_log_level
+                ) and process_metadatas[reader.process].capture:
+                    yield log_record.getMessage()
+
+
+def log_outputs(
+    processes,
+    exclude_stdouts,
+    output_log_level,
+    borg_local_path,
+    borg_exit_codes,
+    capture_stderr=False,
+):
+    '''
+    Given a sequence of subprocess.Popen() instances for multiple processes, log the outputs (stderr
+    and stdout). Use the requested output log level for stdout, but always log stderr to the ERROR
+    log level. Additionally, raise a CalledProcessError if a process exits with an error (or a
+    warning for exit code 1, if that process does not match the Borg local path).
+
+    If the output log level is None, then instead of logging, capture the output for the last
+    process given and yield it one line at a time. This includes stderr if capture stderr is set.
+    But if the output log level is not None, don't yield anything.
+
+    This yielding means that this function is a generator, and must be consumed in order to execute.
+
+    Use the given Borg local path and exit code configuration to decide what's an error and what's a
+    warning. If any stdouts are given to exclude, then for any matching processes, ignore those
+    buffers. Also note that stdout for a process can be None if output is intentionally not
+    captured, in which case it won't be logged.
+    '''
+    # Map from output buffer to Process_metadata instance. By convention, the last process is the
+    # process to capture.
+    process_metadatas = {
+        process: Process_metadata(last_lines=[], capture=bool(process == processes[-1]))
+        for process in processes
+    }
+
+    # Map from buffer to Buffer_reader instance.
+    buffer_readers = {
+        buffer: Buffer_reader(read_lines(buffer, process), process)
+        for process in processes
+        if process.stdout or process.stderr
+        for buffer in output_buffers_for_process(process, exclude_stdouts)
+    }
+
+    # Log output lines for each process until they all exit or one errors.
+    while True:
+        yield from log_buffer_lines(
+            buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
+        )
+
+        if (
+            raise_for_process_errors(
+                buffer_readers,
+                process_metadatas,
+                output_log_level,
+                borg_local_path,
+                borg_exit_codes,
+                capture_stderr,
+            )
+            == Exit_status.WARNING
+        ):
+            break
+
+        if all(process.poll() is not None for process in processes):
+            break
+
+    # Now that all processes have exited, drain and consume any last output.
+    yield from log_remaining_buffer_lines(
+        buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
+    )
+
+
 SECRET_COMMAND_FLAG_NAMES = {'--password'}


@@ -300,16 +614,16 @@ def execute_command(
    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
    '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
-    given log level. If an open output file object is given, then write stdout to the file and only
-    log stderr. If an open input file object is given, then read stdin from the file. If shell is
-    True, execute the command within a shell. If an environment variables dict is given, then pass
-    it into the command. If a working directory is given, use that as the present working directory
-    when running the command. If a Borg local path is given, and the command matches it (regardless
-    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
-    given as a sequence of exit code configuration dicts, then use that configuration to decide
-    what's an error and what's a warning. If run to completion is False, then return the process for
-    the command without executing it to completion.
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
+    the given log level. If an open output file object is given, then write stdout to the file and
+    only log stderr. If an open input file object is given, then read stdin from the file. If shell
+    is True, execute the command within a shell. If an environment variables dict is given, then
+    pass it into the command. If a working directory is given, use that as the present working
+    directory when running the command. If a Borg local path is given, and the command matches it
+    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
+    codes are given as a sequence of exit code configuration dicts, then use that configuration to
+    decide what's an error and what's a warning. If run to completion is False, then return the
+    process for the command without executing it to completion.

    Raise subprocesses.CalledProcessError if an error occurs while running the command.
    '''
@@ -321,7 +635,7 @@ def execute_command(
        command,
        stdin=input_file,
        stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-        stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
+        stderr=None if do_not_capture else subprocess.PIPE,
        shell=shell,
        env=environment,
        cwd=working_directory,
@@ -331,12 +645,14 @@ def execute_command(
        return process

    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        log_outputs(
-            (process,),
-            (input_file, output_file),
-            output_log_level,
-            borg_local_path,
-            borg_exit_codes,
+        tuple(
+            log_outputs(
+                (process,),
+                (input_file, output_file),
+                output_log_level,
+                borg_local_path,
+                borg_exit_codes,
+            )
        )

    return None
@@ -344,6 +660,7 @@ def execute_command(

 def execute_command_and_capture_output(
    full_command,
+    output_log_level=None,
    input_file=None,
    capture_stderr=False,
    shell=False,
@@ -355,14 +672,18 @@ def execute_command_and_capture_output(
 ):
    '''
    Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
-    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
-    execute the command within a shell. If an environment variables dict is given, then pass it into
-    the command. If a working directory is given, use that as the present working directory when
-    running the command. If a Borg local path is given, and the command matches it (regardless of
-    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
-    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
-    error and what's a warning.
+    output (stdout) as a generator that yields one line at a time. The generator must be consumed in
+    order for the called command to execute.
+
+    If an output log level is given, then instead of suppressing log output, also output the
+    captured lines at the given log level.  If an input file descriptor is given, then pipe it to
+    the command's stdin. If capture stderr is True, then capture stderr in addition to stdout. If
+    shell is True, execute the command within a shell. If an environment variables dict is given,
+    then pass it into the command. If a working directory is given, use that as the present working
+    directory when running the command. If a Borg local path is given, and the command matches it
+    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
+    codes are given as a sequence of exit code configuration dicts, then use that configuration to
+    decide what's an error and what's a warning.

    Raise subprocesses.CalledProcessError if an error occurs while running the command.
    '''
@@ -370,10 +691,13 @@ def execute_command_and_capture_output(
    command = ' '.join(full_command) if shell else full_command

    try:
-        output = subprocess.check_output(  # noqa: S603
+        process = subprocess.Popen(  # noqa: S603
            command,
            stdin=input_file,
-            stderr=subprocess.STDOUT if capture_stderr else None,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE
+            if capture_stderr or command_is_borg(command, borg_local_path)
+            else None,
            shell=shell,
            env=environment,
            cwd=working_directory,
@@ -386,9 +710,20 @@ def execute_command_and_capture_output(
        ):
            raise

-        output = error.output
+        if error.output is not None:
+            yield from iter(error.output.decode(locale.getpreferredencoding()).splitlines())

-    return output.decode() if output is not None else None
+        return
+
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        yield from log_outputs(
+            (process,),
+            (input_file,),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+            capture_stderr=capture_stderr,
+        )


 def execute_command_with_processes(
@@ -405,20 +740,23 @@ def execute_command_with_processes(
    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
    '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
-    given log level. Simultaneously, continue to poll one or more active processes so that they
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
+    the given log level. Simultaneously, continue to poll one or more active processes so that they
    run as well. This is useful, for instance, for processes that are streaming output to a named
    pipe that the given command is consuming from.

    If an open output file object is given, then write stdout to the file and only log stderr. But
    if output log level is None, instead suppress logging and return the captured output for (only)
-    the given command. If an open input file object is given, then read stdin from the file. If
-    shell is True, execute the command within a shell. If an environment variables dict is given,
-    then pass it into the command. If a working directory is given, use that as the present working
-    directory when running the command. If a Borg local path is given, then for any matching command
-    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
-    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
-    configuration to decide what's an error and what's a warning.
+    the given command as a generator that yields one line at a time. The generator must be consumed
+    in order for the called command to execute—regardless of the output log level.
+
+    If an open input file object is given, then read stdin from the file. If shell is True, execute
+    the command within a shell. If an environment variables dict is given, then pass it into the
+    command. If a working directory is given, use that as the present working directory when running
+    the command. If a Borg local path is given, then for any matching command or process (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning.

    Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
    upstream process.
@@ -432,9 +770,7 @@ def execute_command_with_processes(
            command,
            stdin=input_file,
            stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-            stderr=(
-                None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
-            ),
+            stderr=None if do_not_capture else subprocess.PIPE,
            shell=shell,
            env=environment,
            cwd=working_directory,
@@ -451,15 +787,10 @@ def execute_command_with_processes(
        raise

    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        captured_outputs = log_outputs(
+        yield from log_outputs(
            (*processes, command_process),
            (input_file, output_file),
            output_log_level,
            borg_local_path,
            borg_exit_codes,
        )
-
-    if output_log_level is None:
-        return captured_outputs.get(command_process)
-
-    return None
@@ -13,39 +13,71 @@ logger = logging.getLogger(__name__)


 SOFT_FAIL_EXIT_CODE = 75
-BORG_PLACEHOLDERS = {
-    '{hostname}',
-    '{fqdn}',
-    '{reverse-fqdn}',
-    '{now}',
-    '{utcnow}',
-    '{unixtime}',
-    '{user}',
-    '{pid}',
-    '{borgversion}',
-    '{borgmajor}',
-    '{borgminor}',
-    '{borgpatch}',
+BORG_PLACEHOLDER_NAMES = {
+    'hostname',
+    'fqdn',
+    'reverse-fqdn',
+    'now',
+    'utcnow',
+    'unixtime',
+    'user',
+    'pid',
+    'borgversion',
+    'borgmajor',
+    'borgminor',
+    'borgpatch',
 }


+VARIABLE_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_variable(match, context, hook_description):
+    '''
+    Given a re.Match instance of VARIABLE_PATTERN representing a matched variable name to be
+    interpolated, a context dict, and a description of the current command hook, lookup the matched
+    variable name within the given context and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given context (and isn't a Borg placeholder), then
+    warn and return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name.
+    if match.group('left_escape') and match.group('right_escape'):
+        return '{' + name + '}'
+
+    value = context.get(name)
+
+    # The matched variable is in the context, so return its value.
+    if value is not None:
+        return shlex.quote(str(value))
+
+    # The matched variable name isn't in the context. Warn about variables unknown to borgmatic, but
+    # don't warn if the variable name happens to be a Borg placeholder, as Borg should hopefully
+    # consume it.
+    if name not in BORG_PLACEHOLDER_NAMES:
+        logger.warning(
+            f'Variable "{name}" is not supported in the {hook_description} hook',
+        )
+
+    # Return the whole string unaltered.
+    return match.group(0)
+
+
 def interpolate_context(hook_description, command, context):
    '''
    Given a config filename, a hook description, a single hook command, and a dict of context
    names/values, interpolate the values by "{name}" into the command and return the result.
    '''
-    for name, value in context.items():
-        command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
-
-    for unsupported_variable in re.findall(r'\{\w+\}', command):
-        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
-        # be a Borg placeholder, as Borg should hopefully consume it.
-        if unsupported_variable not in BORG_PLACEHOLDERS:
-            logger.warning(
-                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook',
-            )
-
-    return command
+    return VARIABLE_PATTERN.sub(
+        functools.partial(resolve_variable, context=context, hook_description=hook_description),
+        command,
+    )


 def make_environment(current_environment, sys_module=sys):
@@ -147,6 +179,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                    shell=True,
                    environment=make_environment(os.environ),
                    working_directory=working_directory,
+                    close_fds=True,
                )
        finally:
            if original_umask:
@@ -7,19 +7,34 @@ import borgmatic.execute
 logger = logging.getLogger(__name__)


+SECRET_SERVICE_DATABASE_PATH = 'secret-service'
+
+
 def load_credential(hook_config, config, credential_parameters):
    '''
    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
    containing a KeePassXC database path and an attribute name to load, run keepassxc-cli to fetch
-    the corresponding KeePassXC credential and return it.
+    the corresponding KeePassXC credential and return it. Or use secret-tool if the database path
+    is "secret-service", indicating that KeePassXC's secret service integration should be used
+    instead.

-    Raise ValueError if keepassxc-cli can't retrieve the credential.
+    Raise ValueError if keepassxc-cli or secret-tool can't retrieve the credential.
    '''
    try:
        (database_path, attribute_name) = credential_parameters
    except ValueError:
        raise ValueError(f'Invalid KeePassXC credential parameters: {credential_parameters}')

+    if database_path == SECRET_SERVICE_DATABASE_PATH:
+        command = (
+            *shlex.split((hook_config or {}).get('secret_tool_command', 'secret-tool')),
+            *('lookup', 'Path', attribute_name),
+        )
+
+        return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+            os.linesep
+        )
+
    expanded_database_path = os.path.expanduser(database_path)

    if not os.path.exists(expanded_database_path):
@@ -29,6 +44,7 @@ def load_credential(hook_config, config, credential_parameters):
    command = (
        tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
        + ('show', '--show-protected', '--attributes', 'Password')
+        + (('--no-password',) if not (hook_config or {}).get('ask_for_password', True) else ())
        + (
            ('--key-file', hook_config['key_file'])
            if hook_config and hook_config.get('key_file')
@@ -42,4 +58,6 @@ def load_credential(hook_config, config, credential_parameters):
        + (expanded_database_path, attribute_name)  # Ensure database and entry are last.
    )

-    return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+    return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+        os.linesep
+    )
@@ -48,7 +48,9 @@ def load_credential(hook_config, config, credential_parameters):
            ),
        )

-        return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+        return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+            os.linesep
+        )

    try:
        with open(
@@ -1,12 +1,14 @@
 import contextlib
 import glob
 import importlib
+import itertools
 import json
 import logging
 import os

 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.data_source.config

 logger = logging.getLogger(__name__)

@@ -18,6 +20,32 @@ def use_streaming(hook_config, config):  # pragma: no cover
    return False


+MAXIMUM_CONFIG_SYMLINKS_TO_FOLLOW = 10
+
+
+def resolve_config_path_symlinks(path):
+    '''
+    Given a path, resolve and yield each successive symlink until the final non-symlink target. If
+    the given path isn't a symlink, then just yield it.
+
+    The purpose of this is to ensure that configuration files that are behind a symbolic link (or
+    several) actually get backed up.
+
+    Raise ValueError if we have to follow too many symlinks without getting to the final target.
+    '''
+    original_path = os.path.normpath(path)
+
+    for _ in range(MAXIMUM_CONFIG_SYMLINKS_TO_FOLLOW):
+        yield path
+
+        if not os.path.islink(path):
+            return
+
+        path = os.path.normpath(os.path.join(os.path.dirname(path), os.readlink(path)))
+
+    raise ValueError(f'Too many symlinks to follow for configuration path: {original_path}')
+
+
 def dump_data_sources(
    hook_config,
    config,
@@ -33,6 +61,9 @@ def dump_data_sources(
    the archive. But skip this if the bootstrap store_config_files option is False or if this is a
    dry run.

+    If any configuration paths are symlinks, then store each symlink along with any destination
+    paths as well.
+
    Return an empty sequence, since there are no ongoing dump processes from this hook.
    '''
    if hook_config and hook_config.get('store_config_files') is False:
@@ -44,6 +75,10 @@ def dump_data_sources(
        'manifest.json',
    )

+    resolved_config_paths = tuple(
+        itertools.chain.from_iterable(resolve_config_path_symlinks(path) for path in config_paths)
+    )
+
    if dry_run:
        return []

@@ -53,25 +88,28 @@ def dump_data_sources(
        json.dump(
            {
                'borgmatic_version': importlib.metadata.version('borgmatic'),
-                'config_paths': config_paths,
+                'config_paths': resolved_config_paths,
            },
            manifest_file,
        )

-    patterns.extend(
-        borgmatic.borg.pattern.Pattern(
-            config_path,
-            source=borgmatic.borg.pattern.Pattern_source.HOOK,
-        )
-        for config_path in config_paths
-    )
-    patterns.append(
+    borgmatic.hooks.data_source.config.inject_pattern(
+        patterns,
        borgmatic.borg.pattern.Pattern(
            os.path.join(borgmatic_runtime_directory, 'bootstrap'),
            source=borgmatic.borg.pattern.Pattern_source.HOOK,
        ),
    )

+    for config_path in resolved_config_paths:
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
+            borgmatic.borg.pattern.Pattern(
+                config_path,
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
+            ),
+        )
+
    return []


@@ -112,6 +150,10 @@ def make_data_source_dump_patterns(
    config,
    borgmatic_runtime_directory,
    name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
    '''
    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
@@ -10,6 +10,7 @@ import subprocess
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.execute
+import borgmatic.hooks.data_source.config
 import borgmatic.hooks.data_source.snapshot

 logger = logging.getLogger(__name__)
@@ -22,33 +23,24 @@ def use_streaming(hook_config, config):  # pragma: no cover
    return False


+BTRFS_SUBVOLUME_INODE_NUMBER = 256
+
+
@functools.cache
-def path_is_a_subvolume(btrfs_command, path):
+def path_is_a_subvolume(path):
    '''
-    Given a btrfs command and a path, return whether the path is a Btrfs subvolume. Return False if
-    the btrfs command errors, which probably indicates there isn't a containing Btrfs subvolume for
-    the given path.
+    Given a path, return whether it is a Btrfs subvolume. Return False if the path doesn't exist.

-    As a performance optimization, multiple calls to this function with the same arguments are
-    cached.
+    As a performance optimization, multiple calls to this function with the same path are cached.
    '''
+    if path == os.environ.get('BTRFS_TEST_SUBVOLUME_PATH'):  # pragma: no cover
+        return True
+
    try:
-        borgmatic.execute.execute_command(
-            (
-                *btrfs_command.split(' '),
-                'subvolume',
-                'show',
-                path,
-            ),
-            output_log_level=None,
-            close_fds=True,
-        )
-    # An error from the command (probably) indicates that the path is not actually a subvolume.
-    except subprocess.CalledProcessError:
+        return os.stat(path).st_ino == BTRFS_SUBVOLUME_INODE_NUMBER
+    except FileNotFoundError:
        return False

-    return True
-

@functools.cache
 def get_subvolume_property(btrfs_command, subvolume_path, property_name):
@@ -61,17 +53,19 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
    As a performance optimization, multiple calls to this function with the same arguments are
    cached.
    '''
-    output = borgmatic.execute.execute_command_and_capture_output(
-        (
-            *btrfs_command.split(' '),
-            'property',
-            'get',
-            '-t',  # Type.
-            'subvol',
-            subvolume_path,
-            property_name,
-        ),
-        close_fds=True,
+    output = '\n'.join(
+        borgmatic.execute.execute_command_and_capture_output(
+            (
+                *btrfs_command.split(' '),
+                'property',
+                'get',
+                '-t',  # Type.
+                'subvol',
+                subvolume_path,
+                property_name,
+            ),
+            close_fds=True,
+        )
    )

    try:
@@ -98,7 +92,7 @@ def get_containing_subvolume_path(btrfs_command, path):
        path,
        *tuple(str(ancestor) for ancestor in pathlib.PurePath(path).parents),
    ):
-        if not path_is_a_subvolume(btrfs_command, candidate_path):
+        if not path_is_a_subvolume(candidate_path):
            continue

        try:
@@ -145,8 +139,8 @@ Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'),
 def get_subvolumes(btrfs_command, patterns):
    '''
    Given a Btrfs command to run and a sequence of configured patterns, find the intersection
-    between the current Btrfs filesystem and subvolume paths and the paths of any patterns. The
-    idea is that these pattern paths represent the requested subvolumes to snapshot.
+    between the current Btrfs filesystem/subvolume paths and the paths of any patterns. The idea is
+    that these pattern paths represent the requested subvolumes to snapshot.

    Only include subvolumes that contain at least one root pattern sourced from borgmatic
    configuration (as opposed to generated elsewhere in borgmatic).
@@ -179,7 +173,7 @@ def get_subvolumes(btrfs_command, patterns):
    return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))


-BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot-'
+BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot'


 def make_snapshot_path(subvolume_path):
@@ -188,7 +182,7 @@ def make_snapshot_path(subvolume_path):
    '''
    return os.path.join(
        subvolume_path,
-        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        f'{BORGMATIC_SNAPSHOT_PREFIX}',
        # Included so that the snapshot ends up in the Borg archive at the "original" subvolume path.
    ) + subvolume_path.rstrip(os.path.sep)

@@ -201,16 +195,16 @@ def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
    directory within the snapshot itself. For instance, if you have a Btrfs subvolume at /mnt and
    make a snapshot of it at:

-        /mnt/.borgmatic-snapshot-1234/mnt
+        /mnt/.borgmatic-snapshot/mnt

    ... then the snapshot itself will have an empty directory at:

-        /mnt/.borgmatic-snapshot-1234/mnt/.borgmatic-snapshot-1234
+        /mnt/.borgmatic-snapshot/mnt/.borgmatic-snapshot

    So to prevent that from ending up in the Borg archive, this function produces an exclude pattern
    to exclude that path.
    '''
-    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}'

    return borgmatic.borg.pattern.Pattern(
        os.path.join(
@@ -243,7 +237,7 @@ def make_borg_snapshot_pattern(subvolume_path, pattern):

    rewritten_path = initial_caret + os.path.join(
        subvolume_path,
-        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        f'{BORGMATIC_SNAPSHOT_PREFIX}',
        # Use the Borg 1.4+ "slashdot" hack to prevent the snapshot path prefix from getting
        # included in the archive—but only if there's not already a slashdot hack present in the
        # pattern.
@@ -273,7 +267,6 @@ def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma:
            *btrfs_command.split(' '),
            'subvolume',
            'snapshot',
-            '-r',  # Read-only.
            subvolume_path,
            snapshot_path,
        ),
@@ -330,14 +323,11 @@ def dump_data_sources(

        for pattern in subvolume.contained_patterns:
            snapshot_pattern = make_borg_snapshot_pattern(subvolume.path, pattern)
+            borgmatic.hooks.data_source.config.replace_pattern(patterns, pattern, snapshot_pattern)

-            # Attempt to update the pattern in place, since pattern order matters to Borg.
-            try:
-                patterns[patterns.index(pattern)] = snapshot_pattern
-            except ValueError:
-                patterns.append(snapshot_pattern)
-
-        patterns.append(make_snapshot_exclude_pattern(subvolume.path))
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns, make_snapshot_exclude_pattern(subvolume.path)
+        )

    return []

@@ -424,6 +414,10 @@ def make_data_source_dump_patterns(
    config,
    borgmatic_runtime_directory,
    name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
    '''
    Restores aren't implemented, because stored files can be extracted directly with "extract".
@@ -3,6 +3,7 @@ import logging
 import shutil
 import subprocess

+import borgmatic.borg.pattern
 from borgmatic.execute import execute_command_and_capture_output

 IS_A_HOOK = False
@@ -71,13 +72,15 @@ def get_ip_from_container(container):
    last_error = None
    for engine in engines:
        try:
-            output = execute_command_and_capture_output(
-                (
-                    engine,
-                    'container',
-                    'inspect',
-                    '--format={{json .NetworkSettings}}',
-                    container,
+            output = '\n'.join(
+                execute_command_and_capture_output(
+                    (
+                        engine,
+                        'container',
+                        'inspect',
+                        '--format={{json .NetworkSettings}}',
+                        container,
+                    )
                )
            )
        except subprocess.CalledProcessError as error:
@@ -102,3 +105,51 @@ def get_ip_from_container(container):
    raise ValueError(
        f"Could not determine ip address for container '{container}'; running in host mode or userspace networking?"
    )
+
+
+def inject_pattern(patterns, data_source_pattern, override_excludes=True):
+    '''
+    Given a list of borgmatic.borg.pattern.Pattern instances representing the configured patterns,
+    insert the given data source pattern at the start of the list. The idea is that borgmatic is
+    injecting its own custom pattern specific to a data source hook into the user's configured
+    patterns so that the hook's data gets included in the backup.
+
+    As part of this injection, if the data source pattern is a root pattern and override_excludes is
+    True, also insert an "include" version of the given root pattern, in an attempt to preempt any
+    of the user's configured exclude patterns that may follow. The is to support use cases like
+    borgmatic injecting its own patterns for things like database dumps or bootstrap metadata, where
+    we don't want them to get accidentally excluded.
+    '''
+    if data_source_pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT and override_excludes:
+        patterns.insert(
+            0,
+            borgmatic.borg.pattern.Pattern(
+                path=data_source_pattern.path,
+                type=borgmatic.borg.pattern.Pattern_type.INCLUDE,
+                style=data_source_pattern.style,
+                device=data_source_pattern.device,
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
+            ),
+        )
+
+    patterns.insert(0, data_source_pattern)
+
+
+def replace_pattern(patterns, pattern_to_replace, data_source_pattern):
+    '''
+    Given a list of borgmatic.borg.pattern.Pattern instances representing the configured patterns,
+    replace the given pattern with the given data source pattern. The idea is that borgmatic is
+    replacing a configured pattern with its own modified pattern specific to a data source hook so
+    that the hook's data gets included in the backup.
+
+    If the pattern to replace can't be found in the given patterns, then just inject the data source
+    pattern at the start of the list.
+    '''
+    try:
+        index = patterns.index(pattern_to_replace)
+    except ValueError:
+        inject_pattern(patterns, data_source_pattern, override_excludes=False)
+
+        return
+
+    patterns[index] = data_source_pattern
--- a/Show More
+++ b/Show More