Bump version for release.

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1259
Reviewed-by: Dan Helfman <witten@torsion.org>

.eleventy.js

@@ -21,7 +21,7 @@ module.exports = function(eleventyConfig) {
             if (process.env.NODE_ENV == "production") {
                 return link;
             }
-            return link.replace('https://torsion.org/borgmatic/', 'http://localhost:8080/');
+            return link.replace('https://torsion.org/', 'http://localhost:8080/');
         }
     };
     let markdownItAnchorOptions = {
@@ -44,7 +44,7 @@ module.exports = function(eleventyConfig) {
         templateFormats: [
           "md",
           "txt"
-        ]
+        ],
     }
 };
 

.gitea/workflows/build.yaml

@@ -29,5 +29,3 @@ jobs:
           PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
       - run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
       - run: podman push "$IMAGE_NAME"
-      - run: scripts/export-docs-from-image
-      - run: curl --user "${{ secrets.REGISTRY_USERNAME }}:${{ secrets.REGISTRY_PASSWORD }}" --upload-file borgmatic-docs.tar.gz https://projects.torsion.org/api/packages/borgmatic-collective/generic/borgmatic-docs/$(head --lines=1 NEWS)/borgmatic-docs.tar.gz

NEWS

@@ -1,10 +1,172 @@
-2.0.9.dev0
+2.1.2
+ * #1231: If a source file is deleted during a "spot" check, consider the file as non-matching
+   and move on instead of immediately failing the entire check.
+ * #1250: Fix a regression in which the "--stats" flag hides statistics at default verbosity.
+ * #1251: Fix a regression in the ntfy monitoring hook in which borgmatic sends tags incorrectly,
+   resulting in "400 Bad Request" from ntfy.
+ * #1258: Fix a "codec can't decode byte" error when running commands that output multi-byte unicode
+   characters.
+ * #1260: Fix for SSH warnings from Borg showing up as JSON logs even without the "--log-json" flag.
+ * #1252: Work around Borg returning a warning exit code when a repository/archive check fails. Now,
+   borgmatic interprets such failures as errors.
+ * Deduplicate overlapping source directories and patterns so they don't throw off "spot" check file
+   counts and cause spurious check failures.
+
+2.1.1
+ * #1241: For the "recreate" action, actually pass the "--dry-run" flag through to Borg instead of
+   just skipping the Borg call.
+ * #1242: Fix a regression in which the "spot" check hung while collecting archive contents.
+ * #1244: When the "unsafe_skip_path_validation_before_create" option is enabled, don't log a
+   warning about it.
+ * #1245: Fix a regression in which the KeePassXC credential hook password prompt was invisible.
+ * #1246: Fix a regression in which the ntfy monitoring hook failed to send a ping when the
+   "priority" option was set.
+
+2.1.0
+ * TL;DR: Many logging, memory, and performance improvements. Mind those breaking changes!
+ * #485: When running commands (database clients, command hooks, etc.), elevate stderr output to
+   show up as borgmatic error logs.
+ * #858: With the "--log-json" flag, log borgmatic's own logs as JSON, not just Borg's.
+ * #1092: BREAKING: Treat most Borg warnings as errors by default, so for instance backups now fail
+   when source directories are missing. You can still override this behavior with the
+   "borg_exit_codes" option. See the documentation for more information:
+   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
+ * #1092: Deprecate the "source_directories_must_exist" option, as borgmatic now treats "backup file
+   not found" warnings from Borg as errors, which accomplishes the same thing.
+ * #1132: BREAKING/SECURITY: For the Healthchecks, Apprise, Pagerduty, and Loki monitoring hooks,
+   disable log sending when not explicitly enabled. This avoids revealing private log information to
+   third-party services. To send logs anyway, set the monitoring hook's "send_logs" option to
+   "true".
+ * #1204: When verbosity levels differ between console/monitoring/syslog/file, log Borg's output to
+   each one at a different level. Previously, it was logged at the maximum level of all the
+   verbosities.
+ * #1208: Fix for the "restore" action incorrectly extracting more database dumps than the
+   "--database" flag specifies.
+ * #1210: Fix an error when running the "spot" check or "extract" action with the "progress" option
+   or "--progress" flag.
+ * #1211: Fix an error about the runtime directory getting excluded by tweaking its logic and
+   lowering the error to a warning.
+ * #1212: Fix an error when restoring multiple directory-format database dumps at once.
+ * #1213: BREAKING: Support disabling both constant and variable interpolation by escaping with
+   backslashes. For instance, interpret "\{name\}" as literally "{name}" instead of trying to
+   resolve it as a constant/variable.
+ * #1220: Cleanup snapshots immediately after ZFS, LVM, or Btrfs hooks error—rather than waiting
+   until the next time borgmatic runs.
+ * #1221: Add an "unsafe_skip_path_validation_before_create" option to skip pre-backup safety
+   validation so as to reduce backup times on large filesystems.
+ * #1224: When running an "extract" check with the "--progress" flag, show file extraction progress.
+ * #1225: Improve performance and greatly reduce memory usage during pre-backup safety validation on
+   large filesystems.
+ * #1230: Fix another warning from LVM about leaked file descriptors, this time when calling
+   "lvcreate" from a command hook.
+ * #1234: Fix for the ntfy monitoring hook erroring on emojis in the "title" option.
+ * When syslog verbosity is enabled, log to systemd's journal (if present) with
+   structured data. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/command-line/logging/#systemd-journal
+ * SECURITY: Prevent shell injection attacks via constant interpolation in command hooks. (This was
+   already implemented for deprecated "before_*"/"after_*" command hooks.)
+ * Fix for an error in the "key import" action when importing a key from stdin.
+ * Fix the "recreate" action to include borgmatic-specific paths (database dumps, etc.) in recreated
+   archives.
+ * Update the "list" action to support the "--json" flag when the "--archive" flag is also used.
+ * Promote the ZFS, LVM, and Btrfs hooks from beta features to stable.
+
+2.0.13
+ * #1054: Allow the Btrfs hook to create and delete snapshots even when running
+   as a non-root user. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#non-root-user
+ * #1179: Add a "file_list_format" option for setting the "list" action's output format and an
+   "archive_list_format" option for setting the "repo-list" action's format.
+ * #1192: Fix for over-aggressive deduplication of source directories that contain the borgmatic
+   runtime directory, potentially resulting in data loss (data not getting backed up) when
+   snapshotting these source directories.
+ * #1192, #1163: Document potential interactions between security settings in borgmatic's sample
+   systemd service file and the ZFS, LVM, and Btrfs hooks.
+ * #1193: In the documentation for the MariaDB/MySQL database hooks, clarify how to set custom
+   command-line flags for database commands.
+ * #1193: For the MariaDB and MySQL database hooks, add a "socket_path" option for Unix socket
+   database connections.
+ * #1193: For the MariaDB and MySQL database hooks, creates a consistent snapshot by dumping all
+   tables in a single transaction.
+ * #1194: Fix for an incorrect diff command shown when running the "generate config" action with a
+   source configuration file. 
+ * #1195: Fix a regression in the ZFS, LVM, and Btrfs hooks in which snapshotted paths ignored
+   global excludes.
+ * #1201: Document a problematic interaction between borgmatic and systemd-tmpfiles:
+   https://torsion.org/borgmatic/reference/configuration/runtime-directory/#systemd-tmpfiles
+ * #1203: Fix that errors and exits when the borgmatic runtime directory is partially excluded by
+   configured excludes. Previously, borgmatic only errored when the runtime directory was completely
+   excluded.
+ * #1206: Adjust Btrfs snapshot paths so that Borg 1.x gets file cache hits when backing them up,
+   improving performance.
+ * Update the sample systemd timer with a shorter random delay when catching up on a missed run.
+
+2.0.12
+ * #1127: Fix for the database hooks not respecting the "working_directory" option.
+ * #1181: Add an "ask_for_password" option to the KeePassXC credential hook for disabling
+   KeePassXC's password prompt, e.g. if you're only using a key file to decrypt your database.
+ * #1184: Fix the fish shell completion's detection of version mismatches.
+ * #1186: Fix a regression in the Btrfs hook in which subvolume snapshots didn't get cleaned up
+   until the start of the next borgmatic run.
+ * In the SQLite database hook, run SQLite such that it exits upon encountering an error instead of,
+   you know, not doing that.
+ * Add documentation on repositories, including SSH, Rclone, S3, and B2:
+   https://torsion.org/borgmatic/reference/configuration/repositories/
+ * Improve documentation search results for individual configuration options.
+ * Add borgmatic release artifacts (wheel and tarball) to each release on the releases page:
+   https://projects.torsion.org/borgmatic-collective/borgmatic/releases
+ * Move the tarball of borgmatic's HTML documentation from the packages page to the releases page.
+
+2.0.11
+ * #957: Document borgmatic's limitations around parallelism—both its own and Borg's. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/how-to/make-per-application-backups/#limitations
+ * #1165: Fix for when the systemd service directories (RuntimeDirectory and StateDirectory) each
+   contain multiple paths.
+ * #1168: Fix for the "list", "info", and "delete" options in "extra_borg_options" being ignored
+   when "--archive" is omitted with Borg 1.x.
+ * #1169: Fix for a regression in the ZFS, LVM, and Btrfs hooks in which partial excludes of
+   snapshot paths were ignored.
+ * #1170: Fix for an inconsistent log level for Borg's last output line before exiting.
+ * #1172: Add an "environment" option to the Sentry monitoring hook.
+ * #1176: Fix the "--repository" flag not applying to command hooks.
+ * Add a "rename" option to "extra_borg_options" to support passing arbitrary flags to "borg
+   rename".
+ * Add documentation on patterns and excludes:
+   https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/
+ * Drop support for Python 3.9, which has been end-of-lifed.
+
+2.0.10
+ * #427: Expand the "extra_borg_options" option to support passing arbitrary Borg flags to every
+   Borg sub-command that borgmatic uses. As part of this, deprecate the "init" option under
+   "borg_extra_options" in favor of "repo_create".
+ * #942: Factor reference material out of the documentation how-to guides. This means there's now a
+   whole reference section in the docs! Check it out: https://torsion.org/borgmatic/
+ * #973: For the MariaDB and MySQL database hooks, add a "skip_names" option to ignore particular
+   databases when dumping "all".
+ * #1150: Fix for a runtime directory error when the "create" action is used with the "--log-json"
+   flag.
+ * #1150: Fix for a runtime directory error when the configured patterns contain a global exclude.
+ * #1161: Fix a traceback (TypeError) in the "check" action with Python 3.14.
+ * #1166: Add a "borg_key_file" option for setting the Borg repository key file path.
+ * Add documentation search.
+ * Change the URL of the local documentation development server to be more like the production URL.
+
+2.0.9
+ * #1105: More accurately collect Btrfs subvolumes to snapshot. As part of this, the Btrfs hook no
+   longer uses "findmnt" and the "findmnt_command" option is deprecated.
  * #1123: Add loading of systemd credentials even when running borgmatic outside of a systemd
    service.
  * #1149: Add support for Python 3.14.
  * #1149: Include automated tests in the source dist tarball uploaded to PyPI.
  * #1151: Fix snapshotting in the ZFS, Btrfs, and LVM hooks to play nicely with the Borg 1.4+
    "slashdot" hack within source directory paths.
+ * #1152: Fix a regression in the Loki monitoring hook in which log messages weren't sending.
+ * #1156: Fix snapshotting in the ZFS, Btrfs, and LVM hooks to snapshot both parent and child
+   volumes/filesystems instead of just the parent. As part of this fix, borgmatic no longer
+   deduplicates patterns except for those containing the borgmatic runtime directory.
+ * Fix a traceback (TypeError) regression in the "spot" check when the "local_path" option isn't
+   set.
 
 2.0.8
  * #1114: Document systemd configuration changes for the ZFS filesystem hook.

README.md

@@ -2,10 +2,7 @@
 title: borgmatic
 permalink: index.html
 ---
-
+<img src="https://torsion.org/borgmatic/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
-## It's your data. Keep it that way.
-
-<img src="docs/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
 
 borgmatic is simple, configuration-driven backup software for servers and
 workstations. Protect your files with client-side encryption. Backup your
@@ -60,45 +57,45 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 
 ### Data
 
-<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.postgresql.org/"><img src="https://torsion.org/borgmatic/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mysql.com/"><img src="https://torsion.org/borgmatic/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://mariadb.com/"><img src="https://torsion.org/borgmatic/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.mongodb.com/"><img src="https://torsion.org/borgmatic/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sqlite.org/"><img src="https://torsion.org/borgmatic/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://openzfs.org/"><img src="https://torsion.org/borgmatic/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="https://torsion.org/borgmatic/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="https://torsion.org/borgmatic/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://rclone.org"><img src="https://torsion.org/borgmatic/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="https://torsion.org/borgmatic/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ### Monitoring
 
-<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://healthchecks.io/"><img src="https://torsion.org/borgmatic/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://uptime.kuma.pet/"><img src="https://torsion.org/borgmatic/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronitor.io/"><img src="https://torsion.org/borgmatic/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://cronhub.io/"><img src="https://torsion.org/borgmatic/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pagerduty.com/"><img src="https://torsion.org/borgmatic/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.pushover.net/"><img src="docs/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pushover.net/"><img src="https://torsion.org/borgmatic/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://ntfy.sh/"><img src="https://torsion.org/borgmatic/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://grafana.com/oss/loki/"><img src="https://torsion.org/borgmatic/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://github.com/caronc/apprise/wiki"><img src="https://torsion.org/borgmatic/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.zabbix.com/"><img src="https://torsion.org/borgmatic/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="https://torsion.org/borgmatic/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ### Credentials
 
-<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://systemd.io/"><img src="https://torsion.org/borgmatic/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="https://torsion.org/borgmatic/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="https://torsion.org/borgmatic/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="https://torsion.org/borgmatic/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ## Getting started
 
 Your first step is to [install and configure
-borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/).
+borgmatic](https://torsion.org/borgmatic/how-to/set-up-backups/).
 
 For additional documentation, check out the links above (left panel on wide screens)
 for <a href="https://torsion.org/borgmatic/#documentation">borgmatic how-to and
@@ -132,7 +129,7 @@ first. If you prefer to use an existing GitHub account, you can skip account
 creation and [login directly](https://projects.torsion.org/user/login).
 
 Also see the [security
-policy](https://torsion.org/borgmatic/docs/security-policy/) for any security
+policy](https://torsion.org/borgmatic/security-policy/) for any security
 issues.
 
 
@@ -177,7 +174,7 @@ discuss your idea. Note that you'll need to
 first. In general, contributions are very welcome. We don't bite!
 
 Also, please check out the [borgmatic development
-how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
+how-to](https://torsion.org/borgmatic/how-to/develop-on-borgmatic/) for
 info on cloning source code, running tests, etc.
 
 ### Recent contributors
@@ -186,4 +183,4 @@ Thanks to all borgmatic contributors! There are multiple ways to contribute to
 this project, so the following includes those who have fixed bugs, contributed
 features, *or* filed tickets.
 
-{% include borgmatic/contributors.html %}
+{% include borgmatic/contributors.html %}

borgmatic/actions/borg.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.borg.borg
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -19,26 +18,22 @@ def run_borg(
     '''
     Run the "borg" action for the given repository.
     '''
-    if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Running arbitrary Borg command')
-        repository,
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-        borg_arguments.repository,
+        repository['path'],
-    ):
+        borg_arguments.archive,
-        logger.info('Running arbitrary Borg command')
+        config,
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        local_borg_version,
-            repository['path'],
+        global_arguments,
-            borg_arguments.archive,
+        local_path,
-            config,
+        remote_path,
-            local_borg_version,
+    )
-            global_arguments,
+    borgmatic.borg.borg.run_arbitrary_borg(
-            local_path,
+        repository['path'],
-            remote_path,
+        config,
-        )
+        local_borg_version,
-        borgmatic.borg.borg.run_arbitrary_borg(
+        options=borg_arguments.options,
-            repository['path'],
+        archive=archive_name,
-            config,
+        local_path=local_path,
-            local_borg_version,
+        remote_path=remote_path,
-            options=borg_arguments.options,
+    )
-            archive=archive_name,
-            local_path=local_path,
-            remote_path=remote_path,
-        )

borgmatic/actions/break_lock.py

@@ -1,7 +1,6 @@
 import logging
 
 import borgmatic.borg.break_lock
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -18,16 +17,12 @@ def run_break_lock(
     '''
     Run the "break-lock" action for the given repository.
     '''
-    if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Breaking repository and cache locks')
-        repository,
+    borgmatic.borg.break_lock.break_lock(
-        break_lock_arguments.repository,
+        repository['path'],
-    ):
+        config,
-        logger.info('Breaking repository and cache locks')
+        local_borg_version,
-        borgmatic.borg.break_lock.break_lock(
+        global_arguments,
-            repository['path'],
+        local_path=local_path,
-            config,
+        remote_path=remote_path,
-            local_borg_version,
+    )
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )

borgmatic/actions/change_passphrase.py

@@ -1,7 +1,6 @@
 import logging
 
 import borgmatic.borg.change_passphrase
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -18,20 +17,13 @@ def run_change_passphrase(
     '''
     Run the "key change-passphrase" action for the given repository.
     '''
-    if (
+    logger.info('Changing repository passphrase')
-        change_passphrase_arguments.repository is None
+    borgmatic.borg.change_passphrase.change_passphrase(
-        or borgmatic.config.validate.repositories_match(
+        repository['path'],
-            repository,
+        config,
-            change_passphrase_arguments.repository,
+        local_borg_version,
-        )
+        change_passphrase_arguments,
-    ):
+        global_arguments,
-        logger.info('Changing repository passphrase')
+        local_path=local_path,
-        borgmatic.borg.change_passphrase.change_passphrase(
+        remote_path=remote_path,
-            repository['path'],
+    )
-            config,
-            local_borg_version,
-            change_passphrase_arguments,
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )

borgmatic/actions/check.py

@@ -9,6 +9,7 @@ import pathlib
 import random
 import shlex
 import shutil
+import subprocess
 import textwrap
 
 import borgmatic.actions.config.bootstrap
@@ -20,11 +21,8 @@ import borgmatic.borg.extract
 import borgmatic.borg.list
 import borgmatic.borg.pattern
 import borgmatic.borg.repo_list
-import borgmatic.borg.state
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.execute
-import borgmatic.hooks.command
 
 DEFAULT_CHECKS = (
     {'name': 'repository', 'frequency': '1 month'},
@@ -246,7 +244,7 @@ def write_check_time(path):  # pragma: no cover
     logger.debug(f'Writing check time at {path}')
 
     os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
-    pathlib.Path(path, mode=0o600).touch()
+    pathlib.Path(path).touch(mode=0o600)
 
 
 def read_check_time(path):
@@ -383,10 +381,7 @@ def collect_spot_check_source_paths(
         dry_run=True,
         repository_path=repository['path'],
         # Omit "progress" because it interferes with "list_details".
-        config=dict(
+        config=dict(config, progress=False, list_details=True),
-            {option: value for option, value in config.items() if option != 'progress'},
-            list_details=True,
-        ),
         patterns=borgmatic.actions.pattern.process_patterns(
             borgmatic.actions.pattern.collect_patterns(config)
             + tuple(
@@ -408,7 +403,7 @@ def collect_spot_check_source_paths(
     )
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    paths_output = borgmatic.execute.execute_command_and_capture_output(
+    path_lines = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
         environment=borgmatic.borg.environment.make_environment(config),
@@ -417,14 +412,20 @@ def collect_spot_check_source_paths(
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
-    paths = tuple(
+    paths = (
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.splitlines()
+        for path_line in path_lines
         if path_line and path_line.startswith(('- ', '+ '))
     )
 
     return tuple(
-        path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+        # Use dict.fromkeys() to deduplicate file paths, which are present in Borg's dry run output
+        # when there are overlapping source patterns. For instance, if both "/foo" and
+        # "/foo/file.txt" are in configured patterns, then "/foo/file.txt" will show up in Borg's
+        # dry run output twice.
+        dict.fromkeys(
+            path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+        )
     )
 
 
@@ -454,24 +455,22 @@ def collect_spot_check_archive_paths(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return tuple(
-        path
+        entry['path']
-        for line in borgmatic.borg.list.capture_archive_listing(
+        for entry in borgmatic.borg.list.capture_archive_listing(
             repository['path'],
             archive,
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} {path}{NUL}',
             local_path=local_path,
             remote_path=remote_path,
         )
-        for (file_type, path) in (line.split(' ', 1),)
+        if entry['type'] not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
-        if file_type not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
+        if pathlib.Path('borgmatic') not in pathlib.Path(entry['path']).parents
-        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
         if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
         if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
     )
 
 
@@ -529,48 +528,91 @@ def compare_spot_check_hashes(
         if not source_sample_paths_subset:
             break
 
-        hash_output = borgmatic.execute.execute_command_and_capture_output(
+        hash_paths = tuple(
-            tuple(
+            path for path in source_sample_paths_subset if path in hashable_source_sample_path
-                shlex.quote(part)
-                for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
-            )
-            + tuple(
-                path for path in source_sample_paths_subset if path in hashable_source_sample_path
-            ),
-            working_directory=working_directory,
         )
 
-        source_hashes.update(
+        try:
-            **dict(
+            hash_lines = borgmatic.execute.execute_command_and_capture_output(
-                (reversed(line.split('  ', 1)) for line in hash_output.splitlines()),
+                tuple(
-                # Represent non-existent files as having empty hashes so the comparison below still
+                    shlex.quote(part)
-                # works. Same thing for filesystem links, since Borg produces empty archive hashes
+                    for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
-                # for them.
+                )
-                **{
+                + hash_paths,
-                    path: ''
+                working_directory=working_directory,
-                    for path in source_sample_paths_subset
+            )
-                    if path not in hashable_source_sample_path
+            source_hashes.update(
-                },
+                **dict(
-            ),
+                    zip(
-        )
+                        # xxh64sum rewrites/escapes the paths that it returns alongside its hashes, for
+                        # instance if they contain special characters. When that happens, they don't
+                        # match the original source paths and therefore hash lookups fail. So when
+                        # building this lookup dict, use the original unaltered paths we provided as
+                        # input to xxh64sum.
+                        hash_paths,
+                        (
+                            # For some reason, xxh64sum prefixes the hash with a backslash if the path
+                            # contains a newline. Work around that.
+                            line.split('  ', 1)[0].lstrip('\\')
+                            for line in hash_lines
+                        ),
+                    ),
+                    # Represent non-existent files as having empty hashes so the comparison below still
+                    # works. Same thing for filesystem links, since Borg produces empty archive hashes
+                    # for them.
+                    **{
+                        path: ''
+                        for path in source_sample_paths_subset
+                        if path not in hashable_source_sample_path
+                    },
+                ),
+            )
+        except subprocess.CalledProcessError:
+            # This can happen if a file we planned to hash gets deleted right before we try to hash
+            # it. Falling back to individual file hashing allows us to find and mark just the
+            # file(s) with problems instead of failing the whole batch.
+            logger.warning(
+                'Bulk source path hashing failed for this batch; falling back to individual file hashing'
+            )
+
+            for hash_path in hash_paths:
+                try:
+                    hash_lines = borgmatic.execute.execute_command_and_capture_output(
+                        (
+                            *(
+                                shlex.quote(part)
+                                for part in shlex.split(
+                                    spot_check_config.get('xxh64sum_command', 'xxh64sum')
+                                )
+                            ),
+                            hash_path,
+                        ),
+                        working_directory=working_directory,
+                    )
+                    source_hashes[hash_path] = next(hash_lines).split('  ', 1)[0].lstrip('\\')
+                except (subprocess.CalledProcessError, StopIteration):  # noqa: PERF203
+                    logger.warning(
+                        f'Source path hashing failed for {hash_path}; treating as missing'
+                    )
+                    source_hashes[hash_path] = ''
 
         # Get the hash for each file in the archive.
         archive_hashes.update(
-            **dict(
+            **{
-                reversed(line.split(' ', 1))
+                entry['path']: entry['xxh64']
-                for line in borgmatic.borg.list.capture_archive_listing(
+                for entry in borgmatic.borg.list.capture_archive_listing(
                     repository['path'],
                     archive,
                     config,
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} {path}{NUL}',
+                    path_format='{xxh64}{path}',
                     local_path=local_path,
                     remote_path=remote_path,
                 )
-                if line
+                if entry
-            ),
+            },
         )
 
     # Compare the source hashes with the archive hashes to see how many match.
@@ -608,8 +650,6 @@ def spot_check(
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    logger.debug('Running spot check')
-
     try:
         spot_check_config = next(
             check for check in config.get('checks', ()) if check.get('name') == 'spot'
@@ -750,12 +790,6 @@ def run_check(
 
     Raise ValueError if the Borg repository ID cannot be determined.
     '''
-    if check_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        check_arguments.repository,
-    ):
-        return
-
     logger.info('Running consistency checks')
 
     repository_id = borgmatic.borg.check.get_repository_id(
@@ -800,6 +834,7 @@ def run_check(
             write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))
 
     if 'extract' in checks:
+        logger.info('Running extract check')
         borgmatic.borg.extract.extract_last_archive_dry_run(
             config,
             local_borg_version,
@@ -812,6 +847,7 @@ def run_check(
         write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
+        logger.info('Running spot check')
         with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
             spot_check(
                 repository,

borgmatic/actions/compact.py

@@ -2,8 +2,6 @@ import logging
 
 import borgmatic.borg.compact
 import borgmatic.borg.feature
-import borgmatic.config.validate
-import borgmatic.hooks.command
 
 logger = logging.getLogger(__name__)
 
@@ -22,12 +20,6 @@ def run_compact(
     '''
     Run the "compact" action for the given repository.
     '''
-    if compact_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        compact_arguments.repository,
-    ):
-        return
-
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
         logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(

borgmatic/actions/config/bootstrap.py

@@ -5,8 +5,6 @@ import os
 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
-import borgmatic.hooks.command
 
 logger = logging.getLogger(__name__)
 
@@ -66,7 +64,7 @@ def load_config_paths_from_archive(
             config,
             local_borg_version,
             global_arguments,
-            local_path=config.get('local_path'),
+            local_path=config.get('local_path', 'borg'),
             remote_path=config.get('remote_path'),
             extract_to_stdout=True,
         )
@@ -129,8 +127,6 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
         bootstrap_arguments.repository,
         archive_name,
         [config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
-        # Only add progress here and not the extract_archive() call above, because progress
-        # conflicts with extract_to_stdout.
         dict(config, progress=bootstrap_arguments.progress or False),
         local_borg_version,
         global_arguments,

borgmatic/actions/config/generate.py

@@ -19,26 +19,33 @@ def run_generate(generate_arguments, global_arguments):
     dry_run_label = ' (dry run; not actually writing anything)' if global_arguments.dry_run else ''
 
     logger.answer(
-        f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}',
+        f'Generating configuration files within: {generate_arguments.destination_path}{dry_run_label}'
+        if generate_arguments.split
+        else f'Generating a configuration file at: {generate_arguments.destination_path}{dry_run_label}'
     )
 
     borgmatic.config.generate.generate_sample_configuration(
         global_arguments.dry_run,
         generate_arguments.source_filename,
-        generate_arguments.destination_filename,
+        generate_arguments.destination_path,
         borgmatic.config.validate.schema_filename(),
         overwrite=generate_arguments.overwrite,
+        split=generate_arguments.split,
     )
 
     if generate_arguments.source_filename:
         logger.answer(
             f'''
-Merged in the contents of configuration file at: {generate_arguments.source_filename}
+Merged the contents of the source configuration file: {generate_arguments.source_filename}'''
-To review the changes made, run:
-
-    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}''',
         )
 
+        if not generate_arguments.split:
+            logger.answer(
+                f'''To review the changes made, run:
+
+    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_path}''',
+            )
+
     logger.answer(
         '''
 This includes all available configuration options with example values, the few

borgmatic/actions/create.py

@@ -1,12 +1,12 @@
 import logging
 
+import borgmatic.actions.dump
 import borgmatic.actions.json
 import borgmatic.borg.create
 import borgmatic.borg.feature
 import borgmatic.borg.rename
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.hooks.dispatch
 from borgmatic.actions import pattern
 
@@ -30,12 +30,6 @@ def run_create(
 
     If create_arguments.json is True, yield the JSON output from creating the archive.
     '''
-    if create_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        create_arguments.repository,
-    ):
-        return
-
     if config.get('list_details') and config.get('progress'):
         raise ValueError(
             'With the create action, only one of --list/--files/list_details and --progress/progress can be used.',
@@ -50,96 +44,93 @@ def run_create(
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            global_arguments.dry_run,
-        )
         patterns = pattern.process_patterns(
             pattern.collect_patterns(config),
             config,
             working_directory,
-        )
-        active_dumps = borgmatic.hooks.dispatch.call_hooks(
-            'dump_data_sources',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            config_paths,
             borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
         )
 
-        # Process the patterns again in case any data source hooks updated them. Without this step,
+        original_patterns = list(patterns)
-        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
-        # the same named pipe twice.
-        patterns = pattern.process_patterns(
-            patterns,
-            config,
-            working_directory,
-            skip_expand_paths=config_paths,
-        )
-        stream_processes = [process for processes in active_dumps.values() for process in processes]
 
-        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+        # Use the original patterns so as to disregard any modifications made by any data source
-        # to make sure we only create a real archive if all the streaming processes completed
+        # hooks, e.g. via dump_data_sources() below.
-        # successfully (create_archive will fail if a streaming process fails, but the archive might
+        with borgmatic.actions.dump.Dump_cleanup(
-        # have already been created at this point).
+            config, borgmatic_runtime_directory, original_patterns, global_arguments.dry_run
-        use_checkpoint = bool(stream_processes)
+        ):
-
+            active_dumps = borgmatic.hooks.dispatch.call_hooks(
-        json_output = borgmatic.borg.create.create_archive(
+                'dump_data_sources',
-            global_arguments.dry_run,
-            repository['path'],
-            config,
-            patterns,
-            local_borg_version,
-            global_arguments,
-            borgmatic_runtime_directory,
-            archive_suffix='.checkpoint' if use_checkpoint else '',
-            local_path=local_path,
-            remote_path=remote_path,
-            json=create_arguments.json,
-            comment=create_arguments.comment,
-            stream_processes=stream_processes,
-        )
-
-        if use_checkpoint:
-            rename_checkpoint_archive(
-                repository['path'],
-                global_arguments,
                 config,
+                borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                config_paths,
+                borgmatic_runtime_directory,
+                patterns,
+                global_arguments.dry_run,
+            )
+
+            # Process the patterns again in case any data source hooks updated them. Without this step,
+            # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+            # the same named pipe twice.
+            patterns = pattern.process_patterns(
+                patterns,
+                config,
+                working_directory,
+                borgmatic_runtime_directory,
+                skip_expand_paths=config_paths,
+            )
+            stream_processes = [
+                process for processes in active_dumps.values() for process in processes
+            ]
+
+            # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+            # to make sure we only create a real archive if all the streaming processes completed
+            # successfully (create_archive will fail if a streaming process fails, but the archive might
+            # have already been created at this point).
+            use_checkpoint = bool(stream_processes)
+
+            json_output = borgmatic.borg.create.create_archive(
+                global_arguments.dry_run,
+                repository['path'],
+                config,
+                patterns,
                 local_borg_version,
-                local_path,
+                global_arguments,
-                remote_path,
+                borgmatic_runtime_directory,
+                archive_suffix='.checkpoint' if use_checkpoint else '',
+                local_path=local_path,
+                remote_path=remote_path,
+                json=create_arguments.json,
+                comment=create_arguments.comment,
+                stream_processes=stream_processes,
             )
 
-        if json_output:
-            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
             if use_checkpoint:
-                # Patch archive name and ID
+                rename_checkpoint_archive(
-                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
                     repository['path'],
+                    global_arguments,
                     config,
                     local_borg_version,
-                    global_arguments,
                     local_path,
                     remote_path,
                 )
 
-                output['archive']['name'] = renamed_archive['name']
+            if json_output:
-                output['archive']['id'] = renamed_archive['id']
+                output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+                if use_checkpoint:
+                    # Patch archive name and ID
+                    renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                        repository['path'],
+                        config,
+                        local_borg_version,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
 
-            yield output
+                    output['archive']['name'] = renamed_archive['name']
+                    output['archive']['id'] = renamed_archive['id']
 
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+                yield output
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            global_arguments.dry_run,
-        )
 
 
 def rename_checkpoint_archive(

borgmatic/actions/delete.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.actions.arguments
 import borgmatic.borg.delete
-import borgmatic.borg.repo_delete
 import borgmatic.borg.repo_list
 
 logger = logging.getLogger(__name__)
@@ -20,32 +19,28 @@ def run_delete(
     '''
     Run the "delete" action for the given repository and archive(s).
     '''
-    if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.answer('Deleting archives')
-        repository,
-        delete_arguments.repository,
-    ):
-        logger.answer('Deleting archives')
 
-        archive_name = (
+    archive_name = (
-            borgmatic.borg.repo_list.resolve_archive_name(
+        borgmatic.borg.repo_list.resolve_archive_name(
-                repository['path'],
+            repository['path'],
-                delete_arguments.archive,
+            delete_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            )
-            if delete_arguments.archive
-            else None
-        )
-
-        borgmatic.borg.delete.delete_archives(
-            repository,
             config,
             local_borg_version,
-            borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
             global_arguments,
             local_path,
             remote_path,
         )
+        if delete_arguments.archive
+        else None
+    )
+
+    borgmatic.borg.delete.delete_archives(
+        repository,
+        config,
+        local_borg_version,
+        borgmatic.actions.arguments.update_arguments(delete_arguments, archive=archive_name),
+        global_arguments,
+        local_path,
+        remote_path,
+    )

borgmatic/actions/dump.py

@@ -0,0 +1,53 @@
+import borgmatic.hooks.dispatch
+
+
+class Dump_cleanup:
+    '''
+    A Python context manager for removing data source dumps before and after wrapped code. Besides
+    doing this for general filesystem cleanliness reasons, leaving old dumps lying around can
+    prevent future dumping from working (in the case of filesystem snapshots) or cause Borg hangs
+    (in the case of database dump named pipes).
+
+    Example use as a context manager:
+
+        with borgmatic.actions.dump.Dump_cleanup(
+            config, borgmatic_runtime_directory, patterns, dry_run,
+        ):
+            do_something_like_perform_a_dump_or_restore()
+    '''
+
+    def __init__(self, config, borgmatic_runtime_directory, patterns, dry_run):
+        '''
+        Given a configuration dict, the borgmatic runtime directory, the configured patterns, and
+        whether this is a dry-run, store these values for use below.
+        '''
+        self.config = config
+        self.borgmatic_runtime_directory = borgmatic_runtime_directory
+        self.patterns = patterns
+        self.dry_run = dry_run
+
+    def __enter__(self):
+        '''
+        Remove all data source dumps that exist prior to the wrapped code running.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Remove all data source dumps, including any created by the wrapped code.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )

borgmatic/actions/export_key.py

@@ -1,7 +1,6 @@
 import logging
 
 import borgmatic.borg.export_key
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -18,17 +17,13 @@ def run_export_key(
     '''
     Run the "key export" action for the given repository.
     '''
-    if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Exporting repository key')
-        repository,
+    borgmatic.borg.export_key.export_key(
-        export_arguments.repository,
+        repository['path'],
-    ):
+        config,
-        logger.info('Exporting repository key')
+        local_borg_version,
-        borgmatic.borg.export_key.export_key(
+        export_arguments,
-            repository['path'],
+        global_arguments,
-            config,
+        local_path=local_path,
-            local_borg_version,
+        remote_path=remote_path,
-            export_arguments,
+    )
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )

borgmatic/actions/export_tar.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.borg.export_tar
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -19,30 +18,26 @@ def run_export_tar(
     '''
     Run the "export-tar" action for the given repository.
     '''
-    if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
-        repository,
+    borgmatic.borg.export_tar.export_tar_archive(
-        export_tar_arguments.repository,
+        global_arguments.dry_run,
-    ):
+        repository['path'],
-        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
+        borgmatic.borg.repo_list.resolve_archive_name(
-        borgmatic.borg.export_tar.export_tar_archive(
-            global_arguments.dry_run,
             repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            export_tar_arguments.archive,
-                repository['path'],
-                export_tar_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            export_tar_arguments.paths,
-            export_tar_arguments.destination,
             config,
             local_borg_version,
             global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-            tar_filter=export_tar_arguments.tar_filter,
+        ),
-            strip_components=export_tar_arguments.strip_components,
+        export_tar_arguments.paths,
-        )
+        export_tar_arguments.destination,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+        tar_filter=export_tar_arguments.tar_filter,
+        strip_components=export_tar_arguments.strip_components,
+    )

borgmatic/actions/extract.py

@@ -2,8 +2,6 @@ import logging
 
 import borgmatic.borg.extract
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
-import borgmatic.hooks.command
 
 logger = logging.getLogger(__name__)
 
@@ -21,29 +19,25 @@ def run_extract(
     '''
     Run the "extract" action for the given repository.
     '''
-    if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info(f'Extracting archive {extract_arguments.archive}')
-        repository,
+    borgmatic.borg.extract.extract_archive(
-        extract_arguments.repository,
+        global_arguments.dry_run,
-    ):
+        repository['path'],
-        logger.info(f'Extracting archive {extract_arguments.archive}')
+        borgmatic.borg.repo_list.resolve_archive_name(
-        borgmatic.borg.extract.extract_archive(
-            global_arguments.dry_run,
             repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            extract_arguments.archive,
-                repository['path'],
-                extract_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            extract_arguments.paths,
             config,
             local_borg_version,
             global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-            destination_path=extract_arguments.destination,
+        ),
-            strip_components=extract_arguments.strip_components,
+        extract_arguments.paths,
-        )
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+        destination_path=extract_arguments.destination,
+        strip_components=extract_arguments.strip_components,
+    )

borgmatic/actions/import_key.py

@@ -1,7 +1,6 @@
 import logging
 
 import borgmatic.borg.import_key
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -18,17 +17,13 @@ def run_import_key(
     '''
     Run the "key import" action for the given repository.
     '''
-    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.info('Importing repository key')
-        repository,
+    borgmatic.borg.import_key.import_key(
-        import_arguments.repository,
+        repository['path'],
-    ):
+        config,
-        logger.info('Importing repository key')
+        local_borg_version,
-        borgmatic.borg.import_key.import_key(
+        import_arguments,
-            repository['path'],
+        global_arguments,
-            config,
+        local_path=local_path,
-            local_borg_version,
+        remote_path=remote_path,
-            import_arguments,
+    )
-            global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )

borgmatic/actions/info.py

@@ -4,7 +4,6 @@ import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.info
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -23,30 +22,26 @@ def run_info(
 
     If info_arguments.json is True, yield the JSON output from the info for the archive.
     '''
-    if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not info_arguments.json:
-        repository,
+        logger.answer('Displaying archive summary information')
-        info_arguments.repository,
-    ):
-        if not info_arguments.json:
-            logger.answer('Displaying archive summary information')
 
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
+        repository['path'],
-            info_arguments.archive,
+        info_arguments.archive,
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        json_output = borgmatic.borg.info.display_archives_info(
+    json_output = borgmatic.borg.info.display_archives_info(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
+        borgmatic.actions.arguments.update_arguments(info_arguments, archive=archive_name),
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/list.py

@@ -3,7 +3,6 @@ import logging
 import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -22,33 +21,29 @@ def run_list(
 
     If list_arguments.json is True, yield the JSON output from listing the archive.
     '''
-    if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not list_arguments.json:
-        repository,
+        if list_arguments.find_paths:  # pragma: no cover
-        list_arguments.repository,
+            logger.answer('Searching archives')
-    ):
+        elif not list_arguments.archive:  # pragma: no cover
-        if not list_arguments.json:
+            logger.answer('Listing archives')
-            if list_arguments.find_paths:  # pragma: no cover
-                logger.answer('Searching archives')
-            elif not list_arguments.archive:  # pragma: no cover
-                logger.answer('Listing archives')
 
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            repository['path'],
+        repository['path'],
-            list_arguments.archive,
+        list_arguments.archive,
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        json_output = borgmatic.borg.list.list_archive(
+    json_output = borgmatic.borg.list.list_archive(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
+        borgmatic.actions.arguments.update_arguments(list_arguments, archive=archive_name),
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/mount.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.borg.mount
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -19,30 +18,26 @@ def run_mount(
     '''
     Run the "mount" action for the given repository.
     '''
-    if mount_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if mount_arguments.archive:
-        repository,
+        logger.info(f'Mounting archive {mount_arguments.archive}')
-        mount_arguments.repository,
+    else:  # pragma: nocover
-    ):
+        logger.info('Mounting repository')
-        if mount_arguments.archive:
-            logger.info(f'Mounting archive {mount_arguments.archive}')
-        else:  # pragma: nocover
-            logger.info('Mounting repository')
 
-        borgmatic.borg.mount.mount_archive(
+    borgmatic.borg.mount.mount_archive(
+        repository['path'],
+        borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
-            borgmatic.borg.repo_list.resolve_archive_name(
+            mount_arguments.archive,
-                repository['path'],
-                mount_arguments.archive,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-            ),
-            mount_arguments,
             config,
             local_borg_version,
             global_arguments,
-            local_path=local_path,
+            local_path,
-            remote_path=remote_path,
+            remote_path,
-        )
+        ),
+        mount_arguments,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
+    )

borgmatic/actions/pattern.py

@@ -227,47 +227,69 @@ def device_map_patterns(patterns, working_directory=None):
     )
 
 
-def deduplicate_patterns(patterns, config):
+def deduplicate_runtime_directory_patterns(patterns, config, borgmatic_runtime_directory=None):
     '''
-    Given a sequence of borgmatic.borg.pattern.Pattern instances and a configuration dict, return
+    Given a sequence of borgmatic.borg.pattern.Pattern instances, the borgmatic runtime directory,
-    them with all duplicate root child patterns removed. For instance, if two root patterns are
+    and a configuration dict, return them without any duplicate root child patterns that contain the
-    given with paths "/foo" and "/foo/bar", return just the one with "/foo". Non-root patterns are
+    runtime directory. For instance, if two root patterns are given with paths "/foo" and
-    passed through without modification.
+    "/foo/bar", and the runtime directory is "/foo/bar", return just the "/foo" pattern. Non-root
+    patterns and patterns not containing the runtime directory are passed through without
+    modification.
 
-    The one exception to deduplication is if two paths are on different filesystems (devices) and
+    One exception to deduplication is if two paths are on different filesystems (devices) and
     "one_file_system" is True in the given configuration. In that case, the paths won't get
     deduplicated, because Borg won't cross filesystem boundaries when "one_file_system" is True.
 
-    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    The idea is that if Borg is given a root parent pattern containing the borgmatic runtime
-    child patterns, because it will naturally spider the contents of the parent pattern's path. And
+    directory, then Borg doesn't also need to be given child patterns, because it will naturally
-    there are cases where Borg coming across the same file twice will result in duplicate reads and
+    spider the contents of the parent pattern's path. Additionally, there are cases where Borg
-    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
+    coming across the same file twice will result in duplicate reads and even hangs, e.g. when a
-    Borg.
+    database hook in the borgmatic runtime directory is using a named pipe for streaming database
+    dumps to Borg.
+
+    This deduplication is limited to the borgmatic runtime directory (where borgmatic's named pipes
+    exist), because there are other legitimate use cases for parent and child patterns to both exist
+    in patterns. For instance, with some snapshotted filesystems, snapshots don't traverse from a
+    parent filesystem to a child and therefore both need to remain in patterns.
+
+    And for the case of named pipes outside of the borgmatic runtime directory, there is code
+    elsewhere (in the "create" action) that auto-excludes special files to prevent Borg hangs.
     '''
+    if borgmatic_runtime_directory is None:
+        return patterns
+
     deduplicated = {}  # Use just the keys as an ordered set.
+    runtime_directory_parents = set(pathlib.PurePath(borgmatic_runtime_directory).parents).union(
+        {pathlib.PurePath(borgmatic_runtime_directory)}
+    )
 
     for pattern in patterns:
         if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
             deduplicated[pattern] = True
             continue
 
-        parents = pathlib.PurePath(pattern.path).parents
+        pattern_parents = pathlib.PurePath(pattern.path).parents
 
-        # If another directory in the given list is a parent of current directory (even n levels up)
+        # If:
-        # and both are on the same filesystem (or one_file_system is not set), then the current
+        #
-        # directory is a duplicate.
+        #   1. another pattern is a parent of the current pattern (even n levels up),
+        #   2. both patterns are parents of the runtime directory (even n levels up),
+        #   3. and both patterns are on the same filesystem (or one_file_system is not set)
+        #
+        # ... then consider the current pattern as a duplicate.
         for other_pattern in patterns:
             if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
                 continue
 
-            if any(
+            device_matches = pattern.device is not None and (
-                pathlib.PurePath(other_pattern.path) == parent
+                other_pattern.device == pattern.device or config.get('one_file_system') is not True
-                and pattern.device is not None
+            )
-                and (
+
-                    other_pattern.device == pattern.device
+            if (
-                    or config.get('one_file_system') is not True
+                pathlib.PurePath(other_pattern.path) in pattern_parents
-                )
+                and pathlib.PurePosixPath(other_pattern.path) in runtime_directory_parents
-                for parent in parents
+                and pathlib.PurePosixPath(pattern.path) in runtime_directory_parents
+                and device_matches
             ):
                 break
         else:
@@ -276,16 +298,22 @@ def deduplicate_patterns(patterns, config):
     return tuple(deduplicated.keys())
 
 
-def process_patterns(patterns, config, working_directory, skip_expand_paths=None):
+def process_patterns(
+    patterns, config, working_directory, borgmatic_runtime_directory=None, skip_expand_paths=None
+):
     '''
-    Given a sequence of Borg patterns, a configuration dict, a configured working directory, and a
+    Given a sequence of Borg patterns, a configuration dict, a configured working directory, the
-    sequence of paths to skip path expansion for, expand and deduplicate any "root" patterns,
+    borgmatic runtime directory, and a sequence of paths to skip path expansion for, expand and
-    returning the resulting root and non-root patterns as a list.
+    deduplicate any "root" patterns, returning the resulting root and non-root patterns as a list.
+
+    If the borgmatic runtime directory is None, then don't deduplicate patterns. Deduplication is
+    really only necessary for the "create" action when the runtime directory might contain named
+    pipes for database dumps.
     '''
     skip_paths = set(skip_expand_paths or ())
 
     return list(
-        deduplicate_patterns(
+        deduplicate_runtime_directory_patterns(
             device_map_patterns(
                 expand_patterns(
                     patterns,
@@ -294,5 +322,6 @@ def process_patterns(patterns, config, working_directory, skip_expand_paths=None
                 ),
             ),
             config,
+            borgmatic_runtime_directory,
         ),
     )

borgmatic/actions/prune.py

@@ -1,8 +1,6 @@
 import logging
 
 import borgmatic.borg.prune
-import borgmatic.config.validate
-import borgmatic.hooks.command
 
 logger = logging.getLogger(__name__)
 
@@ -21,12 +19,6 @@ def run_prune(
     '''
     Run the "prune" action for the given repository.
     '''
-    if prune_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        prune_arguments.repository,
-    ):
-        return
-
     logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,

borgmatic/actions/recreate.py

@@ -1,11 +1,10 @@
 import logging
 import subprocess
 
-import borgmatic.borg.info
+import borgmatic.actions.pattern
+import borgmatic.borg.pattern
 import borgmatic.borg.recreate
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
-from borgmatic.actions.pattern import collect_patterns, process_patterns
 
 logger = logging.getLogger(__name__)
 
@@ -19,70 +18,75 @@ def run_recreate(
     local_borg_version,
     recreate_arguments,
     global_arguments,
+    dry_run_label,
     local_path,
     remote_path,
 ):
     '''
     Run the "recreate" action for the given repository.
     '''
-    if recreate_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if recreate_arguments.archive:
-        repository,
+        logger.answer(f'Recreating archive {recreate_arguments.archive}{dry_run_label}')
-        recreate_arguments.repository,
+    else:
-    ):
+        logger.answer(f'Recreating repository{dry_run_label}')
-        if recreate_arguments.archive:
-            logger.answer(f'Recreating archive {recreate_arguments.archive}')
-        else:
-            logger.answer('Recreating repository')
 
-        # Collect and process patterns.
+    # Collect and process patterns.
-        processed_patterns = process_patterns(
+    processed_patterns = borgmatic.actions.pattern.process_patterns(
-            collect_patterns(config),
+        (
-            config,
+            *borgmatic.actions.pattern.collect_patterns(config),
-            borgmatic.config.paths.get_working_directory(config),
+            # Also add borgmatic-specific paths, so they don't get excluded from the recreated
+            # archive. Note that this doesn't currently work for archives created with Borg 1.2 or
+            # below.
+            borgmatic.borg.pattern.Pattern(
+                '/borgmatic', source=borgmatic.borg.pattern.Pattern_source.INTERNAL
+            ),
+        ),
+        config,
+        borgmatic.config.paths.get_working_directory(config),
+    )
+
+    archive = borgmatic.borg.repo_list.resolve_archive_name(
+        repository['path'],
+        recreate_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+    )
+
+    if archive and archive.endswith('.recreate'):
+        if recreate_arguments.archive == 'latest':
+            raise ValueError(
+                f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
+            )
+
+        raise ValueError(
+            f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
         )
 
-        archive = borgmatic.borg.repo_list.resolve_archive_name(
+    try:
+        borgmatic.borg.recreate.recreate_archive(
             repository['path'],
-            recreate_arguments.archive,
+            archive,
             config,
             local_borg_version,
+            recreate_arguments,
             global_arguments,
-            local_path,
+            local_path=local_path,
-            remote_path,
+            remote_path=remote_path,
+            patterns=processed_patterns,
         )
-
+    except subprocess.CalledProcessError as error:
-        if archive and archive.endswith('.recreate'):
+        if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
-            if recreate_arguments.archive == 'latest':
+            if recreate_arguments.target:
                 raise ValueError(
-                    f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
+                    f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
                 )
 
-            raise ValueError(
+            if archive:
-                f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
+                raise ValueError(
-            )
+                    f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
+                )
 
-        try:
+        raise
-            borgmatic.borg.recreate.recreate_archive(
-                repository['path'],
-                archive,
-                config,
-                local_borg_version,
-                recreate_arguments,
-                global_arguments,
-                local_path=local_path,
-                remote_path=remote_path,
-                patterns=processed_patterns,
-            )
-        except subprocess.CalledProcessError as error:
-            if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
-                if recreate_arguments.target:
-                    raise ValueError(
-                        f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
-                    )
-
-                if archive:
-                    raise ValueError(
-                        f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
-                    )
-
-            raise

borgmatic/actions/repo_create.py

@@ -1,7 +1,6 @@
 import logging
 
 import borgmatic.borg.repo_create
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -18,12 +17,6 @@ def run_repo_create(
     '''
     Run the "repo-create" action for the given repository.
     '''
-    if repo_create_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        repo_create_arguments.repository,
-    ):
-        return
-
     logger.info('Creating repository')
 
     encryption_mode = repo_create_arguments.encryption_mode or repository.get('encryption')

borgmatic/actions/repo_delete.py

@@ -17,20 +17,16 @@ def run_repo_delete(
     '''
     Run the "repo-delete" action for the given repository.
     '''
-    if repo_delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    logger.answer(
-        repository,
+        'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
-        repo_delete_arguments.repository,
+    )
-    ):
-        logger.answer(
-            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
-        )
 
-        borgmatic.borg.repo_delete.delete_repository(
+    borgmatic.borg.repo_delete.delete_repository(
-            repository,
+        repository,
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_delete_arguments,
+        repo_delete_arguments,
-            global_arguments,
+        global_arguments,
-            local_path,
+        local_path,
-            remote_path,
+        remote_path,
-        )
+    )

borgmatic/actions/repo_info.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.actions.json
 import borgmatic.borg.repo_info
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -21,21 +20,17 @@ def run_repo_info(
 
     If repo_info_arguments.json is True, yield the JSON output from the info for the repository.
     '''
-    if repo_info_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not repo_info_arguments.json:
-        repository,
+        logger.answer('Displaying repository summary information')
-        repo_info_arguments.repository,
-    ):
-        if not repo_info_arguments.json:
-            logger.answer('Displaying repository summary information')
 
-        json_output = borgmatic.borg.repo_info.display_repository_info(
+    json_output = borgmatic.borg.repo_info.display_repository_info(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_info_arguments=repo_info_arguments,
+        repo_info_arguments=repo_info_arguments,
-            global_arguments=global_arguments,
+        global_arguments=global_arguments,
-            local_path=local_path,
+        local_path=local_path,
-            remote_path=remote_path,
+        remote_path=remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/repo_list.py

@@ -2,7 +2,6 @@ import logging
 
 import borgmatic.actions.json
 import borgmatic.borg.repo_list
-import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
 
@@ -21,21 +20,17 @@ def run_repo_list(
 
     If repo_list_arguments.json is True, yield the JSON output from listing the repository.
     '''
-    if repo_list_arguments.repository is None or borgmatic.config.validate.repositories_match(
+    if not repo_list_arguments.json:
-        repository,
+        logger.answer('Listing repository')
-        repo_list_arguments.repository,
-    ):
-        if not repo_list_arguments.json:
-            logger.answer('Listing repository')
 
-        json_output = borgmatic.borg.repo_list.list_repository(
+    json_output = borgmatic.borg.repo_list.list_repository(
-            repository['path'],
+        repository['path'],
-            config,
+        config,
-            local_borg_version,
+        local_borg_version,
-            repo_list_arguments=repo_list_arguments,
+        repo_list_arguments=repo_list_arguments,
-            global_arguments=global_arguments,
+        global_arguments=global_arguments,
-            local_path=local_path,
+        local_path=local_path,
-            remote_path=remote_path,
+        remote_path=remote_path,
-        )
+    )
-        if json_output:
+    if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/restore.py

@@ -5,12 +5,12 @@ import pathlib
 import shutil
 import tempfile
 
+import borgmatic.actions.dump
+import borgmatic.actions.pattern
 import borgmatic.borg.extract
 import borgmatic.borg.list
-import borgmatic.borg.mount
 import borgmatic.borg.repo_list
 import borgmatic.config.paths
-import borgmatic.config.validate
 import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch
 
@@ -130,14 +130,14 @@ def get_configured_data_source(config, restore_dump):
 
 
 def strip_path_prefix_from_extracted_dump_destination(
-    destination_path,
+    extract_path,
     borgmatic_runtime_directory,
 ):
     '''
     Directory-format dump files get extracted into a temporary directory containing a path prefix
-    that depends how the files were stored in the archive. So, given the destination path where the
+    that depends how the files were stored in the archive. So, given the path where the dump was
-    dump was extracted and the borgmatic runtime directory, move the dump files such that the
+    extracted and the borgmatic runtime directory, move the dump files such that the restore doesn't
-    restore doesn't have to deal with that varying path prefix.
+    have to deal with that varying path prefix.
 
     For instance, if the dump was extracted to:
 
@@ -151,16 +151,16 @@ def strip_path_prefix_from_extracted_dump_destination(
 
       /run/user/0/borgmatic/postgresql_databases/test/...
     '''
-    for subdirectory_path, _, _ in os.walk(destination_path):
+    for subdirectory_path, _, _ in os.walk(extract_path):
         databases_directory = os.path.basename(subdirectory_path)
 
         if not databases_directory.endswith('_databases'):
             continue
 
-        shutil.move(
+        destination_path = os.path.join(borgmatic_runtime_directory, databases_directory)
-            subdirectory_path,
+        shutil.rmtree(destination_path, ignore_errors=True)
-            os.path.join(borgmatic_runtime_directory, databases_directory),
+        shutil.move(subdirectory_path, destination_path)
-        )
+
         break
 
 
@@ -201,6 +201,10 @@ def restore_single_dump(
         borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         borgmatic_runtime_directory,
         data_source['name'],
+        data_source.get('hostname'),
+        data_source.get('port'),
+        data_source.get('container'),
+        data_source.get('label'),
     )[hook_name.split('_databases', 1)[0]]
 
     destination_path = (
@@ -268,12 +272,12 @@ def collect_dumps_from_archive(
     Given a local or remote repository path, a resolved archive name, a configuration dict, the
     local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
     borgmatic runtime directory, query the archive for the names of data sources dumps it contains
-    and return them as a set of Dump instances.
+    and return them as a tuple of Dump instances.
     '''
-    dumps_from_archive = set()
+    dumps_from_archive = {}  # Use a dict as an ordered set.
 
     # There is (at most) one dump metadata file per data source hook. Load each.
-    for dumps_metadata_path in borgmatic.borg.list.capture_archive_listing(
+    for dumps_metadata_entry in borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
         config,
@@ -296,36 +300,33 @@ def collect_dumps_from_archive(
         local_path=local_path,
         remote_path=remote_path,
     ):
-        if not dumps_metadata_path:
+        if not dumps_metadata_entry.get('path'):
             continue
 
-        dumps_from_archive.update(
+        for dump in borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
-            set(
+            borgmatic.borg.extract.extract_archive(
-                borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
+                global_arguments.dry_run,
-                    borgmatic.borg.extract.extract_archive(
+                repository,
-                        global_arguments.dry_run,
+                archive,
-                        repository,
+                [dumps_metadata_entry['path']],
-                        archive,
+                config,
-                        [dumps_metadata_path],
+                local_borg_version,
-                        config,
+                global_arguments,
-                        local_borg_version,
+                local_path=local_path,
-                        global_arguments,
+                remote_path=remote_path,
-                        local_path=local_path,
+                extract_to_stdout=True,
-                        remote_path=remote_path,
-                        extract_to_stdout=True,
-                    )
-                    .stdout.read()
-                    .decode(),
-                    dumps_metadata_path,
-                )
             )
-        )
+            .stdout.read()
+            .decode(),
+            dumps_metadata_entry['path'],
+        ):
+            dumps_from_archive[dump] = None
 
     # If we've successfully loaded any dumps metadata, we're done.
     if dumps_from_archive:
         logger.debug('Collecting database dumps from archive data source dumps metadata files')
 
-        return dumps_from_archive
+        return tuple(dumps_from_archive.keys())
 
     # No dumps metadata files were found, so for backwards compatibility, fall back to parsing the
     # paths of dumps found in the archive to get their respective dump metadata.
@@ -337,7 +338,7 @@ def collect_dumps_from_archive(
     # Probe for the data source dumps in multiple locations, as the default location has moved to
     # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
     # we still want to support reading dumps from previously created archives as well.
-    dump_paths = borgmatic.borg.list.capture_archive_listing(
+    dump_entries = borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
         config,
@@ -359,7 +360,9 @@ def collect_dumps_from_archive(
         remote_path=remote_path,
     )
 
-    for dump_path in dump_paths:
+    for dump_entry in dump_entries:
+        dump_path = dump_entry.get('path')
+
         if not dump_path:
             continue
 
@@ -389,11 +392,11 @@ def collect_dumps_from_archive(
             except (ValueError, TypeError):
                 port = None
 
-            dumps_from_archive.add(
+            dumps_from_archive[
                 Dump(
                     hook_name, data_source_name, None if hostname == 'localhost' else hostname, port
                 )
-            )
+            ] = None
 
             # We've successfully parsed the dump path, so need to probe any further.
             break
@@ -402,20 +405,21 @@ def collect_dumps_from_archive(
                 f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}',
             )
 
-    return dumps_from_archive
+    return tuple(dumps_from_archive.keys())
 
 
 def get_dumps_to_restore(restore_arguments, dumps_from_archive):
     '''
     Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
-    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    a tuple of Dump instances representing the dumps found in an archive, return a tuple of specific
     Dump instances from the archive to restore. As part of this, replace any Dump having a data
     source name of "all" with multiple named Dump instances as appropriate.
 
     Raise ValueError if any of the requested data source names cannot be found in the archive or if
     there are multiple archive dump matches for a given requested dump.
     '''
-    requested_dumps = (
+    requested_dumps = tuple(
+        # Use a dict comprehension as an ordered set.
         {
             Dump(
                 hook_name=(
@@ -432,16 +436,16 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                 port=restore_arguments.original_port,
                 label=restore_arguments.original_label or UNSPECIFIED,
                 container=restore_arguments.original_container or UNSPECIFIED,
-            )
+            ): None
             for name in restore_arguments.data_sources or (UNSPECIFIED,)
-        }
+        }.keys()
         if restore_arguments.hook
         or restore_arguments.data_sources
         or restore_arguments.original_hostname
         or restore_arguments.original_port
         or restore_arguments.original_label
         or restore_arguments.original_container
-        else {
+        else (
             Dump(
                 hook_name=UNSPECIFIED,
                 data_source_name='all',
@@ -450,14 +454,15 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                 label=UNSPECIFIED,
                 container=UNSPECIFIED,
             ),
-        }
+        )
     )
     missing_dumps = set()
-    dumps_to_restore = set()
+    dumps_to_restore = {}  # Use a dict as an ordered set.
 
     # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
     if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
-        dumps_to_restore.update(dumps_from_archive)
+        for dump in dumps_from_archive:
+            dumps_to_restore[dump] = None
 
     # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
     for requested_dump in requested_dumps:
@@ -473,7 +478,7 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
         if len(matching_dumps) == 0:
             missing_dumps.add(requested_dump)
         elif len(matching_dumps) == 1:
-            dumps_to_restore.add(matching_dumps[0])
+            dumps_to_restore[matching_dumps[0]] = None
         else:
             raise ValueError(
                 f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.',
@@ -488,20 +493,20 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
             f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive",
         )
 
-    return dumps_to_restore
+    return tuple(dumps_to_restore.keys())
 
 
 def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
     '''
-    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    Given a tuple of requested dumps to restore and a set of dumps actually restored, raise
-    if any requested dumps to restore weren't restored, indicating that they were missing from the
+    ValueError if any requested dumps to restore weren't restored, indicating that they were missing
-    configuration.
+    from the configuration.
     '''
-    if not dumps_actually_restored:
+    if not dumps_to_restore:
         raise ValueError('No data source dumps were found to restore')
 
     missing_dumps = sorted(
-        dumps_to_restore - dumps_actually_restored,
+        set(dumps_to_restore) - dumps_actually_restored,
         key=lambda dump: dump.data_source_name,
     )
 
@@ -523,109 +528,96 @@ def run_restore(
     remote_path,
 ):
     '''
-    Run the "restore" action for the given repository, but only if the repository matches the
+    Run the "restore" action for the given repository.
-    requested repository in restore arguments.
 
     Raise ValueError if a configured data source could not be found to restore or there's no
     matching dump in the archive.
     '''
-    if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository,
-        restore_arguments.repository,
-    ):
-        return
-
     logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+        patterns = borgmatic.actions.pattern.process_patterns(
-            'remove_data_source_dumps',
+            borgmatic.actions.pattern.collect_patterns(config),
             config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            working_directory,
-            borgmatic_runtime_directory,
-            global_arguments.dry_run,
         )
 
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        with borgmatic.actions.dump.Dump_cleanup(
-            repository['path'],
+            config, borgmatic_runtime_directory, patterns, global_arguments.dry_run
-            restore_arguments.archive,
+        ):
-            config,
+            archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            local_borg_version,
+                repository['path'],
-            global_arguments,
+                restore_arguments.archive,
-            local_path,
-            remote_path,
-        )
-        dumps_from_archive = collect_dumps_from_archive(
-            repository['path'],
-            archive_name,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-            borgmatic_runtime_directory,
-        )
-        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
-
-        dumps_actually_restored = set()
-        connection_params = {
-            'container': restore_arguments.container,
-            'hostname': restore_arguments.hostname,
-            'port': restore_arguments.port,
-            'username': restore_arguments.username,
-            'password': restore_arguments.password,
-            'restore_path': restore_arguments.restore_path,
-        }
-
-        # Restore each dump.
-        for restore_dump in dumps_to_restore:
-            found_data_source = get_configured_data_source(
-                config,
-                restore_dump,
-            )
-            # For a dump that wasn't found via an exact match in the configuration, try to fallback
-            # to an "all" data source.
-            if not found_data_source:
-                found_data_source = get_configured_data_source(
-                    config,
-                    Dump(
-                        restore_dump.hook_name,
-                        'all',
-                        restore_dump.hostname,
-                        restore_dump.port,
-                        restore_dump.label,
-                        restore_dump.container,
-                    ),
-                )
-
-                if not found_data_source:
-                    continue
-
-                found_data_source = dict(found_data_source)
-                found_data_source['name'] = restore_dump.data_source_name
-
-            dumps_actually_restored.add(restore_dump)
-
-            restore_single_dump(
-                repository,
                 config,
                 local_borg_version,
                 global_arguments,
                 local_path,
                 remote_path,
+            )
+            dumps_from_archive = collect_dumps_from_archive(
+                repository['path'],
                 archive_name,
-                restore_dump.hook_name,
+                config,
-                dict(found_data_source, schemas=restore_arguments.schemas),
+                local_borg_version,
-                connection_params,
+                global_arguments,
+                local_path,
+                remote_path,
                 borgmatic_runtime_directory,
             )
+            dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
 
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            dumps_actually_restored = set()
-            'remove_data_source_dumps',
+            connection_params = {
-            config,
+                'container': restore_arguments.container,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                'hostname': restore_arguments.hostname,
-            borgmatic_runtime_directory,
+                'port': restore_arguments.port,
-            global_arguments.dry_run,
+                'username': restore_arguments.username,
-        )
+                'password': restore_arguments.password,
+                'restore_path': restore_arguments.restore_path,
+            }
+
+            # Restore each dump.
+            for restore_dump in dumps_to_restore:
+                found_data_source = get_configured_data_source(
+                    config,
+                    restore_dump,
+                )
+                # For a dump that wasn't found via an exact match in the configuration, try to fallback
+                # to an "all" data source.
+                if not found_data_source:
+                    found_data_source = get_configured_data_source(
+                        config,
+                        Dump(
+                            restore_dump.hook_name,
+                            'all',
+                            restore_dump.hostname,
+                            restore_dump.port,
+                            restore_dump.label,
+                            restore_dump.container,
+                        ),
+                    )
+
+                    if not found_data_source:
+                        continue
+
+                    found_data_source = dict(found_data_source)
+                    found_data_source['name'] = restore_dump.data_source_name
+
+                dumps_actually_restored.add(restore_dump)
+
+                restore_single_dump(
+                    repository,
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                    archive_name,
+                    restore_dump.hook_name,
+                    dict(found_data_source, schemas=restore_arguments.schemas),
+                    connection_params,
+                    borgmatic_runtime_directory,
+                )
 
     ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)

borgmatic/borg/break_lock.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 from borgmatic.borg import environment, flags
@@ -22,15 +23,17 @@ def break_lock(
     '''
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get('break_lock', '')
 
     full_command = (
         (local_path, 'break-lock')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/change_passphrase.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.execute
@@ -25,15 +26,16 @@ def change_passphrase(
     borgmatic.logger.add_custom_log_levels()
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_change_passphrase', '')
 
     full_command = (
         (local_path, 'key', 'change-passphrase')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(
             repository_path,
             local_borg_version,
@@ -63,5 +65,5 @@ def change_passphrase(
     )
 
     logger.answer(
-        f"{repository_path}: Don't forget to update your encryption_passphrase option (if needed)",
+        "Don't forget to update your encryption_passphrase option (if needed)",
     )

borgmatic/borg/check.py

@@ -149,8 +149,10 @@ def check_archives(
 
     max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')
 
+    # If not configured, elevate Borg's exit code 1 (an ostensible warning) to error, because Borg
+    # returns exit code 1 for repository check errors!
+    borg_exit_codes = [*config.get('borg_exit_codes', []), *[{'code': 1, 'treat_as': 'error'}]]
     umask = config.get('umask')
-    borg_exit_codes = config.get('borg_exit_codes')
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
     if 'data' in checks:
@@ -176,7 +178,13 @@ def check_archives(
             + make_check_name_flags(checks_subset, archive_filter_flags)
             + (('--remote-path', remote_path) if remote_path else ())
             + (('--umask', str(umask)) if umask else ())
-            + (('--log-json',) if config.get('log_json') else ())
+            + (
+                ('--log-json',)
+                if (
+                    config.get('log_json') or not (check_arguments.repair or config.get('progress'))
+                )
+                else ()
+            )
             + (('--lock-wait', str(lock_wait)) if lock_wait else ())
             + verbosity_flags
             + (('--progress',) if config.get('progress') else ())

borgmatic/borg/compact.py

@@ -31,7 +31,7 @@ def compact_segments(
         (local_path, 'compact')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--progress',) if config.get('progress') else ())
         + (('--cleanup-commits',) if cleanup_commits else ())

borgmatic/borg/create.py

@@ -53,6 +53,7 @@ def validate_planned_backup_paths(
     local_path,
     working_directory,
     borgmatic_runtime_directory,
+    find_special_files=False,
 ):
     '''
     Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
@@ -61,18 +62,25 @@ def validate_planned_backup_paths(
     given runtime directory exists, validate that it will be included in a backup and hasn't been
     excluded.
 
+    If find special files is True, then return the subset of planned backup paths that are special
+    files. Otherwise, return an empty tuple.
+
     Raise ValueError if the runtime directory has been excluded via "exclude_patterns" or similar,
-    because any features that rely on the runtime directory getting backed up will break.  For
+    because any features that rely on the runtime directory getting backed up will break. For
     instance, without the runtime directory, Borg can't consume any database dumps and borgmatic may
     hang waiting for them to be consumed.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
     # files including any named pipe we've created. And omit "--filter" because that can break the
     # paths output parsing below such that path lines no longer start with the expected "- ".
-    paths_output = execute_command_and_capture_output(
+    path_lines = execute_command_and_capture_output(
         (
             *flags.omit_flag_and_value(
-                flags.omit_flag(create_command, '--exclude-nodump'), '--filter'
+                flags.omit_flag(
+                    flags.omit_flag(create_command, '--exclude-nodump'),
+                    '--log-json',
+                ),
+                '--filter',
             ),
             '--dry-run',
             '--list',
@@ -86,44 +94,68 @@ def validate_planned_backup_paths(
 
     # These are all the individual files that Borg is planning to backup as determined by the Borg
     # create dry run above.
-    paths = tuple(
+    paths = (
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in path_lines
         if path_line and path_line.startswith(('- ', '+ '))
     )
 
-    # These are the subset of output paths contained within the borgmatic runtime directory.
+    include_pattern_paths = {
-    paths_inside_runtime_directory = {
+        pattern.path
-        path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.INCLUDE
     }
-
+    runtime_directory_root_patterns = tuple(
-    # If the runtime directory isn't present in the source patterns, then we shouldn't expect it to
-    # be in the paths output from the Borg dry run.
-    runtime_directory_present_in_patterns = any(
         pattern
         for pattern in patterns
         if any_parent_directories(pattern.path, (borgmatic_runtime_directory,))
         if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        # Skip root patterns that have corresponding include patterns, because those will "punch
+        # through" any subsequent excludes.
+        if pattern.path not in include_pattern_paths
     )
 
-    # If no paths to backup are inside the runtime directory, it must've been excluded.
+    special_paths = []
-    if (
+    validate_runtime_directory = bool(
-        not paths_inside_runtime_directory
+        not dry_run
-        and runtime_directory_present_in_patterns
-        and not dry_run
         and os.path.exists(borgmatic_runtime_directory)
-    ):
+        and runtime_directory_root_patterns
-        raise ValueError(
+    )
-            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.',
+    runtime_directory_in_path = False
+
+    # Do everything in this one loop because we only want to consume the paths generator once.
+    for path in paths:
+        # If all root patterns in the runtime directory are missing from the paths Borg is planning
+        # to backup, then they must've gotten excluded, e.g. by user-configured excludes. Warn
+        # accordingly (below).
+        if validate_runtime_directory and any(
+            any_parent_directories(path, (pattern.path,))
+            for pattern in runtime_directory_root_patterns
+        ):
+            runtime_directory_in_path = True
+
+        # Return the subset of output paths that are special files but *not* contained within the
+        # borgmatic runtime directory. The intent is to skip runtime paths that borgmatic uses for its
+        # own bookkeeping, instead focusing on user-configured paths.
+        if (
+            find_special_files
+            and not any_parent_directories(path, (borgmatic_runtime_directory,))
+            and special_file(path, working_directory)
+        ):
+            special_paths.append(path)
+
+    if validate_runtime_directory and not runtime_directory_in_path:
+        logger.warning(
+            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes (or the snapshotted source directories are empty). Please ensure the runtime directory is not excluded.'
         )
 
-    return tuple(path for path in paths if path not in paths_inside_runtime_directory)
+    return tuple(special_paths)
 
 
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
 
 
-def make_base_create_command(
+def make_base_create_command(  # noqa: PLR0912
     dry_run,
     repository_path,
     config,
@@ -148,6 +180,9 @@ def make_base_create_command(
     open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
+        logger.warning(
+            'The "source_directories_must_exist" option is deprecated and will be removed from a future release; borgmatic now errors on missing files as Borg runs'
+        )
         borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
 
     patterns_file = borgmatic.borg.pattern.write_patterns_file(
@@ -216,8 +251,8 @@ def make_base_create_command(
         + (('--files-cache', files_cache) if files_cache else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (
             ('--list', '--filter', list_filter_flags)
             if config.get('list_details') and not json and not config.get('progress')
@@ -234,8 +269,16 @@ def make_base_create_command(
     )
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
+    if config.get('unsafe_skip_path_validation_before_create'):
+        logger.debug(
+            'Skipping pre-backup path validation due to "unsafe_skip_path_validation_before_create" option.'
+        )
+
+        return (create_flags, create_positional_arguments, patterns_file)
+
     logger.debug('Checking file paths Borg plans to include')
-    planned_backup_paths = validate_planned_backup_paths(
+
+    special_file_paths = validate_planned_backup_paths(
         dry_run,
         create_flags + create_positional_arguments,
         config,
@@ -243,44 +286,41 @@ def make_base_create_command(
         local_path,
         working_directory,
         borgmatic_runtime_directory=borgmatic_runtime_directory,
+        find_special_files=bool(stream_processes),
     )
 
-    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
+    if stream_processes and config.get('read_special') is False:
-    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
-    if stream_processes and not config.get('read_special'):
         logger.warning(
             'Ignoring configured "read_special" value of false, as true is needed for database hooks.',
         )
 
-        special_file_paths = tuple(
+    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
-            path for path in planned_backup_paths if special_file(path, working_directory)
+    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
+    if special_file_paths:
+        truncated_special_file_paths = textwrap.shorten(
+            ', '.join(special_file_paths),
+            width=MAX_SPECIAL_FILE_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.warning(
+            f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
+        )
+        patterns_file = borgmatic.borg.pattern.write_patterns_file(
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    special_file_path,
+                    borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                )
+                for special_file_path in special_file_paths
+            ),
+            borgmatic_runtime_directory,
+            patterns_file=patterns_file,
         )
 
-        if special_file_paths:
+        if '--patterns-from' not in create_flags:
-            truncated_special_file_paths = textwrap.shorten(
+            create_flags += ('--patterns-from', patterns_file.name)
-                ', '.join(special_file_paths),
-                width=MAX_SPECIAL_FILE_PATHS_LENGTH,
-                placeholder=' ...',
-            )
-            logger.warning(
-                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
-            )
-            patterns_file = borgmatic.borg.pattern.write_patterns_file(
-                tuple(
-                    borgmatic.borg.pattern.Pattern(
-                        special_file_path,
-                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
-                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
-                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
-                    )
-                    for special_file_path in special_file_paths
-                ),
-                borgmatic_runtime_directory,
-                patterns_file=patterns_file,
-            )
-
-            if '--patterns-from' not in create_flags:
-                create_flags += ('--patterns-from', patterns_file.name)
 
     return (create_flags, create_positional_arguments, patterns_file)
 
@@ -352,24 +392,28 @@ def create_archive(
     borg_exit_codes = config.get('borg_exit_codes')
 
     if stream_processes:
-        return execute_command_with_processes(
+        return '\n'.join(
-            create_flags + create_positional_arguments,
+            execute_command_with_processes(
-            stream_processes,
+                create_flags + create_positional_arguments,
-            output_log_level,
+                stream_processes,
-            output_file,
+                output_log_level,
-            working_directory=working_directory,
+                output_file,
-            environment=environment.make_environment(config),
+                working_directory=working_directory,
-            borg_local_path=local_path,
+                environment=environment.make_environment(config),
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     if output_log_level is None:
-        return execute_command_and_capture_output(
+        return '\n'.join(
-            create_flags + create_positional_arguments,
+            execute_command_and_capture_output(
-            working_directory=working_directory,
+                create_flags + create_positional_arguments,
-            environment=environment.make_environment(config),
+                working_directory=working_directory,
-            borg_local_path=local_path,
+                environment=environment.make_environment(config),
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     execute_command(

borgmatic/borg/delete.py

@@ -1,5 +1,6 @@
 import argparse
 import logging
+import shlex
 
 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -28,6 +29,8 @@ def make_delete_command(
     arguments to the delete action as an argparse.Namespace, and global arguments, return a command
     as a tuple to delete archives from the repository.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('delete', '')
+
     return (
         (local_path, 'delete')
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -35,7 +38,7 @@ def make_delete_command(
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
         + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (
@@ -66,6 +69,7 @@ def make_delete_command(
                 'repository',
             ),
         )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
     )
 

borgmatic/borg/environment.py

@@ -10,6 +10,7 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
+    'borg_key_file': 'BORG_KEY_FILE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
 }

borgmatic/borg/export_key.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -31,6 +32,7 @@ def export_key(
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
     working_directory = borgmatic.config.paths.get_working_directory(config)
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_export', '')
 
     if export_arguments.path and export_arguments.path != '-':
         if os.path.exists(os.path.join(working_directory or '', export_arguments.path)):
@@ -46,12 +48,13 @@ def export_key(
         (local_path, 'key', 'export')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if output_file is None else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_flags('paper', export_arguments.paper)
         + flags.make_flags('qr-html', export_arguments.qr_html)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(
             repository_path,
             local_borg_version,

borgmatic/borg/export_tar.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -34,12 +35,13 @@ def export_tar_archive(
     borgmatic.logger.add_custom_log_levels()
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get('export_tar', '')
 
     full_command = (
         (local_path, 'export-tar')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if destination_path != '-' else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--list',) if config.get('list_details') else ())
@@ -47,6 +49,7 @@ def export_tar_archive(
         + (('--dry-run',) if dry_run else ())
         + (('--tar-filter', tar_filter) if tar_filter else ())
         + (('--strip-components', str(strip_components)) if strip_components else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_archive_flags(
             repository_path,
             archive,

borgmatic/borg/extract.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex
 import subprocess
 
 import borgmatic.config.paths
@@ -23,6 +24,7 @@ def extract_last_archive_dry_run(
     Perform an extraction dry-run of the most recent archive. If there are no archives, skip the
     dry-run.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
     verbosity_flags = ()
     if logger.isEnabledFor(logging.DEBUG):
         verbosity_flags = ('--debug', '--show-rc')
@@ -47,10 +49,12 @@ def extract_last_archive_dry_run(
     full_extract_command = (
         (local_path, 'extract', '--dry-run')
         + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not config.get('progress') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + verbosity_flags
+        + (('--progress',) if config.get('progress') else ())
         + list_flag
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_archive_flags(
             repository_path,
             last_archive_name,
@@ -58,13 +62,23 @@ def extract_last_archive_dry_run(
         )
     )
 
-    execute_command(
+    if config.get('progress'):
-        full_extract_command,
+        execute_command(
-        environment=environment.make_environment(config),
+            full_extract_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            output_file=DO_NOT_CAPTURE,
-        borg_local_path=local_path,
+            environment=environment.make_environment(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-    )
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
+    else:
+        execute_command(
+            full_extract_command,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
 
 
 def extract_archive(
@@ -92,9 +106,7 @@ def extract_archive(
     '''
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
-
+    extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
-    if config.get('progress') and extract_to_stdout:
-        raise ValueError('progress and extract to stdout cannot both be set')
 
     if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
         numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
@@ -123,14 +135,15 @@ def extract_archive(
         + (('--remote-path', remote_path) if remote_path else ())
         + numeric_ids_flags
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
         + (('--strip-components', str(strip_components)) if strip_components else ())
-        + (('--progress',) if config.get('progress') else ())
+        + (('--progress',) if config.get('progress') and not extract_to_stdout else ())
         + (('--stdout',) if extract_to_stdout else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_archive_flags(
             # Make the repository path absolute so the destination directory used below via changing
             # the working directory doesn't prevent Borg from finding the repo. But also apply the
@@ -147,6 +160,17 @@ def extract_archive(
         os.path.join(working_directory or '', destination_path) if destination_path else None
     )
 
+    if extract_to_stdout:
+        return execute_command(
+            full_command,
+            output_file=subprocess.PIPE,
+            run_to_completion=False,
+            environment=environment.make_environment(config),
+            working_directory=full_destination_path,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
+
     # The progress output isn't compatible with captured and logged output, as progress messes with
     # the terminal directly.
     if config.get('progress'):
@@ -160,17 +184,6 @@ def extract_archive(
         )
         return None
 
-    if extract_to_stdout:
-        return execute_command(
-            full_command,
-            output_file=subprocess.PIPE,
-            run_to_completion=False,
-            environment=environment.make_environment(config),
-            working_directory=full_destination_path,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
     execute_command(

borgmatic/borg/import_key.py

@@ -1,10 +1,10 @@
 import logging
 import os
+import shlex
 
 import borgmatic.config.paths
-import borgmatic.logger
 from borgmatic.borg import environment, flags
-from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
 
@@ -30,29 +30,30 @@ def import_key(
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
     working_directory = borgmatic.config.paths.get_working_directory(config)
+    extra_borg_options = config.get('extra_borg_options', {}).get('key_import', '')
 
-    if import_arguments.path and import_arguments.path != '-':
+    if (
-        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+        import_arguments.path
-            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+        and import_arguments.path != '-'
-
+        and not os.path.exists(os.path.join(working_directory or '', import_arguments.path))
-        input_file = None
+    ):
-    else:
+        raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
-        input_file = DO_NOT_CAPTURE
 
     full_command = (
         (local_path, 'key', 'import')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_flags('paper', import_arguments.paper)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(
             repository_path,
             local_borg_version,
         )
-        + ((import_arguments.path,) if input_file is None else ())
+        + (import_arguments.path or '-',)
     )
 
     if global_arguments.dry_run:
@@ -61,7 +62,6 @@ def import_key(
 
     execute_command(
         full_command,
-        input_file=input_file,
         output_log_level=logging.INFO,
         environment=environment.make_environment(config),
         working_directory=working_directory,

borgmatic/borg/info.py

@@ -1,5 +1,6 @@
 import argparse
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -23,6 +24,8 @@ def make_info_command(
     arguments to the info action as an argparse.Namespace, and global arguments, return a command
     as a tuple to display summary information for archives in the repository.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('info', '')
+
     return (
         (local_path, 'info')
         + (
@@ -37,7 +40,7 @@ def make_info_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
@@ -58,6 +61,7 @@ def make_info_command(
             info_arguments,
             excludes=('repository', 'archive', 'prefix', 'match_archives'),
         )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
@@ -99,12 +103,14 @@ def display_archives_info(
     borg_exit_codes = config.get('borg_exit_codes')
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    json_info = execute_command_and_capture_output(
+    json_info = '\n'.join(
-        json_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            json_command,
-        working_directory=working_directory,
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=working_directory,
-        borg_exit_codes=borg_exit_codes,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
     )
 
     if info_arguments.json:

borgmatic/borg/list.py

@@ -1,7 +1,9 @@
 import argparse
 import copy
+import json
 import logging
 import re
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -17,6 +19,8 @@ MAKE_FLAGS_EXCLUDES = (
     'archive',
     'paths',
     'find_paths',
+    'format',
+    'json',
     *ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST,
 )
 
@@ -35,6 +39,8 @@ def make_list_command(
     and local and remote Borg paths, return a command as a tuple to list archives or paths within an
     archive.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('list', '')
+
     return (
         (local_path, 'list')
         + (
@@ -49,9 +55,12 @@ def make_list_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
+        + flags.make_flags('json-lines', list_arguments.json)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
+        + flags.make_flags('format', list_arguments.format or config.get('file_list_format'))
         + flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + (
             flags.make_repository_archive_flags(
                 repository_path,
@@ -103,14 +112,15 @@ def capture_archive_listing(
     remote_path=None,
 ):
     '''
-    Given a local or remote repository path, an archive name, a configuration
+    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
-    dict, the local Borg version, global arguments as an argparse.Namespace,
+    version, global arguments as an argparse.Namespace, the archive paths (or Borg patterns) in
-    the archive paths (or Borg patterns) in which to list files, the Borg path
+    which to list files, the Borg path format indicating keys to include in the output, and local
-    format to use for the output, and local and remote Borg paths, capture the
+    and remote Borg paths, capture the output of listing that archive and return it as a sequence of
-    output of listing that archive and return it as a list of file paths.
+    dicts, one per path.
     '''
     return tuple(
-        execute_command_and_capture_output(
+        json.loads(entry)
+        for entry in execute_command_and_capture_output(
             make_list_command(
                 repository_path,
                 config,
@@ -120,8 +130,8 @@ def capture_archive_listing(
                     archive=archive,
                     paths=list(list_paths) if list_paths else None,
                     find_paths=None,
-                    json=None,
+                    json=True,
-                    format=path_format or '{path}{NUL}',
+                    format=path_format or None,
                 ),
                 global_arguments,
                 local_path,
@@ -132,8 +142,6 @@ def capture_archive_listing(
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\0')
-        .split('\0'),
     )
 
 
@@ -190,11 +198,6 @@ def list_archive(
                     f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag.",
                 )
 
-    if list_arguments.json:
-        raise ValueError(
-            'The --json flag on the list action is not supported when using the --archive/--find flags.',
-        )
-
     borg_exit_codes = config.get('borg_exit_codes')
 
     # If there are any paths to find (and there's not a single archive already selected), start by
@@ -213,24 +216,20 @@ def list_archive(
         )
 
         # Ask Borg to list archives. Capture its output for use below.
-        archive_lines = tuple(
+        archive_lines = execute_command_and_capture_output(
-            execute_command_and_capture_output(
+            repo_list.make_repo_list_command(
-                repo_list.make_repo_list_command(
+                repository_path,
-                    repository_path,
+                config,
-                    config,
+                local_borg_version,
-                    local_borg_version,
+                repo_list_arguments,
-                    repo_list_arguments,
+                global_arguments,
-                    global_arguments,
+                local_path,
-                    local_path,
+                remote_path,
-                    remote_path,
+            ),
-                ),
+            environment=environment.make_environment(config),
-                environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-                working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
-                borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
-                borg_exit_codes=borg_exit_codes,
-            )
-            .strip('\n')
-            .splitlines(),
         )
     else:
         archive_lines = (list_arguments.archive,)

borgmatic/borg/mount.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags
@@ -25,12 +26,13 @@ def mount_archive(
     '''
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get('mount', '')
 
     full_command = (
         (local_path, 'mount')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not mount_arguments.foreground else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -39,6 +41,7 @@ def mount_archive(
             excludes=('repository', 'archive', 'mount_point', 'paths', 'options'),
         )
         + (('-o', mount_arguments.options) if mount_arguments.options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + (
             (
                 flags.make_repository_flags(repository_path, local_borg_version)

borgmatic/borg/passcommand.py

@@ -17,9 +17,11 @@ def run_passcommand(passcommand, working_directory):
     Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
     per borgmatic invocation.
     '''
-    return borgmatic.execute.execute_command_and_capture_output(
+    return '\n'.join(
-        shlex.split(passcommand),
+        borgmatic.execute.execute_command_and_capture_output(
-        working_directory=working_directory,
+            shlex.split(passcommand),
+            working_directory=working_directory,
+        )
     )
 
 

borgmatic/borg/prune.py

@@ -72,9 +72,9 @@ def prune_archives(
     full_command = (
         (local_path, 'prune')
         + make_prune_flags(config, prune_arguments, local_borg_version)
+        + ('--log-json',)
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (
             ('--stats',)

borgmatic/borg/recreate.py

@@ -32,6 +32,7 @@ def recreate_archive(
     exclude_flags = flags.make_exclude_flags(config)
     compression = config.get('compression', None)
     chunker_params = config.get('chunker_params', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get('recreate', '')
 
     # Available recompress MODES: "if-different", "always", "never" (default)
     recompress = config.get('recompress', None)
@@ -45,7 +46,7 @@ def recreate_archive(
     recreate_command = (
         (local_path, 'recreate')
         + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -71,6 +72,8 @@ def recreate_archive(
         + (('--chunker-params', chunker_params) if chunker_params else ())
         + (('--recompress', recompress) if recompress else ())
         + exclude_flags
+        + (('--dry-run',) if global_arguments.dry_run else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + (
             (
                 flags.make_repository_flags(repository, local_borg_version)
@@ -92,10 +95,6 @@ def recreate_archive(
         )
     )
 
-    if global_arguments.dry_run:
-        logger.info('Skipping the archive recreation (dry run)')
-        return
-
     borgmatic.execute.execute_command(
         full_command=recreate_command,
         output_log_level=logging.INFO,

borgmatic/borg/rename.py

@@ -1,5 +1,7 @@
 import logging
+import shlex
 
+import borgmatic.borg.environment
 import borgmatic.borg.flags
 
 logger = logging.getLogger(__name__)
@@ -15,6 +17,8 @@ def make_rename_command(
     local_path,
     remote_path,
 ):
+    extra_borg_options = config.get('extra_borg_options', {}).get('rename', '')
+
     return (
         (local_path, 'rename')
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -22,8 +26,9 @@ def make_rename_command(
         + borgmatic.borg.flags.make_flags('dry-run', dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + borgmatic.borg.flags.make_repository_archive_flags(
             repository_name,
             old_archive_name,

borgmatic/borg/repo_create.py

@@ -66,8 +66,14 @@ def create_repository(
 
     lock_wait = config.get('lock_wait')
     umask = config.get('umask')
+    extra_borg_options_from_init = config.get('extra_borg_options', {}).get('init', '')
     extra_borg_options = config.get('extra_borg_options', {}).get('repo-create', '')
 
+    if extra_borg_options_from_init:
+        logger.warning(
+            'The "init" option in "extra_borg_options" is deprecated and will be removed from a future release; use "repo_create" instead.'
+        )
+
     repo_create_command = (
         (local_path,)
         + (
@@ -83,11 +89,11 @@ def create_repository(
         + (('--make-parent-dirs',) if make_parent_directories else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options_from_init)) if extra_borg_options_from_init else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 

borgmatic/borg/repo_delete.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.borg.environment
 import borgmatic.borg.feature
@@ -20,12 +21,23 @@ def make_repo_delete_command(
     global_arguments,
     local_path,
     remote_path,
+    output_file,
 ):
     '''
     Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the repo_delete action as an argparse.Namespace, and global arguments, return a command
+    arguments to the repo_delete action as an argparse.Namespace, and global arguments, the Borg
-    as a tuple to repo_delete the entire repository.
+    local path, the Borg remote path, and an optional output file, return a command as a tuple to
+    repo_delete the entire repository.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_delete'
+        if borgmatic.borg.feature.available(
+            borgmatic.borg.feature.Feature.REPO_DELETE, local_borg_version
+        )
+        else 'delete',
+        '',
+    )
+
     return (
         (local_path,)
         + (
@@ -41,7 +53,7 @@ def make_repo_delete_command(
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if output_file is None else ())
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
         + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (
@@ -56,6 +68,7 @@ def make_repo_delete_command(
             repo_delete_arguments,
             excludes=('list_details', 'force', 'repository'),
         )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
     )
 
@@ -76,6 +89,14 @@ def delete_repository(
     '''
     borgmatic.logger.add_custom_log_levels()
 
+    # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+    # prompt won't work.
+    output_file = (
+        None
+        if repo_delete_arguments.force or repo_delete_arguments.cache_only
+        else borgmatic.execute.DO_NOT_CAPTURE
+    )
+
     command = make_repo_delete_command(
         repository,
         config,
@@ -84,18 +105,13 @@ def delete_repository(
         global_arguments,
         local_path,
         remote_path,
+        output_file,
     )
 
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+        output_file=output_file,
-        # prompt won't work.
-        output_file=(
-            None
-            if repo_delete_arguments.force or repo_delete_arguments.cache_only
-            else borgmatic.execute.DO_NOT_CAPTURE
-        ),
         environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,

borgmatic/borg/repo_info.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -24,6 +25,10 @@ def display_repository_info(
     '''
     borgmatic.logger.add_custom_log_levels()
     lock_wait = config.get('lock_wait', None)
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_info' if feature.available(feature.Feature.REPO_INFO, local_borg_version) else 'info',
+        '',
+    )
 
     full_command = (
         (local_path,)
@@ -32,6 +37,7 @@ def display_repository_info(
             if feature.available(feature.Feature.REPO_INFO, local_borg_version)
             else ('info',)
         )
+        + (('--critical',) if repo_info_arguments.json else ())
         + (
             ('--info',)
             if logger.getEffectiveLevel() == logging.INFO and not repo_info_arguments.json
@@ -44,9 +50,10 @@ def display_repository_info(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', lock_wait)
+        + ('--log-json',)
         + (('--json',) if repo_info_arguments.json else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
@@ -54,12 +61,14 @@ def display_repository_info(
     borg_exit_codes = config.get('borg_exit_codes')
 
     if repo_info_arguments.json:
-        return execute_command_and_capture_output(
+        return '\n'.join(
-            full_command,
+            execute_command_and_capture_output(
-            environment=environment.make_environment(config),
+                full_command,
-            working_directory=working_directory,
+                environment=environment.make_environment(config),
-            borg_local_path=local_path,
+                working_directory=working_directory,
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     execute_command(

borgmatic/borg/repo_list.py

@@ -1,6 +1,7 @@
 import argparse
 import json
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -62,6 +63,10 @@ def get_latest_archive(
 
     Raises ValueError if there are no archives in the repository.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
+        '',
+    )
 
     full_command = (
         local_path,
@@ -72,7 +77,7 @@ def get_latest_archive(
         ),
         *flags.make_flags('remote-path', remote_path),
         *flags.make_flags('umask', config.get('umask')),
-        *flags.make_flags('log-json', config.get('log_json')),
+        *('--log-json',),
         *flags.make_flags('lock-wait', config.get('lock_wait')),
         *(
             flags.make_flags('consider-checkpoints', consider_checkpoints)
@@ -81,15 +86,18 @@ def get_latest_archive(
         ),
         *flags.make_flags('last', 1),
         '--json',
+        *(tuple(shlex.split(extra_borg_options)) if extra_borg_options else ()),
         *flags.make_repository_flags(repository_path, local_borg_version),
     )
 
-    json_output = execute_command_and_capture_output(
+    json_output = '\n'.join(
-        full_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            full_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
     )
 
     archives = json.loads(json_output)['archives']
@@ -104,7 +112,7 @@ def get_latest_archive(
     return latest_archive
 
 
-MAKE_FLAGS_EXCLUDES = ('repository', 'prefix', 'match_archives')
+MAKE_FLAGS_EXCLUDES = ('repository', 'format', 'prefix', 'match_archives')
 
 
 def make_repo_list_command(
@@ -121,6 +129,11 @@ def make_repo_list_command(
     arguments to the repo_list action, global arguments as an argparse.Namespace instance, and local and
     remote Borg paths, return a command as a tuple to list archives with a repository.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get(
+        'repo_list' if feature.available(feature.Feature.REPO_LIST, local_borg_version) else 'list',
+        '',
+    )
+
     return (
         (
             local_path,
@@ -142,7 +155,7 @@ def make_repo_list_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
@@ -159,7 +172,11 @@ def make_repo_list_command(
                 )
             )
         )
+        + flags.make_flags(
+            'format', repo_list_arguments.format or config.get('archive_list_format')
+        )
         + flags.make_flags_from_arguments(repo_list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
@@ -202,12 +219,14 @@ def list_repository(
     working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
-    json_listing = execute_command_and_capture_output(
+    json_listing = '\n'.join(
-        json_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            json_command,
-        working_directory=working_directory,
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=working_directory,
-        borg_exit_codes=borg_exit_codes,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
     )
 
     if repo_list_arguments.json:

borgmatic/borg/state.py

@@ -1 +1 @@
-DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'
+DEFAULT_BORGMATIC_SOURCE_DIRECTORY = '~/.borgmatic'  # pragma: nocover

borgmatic/borg/transfer.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 import borgmatic.logger
@@ -24,6 +25,7 @@ def transfer_archives(
     instance, transfer archives to the given repository.
     '''
     borgmatic.logger.add_custom_log_levels()
+    extra_borg_options = config.get('extra_borg_options', {}).get('transfer', '')
 
     full_command = (
         (local_path, 'transfer')
@@ -31,7 +33,7 @@ def transfer_archives(
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + flags.make_flags('progress', config.get('progress'))
         + (
@@ -53,6 +55,7 @@ def transfer_archives(
                 )
             )
         )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
         + flags.make_flags('other-repo', transfer_arguments.source_repository)
         + flags.make_flags('dry-run', dry_run)

borgmatic/borg/umount.py

@@ -1,4 +1,5 @@
 import logging
+import shlex
 
 import borgmatic.config.paths
 from borgmatic.execute import execute_command
@@ -11,10 +12,13 @@ def unmount_archive(config, mount_point, local_path='borg'):
     Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
     from the mount point.
     '''
+    extra_borg_options = config.get('extra_borg_options', {}).get('umount', '')
     full_command = (
         (local_path, 'umount')
+        + ('--log-json',)
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + (mount_point,)
     )
 

borgmatic/borg/version.py

@@ -16,15 +16,19 @@ def local_borg_version(config, local_path='borg'):
     '''
     full_command = (
         (local_path, '--version')
+        + ('--log-json',)
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
     )
-    output = execute_command_and_capture_output(
+
-        full_command,
+    output = '\n'.join(
-        environment=environment.make_environment(config),
+        execute_command_and_capture_output(
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            full_command,
-        borg_local_path=local_path,
+            environment=environment.make_environment(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
     )
 
     try:

borgmatic/commands/arguments.py

@@ -297,7 +297,14 @@ def parse_arguments_for_actions(unparsed_arguments, action_parsers, global_parse
     )
 
 
-OMITTED_FLAG_NAMES = {'match-archives', 'progress', 'statistics', 'list-details'}
+OMITTED_FLAG_NAMES = {
+    'match-archives',
+    'progress',
+    'statistics',
+    'list-details',
+    'file-list-format',
+    'archive-list-format',
+}
 
 
 def make_argument_description(schema, flag_name):
@@ -568,8 +575,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
     ignoring actions, and the combined parser is handy for displaying help that includes everything:
     global flags, a list of actions, etc.
     '''
-    config_paths = collect.get_default_config_paths(expand_home=True)
+    config_paths = collect.get_default_config_paths()
-    unexpanded_config_paths = collect.get_default_config_paths(expand_home=False)
 
     # Using allow_abbrev=False here prevents the global parser from erroring about "ambiguous"
     # options like --encryption. Such options are intended for an action parser rather than the
@@ -582,7 +588,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
         '--config',
         dest='config_paths',
         action='append',
-        help=f"Configuration filename or directory, can specify flag multiple times, defaults to: -c {' -c '.join(unexpanded_config_paths)}",
+        help='Configuration filename or directory, can specify flag multiple times, defaults to /etc/borgmatic/config.yaml, /etc/borgmatic.d, $XDG_CONFIG_HOME/borgmatic/config.yaml, and $XDG_CONFIG_HOME/borgmatic.d, where $XDG_CONFIG_HOME defaults to $HOME/.config',
     )
     global_group.add_argument(
         '-n',
@@ -641,8 +647,8 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
     repo_create_parser = action_parsers.add_parser(
         'repo-create',
         aliases=ACTION_ALIASES['repo-create'],
-        help='Create a new, empty Borg repository',
+        help='Create a new, empty Borg repository (also known as "init")',
-        description='Create a new, empty Borg repository',
+        description='Create a new, empty Borg repository (also known as "init")',
         add_help=False,
     )
     repo_create_group = repo_create_parser.add_argument_group('repo-create arguments')
@@ -1213,9 +1219,9 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
     config_generate_group.add_argument(
         '-d',
         '--destination',
-        dest='destination_filename',
+        dest='destination_path',
         default=config_paths[0],
-        help=f'Destination configuration file, default: {unexpanded_config_paths[0]}',
+        help='Destination configuration file (or directory if using --split), default: /etc/borgmatic/config.yaml',
     )
     config_generate_group.add_argument(
         '--overwrite',
@@ -1223,6 +1229,11 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
         action='store_true',
         help='Whether to overwrite any existing destination file, defaults to false',
     )
+    config_generate_group.add_argument(
+        '--split',
+        action='store_true',
+        help='Assuming the destination is a directory instead of a file, split the configuration into separate files within it, one per option, useful for documentation',
+    )
     config_generate_group.add_argument(
         '-h',
         '--help',
@@ -1537,7 +1548,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
         action='store_true',
         help='Output only archive names',
     )
-    repo_list_group.add_argument('--format', help='Format for archive listing')
+    repo_list_group.add_argument('--format', help='Borg format for the archive listing')
     repo_list_group.add_argument(
         '--json',
         default=False,
@@ -1640,7 +1651,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
         action='store_true',
         help='Output only path names',
     )
-    list_group.add_argument('--format', help='Format for file listing')
+    list_group.add_argument('--format', help='Borg format for the file listing')
     list_group.add_argument(
         '--json',
         default=False,
@@ -1920,7 +1931,7 @@ def make_parsers(schema, unparsed_arguments):  # noqa: PLR0915
     )
     recreate_group.add_argument(
         '--archive',
-        help='Archive name, hash, or series to recreate',
+        help='Archive name, hash, or series to recreate, defaults to all archives in the repository (if specified), or all archives across all repositories',
     )
     recreate_group.add_argument(
         '--list',
@@ -2010,7 +2021,7 @@ def parse_arguments(schema, *unparsed_arguments):
     )
 
     if not arguments['global'].config_paths:
-        arguments['global'].config_paths = collect.get_default_config_paths(expand_home=True)
+        arguments['global'].config_paths = collect.get_default_config_paths()
 
     for action_name in ('bootstrap', 'generate', 'validate'):
         if action_name in arguments and len(arguments) > HIGHLANDER_ACTION_ARGUMENTS_COUNT:

borgmatic/commands/borgmatic.py

@@ -340,7 +340,7 @@ def run_actions(  # noqa: PLR0912, PLR0915
     '''
     Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
     filename, a configuration dict, a sequence of loaded configuration paths, local and remote paths
-    to Borg, a local Borg version string, and a repository name, run all actions from the
+    to Borg, a local Borg version string, and a repository dict, run all actions from the
     command-line arguments on the given repository.
 
     Yield JSON output strings from executing any actions that produce JSON.
@@ -362,6 +362,22 @@ def run_actions(  # noqa: PLR0912, PLR0915
         'repository': repository_path,
     }
     skip_actions = set(get_skip_actions(config, arguments))
+    requested_repository = next(
+        (
+            repository
+            for action_arguments in arguments.values()
+            for repository in (getattr(action_arguments, 'repository', None),)
+            if repository is not None
+        ),
+        None,
+    )
+
+    if requested_repository and not borgmatic.config.validate.repositories_match(
+        repository,
+        requested_repository,
+    ):
+        logger.debug('Skipping actions because the requested --repository does not match')
+        return
 
     with borgmatic.hooks.command.Before_after_hooks(
         command_hooks=config.get('commands'),
@@ -425,6 +441,7 @@ def run_actions(  # noqa: PLR0912, PLR0915
                         local_borg_version,
                         action_arguments,
                         global_arguments,
+                        dry_run_label,
                         local_path,
                         remote_path,
                     )
@@ -644,6 +661,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                         levelno=logging.DEBUG,
                         levelname='DEBUG',
                         msg=f'{config_filename}: Loading configuration file',
+                        name=logger.name,
                     ),
                 ),
             ],
@@ -666,6 +684,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                             levelno=logging.CRITICAL,
                             levelname='CRITICAL',
                             msg=f'{config_filename}: Insufficient permissions to read configuration file',
+                            name=logger.name,
                         ),
                     ),
                 ],
@@ -678,10 +697,16 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                             levelno=logging.CRITICAL,
                             levelname='CRITICAL',
                             msg=f'{config_filename}: Error parsing configuration file',
+                            name=logger.name,
                         ),
                     ),
                     logging.makeLogRecord(
-                        dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=str(error)),
+                        dict(
+                            levelno=logging.CRITICAL,
+                            levelname='CRITICAL',
+                            msg=str(error),
+                            name=logger.name,
+                        ),
                     ),
                 ],
             )
@@ -694,7 +719,7 @@ def log_record(suppress_log=False, **kwargs):
     Create a log record based on the given makeLogRecord() arguments, one of which must be
     named "levelno". Log the record (unless suppress log is set) and return it.
     '''
-    record = logging.makeLogRecord(kwargs)
+    record = logging.makeLogRecord(dict(kwargs, name=logger.name))
     if suppress_log:
         return record
 
@@ -807,6 +832,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='Bootstrap successful',
+                    name=logger.name,
                 ),
             )
         except (
@@ -829,6 +855,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='Generate successful',
+                    name=logger.name,
                 ),
             )
         except (
@@ -847,6 +874,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.CRITICAL,
                     levelname='CRITICAL',
                     msg='Configuration validation failed',
+                    name=logger.name,
                 ),
             )
 
@@ -860,6 +888,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='All configuration files are valid',
+                    name=logger.name,
                 ),
             )
         except (
@@ -952,6 +981,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                         levelno=logging.INFO,
                         levelname='INFO',
                         msg=f'{config_filename}: Successfully ran configuration file',
+                        name=logger.name,
                     ),
                 )
                 if results:
@@ -1051,6 +1081,23 @@ def get_singular_option_value(configs, option_name):
         return None
 
 
+def display_summary(summary_logs, log_json):  # pragma: no cover
+    summary_logs_max_level = max(log.levelno for log in summary_logs)
+
+    for message in ('summary:',) if log_json else ('', 'summary:'):
+        log_record(
+            levelno=summary_logs_max_level,
+            levelname=logging.getLevelName(summary_logs_max_level),
+            msg=message,
+        )
+
+    for log in summary_logs:
+        logger.handle(log)
+
+    if summary_logs_max_level >= logging.CRITICAL:
+        exit_with_help_link()
+
+
 def main(extra_summary_logs=()):  # pragma: no cover
     configure_signals()
     configure_delayed_logging()
@@ -1110,6 +1157,7 @@ def main(extra_summary_logs=()):  # pragma: no cover
         getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
     )
     log_file_path = get_singular_option_value(configs, 'log_file')
+    log_json = get_singular_option_value(configs, 'log_json')
 
     try:
         configure_logging(
@@ -1119,7 +1167,8 @@ def main(extra_summary_logs=()):  # pragma: no cover
             verbosity_to_log_level(get_verbosity(configs, 'monitoring_verbosity')),
             log_file_path,
             get_singular_option_value(configs, 'log_file_format'),
-            color_enabled=should_do_markup(configs, any_json_flags),
+            log_json,
+            color_enabled=should_do_markup(configs, any_json_flags or log_json),
         )
     except (FileNotFoundError, PermissionError) as error:
         configure_logging(logging.CRITICAL)
@@ -1147,17 +1196,5 @@ def main(extra_summary_logs=()):  # pragma: no cover
             )
         )
     )
-    summary_logs_max_level = max(log.levelno for log in summary_logs)
 
-    for message in ('', 'summary:'):
+    display_summary(summary_logs, log_json)
-        log_record(
-            levelno=summary_logs_max_level,
-            levelname=logging.getLevelName(summary_logs_max_level),
-            msg=message,
-        )
-
-    for log in summary_logs:
-        logger.handle(log)
-
-    if summary_logs_max_level >= logging.CRITICAL:
-        exit_with_help_link()

borgmatic/commands/completion/bash.py

@@ -39,7 +39,7 @@ def bash_completion():
             'check_version() {',
             '    local this_script="$(cat "$BASH_SOURCE" 2> /dev/null)"',
             '    local installed_script="$(borgmatic --bash-completion 2> /dev/null)"',
-            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];'
+            '    if [ "$this_script" != "$installed_script" ] && [ "$installed_script" != "" ];',
             f'''        then cat << EOF\n{borgmatic.commands.completion.actions.upgrade_message(
                     'bash',
                     'sudo sh -c "borgmatic --bash-completion > $BASH_SOURCE"',

borgmatic/config/collect.py

@@ -1,17 +1,14 @@
 import os
 
 
-def get_default_config_paths(expand_home=True):
+def get_default_config_paths():
     '''
-    Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
+    Return a list of default configuration paths. This includes both system-wide
-    default configuration paths. This includes both system-wide configuration and configuration in
+    configuration and configuration in the current user's home directory.
-    the current user's home directory.
-
-    Don't expand the home directory ($HOME) if the expand home flag is False.
     '''
-    user_config_directory = os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
+    user_config_directory = os.path.expandvars(
-    if expand_home:
+        os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
-        user_config_directory = os.path.expandvars(user_config_directory)
+    )
 
     return [
         '/etc/borgmatic/config.yaml',

borgmatic/config/constants.py

@@ -1,4 +1,6 @@
 import contextlib
+import functools
+import re
 import shlex
 
 
@@ -21,7 +23,44 @@ def coerce_scalar(value):
         return value
 
 
-def apply_constants(value, constants, shell_escape=False):
+CONSTANT_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_constant(match, constants, command_hook):
+    '''
+    Given a re.Match instance of CONSTANT_PATTERN representing a matched constant name to be
+    interpolated, a constants dict, and whether this is for a command hook, lookup the matched
+    constant name within the given constants and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given constants, then return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name—unless this is for a command hook, in which case just return the literal
+    # string. That way, subsequent variable interpolation will still see the string as escaped
+    # instead of trying to interpolate it.
+    if match.group('left_escape') and match.group('right_escape'):
+        if command_hook:
+            return match.group(0)
+
+        return '{' + name + '}'
+
+    value = constants.get(name)
+
+    # The matched variable is in the constants, so return its value. And if this is for a command
+    # hook, then shell escape the value so as to prevent shell injection attacks.
+    if value is not None:
+        return shlex.quote(str(value)) if command_hook else str(value)
+
+    # The matched variable name isn't in the constants. Return the whole string unaltered.
+    return match.group(0)
+
+
+def apply_constants(value, constants, command_hook=False):
     '''
     Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
     replace any configuration string values of the form "{constant}" (or containing it) with the
@@ -39,26 +78,25 @@ def apply_constants(value, constants, shell_escape=False):
         return value
 
     if isinstance(value, str):
-        for constant_name, constant_value in constants.items():
-            value = value.replace(
-                '{' + constant_name + '}',
-                shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
-            )
-
         # Support constants within non-string scalars by coercing the value to its appropriate type.
-        value = coerce_scalar(value)
+        value = coerce_scalar(
+            CONSTANT_PATTERN.sub(
+                functools.partial(resolve_constant, constants=constants, command_hook=command_hook),
+                value,
+            )
+        )
     elif isinstance(value, list):
         for index, list_value in enumerate(value):
-            value[index] = apply_constants(list_value, constants, shell_escape)
+            value[index] = apply_constants(list_value, constants, command_hook)
     elif isinstance(value, dict):
         for option_name, option_value in value.items():
             value[option_name] = apply_constants(
                 option_value,
                 constants,
-                shell_escape=(
+                command_hook=(
-                    shell_escape
+                    command_hook
                     or option_name.startswith(('before_', 'after_'))
-                    or option_name == 'on_error'
+                    or option_name in {'on_error', 'run'}
                 ),
             )
 

borgmatic/config/generate.py

@@ -1,6 +1,7 @@
 import collections
 import contextlib
 import io
+import json
 import os
 import re
 
@@ -107,7 +108,7 @@ def comment_out_line(line):
     return '# '.join((indent_spaces, line[count_indent_spaces:]))
 
 
-def comment_out_optional_configuration(rendered_config):
+def transform_optional_configuration(rendered_config, comment_out=True):
     '''
     Post-process a rendered configuration string to comment out optional key/values, as determined
     by a sentinel in the comment before each key.
@@ -117,6 +118,9 @@ def comment_out_optional_configuration(rendered_config):
 
     Ideally ruamel.yaml would support commenting out keys during configuration generation, but it's
     not terribly easy to accomplish that way.
+
+    If comment_out is False, then just strip the comment sentinel without actually commenting
+    anything out.
     '''
     lines = []
     optional = False
@@ -129,6 +133,9 @@ def comment_out_optional_configuration(rendered_config):
         # Upon encountering an optional configuration option, comment out lines until the next blank
         # line.
         if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
+            if comment_out is False:
+                continue
+
             optional = True
             indent_characters_at_sentinel = indent_characters
             continue
@@ -310,19 +317,39 @@ def merge_source_configuration_into_destination(destination_config, source_confi
     return destination_config
 
 
+def get_configuration_subset(config, option_name):  # pragma: no cover
+    '''
+    Given configuration as a ruamel.yaml.CommentedMap and an option name found within it at the top
+    level, return a new CommentedMap containing a subset of the configuration with only the given
+    option and no other top-level options.
+
+    This is useful when generating the sample configuration for a single option instead of a whole
+    configuration file.
+    '''
+    option_config = ruamel.yaml.CommentedMap({option_name: config[option_name]})
+
+    # Due to a quirk of ruamel.yaml, the comment right before a top-level key is not on that key and
+    # needs to get copied separately.
+    option_config.ca.items[option_name] = config.ca.items[option_name]
+
+    return option_config
+
+
 def generate_sample_configuration(
     dry_run,
     source_filename,
-    destination_filename,
+    destination_path,
     schema_filename,
     overwrite=False,
+    split=False,
 ):
     '''
-    Given an optional source configuration filename, and a required destination configuration
+    Given an optional source configuration filename, a required destination configuration path, the
-    filename, the path to a schema filename in a YAML rendition of the JSON Schema format, and
+    path to a schema filename in a YAML rendition of the JSON Schema format, whether to overwrite a
-    whether to overwrite a destination file, write out a sample configuration file based on that
+    destination file, and whether to split the configuration into multiple files (one per option) in
-    schema. If a source filename is provided, merge the parsed contents of that configuration into
+    the assumed destination directory, write out sample configuration file(s) based on that schema.
-    the generated configuration.
+    If a source filename is provided, merge the parsed contents of that configuration into the
+    generated configuration.
     '''
     schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename, encoding='utf-8'))
     source_config = None
@@ -345,8 +372,37 @@ def generate_sample_configuration(
     if dry_run:
         return
 
+    if split:
+        if os.path.exists(destination_path) and not os.path.isdir(destination_path):
+            raise ValueError('With the --split flag, the destination path must be a directory')
+
+        os.makedirs(destination_path, exist_ok=True)
+
+        for option_name in destination_config:
+            write_configuration(
+                os.path.join(destination_path, f'{option_name}.yaml'),
+                transform_optional_configuration(
+                    render_configuration(get_configuration_subset(destination_config, option_name)),
+                    comment_out=False,
+                ).strip(),
+                overwrite=overwrite,
+            )
+
+        # Also dump a manifest listing all the options we've written.
+        json.dump(
+            {'option_names': list(destination_config.keys())},
+            open(os.path.join(destination_path, 'options.json'), 'w', encoding='utf-8'),
+        )
+
+        return
+
+    if os.path.exists(destination_path) and not os.path.isfile(destination_path):
+        raise ValueError('Without the --split flag, the destination path must be a file')
+
     write_configuration(
-        destination_filename,
+        destination_path,
-        comment_out_optional_configuration(render_configuration(destination_config)),
+        transform_optional_configuration(
+            render_configuration(destination_config), comment_out=True
+        ),
         overwrite=overwrite,
     )

borgmatic/config/paths.py

@@ -2,10 +2,19 @@ import contextlib
 import logging
 import os
 import tempfile
+from enum import Enum
 
 logger = logging.getLogger(__name__)
 
 
+class Systemd_directories(Enum):
+    RUNTIME_DIRECTORY = 0
+    STATE_DIRECTORY = 1
+    CACHE_DIRECTORY = 2
+    LOGS_DIRECTORY = 3
+    CONFIGURATION_DIRECTORY = 4
+
+
 def expand_user_in_path(path):
     '''
     Given a directory path, expand any tildes in it.
@@ -16,6 +25,17 @@ def expand_user_in_path(path):
         return None
 
 
+def resolve_systemd_directory(directory):
+    '''
+    Given a systemd directory environment variable enum, read the value if set and return the first
+    configured directory.
+    '''
+    separator = ':'
+
+    paths = os.environ.get(directory.name)
+    return paths.split(separator)[0] if paths else None
+
+
 def get_working_directory(config):  # pragma: no cover
     '''
     Given a configuration dict, get the working directory from it, expanding any tildes.
@@ -96,7 +116,9 @@ class Runtime_directory:
         runtime_directory = (
             config.get('user_runtime_directory')
             or os.environ.get('XDG_RUNTIME_DIR')  # Set by PAM on Linux.
-            or os.environ.get('RUNTIME_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
+                Systemd_directories.RUNTIME_DIRECTORY
+            )  # Set by systemd if configured.
         )
 
         if runtime_directory:
@@ -174,7 +196,9 @@ def get_borgmatic_state_directory(config):
         os.path.join(
             config.get('user_state_directory')
             or os.environ.get('XDG_STATE_HOME')
-            or os.environ.get('STATE_DIRECTORY')  # Set by systemd if configured.
+            or resolve_systemd_directory(
+                Systemd_directories.STATE_DIRECTORY
+            )  # Set by systemd if configured.
             or '~/.local/state',
             'borgmatic',
         ),

borgmatic/config/schema.yaml

@@ -27,6 +27,15 @@ properties:
             - /etc
             - /var/log/syslog*
             - /home/user/path with spaces
+    source_directories_must_exist:
+        type: boolean
+        description: |
+            Deprecated. Replaced by borgmatic treating Borg's "backup file not
+            found" warning as an error by default. But if
+            "source_directories_must_exist" is true, then source directories
+            (and root pattern paths) must exist before a backup begins. If they
+            don't, borgmatic errors. Defaults to false.
+        example: true
     repositories:
         type: array
         items:
@@ -73,7 +82,7 @@ properties:
                         Whether any missing parent directories of the repository
                         path should be created, only used for the repo-create
                         action. Defaults to false. (This option is supported 
-                        for Borg 1.x only)
+                        for Borg 1.x only.)
                     example: true
         description: |
             A required list of local or remote repositories with paths and
@@ -134,6 +143,23 @@ properties:
             false. But when a database hook is used, the setting here is ignored
             and read_special is considered true.
         example: true
+    unsafe_skip_path_validation_before_create:
+        type: boolean
+        description: |
+            Skip pre-backup safety validation: (1) runtime directory exclusion
+            detection, and (2) special file exclusion for database streaming.
+            Can significantly improve performance for large filesystems.
+
+            UNSAFE because, when this option is enabled, Borg may hang on
+            special files (named pipes, devices) and data from borgmatic data
+            source hooks may be silently excluded if your excluded patterns
+            match the runtime directory (see the "user_runtime_directory"
+            option).
+
+            Only enable if not using data source hooks and you manually verify
+            that your excludes don't affect the runtime directory. Defaults
+            to false.
+        example: true
     flags:
         type: boolean
         description: |
@@ -163,9 +189,13 @@ properties:
             type: string
         description: |
             Any paths matching these patterns are included/excluded from
-            backups. Globs are expanded. (Tildes are not.) See the output of
+            backups. Recursion root patterns ("R ...") are effectively the same
-            "borg help patterns" for more details. Quote any value if it
+            as "source_directories"; they tell Borg which paths to backup
-            contains leading punctuation, so it parses correctly.
+            (modulo any excludes). Globs are expanded. (Tildes are not.) See
+            the output of "borg help patterns" for more details. Quote any value
+            if it contains leading punctuation, so it parses correctly. Also use
+            leading slashes in absolute paths, or data source hooks may be
+            unable to rewrite patterns as needed.
         example:
             - 'R /'
             - '- /home/*/.cache'
@@ -232,7 +262,8 @@ properties:
     exclude_nodump:
         type: boolean
         description: |
-            Exclude files with the NODUMP flag. Defaults to false.
+            Exclude files with the NODUMP flag. Defaults to false. (This option
+            is supported for Borg 1.x only.)
         example: true
     borgmatic_source_directory:
         type: string
@@ -248,7 +279,7 @@ properties:
             Path for storing temporary runtime data like streaming database
             dumps and bootstrap metadata. borgmatic automatically creates and
             uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR
-            or or $TMPDIR or $TEMP or /run/user/$UID.
+            or $TMPDIR or $TEMP or /run/user/$UID.
         example: /run/user/1001
     user_state_directory:
         type: string
@@ -259,12 +290,6 @@ properties:
             create the check records again (and therefore re-run checks).
             Defaults to $XDG_STATE_HOME or ~/.local/state.
         example: /var/lib/borgmatic
-    source_directories_must_exist:
-        type: boolean
-        description: |
-            If true, then source directories (and root pattern paths) must
-            exist. If they don't, an error is raised. Defaults to false.
-        example: true
     encryption_passcommand:
         type: string
         description: |
@@ -275,7 +300,7 @@ properties:
             encryption_passphrase takes precedence. This can also be used to
             access encrypted systemd service credentials. Defaults to not set.
             For more details, see:
-            https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+            https://torsion.org/borgmatic/how-to/provide-your-passwords/
         example: "secret-tool lookup borg-repository repo-name"
     encryption_passphrase:
         type: string
@@ -406,14 +431,20 @@ properties:
         type: string
         description: |
             Path for Borg security and encryption nonce files. Defaults to
-            $borg_base_directory/.config/borg/security
+            $borg_config_directory/security
         example: /path/to/base/config/security
     borg_keys_directory:
         type: string
         description: |
             Path for Borg encryption key files. Defaults to
-            $borg_base_directory/.config/borg/keys
+            $borg_config_directory/keys
         example: /path/to/base/config/keys
+    borg_key_file:
+        type: string
+        description: |
+            Path for the Borg repository key file, for use with a repository
+            created with "keyfile" encryption.
+        example: /path/to/base/config/keyfile
     borg_exit_codes:
         type: array
         items:
@@ -437,9 +468,9 @@ properties:
         description: |
             A list of Borg exit codes that should be elevated to errors or
             squashed to warnings as indicated. By default, Borg error exit codes
-            (2 to 99) are treated as errors while warning exit codes (1 and
+            (2 to 99) are treated as errors and most warning exit codes (1 and
-            100+) are treated as warnings. Exit codes other than 1 and 2 are
+            100+) are treated as errors as well. Exit codes other than 1 and 2
-            only present in Borg 1.4.0+.
+            are only present in Borg 1.4.0+.
         example:
             - code: 13
               treat_as: warning
@@ -479,6 +510,27 @@ properties:
             If match_archives is not specified, borgmatic defaults to deriving
             the match_archives value from archive_name_format.
         example: "sh:{hostname}-*"
+    file_list_format:
+        type: string
+        description: |
+            Borg format for the files listing of the "list" action. Corresponds
+            to the "--format" flag. Defaults to "{mode} {user:6} {group:6}
+            {size:8} {mtime} {path}{extra}{NL}". With "--json", the form of the
+            format is ignored, but the keys used in it are added to the JSON
+            output. See
+            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
+            details.
+        example: "{path}{extra}{NL}"
+    archive_list_format:
+        type: string
+        description: |
+            Borg format for the archives listing of the "repo-list" action.
+            Corresponds to the "--format" flag. Defaults to "{archive:<36}
+            {time} [{id}]{NL}". With "--json", the form of the format is
+            ignored, but the keys used in it are added to the JSON output. See
+            https://borgbackup.readthedocs.io/en/stable/usage/list.html for
+            details.
+        example: "{archive}{NL}"
     relocated_repo_access_is_ok:
         type: boolean
         description: |
@@ -514,30 +566,123 @@ properties:
         type: object
         additionalProperties: false
         properties:
-            init:
+            break_lock:
                 type: string
                 description: |
-                  Extra command-line options to pass to "borg init".
+                  Extra command-line options to pass to "borg break-lock".
                 example: "--extra-option"
-            create:
+            check:
                 type: string
                 description: |
-                  Extra command-line options to pass to "borg create".
+                  Extra command-line options to pass to "borg check".
-                example: "--extra-option"
-            prune:
-                type: string
-                description: |
-                  Extra command-line options to pass to "borg prune".
                 example: "--extra-option"
             compact:
                 type: string
                 description: |
                   Extra command-line options to pass to "borg compact".
                 example: "--extra-option"
-            check:
+            create:
                 type: string
                 description: |
-                  Extra command-line options to pass to "borg check".
+                  Extra command-line options to pass to "borg create".
+                example: "--extra-option"
+            delete:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg delete".
+                example: "--extra-option"
+            export_tar:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg export-tar".
+                example: "--extra-option"
+            extract:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg extract".
+                example: "--extra-option"
+            key_export:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key export".
+                example: "--extra-option"
+            key_import:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key import".
+                example: "--extra-option"
+            key_change_passphrase:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg key
+                  change-passphrase".
+                example: "--extra-option"
+            info:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg info".
+                example: "--extra-option"
+            init:
+                type: string
+                description: |
+                  Deprecated. Use "repo_create" instead. Extra command-line
+                  options to pass to "borg init" / "borg repo-create".
+                example: "--extra-option"
+            list:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg list".
+                example: "--extra-option"
+            mount:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg mount".
+                example: "--extra-option"
+            prune:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg prune".
+                example: "--extra-option"
+            recreate:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg recreate".
+                example: "--extra-option"
+            rename:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg rename".
+                example: "--extra-option"
+            repo_create:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg init" / "borg
+                  repo-create".
+                example: "--extra-option"
+            repo_delete:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-delete".
+                example: "--extra-option"
+            repo_info:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-info".
+                example: "--extra-option"
+            repo_list:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg repo-list".
+                example: "--extra-option"
+            transfer:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg transfer".
+                example: "--extra-option"
+            umount:
+                type: string
+                description: |
+                  Extra command-line options to pass to "borg umount".
                 example: "--extra-option"
         description: |
             Additional options to pass directly to particular Borg commands,
@@ -892,6 +1037,8 @@ properties:
         type: string
         description: |
             Python format string used for log messages written to the log file.
+            See https://docs.python.org/3/library/logging.html (and specifically
+            the LogRecord attributes with "{}-formatting") for details.
         example: "[{asctime}] {levelname}: {prefix}{message}"
     monitoring_verbosity:
         type: integer
@@ -1541,6 +1688,15 @@ properties:
                         database hook implicitly enables read_special (see
                         above) to support dump and restore streaming.
                     example: users
+                skip_names:
+                    type: array
+                    items:
+                        type: string
+                    description: |
+                        Database names to skip when dumping "all" databases.
+                        Ignored when the database name is not "all".
+                    example:
+                        - cache
                 label:
                     type: string
                     description: |
@@ -1579,6 +1735,20 @@ properties:
                     description: |
                         Port to restore to. Defaults to the "port" option.
                     example: 5433
+                socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
+                    example: database.example.org
+                restore_socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to the "socket_path" option.
+                    example: database.example.org
                 username:
                     type: string
                     description: |
@@ -1642,18 +1812,21 @@ properties:
                 mariadb_dump_command:
                     type: string
                     description: |
-                        Command to use instead of "mariadb-dump". This can be
+                        Command to use instead of "mariadb-dump". But set any
-                        used to run a specific mariadb_dump version (e.g., one
+                        additional command-line flags in "options", not here.
-                        inside a running container). If you run it from within a
+                        This command can be used to run a specific mariadb_dump
-                        container, make sure to mount the path in the
+                        version (e.g., one inside a running container). If you
-                        "user_runtime_directory" option from the host into the
+                        run it from within a container, make sure to mount the
-                        container at the same location. Defaults to
+                        path in the "user_runtime_directory" option from the
-                        "mariadb-dump".
+                        host into the container at the same location. Defaults
+                        to "mariadb-dump".
                     example: docker exec mariadb_container mariadb-dump
                 mariadb_command:
                     type: string
                     description: |
-                        Command to run instead of "mariadb". This can be used to
+                        Command to run instead of "mariadb". But set any
+                        additional command-line flags in "list_options" or
+                        "restore_options", not here. This command can be used to
                         run a specific mariadb version (e.g., one inside a
                         running container). Defaults to "mariadb".
                     example: docker exec mariadb_container mariadb           
@@ -1723,6 +1896,15 @@ properties:
                         database hook implicitly enables read_special (see
                         above) to support dump and restore streaming.
                     example: users
+                skip_names:
+                    type: array
+                    items:
+                        type: string
+                    description: |
+                        Database names to skip when dumping "all" databases.
+                        Ignored when the database name is not "all".
+                    example:
+                        - cache
                 label:
                     type: string
                     description: |
@@ -1761,6 +1943,20 @@ properties:
                     description: |
                         Port to restore to. Defaults to the "port" option.
                     example: 5433
+                socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to "/run/mysqld/mysqld.sock".
+                    example: database.example.org
+                restore_socket_path:
+                    type: string
+                    description: |
+                        Path of a Unix socket to connect to instead of a remote
+                        hostname. Ignored when "hostname" is set and not
+                        "localhost". Defaults to the "socket_path" option.
+                    example: database.example.org
                 username:
                     type: string
                     description: |
@@ -1824,17 +2020,21 @@ properties:
                 mysql_dump_command:
                     type: string
                     description: |
-                        Command to use instead of "mysqldump". This can be used
+                        Command to use instead of "mysqldump". But set any
-                        to run a specific mysql_dump version (e.g., one inside a
+                        additional command-line flags in "options", not here.
-                        running container). If you run it from within a
+                        This command can be used to run a specific mysql_dump
-                        container, make sure to mount the path in the
+                        version (e.g., one inside a running container). If you
-                        "user_runtime_directory" option from the host into the
+                        run it from within a container, make sure to mount the
-                        container at the same location. Defaults to "mysqldump".
+                        path in the "user_runtime_directory" option from the
+                        host into the container at the same location. Defaults
+                        to "mysqldump".
                     example: docker exec mysql_container mysqldump
                 mysql_command:
                     type: string
                     description: |
-                        Command to run instead of "mysql". This can be used to
+                        Command to run instead of "mysql". But set any
+                        additional command-line flags in "list_options" or
+                        "restore_options", not here. This command can be used to
                         run a specific mysql version (e.g., one inside a running
                         container). Defaults to "mysql".
                     example: docker exec mysql_container mysql
@@ -2139,17 +2339,24 @@ properties:
                         type: string
                         description: |
                             The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have started.
                     priority:
                         type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                         description: |
                             The priority to set.
-                        example: urgent
+                        example: min
                     tags:
                         type: string
                         description: |
                             Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic
             finish:
                 type: object
                 additionalProperties: false
@@ -2163,17 +2370,24 @@ properties:
                         type: string
                         description: |
                             The message body to publish.
-                        example: Your backups have failed.
+                        example: Your backups have finished.
                     priority:
                         type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                         description: |
                             The priority to set.
-                        example: urgent
+                        example: min
                     tags:
                         type: string
                         description: |
                             Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,+1
             fail:
                 type: object
                 additionalProperties: false
@@ -2190,14 +2404,21 @@ properties:
                         example: Your backups have failed.
                     priority:
                         type: string
+                        enum:
+                            - max
+                            - urgent
+                            - high
+                            - default
+                            - low
+                            - min
                         description: |
                             The priority to set.
-                        example: urgent
+                        example: max
                     tags:
                         type: string
                         description: |
                             Tags to attach to the message.
-                        example: incoming_envelope
+                        example: borgmatic,-1,skull
             states:
                 type: array
                 items:
@@ -2606,8 +2827,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to Apprise services as part of the
-                    "finish", "fail", and "log" states. Defaults to true.
+                    "finish", "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
             logs_size_limit:
                 type: integer
                 description: |
@@ -2721,8 +2942,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to Healthchecks as part of the "finish",
-                    "fail", and "log" states. Defaults to true.
+                    "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
             ping_body_limit:
                 type: integer
                 description: |
@@ -2828,8 +3049,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to PagerDuty when a backup errors.
-                    Defaults to true.
+                    Defaults to false.
-                example: false
+                example: true
         description: |
             Configuration for a monitoring integration with PagerDuty. Create an
             account at https://www.pagerduty.com if you'd like to use this
@@ -2857,7 +3078,7 @@ properties:
             url:
                 type: string
                 description: |
-                    Grafana loki log URL to notify when a backup begins,
+                    Grafana Loki log URL to notify when a backup begins,
                     ends, or fails.
                 example: "http://localhost:3100/loki/api/v1/push"
             labels:
@@ -2874,6 +3095,11 @@ properties:
                     app: "borgmatic"
                     config: "__config"
                     hostname: "__hostname"
+            send_logs:
+                type: boolean
+                description: |
+                    Send borgmatic logs to Loki. Defaults to false.
+                example: true
         description: |
             Configuration for a monitoring integration with Grafana Loki. You
             can send the logs to a self-hosted instance or create an account at
@@ -2898,6 +3124,12 @@ properties:
                     project monitor. Used along with the data source name URL to
                     construct a cron URL.
                 example: mymonitor
+            environment:
+                type: string
+                description: |
+                    Sentry monitor environment used in the call to Sentry. If
+                    not set, the Sentry default is used.
+                example: production
             states:
                 type: array
                 items:
@@ -2951,7 +3183,8 @@ properties:
             findmnt_command:
                 type: string
                 description: |
-                    Command to use instead of "findmnt".
+                    Deprecated and unused. Was the command to use instead of
+                    "findmnt".
                 example: /usr/local/bin/findmnt
         description: |
             Configuration for integration with the Btrfs filesystem.
@@ -3041,6 +3274,14 @@ properties:
                 description: |
                     Command to use instead of "keepassxc-cli".
                 example: /usr/local/bin/keepassxc-cli
+            ask_for_password:
+                type: boolean
+                description: |
+                    Whether keepassxc-cli should prompt the user for a password.
+                    Disabling this is only really useful if you're unlocking
+                    your KeePassXC database with a key file instead of a
+                    password. Defaults to true.
+                example: false
             key_file:
                 type: string
                 description: |

borgmatic/execute.py

@@ -1,9 +1,13 @@
 import collections
+import contextlib
 import enum
+import json
 import logging
+import os
 import select
 import subprocess
 import textwrap
+import time
 
 import borgmatic.logger
 
@@ -14,6 +18,9 @@ ERROR_OUTPUT_MAX_LINE_COUNT = 25
 BORG_ERROR_EXIT_CODE_START = 2
 BORG_ERROR_EXIT_CODE_END = 99
 
+# See https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids
+BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS = {101, 102, 104, 105, 106, 107}
+
 
 class Exit_status(enum.Enum):
     STILL_RUNNING = 1
@@ -22,7 +29,20 @@ class Exit_status(enum.Enum):
     ERROR = 4
 
 
-def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
+def command_is_borg(command, borg_local_path):
+    '''
+    Given a command as a sequence and the Borg local path, return whether that command is a call to
+    Borg.
+    '''
+    parsed_command = command.split(' ', 1) if isinstance(command, str) else command
+
+    if not parsed_command:
+        return False
+
+    return bool(borg_local_path and parsed_command[0] == borg_local_path)
+
+
+def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):  # noqa: PLR0911
     '''
     Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
     exit code. If a Borg local path is given and matches the process' command, then interpret the
@@ -35,7 +55,7 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
     if exit_code == 0:
         return Exit_status.SUCCESS
 
-    if borg_local_path and command[0] == borg_local_path:
+    if command_is_borg(command, borg_local_path):
         # First try looking for the exit code in the borg_exit_codes configuration.
         for entry in borg_exit_codes or ():
             if entry.get('code') == exit_code:
@@ -53,8 +73,15 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
                     )
                     return Exit_status.WARNING
 
-        # If the exit code doesn't have explicit configuration, then fall back to the default Borg
+        # If the exit code doesn't have explicit configuration, then fall back to the default
-        # behavior.
+        # behavior of treating Borg errors as errors and some Borg warnings as errors.
+        if exit_code in BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS:
+            logger.error(
+                f'Treating exit code {exit_code} as an error, as per borgmatic defaults',
+            )
+
+            return Exit_status.ERROR
+
         return (
             Exit_status.ERROR
             if (
@@ -78,158 +105,398 @@ def command_for_process(process):
     return process.args if isinstance(process.args, str) else ' '.join(process.args)
 
 
-def output_buffer_for_process(process, exclude_stdouts):
+def output_buffers_for_process(process, exclude_stdouts):
     '''
     Given a process as an instance of subprocess.Popen and a sequence of stdouts to exclude, return
-    either the process's stdout or stderr. The idea is that if stdout is excluded for a process, we
+    the process stdout and stderr as a tuple—but exclude the stdout if it's in the given stdouts to
-    still have stderr to log.
+    exclude.
     '''
-    return process.stderr if process.stdout in exclude_stdouts else process.stdout
+    return tuple(
+        buffer for buffer in (process.stdout, process.stderr) if buffer not in exclude_stdouts
+    )
 
 
-def append_last_lines(last_lines, captured_output, line, output_log_level):
+BORG_LOG_LEVEL_ELEVATION_THRESHOLD = 10
+
+
+def borg_json_log_line_to_record(line, log_level):
     '''
-    Given a rolling list of last lines, a list of captured output, a line to append, and an output
+    Given a single Borg "--log-json"-style log line and a log level, return the line converted to a
-    log level, append the line to the last lines and (if necessary) the captured output. Then log
+    logging.LogRecord instance. Return None if the line can't be parsed as JSON.
-    the line at the requested output log level.
+
+    If Borg provides a log level in its JSON, prefer logging at that level. But if Borg doesn't
+    provide a log level—or the log level given to this function is just a little bit higher than
+    Borg's—elevate to that level. This supports use cases like elevating Borg's INFO level logs to
+    borgmatic's custom ANSWER level so that requested data shows up even at the default verbosity.
     '''
-    last_lines.append(line)
+    with contextlib.suppress(json.JSONDecodeError, TypeError, KeyError, AttributeError):
+        log_data = json.loads(line)
+        log_type = log_data.get('type')
 
-    if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
+        if log_type == 'log_message':
-        last_lines.pop(0)
+            borg_log_level = logging._nameToLevel.get(log_data.get('levelname'))
+            log_level_delta = 0 if log_level is None else log_level - borg_log_level
 
-    if output_log_level is None:
+            if log_level_delta > 0 and log_level_delta < BORG_LOG_LEVEL_ELEVATION_THRESHOLD:
-        captured_output.append(line)
+                return logging.makeLogRecord(
-    else:
+                    dict(
-        logger.log(output_log_level, line)
+                        levelno=log_level,
-
+                        created=log_data.get('time'),
-
+                        msg=log_data.get('message'),
-def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):  # noqa: PLR0912
+                        levelname=logging.getLevelName(log_level),
-    '''
+                        name=log_data.get('name'),
-    Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
-    process with the requested log level. Additionally, raise a CalledProcessError if a process
-    exits with an error (or a warning for exit code 1, if that process does not match the Borg local
-    path).
-
-    If output log level is None, then instead of logging, capture output for each process and return
-    it as a dict from the process to its output. Use the given Borg local path and exit code
-    configuration to decide what's an error and what's a warning.
-
-    For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
-    stdouts are given to exclude, then for any matching processes, log from their stderr instead.
-
-    Note that stdout for a process can be None if output is intentionally not captured. In which
-    case it won't be logged.
-    '''
-    # Map from output buffer to sequence of last lines.
-    buffer_last_lines = collections.defaultdict(list)
-    process_for_output_buffer = {
-        output_buffer_for_process(process, exclude_stdouts): process
-        for process in processes
-        if process.stdout or process.stderr
-    }
-    output_buffers = list(process_for_output_buffer.keys())
-    captured_outputs = collections.defaultdict(list)
-    still_running = True
-
-    # Log output for each process until they all exit.
-    while True:  # noqa: PLR1702
-        if output_buffers:
-            (ready_buffers, _, _) = select.select(output_buffers, [], [])
-
-            for ready_buffer in ready_buffers:
-                ready_process = process_for_output_buffer.get(ready_buffer)
-
-                # The "ready" process has exited, but it might be a pipe destination with other
-                # processes (pipe sources) waiting to be read from. So as a measure to prevent
-                # hangs, vent all processes when one exits.
-                if ready_process and ready_process.poll() is not None:
-                    for other_process in processes:
-                        if (
-                            other_process.poll() is None
-                            and other_process.stdout
-                            and other_process.stdout not in output_buffers
-                        ):
-                            # Add the process's output to output_buffers to ensure it'll get read.
-                            output_buffers.append(other_process.stdout)
-
-                while True:
-                    line = ready_buffer.readline().rstrip().decode()
-                    if not line or not ready_process:
-                        break
-
-                    # Keep the last few lines of output in case the process errors, and we need the output for
-                    # the exception below.
-                    append_last_lines(
-                        buffer_last_lines[ready_buffer],
-                        captured_outputs[ready_process],
-                        line,
-                        output_log_level,
                     )
+                )
 
-        if not still_running:
+            return logging.makeLogRecord(
-            break
+                dict(
+                    levelno=borg_log_level,
+                    created=log_data.get('time'),
+                    msg=log_data.get('message'),
+                    levelname=log_data.get('levelname'),
+                    name=log_data.get('name'),
+                )
+            )
 
-        still_running = False
+        if log_type == 'file_status':
-
+            return logging.makeLogRecord(
-        for process in processes:
+                dict(
-            exit_code = process.poll() if output_buffers else process.wait()
+                    levelno=log_level,
-
+                    created=time.time(),
-            if exit_code is None:
+                    msg=f'{log_data.get("status")} {log_data.get("path")}',
-                still_running = True
+                    levelname=logging.getLevelName(log_level),
-                command = process.args.split(' ') if isinstance(process.args, str) else process.args
+                    name='borg.file_status',
-                continue
+                )
-
+            )
-            command = process.args.split(' ') if isinstance(process.args, str) else process.args
-            exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
-
-            if exit_status in {Exit_status.ERROR, Exit_status.WARNING}:
-                # If an error occurs, include its output in the raised exception so that we don't
-                # inadvertently hide error output.
-                output_buffer = output_buffer_for_process(process, exclude_stdouts)
-                last_lines = buffer_last_lines[output_buffer] if output_buffer else []
-
-                # Collect any straggling output lines that came in since we last gathered output.
-                while output_buffer:  # pragma: no cover
-                    line = output_buffer.readline().rstrip().decode()
-                    if not line:
-                        break
-
-                    append_last_lines(
-                        last_lines,
-                        captured_outputs[process],
-                        line,
-                        output_log_level=logging.ERROR,
-                    )
-
-                if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
-                    last_lines.insert(0, '...')
-
-                # Something has gone wrong. So vent each process' output buffer to prevent it from
-                # hanging. And then kill the process.
-                for other_process in processes:
-                    if other_process.poll() is None:
-                        other_process.stdout.read(0)
-                        other_process.kill()
-
-                if exit_status == Exit_status.ERROR:
-                    raise subprocess.CalledProcessError(
-                        exit_code,
-                        command_for_process(process),
-                        '\n'.join(last_lines),
-                    )
-
-                still_running = False
-                break
-
-    if captured_outputs:
-        return {
-            process: '\n'.join(output_lines) for process, output_lines in captured_outputs.items()
-        }
 
     return None
 
 
+def log_line_to_record(line, log_level):
+    '''
+    Given a log data dict for a single Borg log entry and a log level, return it converted to a
+    logging.LogRecord instance.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            msg=line,
+            levelno=log_level,
+            levelname=logging.getLevelName(log_level),
+        )
+    )
+
+
+def parse_log_line(line, log_level, elevate_stderr, borg_local_path, command):
+    '''
+    Given a raw output line from an external program, whether this line came from stderr and should
+    be elevated to error/warning, the Borg local path, and the command as a sequence, return a
+    logging.LogRecord instance containing its parsed data.
+
+    If the command being run is Borg, and the log line is JSON-formatted log data, then grab the log
+    level from it and log the parsed JSON to be consumed later by a Python logging.Formatter.
+
+    But for non-Borg commands, elevate stderr-sourced logs to ERROR. The one exception is if the log
+    came from stderr and the string "warning:" appears at the start of the log line. In that case,
+    just elevate the log level to a WARN.
+    '''
+    if command_is_borg(command, borg_local_path):
+        log_record = borg_json_log_line_to_record(line, log_level)
+
+        if log_record:
+            return log_record
+
+    if elevate_stderr:
+        return log_line_to_record(
+            line, logging.WARNING if line.lower().startswith('warning:') else logging.ERROR
+        )
+
+    return log_line_to_record(line, log_level)
+
+
+def handle_log_record(log_record, last_lines=None):
+    '''
+    Given a log record to be logged and a rolling list of last lines, append the record's message to
+    the last lines (if given). Then (if the log level is not None), log the record.
+
+    Return the log record.
+    '''
+    log_message = log_record.getMessage()
+
+    if last_lines is not None:
+        last_lines.append(log_message)
+
+        if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
+            last_lines.pop(0)
+
+    if log_record.levelno is not None:
+        logger.handle(log_record)
+
+    return log_record
+
+
+READ_CHUNK_SIZE = 4096
+
+
+def read_lines(buffer, process, line_separator='\n'):
+    '''
+    Given a Python buffer (like stdout) ready for reading, its process, and a line separator,
+    repeatedly yield a tuple of (decoded) lines from the buffer until the process has exited.
+
+    It is assumed that this function's generator is used in conjunction with an external select()
+    call to know when to read more lines. Otherwise, the generator will busywait if it's called in a
+    tight loop.
+    '''
+    data = b''
+    encoded_separator = line_separator.encode()
+    separator_size = len(encoded_separator)
+
+    while True:
+        chunk = os.read(buffer.fileno(), READ_CHUNK_SIZE)
+
+        if not chunk:  # EOF
+            # The process is still running, so we keep running too.
+            if process.poll() is None:  # pragma: no cover
+                continue
+
+            break
+
+        data += chunk
+        lines = []
+
+        # Split the data into lines, holding back anything leftover that might
+        # be a partial line.
+        while True:
+            separator_position = data.find(encoded_separator)
+
+            if separator_position == -1:
+                break
+
+            lines.append(data[:separator_position].decode())
+            data = data[separator_position + separator_size :]
+
+        yield tuple(lines)
+
+    # Yield any leftover data from the end of the buffer.
+    if data:
+        yield (data.decode().rstrip(),)
+
+
+Buffer_reader = collections.namedtuple(
+    'Buffer_reader',
+    ('lines', 'process'),
+)
+
+
+Process_metadata = collections.namedtuple(
+    'Process_metadata',
+    ('last_lines', 'capture'),
+)
+
+
+def log_buffer_lines(
+    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
+):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
+    whether to capture stderr, read and log any ready output lines from the buffers.  Additionally,
+    if the log level is None for any log record, then yield those log messages for capture.
+
+    This function just does one "turn of the crank" of logging buffer output. It is intended to be
+    called repeatedly to continue to process buffers.
+    '''
+    if not buffer_readers:
+        return
+
+    (ready_buffers, _, _) = select.select(buffer_readers.keys(), [], [])
+
+    for ready_buffer in ready_buffers:
+        reader = buffer_readers[ready_buffer]
+
+        # The "ready" process has exited, but it might be a pipe destination with other
+        # processes (pipe sources) waiting to be read from. So as a measure to prevent
+        # hangs, vent all processes when one exits.
+        if reader.process and reader.process.poll() is not None:
+            for other_process in process_metadatas:
+                if (
+                    other_process.poll() is None
+                    and other_process.stdout
+                    and other_process.stdout not in buffer_readers
+                ):
+                    # Add the process's output to buffer_readers to ensure it'll get read.
+                    buffer_readers[other_process.stdout] = Buffer_reader(
+                        read_lines(other_process.stdout, other_process), other_process
+                    )
+
+        try:
+            lines = next(reader.lines)
+        except StopIteration:
+            continue
+
+        for line in lines:
+            if not line or not reader.process:
+                continue
+
+            # Keep the last few lines of output in case the process errors and we need the
+            # output for the exception below.
+            log_record = handle_log_record(
+                parse_log_line(
+                    line=line,
+                    log_level=output_log_level,
+                    elevate_stderr=(ready_buffer == reader.process.stderr and not capture_stderr),
+                    borg_local_path=borg_local_path,
+                    command=reader.process.args,
+                ),
+                last_lines=process_metadatas[reader.process].last_lines,
+            )
+
+            if log_record.levelno is None and process_metadatas[reader.process].capture:
+                yield log_record.getMessage()
+
+
+def raise_for_process_errors(buffer_readers, process_metadatas, borg_local_path, borg_exit_codes):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, Borg's local path, a sequence of exit code configuration dicts, check
+    the given processes for error or warning exit codes. If found, vent or kill any running
+    processes. In the case of an error exit code, raise. In the case of warning, return
+    Exit_status.WARNING. Otherwise, return None.
+    '''
+    result_status = None
+
+    for process in process_metadatas:
+        exit_code = process.poll() if buffer_readers else process.wait()
+
+        if exit_code is None:
+            continue
+
+        exit_status = interpret_exit_code(process.args, exit_code, borg_local_path, borg_exit_codes)
+
+        if exit_status not in {Exit_status.ERROR, Exit_status.WARNING}:
+            continue
+
+        # Something has gone wrong. So vent each process' output buffer to prevent it from
+        # hanging. And then kill the process.
+        for other_process in process_metadatas:
+            if other_process.poll() is None:
+                other_process.stdout.read(0)
+                other_process.kill()
+
+        if exit_status == Exit_status.WARNING:
+            result_status = Exit_status.WARNING
+            continue
+
+        last_lines = process_metadatas[process].last_lines
+
+        # If an error occurs, include its output in the raised exception so that we don't
+        # inadvertently hide error output.
+        if len(last_lines) >= ERROR_OUTPUT_MAX_LINE_COUNT:
+            last_lines.insert(0, '...')
+
+        raise subprocess.CalledProcessError(
+            exit_code,
+            command_for_process(process),
+            '\n'.join(last_lines),
+        )
+
+    return result_status
+
+
+def log_remaining_buffer_lines(
+    buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr=False
+):
+    '''
+    Given a dict from buffer object to Buffer_reader, a dict from subprocess.Popen() instance to
+    Process_metadata instance, a requested output log level for stdout, Borg's local path, and
+    whether to capture stderr, drain and log any remaining output lines from the buffers until
+    they're empty. Additionally, if the log level is None for any log record, then yield those log
+    messages for capture.
+    '''
+    for output_buffer, reader in buffer_readers.items():
+        if not reader.process:
+            continue
+
+        for lines in reader.lines:
+            for line in lines:
+                log_record = handle_log_record(
+                    parse_log_line(
+                        line=line.rstrip(),
+                        log_level=output_log_level,
+                        elevate_stderr=(
+                            output_buffer == reader.process.stderr and not capture_stderr
+                        ),
+                        borg_local_path=borg_local_path,
+                        command=reader.process.args,
+                    ),
+                )
+
+                if log_record.levelno is None and process_metadatas[reader.process].capture:
+                    yield log_record.getMessage()
+
+
+def log_outputs(
+    processes,
+    exclude_stdouts,
+    output_log_level,
+    borg_local_path,
+    borg_exit_codes,
+    capture_stderr=False,
+):
+    '''
+    Given a sequence of subprocess.Popen() instances for multiple processes, log the outputs (stderr
+    and stdout). Use the requested output log level for stdout, but always log stderr to the ERROR
+    log level. Additionally, raise a CalledProcessError if a process exits with an error (or a
+    warning for exit code 1, if that process does not match the Borg local path).
+
+    If the output log level is None, then instead of logging, capture the output for the last
+    process given and yield it one line at a time. This includes stderr if capture stderr is set.
+    But if the output log level is not None, don't yield anything.
+
+    This yielding means that this function is a generator, and must be consumed in order to execute.
+
+    Use the given Borg local path and exit code configuration to decide what's an error and what's a
+    warning. If any stdouts are given to exclude, then for any matching processes, ignore those
+    buffers. Also note that stdout for a process can be None if output is intentionally not
+    captured, in which case it won't be logged.
+    '''
+    # Map from output buffer to Process_metadata instance. By convention, the last process is the
+    # process to capture.
+    process_metadatas = {
+        process: Process_metadata(last_lines=[], capture=bool(process == processes[-1]))
+        for process in processes
+    }
+
+    # Map from buffer to Buffer_reader instance.
+    buffer_readers = {
+        buffer: Buffer_reader(read_lines(buffer, process), process)
+        for process in processes
+        if process.stdout or process.stderr
+        for buffer in output_buffers_for_process(process, exclude_stdouts)
+    }
+
+    # Log output lines for each process until they all exit.
+    while True:
+        yield from log_buffer_lines(
+            buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
+        )
+
+        if (
+            raise_for_process_errors(
+                buffer_readers, process_metadatas, borg_local_path, borg_exit_codes
+            )
+            == Exit_status.WARNING
+        ):
+            break
+
+        if all(process.poll() is not None for process in processes):
+            break
+
+    # Now that all processes have exited, drain and consume any last output.
+    yield from log_remaining_buffer_lines(
+        buffer_readers, process_metadatas, output_log_level, borg_local_path, capture_stderr
+    )
+
+
 SECRET_COMMAND_FLAG_NAMES = {'--password'}
 
 
@@ -300,16 +567,16 @@ def execute_command(
     close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. If an open output file object is given, then write stdout to the file and only
+    the given log level. If an open output file object is given, then write stdout to the file and
-    log stderr. If an open input file object is given, then read stdin from the file. If shell is
+    only log stderr. If an open input file object is given, then read stdin from the file. If shell
-    True, execute the command within a shell. If an environment variables dict is given, then pass
+    is True, execute the command within a shell. If an environment variables dict is given, then
-    it into the command. If a working directory is given, use that as the present working directory
+    pass it into the command. If a working directory is given, use that as the present working
-    when running the command. If a Borg local path is given, and the command matches it (regardless
+    directory when running the command. If a Borg local path is given, and the command matches it
-    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
-    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    codes are given as a sequence of exit code configuration dicts, then use that configuration to
-    what's an error and what's a warning. If run to completion is False, then return the process for
+    decide what's an error and what's a warning. If run to completion is False, then return the
-    the command without executing it to completion.
+    process for the command without executing it to completion.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
@@ -321,7 +588,7 @@ def execute_command(
         command,
         stdin=input_file,
         stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-        stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
+        stderr=None if do_not_capture else subprocess.PIPE,
         shell=shell,
         env=environment,
         cwd=working_directory,
@@ -331,12 +598,14 @@ def execute_command(
         return process
 
     with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        log_outputs(
+        tuple(
-            (process,),
+            log_outputs(
-            (input_file, output_file),
+                (process,),
-            output_log_level,
+                (input_file, output_file),
-            borg_local_path,
+                output_log_level,
-            borg_exit_codes,
+                borg_local_path,
+                borg_exit_codes,
+            )
         )
 
     return None
@@ -355,14 +624,16 @@ def execute_command_and_capture_output(
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    output (stdout) as a generator that yields one line at a time. The generator must be consumed in
-    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    order for the called command to execute.
-    execute the command within a shell. If an environment variables dict is given, then pass it into
+
-    the command. If a working directory is given, use that as the present working directory when
+    If an input file descriptor is given, then pipe it to the command's stdin. If capture stderr is
-    running the command. If a Borg local path is given, and the command matches it (regardless of
+    True, then capture stderr in addition to stdout. If shell is True, execute the command within a
-    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    shell. If an environment variables dict is given, then pass it into the command. If a working
-    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    directory is given, use that as the present working directory when running the command. If a
-    error and what's a warning.
+    Borg local path is given, and the command matches it (regardless of arguments), treat exit code
+    1 as a warning instead of an error. But if Borg exit codes are given as a sequence of exit code
+    configuration dicts, then use that configuration to decide what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
@@ -370,10 +641,13 @@ def execute_command_and_capture_output(
     command = ' '.join(full_command) if shell else full_command
 
     try:
-        output = subprocess.check_output(  # noqa: S603
+        process = subprocess.Popen(  # noqa: S603
             command,
             stdin=input_file,
-            stderr=subprocess.STDOUT if capture_stderr else None,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE
+            if capture_stderr or command_is_borg(command, borg_local_path)
+            else None,
             shell=shell,
             env=environment,
             cwd=working_directory,
@@ -386,9 +660,22 @@ def execute_command_and_capture_output(
         ):
             raise
 
-        output = error.output
+        if error.output is not None:
+            yield from iter(error.output.decode().splitlines())
 
-    return output.decode() if output is not None else None
+        return
+
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        captured_lines = log_outputs(
+            (process,),
+            (input_file,),
+            None,
+            borg_local_path,
+            borg_exit_codes,
+            capture_stderr=capture_stderr,
+        )
+
+    yield from captured_lines
 
 
 def execute_command_with_processes(
@@ -405,20 +692,23 @@ def execute_command_with_processes(
     close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. Simultaneously, continue to poll one or more active processes so that they
+    the given log level. Simultaneously, continue to poll one or more active processes so that they
     run as well. This is useful, for instance, for processes that are streaming output to a named
     pipe that the given command is consuming from.
 
     If an open output file object is given, then write stdout to the file and only log stderr. But
     if output log level is None, instead suppress logging and return the captured output for (only)
-    the given command. If an open input file object is given, then read stdin from the file. If
+    the given command as a generator that yields one line at a time. The generator must be consumed
-    shell is True, execute the command within a shell. If an environment variables dict is given,
+    in order for the called command to execute—regardless of the output log level.
-    then pass it into the command. If a working directory is given, use that as the present working
+
-    directory when running the command. If a Borg local path is given, then for any matching command
+    If an open input file object is given, then read stdin from the file. If shell is True, execute
-    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    the command within a shell. If an environment variables dict is given, then pass it into the
-    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    command. If a working directory is given, use that as the present working directory when running
-    configuration to decide what's an error and what's a warning.
+    the command. If a Borg local path is given, then for any matching command or process (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
     upstream process.
@@ -432,9 +722,7 @@ def execute_command_with_processes(
             command,
             stdin=input_file,
             stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-            stderr=(
+            stderr=None if do_not_capture else subprocess.PIPE,
-                None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
-            ),
             shell=shell,
             env=environment,
             cwd=working_directory,
@@ -451,7 +739,7 @@ def execute_command_with_processes(
         raise
 
     with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        captured_outputs = log_outputs(
+        captured_lines = log_outputs(
             (*processes, command_process),
             (input_file, output_file),
             output_log_level,
@@ -459,7 +747,4 @@ def execute_command_with_processes(
             borg_exit_codes,
         )
 
-    if output_log_level is None:
+    yield from captured_lines
-        return captured_outputs.get(command_process)
-
-    return None

borgmatic/hooks/command.py

@@ -13,39 +13,71 @@ logger = logging.getLogger(__name__)
 
 
 SOFT_FAIL_EXIT_CODE = 75
-BORG_PLACEHOLDERS = {
+BORG_PLACEHOLDER_NAMES = {
-    '{hostname}',
+    'hostname',
-    '{fqdn}',
+    'fqdn',
-    '{reverse-fqdn}',
+    'reverse-fqdn',
-    '{now}',
+    'now',
-    '{utcnow}',
+    'utcnow',
-    '{unixtime}',
+    'unixtime',
-    '{user}',
+    'user',
-    '{pid}',
+    'pid',
-    '{borgversion}',
+    'borgversion',
-    '{borgmajor}',
+    'borgmajor',
-    '{borgminor}',
+    'borgminor',
-    '{borgpatch}',
+    'borgpatch',
 }
 
 
+VARIABLE_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_variable(match, context, hook_description):
+    '''
+    Given a re.Match instance of VARIABLE_PATTERN representing a matched variable name to be
+    interpolated, a context dict, and a description of the current command hook, lookup the matched
+    variable name within the given context and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given context (and isn't a Borg placeholder), then
+    warn and return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name.
+    if match.group('left_escape') and match.group('right_escape'):
+        return '{' + name + '}'
+
+    value = context.get(name)
+
+    # The matched variable is in the context, so return its value.
+    if value is not None:
+        return shlex.quote(str(value))
+
+    # The matched variable name isn't in the context. Warn about variables unknown to borgmatic, but
+    # don't warn if the variable name happens to be a Borg placeholder, as Borg should hopefully
+    # consume it.
+    if name not in BORG_PLACEHOLDER_NAMES:
+        logger.warning(
+            f'Variable "{name}" is not supported in the {hook_description} hook',
+        )
+
+    # Return the whole string unaltered.
+    return match.group(0)
+
+
 def interpolate_context(hook_description, command, context):
     '''
     Given a config filename, a hook description, a single hook command, and a dict of context
     names/values, interpolate the values by "{name}" into the command and return the result.
     '''
-    for name, value in context.items():
+    return VARIABLE_PATTERN.sub(
-        command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
+        functools.partial(resolve_variable, context=context, hook_description=hook_description),
-
+        command,
-    for unsupported_variable in re.findall(r'\{\w+\}', command):
+    )
-        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
-        # be a Borg placeholder, as Borg should hopefully consume it.
-        if unsupported_variable not in BORG_PLACEHOLDERS:
-            logger.warning(
-                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook',
-            )
-
-    return command
 
 
 def make_environment(current_environment, sys_module=sys):
@@ -147,6 +179,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                     shell=True,
                     environment=make_environment(os.environ),
                     working_directory=working_directory,
+                    close_fds=True,
                 )
         finally:
             if original_umask:

borgmatic/hooks/credential/keepassxc.py

@@ -29,6 +29,7 @@ def load_credential(hook_config, config, credential_parameters):
     command = (
         tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
         + ('show', '--show-protected', '--attributes', 'Password')
+        + (('--no-password',) if not (hook_config or {}).get('ask_for_password', True) else ())
         + (
             ('--key-file', hook_config['key_file'])
             if hook_config and hook_config.get('key_file')
@@ -42,4 +43,6 @@ def load_credential(hook_config, config, credential_parameters):
         + (expanded_database_path, attribute_name)  # Ensure database and entry are last.
     )
 
-    return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+    return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+        os.linesep
+    )

borgmatic/hooks/credential/systemd.py

@@ -48,7 +48,9 @@ def load_credential(hook_config, config, credential_parameters):
             ),
         )
 
-        return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+        return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+            os.linesep
+        )
 
     try:
         with open(

borgmatic/hooks/data_source/bootstrap.py

@@ -7,6 +7,7 @@ import os
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.data_source.config
 
 logger = logging.getLogger(__name__)
 
@@ -58,28 +59,31 @@ def dump_data_sources(
             manifest_file,
         )
 
-    patterns.extend(
+    borgmatic.hooks.data_source.config.inject_pattern(
-        borgmatic.borg.pattern.Pattern(
+        patterns,
-            config_path,
-            source=borgmatic.borg.pattern.Pattern_source.HOOK,
-        )
-        for config_path in config_paths
-    )
-    patterns.append(
         borgmatic.borg.pattern.Pattern(
             os.path.join(borgmatic_runtime_directory, 'bootstrap'),
             source=borgmatic.borg.pattern.Pattern_source.HOOK,
         ),
     )
 
+    for config_path in config_paths:
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
+            borgmatic.borg.pattern.Pattern(
+                config_path,
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
+            ),
+        )
+
     return []
 
 
-def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):
     '''
-    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, and
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, the
-    whether this is a dry run, then remove the manifest file created above. If this is a dry run,
+    configured patterns, and whether this is a dry run, then remove the manifest file created above.
-    then don't actually remove anything.
+    If this is a dry run, then don't actually remove anything.
     '''
     dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
 
@@ -112,6 +116,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the

borgmatic/hooks/data_source/btrfs.py

@@ -1,15 +1,16 @@
 import collections
+import functools
 import glob
-import itertools
-import json
 import logging
 import os
+import pathlib
 import shutil
 import subprocess
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.execute
+import borgmatic.hooks.data_source.config
 import borgmatic.hooks.data_source.snapshot
 
 logger = logging.getLogger(__name__)
@@ -22,105 +23,49 @@ def use_streaming(hook_config, config):  # pragma: no cover
     return False
 
 
-def get_contained_subvolume_paths(btrfs_command, subvolume_path):
+BTRFS_SUBVOLUME_INODE_NUMBER = 256
+
+
+@functools.cache
+def path_is_a_subvolume(path):
     '''
-    Given the path of a Btrfs subvolume, return it in a sequence along with the paths of its
+    Given a path, return whether it is a Btrfs subvolume. Return False if the path doesn't exist.
-    contained subvolumes.
 
-    If the btrfs command errors, log that error and return an empty sequence.
+    As a performance optimization, multiple calls to this function with the same path are cached.
     '''
-    try:
+    if path == os.environ.get('BTRFS_TEST_SUBVOLUME_PATH'):  # pragma: no cover
-        btrfs_output = borgmatic.execute.execute_command_and_capture_output(
+        return True
-            (
-                *btrfs_command.split(' '),
-                'subvolume',
-                'list',
-                subvolume_path,
-            ),
-            close_fds=True,
-        )
-    except subprocess.CalledProcessError as error:
-        logger.debug(
-            f'Ignoring Btrfs subvolume {subvolume_path} because of error listing its subvolumes: {error}',
-        )
-
-        return ()
-
-    return (
-        subvolume_path,
-        *tuple(
-            os.path.join(subvolume_path, line.split(' ')[-1])
-            for line in btrfs_output.splitlines()
-            if line.strip()
-        ),
-    )
-
-
-FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION = 'subvolid=5'
-
-
-def get_all_subvolume_paths(btrfs_command, findmnt_command):
-    '''
-    Given btrfs and findmnt commands to run, get the sorted paths for all Btrfs subvolumes on the
-    system.
-    '''
-    findmnt_output = borgmatic.execute.execute_command_and_capture_output(
-        (
-            *findmnt_command.split(' '),
-            '-t',  # Filesystem type.
-            'btrfs',
-            '--json',
-            '--list',  # Request a flat list instead of a nested subvolume hierarchy.
-        ),
-        close_fds=True,
-    )
 
     try:
-        return tuple(
+        return os.stat(path).st_ino == BTRFS_SUBVOLUME_INODE_NUMBER
-            sorted(
+    except FileNotFoundError:
-                itertools.chain.from_iterable(
+        return False
-                    # If findmnt gave us a Btrfs root filesystem, list the subvolumes within it.
-                    # This is necessary because findmnt only returns a subvolume's mount point
-                    # rather than its original subvolume path (which can differ). For instance,
-                    # a subvolume might exist at /mnt/subvolume but be mounted at /home/myuser.
-                    # findmnt is still useful though because it's a global way to discover all
-                    # Btrfs subvolumes—even if we have to do some additional legwork ourselves.
-                    (
-                        get_contained_subvolume_paths(btrfs_command, filesystem['target'])
-                        if FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION in filesystem['options'].split(',')
-                        else (filesystem['target'],)
-                    )
-                    for filesystem in json.loads(findmnt_output)['filesystems']
-                ),
-            ),
-        )
-    except json.JSONDecodeError as error:
-        raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
-    except KeyError as error:
-        raise ValueError(f'Invalid {findmnt_command} output: Missing key "{error}"')
-
-
-Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
 
 
+@functools.cache
 def get_subvolume_property(btrfs_command, subvolume_path, property_name):
     '''
     Given a btrfs command, a subvolume path, and a property name to lookup, return the value of the
     corresponding property.
 
     Raise subprocess.CalledProcessError if the btrfs command errors.
+
+    As a performance optimization, multiple calls to this function with the same arguments are
+    cached.
     '''
-    output = borgmatic.execute.execute_command_and_capture_output(
+    output = '\n'.join(
-        (
+        borgmatic.execute.execute_command_and_capture_output(
-            *btrfs_command.split(' '),
+            (
-            'property',
+                *btrfs_command.split(' '),
-            'get',
+                'property',
-            '-t',  # Type.
+                'get',
-            'subvol',
+                '-t',  # Type.
-            subvolume_path,
+                'subvol',
-            property_name,
+                subvolume_path,
-        ),
+                property_name,
-        close_fds=True,
+            ),
+            close_fds=True,
+        )
     )
 
     try:
@@ -134,37 +79,71 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
     }.get(value, value)
 
 
-def omit_read_only_subvolume_paths(btrfs_command, subvolume_paths):
+def get_containing_subvolume_path(btrfs_command, path):
     '''
-    Given a Btrfs command to run and a sequence of Btrfs subvolume paths, filter them down to just
+    Given a btrfs command and a path, return the subvolume path that contains the given path (or is
-    those that are read-write. The idea is that Btrfs can't actually snapshot a read-only subvolume,
+    the same as the path).
-    so we should just ignore them.
+
+    If there is no such subvolume path or the containing subvolume is read-only, return None.
     '''
-    retained_subvolume_paths = []
+    # Probe the given pattern's path and all of its parents, grandparents, etc. to try to find a
+    # Btrfs subvolume.
+    for candidate_path in (
+        path,
+        *tuple(str(ancestor) for ancestor in pathlib.PurePath(path).parents),
+    ):
+        if not path_is_a_subvolume(candidate_path):
+            continue
 
-    for subvolume_path in subvolume_paths:
         try:
-            if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
+            if get_subvolume_property(btrfs_command, candidate_path, 'ro'):
-                logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
+                logger.debug(f'Ignoring Btrfs subvolume {candidate_path} because it is read-only')
-            else:
+
-                retained_subvolume_paths.append(subvolume_path)
+                return None
-        except subprocess.CalledProcessError as error:  # noqa: PERF203
+
+            logger.debug(f'Path {candidate_path} is a Btrfs subvolume')
+
+            return candidate_path
+        except subprocess.CalledProcessError as error:
             logger.debug(
-                f'Error determining read-only status of Btrfs subvolume {subvolume_path}: {error}',
+                f'Error determining read-only status of Btrfs subvolume {candidate_path}: {error}',
             )
 
-    return tuple(retained_subvolume_paths)
+            return None
+
+    return None
 
 
-def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
+def get_all_subvolume_paths(btrfs_command, patterns):
+    '''
+    Given a btrfs command and a sequence of patterns, get the sorted paths for all Btrfs subvolumes
+    containing those patterns.
+    '''
+    return tuple(
+        sorted(
+            {
+                subvolume_path
+                for pattern in patterns
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                if pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for subvolume_path in (get_containing_subvolume_path(btrfs_command, pattern.path),)
+                if subvolume_path
+            }
+        ),
+    )
+
+
+Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
+
+
+def get_subvolumes(btrfs_command, patterns):
     '''
     Given a Btrfs command to run and a sequence of configured patterns, find the intersection
-    between the current Btrfs filesystem and subvolume paths and the paths of any patterns.  The
+    between the current Btrfs filesystem/subvolume paths and the paths of any patterns. The idea is
-    idea is that these pattern paths represent the requested subvolumes to snapshot.
+    that these pattern paths represent the requested subvolumes to snapshot.
 
     Only include subvolumes that contain at least one root pattern sourced from borgmatic
-    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
+    configuration (as opposed to generated elsewhere in borgmatic).
-    return all subvolumes instead, sorted by path.
 
     Return the result as a sequence of matching Subvolume instances.
     '''
@@ -172,15 +151,10 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     subvolumes = []
 
     # For each subvolume path, match it against the given patterns to find the subvolumes to
-    # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
+    # backup. Sort the subvolumes from longest to shortest mount points, so longer subvolumes get
     # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
     # this process, so no two subvolumes end up with the same contained patterns.)
-    for subvolume_path in reversed(
+    for subvolume_path in reversed(get_all_subvolume_paths(btrfs_command, patterns)):
-        omit_read_only_subvolume_paths(
-            btrfs_command,
-            get_all_subvolume_paths(btrfs_command, findmnt_command),
-        ),
-    ):
         subvolumes.extend(
             Subvolume(subvolume_path, contained_patterns)
             for contained_patterns in (
@@ -189,8 +163,7 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
                     candidate_patterns,
                 ),
             )
-            if patterns is None
+            if any(
-            or any(
                 pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
                 and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
                 for pattern in contained_patterns
@@ -200,7 +173,7 @@ def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
     return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))
 
 
-BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot-'
+BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot'
 
 
 def make_snapshot_path(subvolume_path):
@@ -209,7 +182,7 @@ def make_snapshot_path(subvolume_path):
     '''
     return os.path.join(
         subvolume_path,
-        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        f'{BORGMATIC_SNAPSHOT_PREFIX}',
         # Included so that the snapshot ends up in the Borg archive at the "original" subvolume path.
     ) + subvolume_path.rstrip(os.path.sep)
 
@@ -222,16 +195,16 @@ def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
     directory within the snapshot itself. For instance, if you have a Btrfs subvolume at /mnt and
     make a snapshot of it at:
 
-        /mnt/.borgmatic-snapshot-1234/mnt
+        /mnt/.borgmatic-snapshot/mnt
 
     ... then the snapshot itself will have an empty directory at:
 
-        /mnt/.borgmatic-snapshot-1234/mnt/.borgmatic-snapshot-1234
+        /mnt/.borgmatic-snapshot/mnt/.borgmatic-snapshot
 
     So to prevent that from ending up in the Borg archive, this function produces an exclude pattern
     to exclude that path.
     '''
-    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}'
 
     return borgmatic.borg.pattern.Pattern(
         os.path.join(
@@ -264,7 +237,7 @@ def make_borg_snapshot_pattern(subvolume_path, pattern):
 
     rewritten_path = initial_caret + os.path.join(
         subvolume_path,
-        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        f'{BORGMATIC_SNAPSHOT_PREFIX}',
         # Use the Borg 1.4+ "slashdot" hack to prevent the snapshot path prefix from getting
         # included in the archive—but only if there's not already a slashdot hack present in the
         # pattern.
@@ -294,7 +267,6 @@ def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma:
             *btrfs_command.split(' '),
             'subvolume',
             'snapshot',
-            '-r',  # Read-only.
             subvolume_path,
             snapshot_path,
         ),
@@ -325,11 +297,15 @@ def dump_data_sources(
     dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
     logger.info(f'Snapshotting Btrfs subvolumes{dry_run_label}')
 
+    if 'findmnt_command' in hook_config:
+        logger.warning(
+            'The Btrfs "findmnt_command" option is deprecated and will be removed from a future release; findmnt is no longer used',
+        )
+
     # Based on the configured patterns, determine Btrfs subvolumes to backup. Only consider those
     # patterns that came from actual user configuration (as opposed to, say, other hooks).
     btrfs_command = hook_config.get('btrfs_command', 'btrfs')
-    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
+    subvolumes = get_subvolumes(btrfs_command, patterns)
-    subvolumes = get_subvolumes(btrfs_command, findmnt_command, patterns)
 
     if not subvolumes:
         logger.warning(f'No Btrfs subvolumes found to snapshot{dry_run_label}')
@@ -347,14 +323,11 @@ def dump_data_sources(
 
         for pattern in subvolume.contained_patterns:
             snapshot_pattern = make_borg_snapshot_pattern(subvolume.path, pattern)
+            borgmatic.hooks.data_source.config.replace_pattern(patterns, pattern, snapshot_pattern)
 
-            # Attempt to update the pattern in place, since pattern order matters to Borg.
+        borgmatic.hooks.data_source.config.inject_pattern(
-            try:
+            patterns, make_snapshot_exclude_pattern(subvolume.path)
-                patterns[patterns.index(pattern)] = snapshot_pattern
+        )
-            except ValueError:
-                patterns.append(snapshot_pattern)
-
-        patterns.append(make_snapshot_exclude_pattern(subvolume.path))
 
     return []
 
@@ -375,11 +348,12 @@ def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
     )
 
 
-def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):
     '''
-    Given a Btrfs configuration dict, a configuration dict, the borgmatic runtime directory, and
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic runtime directory, the
-    whether this is a dry run, delete any Btrfs snapshots created by borgmatic. If this is a dry run
+    configured patterns, and whether this is a dry run, delete any Btrfs snapshots created by
-    or Btrfs isn't configured in borgmatic's configuration, then don't actually remove anything.
+    borgmatic. If this is a dry run or Btrfs isn't configured in borgmatic's configuration, then
+    don't actually remove anything.
     '''
     if hook_config is None:
         return
@@ -387,10 +361,9 @@ def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, d
     dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
 
     btrfs_command = hook_config.get('btrfs_command', 'btrfs')
-    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
 
     try:
-        all_subvolumes = get_subvolumes(btrfs_command, findmnt_command)
+        all_subvolumes = get_subvolumes(btrfs_command, patterns)
     except FileNotFoundError as error:
         logger.debug(f'Could not find "{error.filename}" command')
         return
@@ -441,6 +414,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/data_source/config.py

@@ -3,6 +3,7 @@ import logging
 import shutil
 import subprocess
 
+import borgmatic.borg.pattern
 from borgmatic.execute import execute_command_and_capture_output
 
 IS_A_HOOK = False
@@ -71,13 +72,15 @@ def get_ip_from_container(container):
     last_error = None
     for engine in engines:
         try:
-            output = execute_command_and_capture_output(
+            output = '\n'.join(
-                (
+                execute_command_and_capture_output(
-                    engine,
+                    (
-                    'container',
+                        engine,
-                    'inspect',
+                        'container',
-                    '--format={{json .NetworkSettings}}',
+                        'inspect',
-                    container,
+                        '--format={{json .NetworkSettings}}',
+                        container,
+                    )
                 )
             )
         except subprocess.CalledProcessError as error:
@@ -102,3 +105,51 @@ def get_ip_from_container(container):
     raise ValueError(
         f"Could not determine ip address for container '{container}'; running in host mode or userspace networking?"
     )
+
+
+def inject_pattern(patterns, data_source_pattern, override_excludes=True):
+    '''
+    Given a list of borgmatic.borg.pattern.Pattern instances representing the configured patterns,
+    insert the given data source pattern at the start of the list. The idea is that borgmatic is
+    injecting its own custom pattern specific to a data source hook into the user's configured
+    patterns so that the hook's data gets included in the backup.
+
+    As part of this injection, if the data source pattern is a root pattern and override_excludes is
+    True, also insert an "include" version of the given root pattern, in an attempt to preempt any
+    of the user's configured exclude patterns that may follow. The is to support use cases like
+    borgmatic injecting its own patterns for things like database dumps or bootstrap metadata, where
+    we don't want them to get accidentally excluded.
+    '''
+    if data_source_pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT and override_excludes:
+        patterns.insert(
+            0,
+            borgmatic.borg.pattern.Pattern(
+                path=data_source_pattern.path,
+                type=borgmatic.borg.pattern.Pattern_type.INCLUDE,
+                style=data_source_pattern.style,
+                device=data_source_pattern.device,
+                source=borgmatic.borg.pattern.Pattern_source.HOOK,
+            ),
+        )
+
+    patterns.insert(0, data_source_pattern)
+
+
+def replace_pattern(patterns, pattern_to_replace, data_source_pattern):
+    '''
+    Given a list of borgmatic.borg.pattern.Pattern instances representing the configured patterns,
+    replace the given pattern with the given data source pattern. The idea is that borgmatic is
+    replacing a configured pattern with its own modified pattern specific to a data source hook so
+    that the hook's data gets included in the backup.
+
+    If the pattern to replace can't be found in the given patterns, then just inject the data source
+    pattern at the start of the list.
+    '''
+    try:
+        index = patterns.index(pattern_to_replace)
+    except ValueError:
+        inject_pattern(patterns, data_source_pattern, override_excludes=False)
+
+        return
+
+    patterns[index] = data_source_pattern

borgmatic/hooks/data_source/dump.py

@@ -2,6 +2,7 @@ import fnmatch
 import json
 import logging
 import os
+import re
 import shutil
 
 import borgmatic.actions.restore
@@ -114,16 +115,33 @@ def remove_data_source_dumps(dump_path, data_source_type_name, dry_run):
         shutil.rmtree(dump_path)
 
 
+END_OF_STRING_PATTTERN = re.compile(r'\\z', flags=re.IGNORECASE)
+
+
 def convert_glob_patterns_to_borg_pattern(patterns):
     '''
     Convert a sequence of shell glob patterns like "/etc/*", "/tmp/*" to the corresponding Borg
     regular expression archive pattern as a single string like "re:etc/.*|tmp/.*".
     '''
-    # Remove the "\z" or "\Z" generated by fnmatch.translate() because we don't want the pattern to
+    # This deserves some explanation. Here's what this code is doing for each shell glob pattern:
-    # match only at the end of a path, as directory format dumps require extracting files with paths
+    #
-    # longer than the pattern. E.g., a pattern of "borgmatic/*/foo_databases/test" should also match
+    #  * Strip off the leading slash, since Borg doesn't store leading slashes in archives.
-    # paths like "borgmatic/*/foo_databases/test/toc.dat"
+    #  * Use fnmatch.translate() to convert the pattern to a Borg pattern.
+    #  * Remove the "\z" or \"Z" end-of-string special character generated by fnmatch.translate(),
+    #    because Borg doesn't like it. Replace it with a "$" end-of-string special character
+    #    instead. And yes, this is using a regular expression to modify a regular expression.
+    #  * Do the above for each of:
+    #    * The plain pattern. This supports the use case of a standard database where its name is a
+    #      filename in the path. Example: borgmatic/foo_databases/hostname/test
+    #    * The pattern with "/*" tacked onto the end of it. This supports the use case of a
+    #      directory-format database where its name is a directory name in the path. Example:
+    #      borgmatic/foo_databases/hostname/test/toc.dat
+    #
+    # Join the resulting transformed patterns together with "|" and return them as a string.
     return 're:' + '|'.join(
-        fnmatch.translate(pattern.lstrip('/')).replace('\\z', '').replace('\\Z', '')
+        re.sub(END_OF_STRING_PATTTERN, '$', fnmatch.translate(stripped))
+        + '|'
+        + re.sub(END_OF_STRING_PATTTERN, '$', fnmatch.translate(stripped + '/*'))
         for pattern in patterns
+        for stripped in (pattern.lstrip('/'),)
     )

borgmatic/hooks/data_source/lvm.py

@@ -10,6 +10,7 @@ import subprocess
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.execute
+import borgmatic.hooks.data_source.config
 import borgmatic.hooks.data_source.snapshot
 
 logger = logging.getLogger(__name__)
@@ -43,17 +44,19 @@ def get_logical_volumes(lsblk_command, patterns=None):
     '''
     try:
         devices_info = json.loads(
-            borgmatic.execute.execute_command_and_capture_output(
+            '\n'.join(
-                # Use lsblk instead of lvs here because lvs can't show active mounts.
+                borgmatic.execute.execute_command_and_capture_output(
-                (
+                    # Use lsblk instead of lvs here because lvs can't show active mounts.
-                    *lsblk_command.split(' '),
+                    (
-                    '--output',
+                        *lsblk_command.split(' '),
-                    'name,path,mountpoint,type',
+                        '--output',
-                    '--json',
+                        'name,path,mountpoint,type',
-                    '--list',
+                        '--json',
+                        '--list',
+                    ),
+                    close_fds=True,
                 ),
-                close_fds=True,
+            )
-            ),
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {lsblk_command} JSON output: {error}')
@@ -275,11 +278,7 @@ def dump_data_sources(
                 normalized_runtime_directory,
             )
 
-            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            borgmatic.hooks.data_source.config.replace_pattern(patterns, pattern, snapshot_pattern)
-            try:
-                patterns[patterns.index(pattern)] = snapshot_pattern
-            except ValueError:
-                patterns.append(snapshot_pattern)
 
     return []
 
@@ -324,19 +323,21 @@ def get_snapshots(lvs_command, snapshot_name=None):
     '''
     try:
         snapshot_info = json.loads(
-            borgmatic.execute.execute_command_and_capture_output(
+            '\n'.join(
-                # Use lvs instead of lsblk here because lsblk can't filter to just snapshots.
+                borgmatic.execute.execute_command_and_capture_output(
-                (
+                    # Use lvs instead of lsblk here because lsblk can't filter to just snapshots.
-                    *lvs_command.split(' '),
+                    (
-                    '--report-format',
+                        *lvs_command.split(' '),
-                    'json',
+                        '--report-format',
-                    '--options',
+                        'json',
-                    'lv_name,lv_path',
+                        '--options',
-                    '--select',
+                        'lv_name,lv_path',
-                    'lv_attr =~ ^s',  # Filter to just snapshots.
+                        '--select',
+                        'lv_attr =~ ^s',  # Filter to just snapshots.
+                    ),
+                    close_fds=True,
                 ),
-                close_fds=True,
+            )
-            ),
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {lvs_command} JSON output: {error}')
@@ -353,12 +354,12 @@ def get_snapshots(lvs_command, snapshot_name=None):
         raise ValueError(f'Invalid {lvs_command} output: Missing key "{error}"')
 
 
-def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):  # noqa: PLR0912
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):  # noqa: PLR0912
     '''
-    Given an LVM configuration dict, a configuration dict, the borgmatic runtime directory, and
+    Given an LVM configuration dict, a configuration dict, the borgmatic runtime directory, the
-    whether this is a dry run, unmount and delete any LVM snapshots created by borgmatic. If this is
+    configured patterns, and whether this is a dry run, unmount and delete any LVM snapshots created
-    a dry run or LVM isn't configured in borgmatic's configuration, then don't actually remove
+    by borgmatic. If this is a dry run or LVM isn't configured in borgmatic's configuration, then
-    anything.
+    don't actually remove anything.
     '''
     if hook_config is None:
         return
@@ -455,6 +456,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/data_source/mariadb.py

@@ -7,6 +7,7 @@ import shlex
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.config
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -112,7 +113,14 @@ def database_names_to_dump(database, config, username, password, environment, dr
     names to dump. In the case of "all", query for the names of databases on the configured host and
     return them, excluding any system databases that will cause problems during restore.
     '''
+    skip_names = database.get('skip_names')
+
     if database['name'] != 'all':
+        if skip_names:
+            logger.warning(
+                f'For MariaDB database {database["name"]}, ignoring the "skip_names" option, which is only supported for database "all"'
+            )
+
         return (database['name'],)
 
     if dry_run:
@@ -124,6 +132,8 @@ def database_names_to_dump(database, config, username, password, environment, dr
     extra_options, defaults_extra_filename = parse_extra_options(database.get('list_options'))
     password_transport = database.get('password_transport', 'pipe')
     hostname = database_config.resolve_database_option('hostname', database)
+    socket_path = database.get('socket_path')
+
     show_command = (
         mariadb_show_command
         + (
@@ -135,6 +145,7 @@ def database_names_to_dump(database, config, username, password, environment, dr
         + (('--host', hostname) if hostname else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if hostname or 'port' in database else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
@@ -144,12 +155,20 @@ def database_names_to_dump(database, config, username, password, environment, dr
 
     logger.debug('Querying for "all" MariaDB databases to dump')
 
-    show_output = execute_command_and_capture_output(show_command, environment=environment)
+    if skip_names:
+        logger.debug(f'Skipping database names: {", ".join(skip_names)}')
+
+    show_lines = execute_command_and_capture_output(
+        show_command,
+        environment=environment,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+    )
 
     return tuple(
-        show_name
+        show_name.strip()
-        for show_name in show_output.strip().splitlines()
+        for show_name in show_lines
         if show_name not in SYSTEM_DATABASE_NAMES
+        if not skip_names or show_name not in skip_names
     )
 
 
@@ -197,6 +216,8 @@ def execute_dump_command(
     extra_options, defaults_extra_filename = parse_extra_options(database.get('options'))
     password_transport = database.get('password_transport', 'pipe')
     hostname = database_config.resolve_database_option('hostname', database)
+    socket_path = database.get('socket_path')
+
     dump_command = (
         mariadb_dump_command
         + (
@@ -206,9 +227,11 @@ def execute_dump_command(
         )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+        + ('--single-transaction',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if hostname or 'port' in database else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
@@ -227,6 +250,7 @@ def execute_dump_command(
         dump_command,
         environment=environment,
         run_to_completion=False,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
     )
 
 
@@ -355,7 +379,8 @@ def dump_data_sources(
         dump.write_data_source_dumps_metadata(
             borgmatic_runtime_directory, 'mariadb_databases', dumps_metadata
         )
-        patterns.append(
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
             borgmatic.borg.pattern.Pattern(
                 os.path.join(borgmatic_runtime_directory, 'mariadb_databases'),
                 source=borgmatic.borg.pattern.Pattern_source.HOOK,
@@ -369,6 +394,7 @@ def remove_data_source_dumps(
     databases,
     config,
     borgmatic_runtime_directory,
+    patterns,
     dry_run,
 ):  # pragma: no cover
     '''
@@ -384,6 +410,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -393,16 +423,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -429,6 +467,7 @@ def restore_data_source_dump(
     port = database_config.resolve_database_option(
         'port', data_source, connection_params, restore=True
     )
+    socket_path = database_config.resolve_database_option('socket_path', data_source, restore=True)
     tls = database_config.resolve_database_option('tls', data_source, restore=True)
     username = borgmatic.hooks.credential.parse.resolve_credential(
         database_config.resolve_database_option(
@@ -448,6 +487,7 @@ def restore_data_source_dump(
     )
     extra_options, defaults_extra_filename = parse_extra_options(data_source.get('restore_options'))
     password_transport = data_source.get('password_transport', 'pipe')
+
     restore_command = (
         mariadb_restore_command
         + (
@@ -460,6 +500,7 @@ def restore_data_source_dump(
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
@@ -475,10 +516,14 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
-        environment=environment,
+            input_file=extract_process.stdout,
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/mongodb.py

@@ -5,6 +5,7 @@ import shlex
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.config
 from borgmatic.execute import execute_command, execute_command_with_processes
 from borgmatic.hooks.data_source import config as database_config
 from borgmatic.hooks.data_source import dump
@@ -89,18 +90,28 @@ def dump_data_sources(
 
         if dump_format == 'directory':
             dump.create_parent_directory_for_dump(dump_filename)
-            execute_command(command, shell=True)  # noqa: S604
+            execute_command(  # noqa: S604
+                command,
+                shell=True,
+                working_directory=borgmatic.config.paths.get_working_directory(config),
+            )
         else:
             dump.create_named_pipe_for_dump(dump_filename)
             processes.append(
-                execute_command(command, shell=True, run_to_completion=False),  # noqa: S604
+                execute_command(  # noqa: S604
+                    command,
+                    shell=True,
+                    run_to_completion=False,
+                    working_directory=borgmatic.config.paths.get_working_directory(config),
+                ),
             )
 
     if not dry_run:
         dump.write_data_source_dumps_metadata(
             borgmatic_runtime_directory, 'mongodb_databases', dumps_metadata
         )
-        patterns.append(
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
             borgmatic.borg.pattern.Pattern(
                 os.path.join(borgmatic_runtime_directory, 'mongodb_databases'),
                 source=borgmatic.borg.pattern.Pattern_source.HOOK,
@@ -185,6 +196,7 @@ def remove_data_source_dumps(
     databases,
     config,
     borgmatic_runtime_directory,
+    patterns,
     dry_run,
 ):  # pragma: no cover
     '''
@@ -200,6 +212,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -209,16 +225,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -265,11 +289,15 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process] if extract_process else [],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process] if extract_process else [],
-        input_file=extract_process.stdout if extract_process else None,
+            output_log_level=logging.DEBUG,
+            input_file=extract_process.stdout if extract_process else None,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )
 
 

borgmatic/hooks/data_source/mysql.py

@@ -6,6 +6,7 @@ import shlex
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.config
 import borgmatic.hooks.data_source.mariadb
 from borgmatic.execute import (
     execute_command,
@@ -35,7 +36,14 @@ def database_names_to_dump(database, config, username, password, environment, dr
     names to dump. In the case of "all", query for the names of databases on the configured host and
     return them, excluding any system databases that will cause problems during restore.
     '''
+    skip_names = database.get('skip_names')
+
     if database['name'] != 'all':
+        if skip_names:
+            logger.warning(
+                f'For MySQL database {database["name"]}, ignoring the "skip_names" option, which is only supported for database "all"'
+            )
+
         return (database['name'],)
 
     if dry_run:
@@ -49,6 +57,8 @@ def database_names_to_dump(database, config, username, password, environment, dr
     )
     password_transport = database.get('password_transport', 'pipe')
     hostname = database_config.resolve_database_option('hostname', database)
+    socket_path = database.get('socket_path')
+
     show_command = (
         mysql_show_command
         + (
@@ -64,6 +74,7 @@ def database_names_to_dump(database, config, username, password, environment, dr
         + (('--host', hostname) if hostname else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if hostname or 'port' in database else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
@@ -73,12 +84,20 @@ def database_names_to_dump(database, config, username, password, environment, dr
 
     logger.debug('Querying for "all" MySQL databases to dump')
 
-    show_output = execute_command_and_capture_output(show_command, environment=environment)
+    if skip_names:
+        logger.debug(f'Skipping database names: {", ".join(skip_names)}')
+
+    show_lines = execute_command_and_capture_output(
+        show_command,
+        environment=environment,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+    )
 
     return tuple(
-        show_name
+        show_name.strip()
-        for show_name in show_output.strip().splitlines()
+        for show_name in show_lines
         if show_name not in SYSTEM_DATABASE_NAMES
+        if not skip_names or show_name not in skip_names
     )
 
 
@@ -124,6 +143,8 @@ def execute_dump_command(
     )
     password_transport = database.get('password_transport', 'pipe')
     hostname = database_config.resolve_database_option('hostname', database)
+    socket_path = database.get('socket_path')
+
     dump_command = (
         mysql_dump_command
         + (
@@ -137,9 +158,11 @@ def execute_dump_command(
         )
         + extra_options
         + (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+        + ('--single-transaction',)
         + (('--host', hostname) if hostname else ())
         + (('--port', str(database['port'])) if 'port' in database else ())
         + (('--protocol', 'tcp') if hostname or 'port' in database else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if database.get('tls') is True else ())
         + (('--skip-ssl',) if database.get('tls') is False else ())
@@ -158,6 +181,7 @@ def execute_dump_command(
         dump_command,
         environment=environment,
         run_to_completion=False,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
     )
 
 
@@ -286,7 +310,8 @@ def dump_data_sources(
         dump.write_data_source_dumps_metadata(
             borgmatic_runtime_directory, 'mysql_databases', dumps_metadata
         )
-        patterns.append(
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
             borgmatic.borg.pattern.Pattern(
                 os.path.join(borgmatic_runtime_directory, 'mysql_databases'),
                 source=borgmatic.borg.pattern.Pattern_source.HOOK,
@@ -300,6 +325,7 @@ def remove_data_source_dumps(
     databases,
     config,
     borgmatic_runtime_directory,
+    patterns,
     dry_run,
 ):  # pragma: no cover
     '''
@@ -315,6 +341,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -324,16 +354,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -360,6 +398,7 @@ def restore_data_source_dump(
     port = database_config.resolve_database_option(
         'port', data_source, connection_params, restore=True
     )
+    socket_path = database_config.resolve_database_option('socket_path', data_source, restore=True)
     tls = database_config.resolve_database_option('tls', data_source, restore=True)
     username = borgmatic.hooks.credential.parse.resolve_credential(
         database_config.resolve_database_option(
@@ -381,6 +420,7 @@ def restore_data_source_dump(
         borgmatic.hooks.data_source.mariadb.parse_extra_options(data_source.get('restore_options'))
     )
     password_transport = data_source.get('password_transport', 'pipe')
+
     restore_command = (
         mysql_restore_command
         + (
@@ -397,6 +437,7 @@ def restore_data_source_dump(
         + (('--host', hostname) if hostname else ())
         + (('--port', str(port)) if port else ())
         + (('--protocol', 'tcp') if hostname or port else ())
+        + (('--socket', socket_path) if socket_path else ())
         + (('--user', username) if username and password_transport == 'environment' else ())
         + (('--ssl',) if tls is True else ())
         + (('--skip-ssl',) if tls is False else ())
@@ -412,10 +453,14 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
-        environment=environment,
+            input_file=extract_process.stdout,
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/postgresql.py

@@ -8,6 +8,7 @@ import shlex
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.hooks.credential.parse
+import borgmatic.hooks.data_source.config
 from borgmatic.execute import (
     execute_command,
     execute_command_and_capture_output,
@@ -102,11 +103,15 @@ def database_names_to_dump(database, config, environment, dry_run):
         + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
     )
     logger.debug('Querying for "all" PostgreSQL databases to dump')
-    list_output = execute_command_and_capture_output(list_command, environment=environment)
+    list_lines = execute_command_and_capture_output(
+        list_command,
+        environment=environment,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+    )
 
     return tuple(
         row[0]
-        for row in csv.reader(list_output.splitlines(), delimiter=',', quotechar='"')
+        for row in csv.reader(list_lines, delimiter=',', quotechar='"')
         if row[0] not in EXCLUDED_DATABASE_NAMES
     )
 
@@ -244,6 +249,7 @@ def dump_data_sources(
                     command,
                     shell=True,
                     environment=environment,
+                    working_directory=borgmatic.config.paths.get_working_directory(config),
                 )
             else:
                 dump.create_named_pipe_for_dump(dump_filename)
@@ -253,6 +259,7 @@ def dump_data_sources(
                         shell=True,
                         environment=environment,
                         run_to_completion=False,
+                        working_directory=borgmatic.config.paths.get_working_directory(config),
                     ),
                 )
 
@@ -260,7 +267,8 @@ def dump_data_sources(
         dump.write_data_source_dumps_metadata(
             borgmatic_runtime_directory, 'postgresql_databases', dumps_metadata
         )
-        patterns.append(
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
             borgmatic.borg.pattern.Pattern(
                 os.path.join(borgmatic_runtime_directory, 'postgresql_databases'),
                 source=borgmatic.borg.pattern.Pattern_source.HOOK,
@@ -274,6 +282,7 @@ def remove_data_source_dumps(
     databases,
     config,
     borgmatic_runtime_directory,
+    patterns,
     dry_run,
 ):  # pragma: no cover
     '''
@@ -293,6 +302,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -302,16 +315,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -413,11 +434,19 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process] if extract_process else [],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process] if extract_process else [],
-        input_file=extract_process.stdout if extract_process else None,
+            output_log_level=logging.DEBUG,
+            input_file=extract_process.stdout if extract_process else None,
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
+    )
+    execute_command(
+        analyze_command,
         environment=environment,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
     )
-    execute_command(analyze_command, environment=environment)

borgmatic/hooks/data_source/sqlite.py

@@ -4,6 +4,7 @@ import shlex
 
 import borgmatic.borg.pattern
 import borgmatic.config.paths
+import borgmatic.hooks.data_source.config
 from borgmatic.execute import execute_command, execute_command_with_processes
 from borgmatic.hooks.data_source import dump
 
@@ -85,6 +86,7 @@ def dump_data_sources(
         )
         command = (
             *sqlite_command,
+            '-bail',
             shlex.quote(database_path),
             '.dump',
             '>',
@@ -99,14 +101,20 @@ def dump_data_sources(
 
         dump.create_named_pipe_for_dump(dump_filename)
         processes.append(
-            execute_command(command, shell=True, run_to_completion=False),  # noqa: S604
+            execute_command(  # noqa: S604
+                command,
+                shell=True,
+                run_to_completion=False,
+                working_directory=borgmatic.config.paths.get_working_directory(config),
+            ),
         )
 
     if not dry_run:
         dump.write_data_source_dumps_metadata(
             borgmatic_runtime_directory, 'sqlite_databases', dumps_metadata
         )
-        patterns.append(
+        borgmatic.hooks.data_source.config.inject_pattern(
+            patterns,
             borgmatic.borg.pattern.Pattern(
                 os.path.join(borgmatic_runtime_directory, 'sqlite_databases'),
                 source=borgmatic.borg.pattern.Pattern_source.HOOK,
@@ -120,6 +128,7 @@ def remove_data_source_dumps(
     databases,
     config,
     borgmatic_runtime_directory,
+    patterns,
     dry_run,
 ):  # pragma: no cover
     '''
@@ -135,6 +144,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -144,16 +157,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -193,12 +214,17 @@ def restore_data_source_dump(
         shlex.quote(part)
         for part in shlex.split(data_source.get('sqlite_restore_command') or 'sqlite3')
     )
-    restore_command = (*sqlite_restore_command, shlex.quote(database_path))
+    restore_command = (*sqlite_restore_command, '-bail', shlex.quote(database_path))
+
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
+            input_file=extract_process.stdout,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/zfs.py

@@ -9,6 +9,7 @@ import subprocess
 import borgmatic.borg.pattern
 import borgmatic.config.paths
 import borgmatic.execute
+import borgmatic.hooks.data_source.config
 import borgmatic.hooks.data_source.snapshot
 
 logger = logging.getLogger(__name__)
@@ -44,7 +45,7 @@ def get_datasets_to_backup(zfs_command, patterns):
 
     Return the result as a sequence of Dataset instances, sorted by mount point.
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *zfs_command.split(' '),
             'list',
@@ -64,7 +65,7 @@ def get_datasets_to_backup(zfs_command, patterns):
         datasets = sorted(
             (
                 Dataset(dataset_name, mount_point, (user_property_value == 'auto'), ())
-                for line in list_output.splitlines()
+                for line in list_lines
                 for (dataset_name, mount_point, can_mount, user_property_value) in (
                     line.rstrip().split('\t'),
                 )
@@ -124,7 +125,7 @@ def get_all_dataset_mount_points(zfs_command):
     '''
     Given a ZFS command to run, return all ZFS datasets as a sequence of sorted mount points.
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *zfs_command.split(' '),
             'list',
@@ -141,7 +142,7 @@ def get_all_dataset_mount_points(zfs_command):
         sorted(
             {
                 mount_point
-                for line in list_output.splitlines()
+                for line in list_lines
                 for mount_point in (line.rstrip(),)
                 if mount_point != 'none'
             },
@@ -306,11 +307,7 @@ def dump_data_sources(
                 normalized_runtime_directory,
             )
 
-            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            borgmatic.hooks.data_source.config.replace_pattern(patterns, pattern, snapshot_pattern)
-            try:
-                patterns[patterns.index(pattern)] = snapshot_pattern
-            except ValueError:
-                patterns.append(snapshot_pattern)
 
     return []
 
@@ -347,7 +344,7 @@ def get_all_snapshots(zfs_command):
     Given a ZFS command to run, return all ZFS snapshots as a sequence of full snapshot names of the
     form "dataset@snapshot".
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *tuple(zfs_command.split(' ')),
             'list',
@@ -360,15 +357,15 @@ def get_all_snapshots(zfs_command):
         close_fds=True,
     )
 
-    return tuple(line.rstrip() for line in list_output.splitlines())
+    return tuple(line.rstrip() for line in list_lines)
 
 
-def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):  # noqa: PLR0912
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):  # noqa: PLR0912
     '''
-    Given a ZFS configuration dict, a configuration dict, the borgmatic runtime directory, and
+    Given a ZFS configuration dict, a configuration dict, the borgmatic runtime directory, the
-    whether this is a dry run, unmount and destroy any ZFS snapshots created by borgmatic. If this
+    configured patterns, and whether this is a dry run, unmount and destroy any ZFS snapshots
-    is a dry run or ZFS isn't configured in borgmatic's configuration, then don't actually remove
+    created by borgmatic. If this is a dry run or ZFS isn't configured in borgmatic's configuration,
-    anything.
+    then don't actually remove anything.
     '''
     if hook_config is None:
         return
@@ -456,6 +453,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/dispatch.py

@@ -3,7 +3,6 @@ import importlib
 import logging
 import pkgutil
 
-import borgmatic.hooks.command
 import borgmatic.hooks.credential
 import borgmatic.hooks.data_source
 import borgmatic.hooks.monitoring

borgmatic/hooks/monitoring/apprise.py

@@ -17,7 +17,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to an Apprise notification service upon a finish or failure state. But skip
     this if the "send_logs" option is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     logs_size_limit = max(

borgmatic/hooks/monitoring/healthchecks.py

@@ -26,7 +26,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to Healthchecks upon a finish or failure state. But skip this if the
     "send_logs" option is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     ping_body_limit = max(

borgmatic/hooks/monitoring/loki.py

@@ -23,8 +23,8 @@ MAX_BUFFER_LINES = 100
 
 class Loki_log_buffer:
     '''
-    A log buffer that allows to output the logs as loki requests in json. Allows
+    A log buffer that allows to output the logs as Loki requests in json. Allows
-    adding labels to the log stream and takes care of communication with loki.
+    adding labels to the log stream and takes care of communication with Loki.
     '''
 
     def __init__(self, url, dry_run):
@@ -58,19 +58,20 @@ class Loki_log_buffer:
         if self.dry_run:
             # Just empty the buffer and skip
             self.root['streams'][0]['values'] = []
-            logger.info('Skipped uploading logs to loki due to dry run')
+            logger.info('Skipped uploading logs to Loki due to dry run')
             return
 
         if len(self) == 0:
             # Skip as there are not logs to send yet
             return
 
+        request_body = self.to_request()
         self.root['streams'][0]['values'] = []
 
         try:
             result = requests.post(
                 self.url,
-                data=self.to_request(),
+                data=request_body,
                 timeout=TIMEOUT_SECONDS,
                 headers={
                     'Content-Type': 'application/json',
@@ -79,23 +80,32 @@ class Loki_log_buffer:
             )
             result.raise_for_status()
         except requests.RequestException:
-            logger.warning('Failed to upload logs to loki')
+            logger.warning('Failed to upload logs to Loki')
 
 
 class Loki_log_handler(logging.Handler):
     '''
-    A log handler that sends logs to loki.
+    A log handler that sends logs to Loki.
     '''
 
-    def __init__(self, url, dry_run):
+    def __init__(self, url, send_logs, dry_run):
+        '''
+        Given a URL to send logs to, whether all borgmatic logs should be sent (or just explicitly
+        added messages from this hook), and whether this is a dry run, create an instance of
+        Loki_log_buffer.
+        '''
         super().__init__()
+
         self.buffer = Loki_log_buffer(url, dry_run)
+        self.send_logs = send_logs
 
     def emit(self, record):
         '''
-        Add a log record from the logging module to the stream.
+        Add a general log record from the logging module to the stream—but only if send logs is
+        enabled.
         '''
-        self.raw(record.getMessage())
+        if self.send_logs:
+            self.raw(record.getMessage())
 
     def add_label(self, key, value):
         '''
@@ -109,22 +119,26 @@ class Loki_log_handler(logging.Handler):
         '''
         self.buffer.add_value(msg)
 
-        if len(self.buffer) > MAX_BUFFER_LINES:
+        # If log sending is enabled, flush the buffer (and send data to Loki) once we accumulate
+        # enough log data in the buffer. But if log sending is disabled, flush immediately so that,
+        # for instance, start backup notifications are sent when the backup starts instead of after
+        # it finishes!
+        if len(self.buffer) > MAX_BUFFER_LINES or not self.send_logs:
             self.buffer.flush()
 
     def flush(self):
         '''
-        Send the logs to loki and empty the buffer.
+        Send the logs to Loki and empty the buffer.
         '''
         self.buffer.flush()
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
     '''
-    Add a handler to the root logger to regularly send the logs to loki.
+    Add a handler to the root logger to regularly send the logs to Loki.
     '''
     url = hook_config.get('url')
-    loki = Loki_log_handler(url, dry_run)
+    loki = Loki_log_handler(url, hook_config.get('send_logs', False), dry_run)
 
     for key, value in hook_config.get('labels').items():
         if value == '__hostname':
@@ -141,7 +155,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
 
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
     '''
-    Add an entry to the loki logger with the current state.
+    Add an entry to the Loki logger with the current state.
     '''
     for handler in tuple(logging.getLogger().handlers):
         if isinstance(handler, Loki_log_handler) and state in MONITOR_STATE_TO_LOKI:

borgmatic/hooks/monitoring/ntfy.py

@@ -22,6 +22,29 @@ def initialize_monitor(
     '''
 
 
+def convert_string_to_array(value):
+    value = '' if value is None else str(value)
+    items = []
+
+    for item in value.split(','):
+        stripped = item.strip()
+
+        if stripped:
+            items.append(stripped)
+
+    return items
+
+
+PRIORITY_NAME_TO_ID = {
+    'max': 5,
+    'urgent': 5,
+    'high': 4,
+    'default': 3,
+    'low': 2,
+    'min': 1,
+}
+
+
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
     '''
     Ping the configured Ntfy topic. Use the given configuration filename in any log entries.
@@ -31,13 +54,13 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
 
     if state.name.lower() in run_states:
         dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
-
+        default_priority = PRIORITY_NAME_TO_ID['default']
         state_config = hook_config.get(
             state.name.lower(),
             {
                 'title': f'A borgmatic {state.name} event happened',
                 'message': f'A borgmatic {state.name} event happened',
-                'priority': 'default',
+                'priority': default_priority,
                 'tags': 'borgmatic',
             },
         )
@@ -46,14 +69,17 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         topic = hook_config.get('topic')
 
         logger.info(f'Pinging ntfy topic {topic}{dry_run_label}')
-        logger.debug(f'Using Ntfy ping URL {base_url}/{topic}')
+        logger.debug(f'Using ntfy ping URL {base_url}')
 
         headers = {
             'User-Agent': 'borgmatic',
-            'X-Title': state_config.get('title'),
+        }
-            'X-Message': state_config.get('message'),
+        payload = {
-            'X-Priority': state_config.get('priority'),
+            'topic': topic,
-            'X-Tags': state_config.get('tags'),
+            'title': state_config.get('title'),
+            'message': state_config.get('message'),
+            'priority': PRIORITY_NAME_TO_ID.get(state_config.get('priority'), default_priority),
+            'tags': convert_string_to_array(state_config.get('tags')),
         }
 
         try:
@@ -81,7 +107,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                     'ntfy access_token is set but so is username/password, only using access_token',
                 )
 
-            auth = requests.auth.HTTPBasicAuth('', access_token)
+            headers['Authorization'] = f'Bearer {access_token}'
         elif (username and password) is not None:
             auth = requests.auth.HTTPBasicAuth(username, password)
             logger.info(f'Using basic auth with user {username} for ntfy')
@@ -94,10 +120,11 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             logging.getLogger('urllib3').setLevel(logging.ERROR)
             try:
                 response = requests.post(
-                    f'{base_url}/{topic}',
+                    base_url,
                     auth=auth,
                     timeout=TIMEOUT_SECONDS,
                     headers=headers,
+                    json=payload,
                 )
                 if not response.ok:
                     response.raise_for_status()

borgmatic/hooks/monitoring/pagerduty.py

@@ -23,7 +23,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to PagerDuty upon a failure state. But skip this if the "send_logs" option
     is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     ping_body_limit = max(

borgmatic/hooks/monitoring/sentry.py

@@ -1,5 +1,6 @@
 import logging
 import re
+import urllib
 
 import requests
 
@@ -40,6 +41,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
 
     data_source_name_url = hook_config.get('data_source_name_url')
+    environment = hook_config.get('environment')
     monitor_slug = hook_config.get('monitor_slug')
     match = DATA_SOURCE_NAME_URL_PATTERN.match(data_source_name_url)
 
@@ -65,10 +67,12 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
     if dry_run:
         return
 
+    environment_query = f'&environment={urllib.parse.quote(environment)}' if environment else ''
+
     logging.getLogger('urllib3').setLevel(logging.ERROR)
     try:
         response = requests.post(
-            f'{cron_url}?status={status}',
+            f'{cron_url}?status={status}{environment_query}',
             timeout=TIMEOUT_SECONDS,
             headers={'User-Agent': 'borgmatic'},
         )

borgmatic/logger.py

@@ -1,7 +1,9 @@
 import enum
+import json
 import logging
 import logging.handlers
 import os
+import socket
 import sys
 
 
@@ -85,6 +87,57 @@ class Multi_stream_handler(logging.Handler):
             handler.setLevel(level)
 
 
+DEFAULT_JOURNALD_PRIORITY = 6
+
+
+class JournaldHandler(logging.Handler):
+    def __init__(self, journald_socket_path):
+        super().__init__()
+
+        add_custom_log_levels()
+
+        self.journald_socket_path = journald_socket_path
+        self.log_level_to_journald_priority = {
+            logging.CRITICAL: 2,
+            logging.ERROR: 3,
+            logging.WARNING: 4,
+            logging.ANSWER: 5,
+            logging.INFO: 6,
+            logging.DEBUG: 7,
+        }
+
+    def emit(self, record):
+        sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+
+        try:
+            message_parts = []
+            entry = dict(
+                MESSAGE=record.getMessage(),
+                PRIORITY=self.log_level_to_journald_priority.get(
+                    record.levelno, DEFAULT_JOURNALD_PRIORITY
+                ),
+                SYSLOG_IDENTIFIER='borgmatic',
+                SYSLOG_PID=os.getpid(),
+            )
+
+            for key, value in entry.items():
+                encoded_key = key.upper().encode('utf-8')
+                encoded_value = str(value).encode('utf-8')
+
+                # Multi-line and single-line values use different formats on the wire.
+                if b'\n' in encoded_value:
+                    message_parts.extend((encoded_key, b'\n'))
+                    message_parts.extend(
+                        (len(encoded_value).to_bytes(8, 'little'), encoded_value, b'\n')
+                    )
+                else:
+                    message_parts.extend((encoded_key, b'=', encoded_value, b'\n'))
+
+            sock.sendto(b''.join(message_parts), self.journald_socket_path)
+        finally:
+            sock.close()
+
+
 class Log_prefix_formatter(logging.Formatter):
     def __init__(self, fmt='{prefix}{message}', *args, style='{', **kwargs):
         self.prefix = None
@@ -97,6 +150,29 @@ class Log_prefix_formatter(logging.Formatter):
         return super().format(record)
 
 
+def log_record_to_json(record):
+    '''
+    Given a logging.LogRecord, return it as a JSON-encoded string containing relevant attributes.
+    '''
+    return json.dumps(
+        dict(
+            type='log_message',
+            time=record.created,
+            message=record.getMessage(),
+            levelname=record.levelname,
+            name=record.name,
+        )
+    )
+
+
+class Json_formatter(logging.Formatter):
+    def __init__(self, fmt='{message}', *args, style='{', **kwargs):
+        super().__init__(*args, fmt=fmt, style=style, **kwargs)
+
+    def format(self, record):  # noqa: PLR6301
+        return log_record_to_json(record)
+
+
 class Color(enum.Enum):
     RESET = 0
     RED = 31
@@ -321,6 +397,10 @@ def flush_delayed_logging(target_handlers):
         root_logger.removeHandler(delayed_handler)
 
 
+JOURNALD_SOCKET_PATH = '/run/systemd/journal/socket'
+SYSLOG_PATHS = ('/dev/log', '/var/run/syslog', '/var/run/log')
+
+
 def configure_logging(
     console_log_level,
     syslog_log_level=None,
@@ -328,6 +408,7 @@ def configure_logging(
     monitoring_log_level=None,
     log_file=None,
     log_file_format=None,
+    log_json=False,
     color_enabled=True,
 ):
     '''
@@ -364,7 +445,9 @@ def configure_logging(
         },
     )
 
-    if color_enabled:
+    if log_json:
+        console_handler.setFormatter(Json_formatter())
+    elif color_enabled:
         console_handler.setFormatter(Console_color_formatter())
     else:
         console_handler.setFormatter(Log_prefix_formatter())
@@ -373,29 +456,32 @@ def configure_logging(
     handlers = [console_handler]
 
     if syslog_log_level != logging.DISABLED:
-        syslog_path = None
+        if os.path.exists(JOURNALD_SOCKET_PATH):
-
+            journald_handler = JournaldHandler(JOURNALD_SOCKET_PATH)
-        if os.path.exists('/dev/log'):
+            journald_handler.setLevel(syslog_log_level)
-            syslog_path = '/dev/log'
+            handlers.append(journald_handler)
-        elif os.path.exists('/var/run/syslog'):
+        else:
-            syslog_path = '/var/run/syslog'
+            syslog_path = next(
-        elif os.path.exists('/var/run/log'):
+                (path for path in SYSLOG_PATHS if os.path.exists(path)),
-            syslog_path = '/var/run/log'
+                None,
-
-        if syslog_path:
-            syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
-            syslog_handler.setFormatter(
-                Log_prefix_formatter(
-                    'borgmatic: {levelname} {prefix}{message}',
-                ),
             )
-            syslog_handler.setLevel(syslog_log_level)
+
-            handlers.append(syslog_handler)
+            if syslog_path:
+                syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
+                syslog_handler.setFormatter(
+                    Log_prefix_formatter(
+                        'borgmatic: {levelname} {prefix}{message}',
+                    ),
+                )
+                syslog_handler.setLevel(syslog_log_level)
+                handlers.append(syslog_handler)
 
     if log_file and log_file_log_level != logging.DISABLED:
         file_handler = logging.handlers.WatchedFileHandler(log_file)
         file_handler.setFormatter(
-            Log_prefix_formatter(
+            Json_formatter()
+            if log_json
+            else Log_prefix_formatter(
                 log_file_format or '[{asctime}] {levelname}: {prefix}{message}',
             ),
         )

docs/Dockerfile

@@ -2,11 +2,11 @@ FROM docker.io/alpine:3.20.1 AS borgmatic
 
 COPY . /app
 RUN apk add --no-cache py3-pip py3-ruamel.yaml py3-ruamel.yaml.clib
-RUN pip install --break-system-packages --no-cache /app && borgmatic config generate && chmod +r /etc/borgmatic/config.yaml
+RUN pip install --break-system-packages --no-cache /app && borgmatic config generate && borgmatic config generate --destination /etc/borgmatic --split && chmod +r /etc/borgmatic/*.yaml
-RUN borgmatic --help > /command-line.txt \
+RUN mkdir /command-line \
+    && borgmatic --help > /command-line/global.txt \
     && for action in repo-create transfer create prune compact check delete extract config "config bootstrap" "config generate" "config validate" export-tar mount umount repo-delete restore repo-list list repo-info info break-lock "key export" "key import" "key change-passphrase" recreate borg; do \
-           echo -e "\n--------------------------------------------------------------------------------\n" >> /command-line.txt \
+           borgmatic $action --help > /command-line/${action/ /-}.txt; done
-           && borgmatic $action --help >> /command-line.txt; done
 RUN /app/docs/fetch-contributors >> /contributors.html
 
 FROM docker.io/node:22.4.0-alpine AS html
@@ -23,14 +23,15 @@ RUN npm install @11ty/eleventy \
     markdown-it \
     markdown-it-anchor \
     markdown-it-replace-link
-COPY --from=borgmatic /etc/borgmatic/config.yaml /source/docs/_includes/borgmatic/config.yaml
+COPY --from=borgmatic /etc/borgmatic/* /source/docs/_includes/borgmatic/
-COPY --from=borgmatic /command-line.txt /source/docs/_includes/borgmatic/command-line.txt
+COPY --from=borgmatic /etc/borgmatic/options.json /source/docs/reference/configuration/index.json
+COPY --from=borgmatic /command-line/* /source/docs/_includes/borgmatic/command-line/
 COPY --from=borgmatic /contributors.html /source/docs/_includes/borgmatic/contributors.html
 COPY . /source
-RUN NODE_ENV=${ENVIRONMENT} npx eleventy --input=/source/docs --output=/output/docs \
+RUN NODE_ENV=${ENVIRONMENT} npx eleventy --input=/source/docs --output=/output
-  && mv /output/docs/index.html /output/index.html
+RUN npx -y pagefind --site /output
 
 FROM docker.io/nginx:1.26.1-alpine
 
 COPY --from=html /output /usr/share/nginx/html
-COPY --from=borgmatic /etc/borgmatic/config.yaml /usr/share/nginx/html/docs/reference/config.yaml
+COPY --from=borgmatic /etc/borgmatic/config.yaml /usr/share/nginx/html/reference/config.yaml

docs/_includes/asciinema.css

@@ -1,3 +0,0 @@
-.asciicast > iframe {
-    width: 100% !important;
-}

docs/_includes/components/toc.css

@@ -48,9 +48,20 @@
 	padding-bottom: 0;
 	padding-left: 0.625rem; /* 10px /16 */
 }
-/* Hide inactive menus 3 or more deep */
+
-.elv-toc-list ul ul > li:not(.elv-toc-active) > ul > li:not(.elv-toc-active) {
+/* Display inline menus 4 or more deep */
-	display: none;
+.elv-toc-list ul ul ul > li {
+  display: inline-block;
+}
+
+.elv-toc-list ul ul ul > li::before {
+	content: "•";
+	margin-left: 0.4em;
+	margin-right: -0.2em;
+}
+
+.elv-toc-list ul ul ul > li > a {
+	display: inline-block;
 }
 
 /* List items */
@@ -98,3 +109,29 @@
 .elv-cat-list-active {
 	font-weight: 600;
 }
+
+.breadcrumb-item.active {
+  color: var(--primary-color);
+}
+
+#breadcrumb ol,
+#breadcrumb ul {
+  margin: 0;
+  padding: 0 0 1em;
+}
+
+#breadcrumb li {
+  display: inline;
+}
+
+#breadcrumb li:not(:last-child)::after {
+  content: " ˃"
+}
+
+#breadcrumb a:not(:hover) {
+	text-decoration: none;
+}
+
+#breadcrumb a {
+	text-decoration-color: #00bcd4;
+}

docs/_includes/header.njk

@@ -1,4 +1,13 @@
 <header class="elv-layout elv-layout-full elv-header{% if headerClass %} {{ headerClass }}{% endif %}">
     {% if page.url != '/' %}<h3><a href="https://torsion.org/borgmatic/">borgmatic</a></h3>{% endif %}
+    <div class="container" id="breadcrumb">
+        {% set breadcrumb = collections.all | eleventyNavigationBreadcrumb(eleventyNavigation.key, {allowMissing: true}) %}
+        {# The replace() is a work-around for https://github.com/11ty/eleventy-navigation/issues/56 #}
+        {{ breadcrumb | eleventyNavigationToHtml | replace('href="/reference/', 'href="/borgmatic/reference/') | safe }}
+        <div data-pagefind-meta="title" aria-hidden="true" style="display: none">
+           {% for entry in breadcrumb %}{{ entry.title }} > {% endfor %}{{ title | safe }}
+        </div>
+    </div>
     <h1 class="elv-hed">{{ title | safe }}</h1>
+    {% if page.url == '/' %}<h3>It's your data. Keep it that way.</h3>{% endif %}
 </header>

docs/_includes/index.css

@@ -112,7 +112,7 @@ h5 {
 }
 h1 {
 	font-size: 2.666666666667em; /* 48px /18 */
-	margin: 0 0 .5em;
+	margin: 0;
 }
 main .elv-toc + h1 {
 	margin-top: 1em;
@@ -229,7 +229,7 @@ pre + .note {
 
 /* Layout */
 .elv-layout {
-	padding: 1rem;
+	padding: 0 1rem 1rem;
 	margin: 0 auto;
 	max-width: 42rem;
 	clear: both;
@@ -548,3 +548,15 @@ main .elv-toc + h1 .direct-link {
 .mdi.mdi-content-copy {
     --svg: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' width='24' height='24'%3E%3Cpath fill='black' d='M19 21H8V7h11m0-2H8a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h11a2 2 0 0 0 2-2V7a2 2 0 0 0-2-2m-3-4H4a2 2 0 0 0-2 2v14h2V3h12V1Z'/%3E%3C/svg%3E");
 }
+
+#search-entrance {
+		padding-bottom: 1em;
+}
+
+#search-entrance form {
+		display: inline-block;
+}
+
+#search {
+		padding-top: 1em;
+}

docs/_includes/layouts/base.njk

@@ -4,8 +4,12 @@
 		<meta charset="utf-8">
 		<meta name="viewport" content="width=device-width, initial-scale=1.0">
                 <meta name="generator" content="{{ eleventy.generator }}">
-		<link rel="icon" href="https://torsion.org/borgmatic/docs/static/borgmatic.png" type="image/x-icon">
+		<link rel="icon" href="https://torsion.org/borgmatic/static/borgmatic.png" type="image/x-icon">
-		<title>{{ subtitle + ' - ' if subtitle}}{{ title }}</title>
+		{% if page.url == '/' %}
+		<title>borgmatic</title>
+		{% else %}
+		<title>borgmatic{% if subtitle or title %} - {% endif %}{{ subtitle + ' - ' if subtitle}}{{ title }}</title>
+		{% endif %}
 {%- set css %}
 {% include 'index.css' %}
 {% include 'components/lists.css' %}
@@ -14,7 +18,6 @@
 {% include 'components/toc.css' %}
 {% include 'components/info-blocks.css' %}
 {% include 'prism-theme.css' %}
-{% include 'asciinema.css' %}
 {% endset %}
 		<style>{{ css | safe }}</style>
 {% if feedTitle and feedUrl %}

docs/_includes/layouts/main.njk

@@ -7,11 +7,21 @@ headerClass: elv-header-default
 
 <main class="elv-layout{% if layoutClass %} {{ layoutClass }}{% endif %}">
     <div id="documentation" class="elv-toc">
+        {% if page.url != '/search/' %}
+        <div id="search-entrance">
+            🔍
+            <form action="/borgmatic/search/">
+                <label>
+                    <input type="search" name="query" placeholder="Search">
+                </label>
+            </form>
+        </div>
+        {% endif %}
         <div>
             {% set navPages = collections.all | eleventyNavigation %}
             {% macro renderNavListItem(entry) -%}
             <li{% if entry.url == page.url %} class="elv-toc-active"{% endif %}>
-              <a {% if entry.url %}href="{% if borgmatic.environment == "production" %}https://torsion.org/borgmatic/docs{% else %}http://localhost:8080/docs{% endif %}{{ entry.url | url }}"{% endif %}>{{ entry.title }}</a>
+              <a {% if entry.url %}href="{% if borgmatic.environment == "production" %}https://torsion.org/borgmatic{% else %}http://localhost:8080/borgmatic{% endif %}{{ entry.url | url }}"{% endif %}>{{ entry.title }}</a>
             {%- if entry.children.length -%}
               <ul>
                 {%- for child in entry.children %}{{ renderNavListItem(child) }}{% endfor -%}
@@ -26,7 +36,9 @@ headerClass: elv-header-default
         </div>
     </div>
 
+    <div data-pagefind-body>
     {{ content | safe }}
+    </div>
 
     {% include 'components/suggestion-link.html' %}
 </main>

docs/_includes/snippet/command-line/sample.md

@@ -0,0 +1,5 @@
+Here's the command-line help for this action in the [most recent version of
+borgmatic](https://projects.torsion.org/borgmatic-collective/borgmatic/releases).
+If you're using an older version, some of these flags may not work, and you
+should instead run the action with `--help` to see the flags specific to your
+borgmatic version.

docs/_includes/snippet/configuration/sample.md

@@ -0,0 +1,6 @@
+Here's an example configuration with all of the available options for this
+feature in the [most recent version of
+borgmatic](https://projects.torsion.org/borgmatic-collective/borgmatic/releases).
+If you're using an older version, some of these options may not work, and you
+should instead [generate a sample configuration file specific to your borgmatic
+version](https://torsion.org/borgmatic/how-to/set-up-backups/#configuration).

docs/docker-compose.yaml

@@ -1,9 +1,37 @@
 services:
+  traefik:
+    image: public.ecr.aws/docker/library/traefik:3.5.3
+    container_name: borgmatic-docs-traefik
+    command:
+      - "--global.checkNewVersion=false"
+      - "--global.sendAnonymousUsage=false"
+      - "--entrypoints.web.address=:8080"
+      - "--accesslog"
+      - "--accesslog.fields.headers.defaultmode=keep"
+      - "--providers.docker"
+      - "--providers.docker.exposedbydefault=false"
+      - "--api.insecure=false"
+      - "--api.dashboard=false"
+      - "--log.level=WARN"
+    ports:
+      - "127.0.0.1:8080:8080"
+    volumes:
+      - ${CONTAINER_SOCKET_PATH:-/run/user/docker.sock}:/var/run/docker.sock:ro
   docs:
     image: borgmatic-docs
     container_name: borgmatic-docs
-    ports:
+    labels:
-      - 8080:80
+      - "traefik.enable=true"
+      - "traefik.http.routers.borgmatic-docs.rule=PathPrefix(`/borgmatic`)"
+      - "traefik.http.routers.borgmatic-docs.middlewares=borgmatic-trailing-slash-redirectregex,borgmatic-docs-redirectregex,borgmatic-stripprefix"
+      - "traefik.http.middlewares.borgmatic-trailing-slash-redirectregex.redirectregex.regex=^(.*)/borgmatic$$"
+      - "traefik.http.middlewares.borgmatic-trailing-slash-redirectregex.redirectregex.replacement=$${1}/borgmatic/"
+      - "traefik.http.middlewares.borgmatic-trailing-slash-redirectregex.redirectregex.permanent=true"
+      - "traefik.http.middlewares.borgmatic-docs-redirectregex.redirectregex.regex=^(.*)/borgmatic/docs/(.*)$$"
+      - "traefik.http.middlewares.borgmatic-docs-redirectregex.redirectregex.replacement=$${1}/borgmatic/$${2}"
+      - "traefik.http.middlewares.borgmatic-docs-redirectregex.redirectregex.permanent=true"
+      - "traefik.http.middlewares.borgmatic-stripprefix.stripprefix.prefixes=/borgmatic"
+      - "traefik.http.routers.borgmatic-docs.entrypoints=web"
     build:
       dockerfile: docs/Dockerfile
       context: ..
@@ -16,6 +44,6 @@ services:
       - sh
       - -c
       - |
-        echo; echo "You can view dev docs at http://localhost:8080"; echo
+        echo; echo "You can view dev docs at http://localhost:8080/borgmatic/"; echo
     depends_on:
       - docs

docs/how-to/add-preparation-and-cleanup-steps-to-backups.md

@@ -1,12 +1,10 @@
 ---
-title: How to add preparation and cleanup steps to backups
+title: 🧹 How to add preparation and cleanup steps to backups
 eleventyNavigation:
   key: 🧹 Add preparation and cleanup steps
   parent: How-to guides
   order: 10
 ---
-## Preparation and cleanup hooks
-
 If you find yourself performing preparation tasks before your backup runs or
 doing cleanup work afterwards, borgmatic command hooks may be of interest. These
 are custom shell commands you can configure borgmatic to execute at various
@@ -14,7 +12,7 @@ points as it runs.
 
 (But if you're looking to backup a database, it's probably easier to use the
 [database backup
-feature](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/)
+feature](https://torsion.org/borgmatic/how-to/backup-your-databases/)
 instead.)
 
 <span class="minilink minilink-addedin">New in version 2.0.0</span> Command
@@ -40,7 +38,7 @@ commands:
 
 If you're coming from an older version of borgmatic, there is tooling to help
 you [upgrade your
-configuration](https://torsion.org/borgmatic/docs/how-to/upgrade/#upgrading-your-configuration)
+configuration](https://torsion.org/borgmatic/how-to/upgrade/#upgrading-your-configuration)
 to this new command hook format.
 
 Note that if a `run:` command contains a special YAML character such as a colon,
@@ -74,69 +72,9 @@ commands:
           - echo "After successful create!"
 ```
 
-Each command in the `commands:` list has the following options:
+See the [command hooks
-
+documentation](https://torsion.org/borgmatic/reference/configuration/command-hooks/)
- * `before` or `after`: Name for the point in borgmatic's execution that the commands should be run before or after, one of:
+for additional details about how to configure command hooks.
-    * `action` runs before or after each action for each repository. This replaces the deprecated `before_create`, `after_prune`, etc.
-    * `repository` runs before or after all actions for each repository. This replaces the deprecated `before_actions` and `after_actions`.
-    * `configuration` runs before or after all actions and repositories in the current configuration file.
-    * `everything` runs before or after all configuration files. Errors here do not trigger `error` hooks or the `fail` state in monitoring hooks. This replaces the deprecated `before_everything` and `after_everything`.
-    * `error` runs after an error occurs—and it's only available for `after`. This replaces the deprecated `on_error` hook.
- * `when`: Only trigger the hook when borgmatic is run with particular actions (`create`, `prune`, etc.) listed here. Defaults to running for all actions.
- * `states`: <span class="minilink minilink-addedin">New in version 2.0.3</span> Only trigger the hook if borgmatic encounters one of the states (execution results) listed here. This state is evaluated only for the scope of the configured `action`, `repository`, etc., rather than for the entire borgmatic run. Only available for `after` hooks. Defaults to running the hook for all states. One or more of:
-    * `finish`: No errors occurred.
-    * `fail`: An error occurred.
- * `run`: List of one or more shell commands or scripts to run when this command hook is triggered.
-
-When command hooks run, they respect the `working_directory` option if it is
-configured, meaning that the hook commands are run in that directory.
-
-<span class="minilink minilink-addedin">New in version 2.0.4</span>If the exact
-same `everything` command hook is present in multiple configuration files,
-borgmatic only runs it once.
-
-borgmatic's `--repository` flag does not impact which command hooks get run. But
-you can use the `--config` flag to limit the configuration files (and thus
-command hooks) used.
-
-
-### Order of execution
-
-Here's a way of visualizing how all of these command hooks slot into borgmatic's
-execution.
-
-Let's say you've got a borgmatic configuration file with a configured
-repository. And suppose you configure several command hooks and then run
-borgmatic for the `create` and `prune` actions. Here's the order of execution:
-
- * Run `before: everything` hooks (from all configuration files).
-    * Run `before: configuration` hooks (from the first configuration file).
-        * Run `before: repository` hooks (for the first repository).
-            * Run `before: action` hooks for `create`.
-            * Actually run the `create` action (e.g. `borg create`).
-            * Run `after: action` hooks for `create`.
-            * Run `before: action` hooks for `prune`.
-            * Actually run the `prune` action (e.g. `borg prune`).
-            * Run `after: action` hooks for `prune`.
-        * Run `after: repository` hooks (for the first repository).
-    * Run `after: configuration` hooks (from the first configuration file).
-    * Run `after: error` hooks (if an error occurs).
- * Run `after: everything` hooks (from all configuration files).
-
-This same order of execution extends to multiple repositories and/or
-configuration files.
-
-Based on the above, you can see the difference between, say, an `after: action`
-hook with `states: [fail]` and an `after: error` hook. The `after: action hook`
-runs immediately after the create action fails for a particular repository—so
-before any subsequent actions for that repository or other repositories even
-have a chance to run. Whereas the `after: error` hook doesn't run until all
-actions for—and repositories in—a configuration file have had a chance to
-execute.
-
-And if there are multiple hooks defined for a particular step (e.g. `before:
-action` for `create`), then those hooks are run in the order they're defined in
-configuration.
 
 
 ### Deprecated command hooks
@@ -247,83 +185,12 @@ borgmatic does not run `on_error` hooks if an error occurs within a
 `before_everything` or `after_everything` hook.
 
 
-## Variable interpolation
-
-The command action hooks support interpolating particular runtime variables into
-the commands that are run. Here's are a couple examples that assume you provide
-separate shell scripts:
-
-```yaml
-commands:
-    - after: action
-      when: [prune]
-      run:
-          - record-prune.sh {configuration_filename} {repository}
-    - after: error
-      when: [create]
-      run:
-          - send-text-message.sh {configuration_filename} {repository}
-```
-
-In this example, when the hook is triggered, borgmatic interpolates runtime
-values into each hook command: the borgmatic configuration filename and the
-paths of the current Borg repository.
-
-Here's the full set of supported variables you can use here:
-
- * `configuration_filename`: borgmatic configuration filename in which the
-   hook was defined
- * `log_file`
-   <span class="minilink minilink-addedin">New in version 1.7.12</span>:
-   path of the borgmatic log file, only set when the `--log-file` flag is used
- * `repository`: path of the current repository as configured in the current
-   borgmatic configuration file, if applicable to the current hook
- * `repository_label` <span class="minilink minilink-addedin">New in version
-   1.8.12</span>: label of the current repository as configured in the current
-   borgmatic configuration file, if applicable to the current hook
- * `error`: the error message itself, only applies to `error` hooks
- * `output`: output of the command that failed, only applies to `error` hooks
-   (may be blank if an error occurred without running a command)
-
-Not all command hooks support all variables. For instance, the `everything` and
-`configuration` hooks don't support repository variables because those hooks
-don't run in the context of a single repository. But the deprecated command
-hooks (`before_backup`, `on_error`, etc.) do generally support variable
-interpolation.
-
-borgmatic automatically escapes these interpolated values to prevent shell
-injection attacks. One implication is that you shouldn't wrap the interpolated
-values in your own quotes, as that will interfere with the quoting performed by
-borgmatic and result in your command receiving incorrect arguments. For
-instance, this won't work:
-
-```yaml
-commands:
-    - after: error
-      run:
-          # Don't do this! It won't work, as the {error} value is already quoted.
-          - send-text-message.sh "Uh oh: {error}"
-```
-
-Do this instead:
-
-```yaml
-commands:
-    - after: error
-      run:
-          - send-text-message.sh {error}
-```
-
-Note that you can also interpolate [arbitrary environment
-variables](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
-
-
 ## Hook output
 
 Any output produced by your hooks shows up both at the console and in syslog
-(when enabled). For more information, read about <a
+(when enabled). For more information, see the <a
-href="https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/">inspecting
+href="https://torsion.org/borgmatic/reference/command-line/logging/">logging
-your backups</a>.
+documentation</a>.
 
 
 ## Security

docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server.md

@@ -1,12 +1,10 @@
 ---
-title: How to backup to a removable drive or an intermittent server
+title: 💾 How to backup to a removable drive or an intermittent server
 eleventyNavigation:
   key: 💾 Backup to a removable drive/server
   parent: How-to guides
   order: 11
 ---
-## Occasional backups
-
 A common situation is backing up to a repository that's only sometimes online.
 For instance, you might send most of your backups to the cloud, but
 occasionally you want to plug in an external hard drive or backup to your
@@ -25,16 +23,19 @@ or an offline server or a low battery—and exit gracefully? That's where the
 concept of "soft failure" come in.
 
 
+<a id="caveats-and-details"></a>
+
+
 ## Soft failure command hooks
 
 This feature leverages [borgmatic command
-hooks](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/),
+hooks](https://torsion.org/borgmatic/how-to/add-preparation-and-cleanup-steps-to-backups/),
 so familiarize yourself with them first. The idea is that you write a simple
 test in the form of a borgmatic command hook to see if backups should proceed or
 not.
 
 The way the test works is that if any of your hook commands return a special
-exit status of 75, that indicates to borgmatic that it's a temporary failure,
+exit status of 75, that indicates to borgmatic that it's a temporary failure
 and borgmatic should skip all subsequent actions for the current repository.
 
 If you return any status besides 75, then it's a standard success or error.
@@ -42,7 +43,7 @@ If you return any status besides 75, then it's a standard success or error.
 
 So for instance, if you have an external drive that's only sometimes mounted,
 declare its repository in its own [separate configuration
-file](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/),
+file](https://torsion.org/borgmatic/how-to/make-per-application-backups/),
 say at `/etc/borgmatic.d/removable.yaml`:
 
 ```yaml
@@ -81,7 +82,7 @@ before_actions:
 option in the `hooks:` section of your configuration.
 
 <span class="minilink minilink-addedin">Prior to version 1.7.0</span> Use
-`before_create` or similar instead of `before_actions`, which was introduced in
+`before_backup` or similar instead of `before_actions`, which was introduced in
 borgmatic 1.7.0.
 
 What this does is check if the `findmnt` command errors when probing for a
@@ -116,33 +117,6 @@ commands:
 
 Writing the battery script is left as an exercise to the reader.
 
-
+See the [soft failure
-## Caveats and details
+documentation](https://torsion.org/borgmatic/reference/configuration/command-hooks/#soft-failure)
-
+for additional details.
-There are some caveats you should be aware of with this feature.
-
- * You'll generally want to put a soft failure command in a `before` command
-   hook, so as to gate whether the backup action occurs. While a soft failure is
-   also supported in an `after` command hook, returning a soft failure there
-   won't prevent any actions from occurring, because they've already occurred!
-   Similarly, you can return a soft failure from an `error` command hook, but at
-   that point it's too late to prevent the error.
- * Returning a soft failure does prevent further commands in the same hook from
-   executing. So, like a standard error, it is an "early out." Unlike a standard
-   error, borgmatic does not display it in angry red text or consider it a
-   failure.
- * <span class="minilink minilink-addedin">New in version 1.9.0</span> Soft
-   failures in `action` or `before_*` command hooks only skip the current
-   repository rather than all repositories in a configuration file.
- * If you're writing a soft failure script that you want to vary based on the
-   current repository, for instance so you can have multiple repositories in a
-   single configuration file, have a look at [command hook variable
-   interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation).
-   And there's always still the option of putting anything that you don't want
-   soft-failed (like always-online cloud backups) in separate configuration
-   files from your soft-failing repositories.
- * The soft failure doesn't have to test anything related to a repository. You
-   can even perform a test that individual source directories are mounted and
-   available. Use your imagination!
- * Soft failures are not currently implemented for `everything`,
-   `before_everything`, or `after_everything` command hooks.

docs/how-to/backup-your-databases.md

@@ -1,12 +1,10 @@
 ---
-title: How to backup your databases
+title: 🗄️ How to backup your databases
 eleventyNavigation:
   key: 🗄️ Backup your databases
   parent: How-to guides
   order: 8
 ---
-## Database dump hooks
-
 If you want to backup a database, it's best practice with most database
 systems to backup an exported database dump, rather than backing up your
 database's internal file storage. That's because the internal storage can
@@ -29,33 +27,6 @@ mysql_databases:
 these and other database options in the `hooks:` section of your
 configuration.
 
-<span class="minilink minilink-addedin">New in version 1.5.22</span> You can
-also dump MongoDB databases. For example:
-
-```yaml
-mongodb_databases:
-    - name: messages
-```
-
-<span class="minilink minilink-addedin">New in version 1.7.9</span>
-Additionally, you can dump SQLite databases. For example:
-
-```yaml
-sqlite_databases:
-    - name: mydb
-      path: /var/lib/sqlite3/mydb.sqlite
-```
-
-<span class="minilink minilink-addedin">New in version 1.8.2</span> If you're
-using MariaDB, use the MariaDB database hook instead of `mysql_databases:` as
-the MariaDB hook calls native MariaDB commands instead of the deprecated MySQL
-ones. For instance:
-
-```yaml
-mariadb_databases:
-    - name: comments
-```
-
 As part of each backup, borgmatic streams a database dump for each configured
 database directly to Borg, so it's included in the backup without consuming
 additional disk space. (The exceptions are the PostgreSQL/MongoDB `directory`
@@ -109,47 +80,17 @@ sqlite_databases:
       path: /var/lib/sqlite3/mydb.sqlite
 ```
 
-See your [borgmatic configuration
+See the [data sources
-file](https://torsion.org/borgmatic/docs/reference/configuration/) for
+documentation](https://torsion.org/borgmatic/reference/configuration/data-sources/)
-additional customization of the options passed to database commands (when
+for details on additional options, including customizing the flags passed to
-listing databases, restoring databases, etc.).
+database commands when listing databases, restoring databases, etc.
 
+<a id="runtime-directory"></a>
 
-### Runtime directory
+To support streaming database dumps to Borg, borgmatic uses a runtime directory
-
+for temporary file storage. See the [runtime directory
-<span class="minilink minilink-addedin">New in version 1.9.0</span> To support
+documentation](https://torsion.org/borgmatic/reference/configuration/runtime-directory/)
-streaming database dumps to Borg, borgmatic uses a runtime directory for
+for details.
-temporary file storage, probing the following locations (in order) to find it:
-
- 1. The `user_runtime_directory` borgmatic configuration option.
- 2. The `XDG_RUNTIME_DIR` environment variable, usually `/run/user/$UID`
-    (where `$UID` is the current user's ID), automatically set by PAM on Linux
-    for a user with a session.
- 3. <span class="minilink minilink-addedin">New in version 1.9.2</span>The
-    `RUNTIME_DIRECTORY` environment variable, set by systemd if
-    `RuntimeDirectory=borgmatic` is added to borgmatic's systemd service file.
- 4. <span class="minilink minilink-addedin">New in version 1.9.1</span>The
-    `TMPDIR` environment variable, set on macOS for a user with a session,
-    among other operating systems.
- 5. <span class="minilink minilink-addedin">New in version 1.9.1</span>The
-    `TEMP` environment variable, set on various systems.
- 6. <span class="minilink minilink-addedin">New in version 1.9.2</span>
-    Hard-coded `/tmp`. <span class="minilink minilink-addedin">Prior to
-    version 1.9.2</span>This was instead hard-coded to `/run/user/$UID`.
-
-You can see the runtime directory path that borgmatic selects by running with
-`--verbosity 2` and looking for "Using runtime directory" in the output.
-
-Regardless of the runtime directory selected, borgmatic stores its files
-within a `borgmatic` subdirectory of the runtime directory. Additionally, in
-the case of `TMPDIR`, `TEMP`, and the hard-coded `/tmp`, borgmatic creates a
-randomly named subdirectory in an effort to reduce path collisions in shared
-system temporary directories.
-
-<span class="minilink minilink-addedin">Prior to version 1.9.0</span>
-borgmatic created temporary streaming database dumps within the `~/.borgmatic`
-directory by default. At that time, the path was configurable by the
-`borgmatic_source_directory` configuration option (now deprecated).
 
 
 ### All databases
@@ -269,9 +210,11 @@ the `pg_restore_command` and `psql_command` options.
 
 If you choose to use the `pg_dump` command within the container, and you're
 using the `directory` format in particular, you'll also need to mount the
-[runtime directory](#runtime-directory) from your host into the container at the
+[runtime
-same path. Otherwise, the `directory` format dump will remain locked away inside
+directory](https://torsion.org/borgmatic/reference/configuration/runtime-directory/)
-the database container where Borg can't read it.
+from your host into the container at the same path. Otherwise, the `directory`
+format dump will remain locked away inside the database container where Borg
+can't read it.
 
 For example, with Docker Compose and a runtime directory located at
 `/run/user/1000`:
@@ -306,6 +249,10 @@ container separate from borgmatic.
 A similar approach can work with MySQL, using `mysql_dump_command` instead of
 `mariadb_dump_command` to run `mysqldump` in a container.
 
+Note: With MariaDB and MySQL, use `options`, `list_options`, and/or
+`restore_options` to override command-line flags rather than putting flags into
+`mariadb_dump_command`, `mysql_dump_command`, etc.
+
 
 #### Database client in a temporary container
 
@@ -328,7 +275,7 @@ password to the temporary `pg_dump` container.
 
 Similar command override options are available for (some of) the other
 supported database types as well. See the [configuration
-reference](https://torsion.org/borgmatic/docs/reference/configuration/) for
+reference](https://torsion.org/borgmatic/reference/configuration/) for
 details.
 
 
@@ -355,8 +302,10 @@ hooks:
 ### External passwords
 
 If you don't want to keep your database passwords in your borgmatic
-configuration file, you can instead pass them in [from external credential
+configuration file, you can instead pass them in from [external credential
-sources](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
+sources](https://torsion.org/borgmatic/reference/configuration/credentials/) or
+[environment
+variables](https://torsion.org/borgmatic/reference/configuration/environment-variables/).
 
 
 ### Configuration backups
@@ -370,17 +319,23 @@ bring back any missing configuration files in order to restore a database.
 <span class="minilink minilink-addedin">New in version 1.7.15</span> borgmatic
 automatically includes configuration files in your backup. See [the
 documentation on the `config bootstrap`
-action](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive)
+action](https://torsion.org/borgmatic/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive)
 for more information.
 
 
 ## Supported databases
 
-As of now, borgmatic supports PostgreSQL, MariaDB, MySQL, MongoDB, and SQLite
+borgmatic directly supports [several database
-databases directly. But see below about general-purpose preparation and
+systems](https://torsion.org/borgmatic/reference/configuration/data-sources/).
-cleanup hooks as a work-around with other database systems. Also, please [file
+But if you're looking to backup an unsupported databases system, you can use
-a ticket](https://torsion.org/borgmatic/#issues) for additional database
+general-purpose [preparation and cleanup
-systems that you'd like supported.
+hooks](https://torsion.org/borgmatic/how-to/add-preparation-and-cleanup-steps-to-backups/)
+as a work-around. These hooks allows you to trigger arbitrary commands or
+scripts before and after backups to create and cleanup database dumps for any
+database system.
+
+Also, please [file a ticket](https://torsion.org/borgmatic/#issues) for
+additional database systems that you'd like borgmatic to officially support.
 
 
 ## Database restoration
@@ -414,8 +369,6 @@ most up-to-date files and therefore the latest timestamp, run a command like:
 borgmatic restore --archive host-2023-01-02T04:06:07.080910
 ```
 
-(No borgmatic `restore` action? Upgrade borgmatic!)
-
 Or you can simplify this to:
 
 ```bash
@@ -431,6 +384,20 @@ restoring dumps from the selected archive. So be very careful when and where
 you run it.
 
 
+### Configuration file selection
+
+If you have a multi-configuration-file setup and you want to restore a database
+from a single configuration file, use the `--config` flag with the path of the
+configuration file to use for the restore. For example:
+
+```bash
+borgmatic restore --config /path/to/config.yaml --archive latest
+```
+
+Without `--config`, borgmatic tries to run the `restore` action once for each
+configuration file it finds.
+
+
 ### Repository selection
 
 If you have a single repository in your borgmatic configuration file(s), no
@@ -581,16 +548,17 @@ Or from the configuration file:
 postgresql_databases:
     - name: users
         hostname: database1.example.org
-        restore_hostname: database1.example.org
+        restore_hostname: database2.example.org
         restore_port: 5433
         restore_username: postgres
         restore_password: trustsome1
 ```
 
+
 ### Manual restoration
 
 If you prefer to restore a database without the help of borgmatic, first
-[extract](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/) an
+[extract](https://torsion.org/borgmatic/how-to/extract-a-backup/) an
 archive containing a database dump.
 
 borgmatic extracts the dump file into the `borgmatic/` directory within the
@@ -607,7 +575,7 @@ After extraction, you can manually restore the dump file using native database
 commands like `pg_restore`, `mysql`, `mongorestore`, `sqlite`, or similar.
 
 Also see the documentation on [listing database
-dumps](https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#listing-database-dumps).
+dumps](https://torsion.org/borgmatic/how-to/inspect-your-backups/#listing-database-dumps).
 
 
 ## Limitations
@@ -653,16 +621,6 @@ starting from version 1.7.15, borgmatic includes your configuration files
 automatically.
 
 
-## Preparation and cleanup hooks
-
-If this database integration is too limited for needs, borgmatic also supports
-general-purpose [preparation and cleanup
-hooks](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/).
-These hooks allows you to trigger arbitrary commands or scripts before and
-after backups. So if necessary, you can use these hooks to create database
-dumps with any database system.
-
-
 ## Troubleshooting
 
 ### Authentication errors
@@ -701,9 +659,14 @@ configuration. Here's an example with MariaDB:
 ```yaml
 mariadb_databases:
     - name: posts
-      options: "--single-transaction --quick"
+      options: "--single-transaction"
 ```
 
+<span class="minilink minilink-addedin">New in version 2.0.13</span> borgmatic
+passes `--single-transaction` to MariaDB/MySQL by default, and you no longer
+need to set that in `options:`.
+
+
 ### borgmatic hangs during backup
 
 See Limitations above about `read_special`. You may need to exclude certain

docs/how-to/customize-warnings-and-errors.md

@@ -1,46 +1,48 @@
 ---
-title: How to customize warnings and errors
+title: 💥 How to customize warnings and errors
 eleventyNavigation:
   key: 💥 Customize warnings/errors
   parent: How-to guides
   order: 13
 ---
-## When things go wrong
 
-After Borg runs, it indicates whether it succeeded via its exit code, a
+After Borg runs, it indicates whether it succeeded via its exit code, a numeric
-numeric ID indicating success, warning, or error. borgmatic consumes this exit
+ID indicating success, warning, or error. borgmatic consumes this exit code to
-code to decide how to respond. Normally, a Borg error results in a borgmatic
+decide how to respond. By default, Borg errors (and some warnings) result
-error, while a Borg warning or success doesn't.
+in a borgmatic error, while Borg successes don't.
+
+<span class="minilink minilink-addedin">New in borgmatic version 2.1.0</span>
+borgmatic elevates most Borg warnings to errors by default. For instance, if a
+source directory is missing during backup, Borg indicates that with a warning
+exit code (`107`). And starting in borgmatic 2.1.0, that exit code is considered
+an error, so you'll actually find out about missing files.
 
 <span class="minilink minilink-addedin">With Borg version 1.4+</span> If the
 default behavior isn't sufficient for your needs, you can customize how
 borgmatic interprets [Borg's exit
-codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
+codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
 
-For instance, this borgmatic configuration elevates all Borg backup file
+For instance, this borgmatic configuration elevates a Borg warning about source files
-permission warnings (exit code `105`)—and only those warnings—to errors:
+changes during backup (exit code `100`)—and only those warnings—to
+errors:
 
 ```yaml
 borg_exit_codes:
-    - code: 105
+    - code: 100
       treat_as: error
 ```
 
-The following configuration does that *and* elevates backup file not found
+The following configuration does that *and* treats Borg's backup file not found
-warnings (exit code `107`) to errors as well:
+(exit code `107`) as a warning:
 
 ```yaml
 borg_exit_codes:
-    - code: 105
+    - code: 100
       treat_as: error
     - code: 107
-      treat_as: error
+      treat_as: warning
 ```
 
-See the full list of [Borg 1.4 error and warning exit
-codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
-The `rc:` numeric value there tells you the exit code for each.
-
 If you don't know the exit code for a particular Borg error or warning you're
 experiencing, you can usually find it in your borgmatic output when `--verbosity
 2` is enabled. For instance, here's a snippet of that output when a backup file
@@ -52,8 +54,8 @@ is not found:
 terminating with warning status, rc 107
 ```
 
-So if you want to configure borgmatic to treat this as an error instead of a
+So if you want to configure borgmatic to treat this as an warning instead of an
-warning, the exit status to use is `107`.
+error, the exit status to use is `107`.
 
 <span class="minilink minilink-addedin">With Borg version 1.2 and earlier</span>
 Older versions of Borg didn't support granular exit codes, but still
@@ -69,15 +71,4 @@ borg_exit_codes:
 
 Be aware though that Borg exits with a warning code for a variety of benign
 situations such as files changing while they're being read, so this example
-may not meet your needs.
+may not meet your needs. Upgrading to Borg 1.4+ is recommended.
-
-Here's another Borg 1.2 example that squashes Borg errors to warnings:
-
-```yaml
-borg_exit_codes:
-    - code: 2
-      treat_as: warning
-```
-
-Be careful with this example though, because it prevents borgmatic from
-erroring when Borg errors, which may not be desirable.