Merge branch 'main' into command-hook-steps

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1121
Reviewed-by: Dan Helfman <witten@torsion.org>

.bandit

@@ -0,0 +1,3 @@
+[bandit]
+exclude=tests
+skips=S105,S404

.flake8

@@ -1 +0,0 @@
-select = Q0

.gitea/pull_request_template.md

@@ -0,0 +1,4 @@
+---
+name: "Pull Request"
+about: "Pull Request"
+---

.gitea/workflows/build.yaml

@@ -3,6 +3,8 @@ run-name: ${{ gitea.actor }} is building
 on:
   push:
     branches: [main]
+  pull_request:
+    branches: [main]
 
 jobs:
   test:
@@ -15,6 +17,7 @@ jobs:
   docs:
     needs: [test]
     runs-on: host
+    if: gitea.event_name == 'push'
     env:
       IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
 
@@ -26,3 +29,5 @@ jobs:
           PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
       - run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
       - run: podman push "$IMAGE_NAME"
+      - run: scripts/export-docs-from-image
+      - run: curl --user "${{ secrets.REGISTRY_USERNAME }}:${{ secrets.REGISTRY_PASSWORD }}" --upload-file borgmatic-docs.tar.gz https://projects.torsion.org/api/packages/borgmatic-collective/generic/borgmatic-docs/$(head --lines=1 NEWS)/borgmatic-docs.tar.gz

.github/pull_request_template.md

@@ -0,0 +1,10 @@
+## Hold up, GitHub users
+
+Thanks for your contribution! 
+
+Unfortunately, we don't use GitHub pull requests to manage code contributions to this repository (and GitHub doesn't have any way to disable pull requests entirely). Instead, please see:
+
+https://torsion.org/borgmatic/#contributing
+
+... which provides full instructions on how to submit pull requests. You can even use your GitHub account to login.
+

MANIFEST.in

@@ -1,2 +1,7 @@
+# This file only applies to the source dist tarball, not the built wheel.
 include borgmatic/config/schema.yaml
-graft sample/systemd
+graft docs
+graft sample
+graft scripts
+graft tests
+global-exclude *.py[co]

NEWS

@@ -1,3 +1,410 @@
+2.0.9.dev0
+ * #1123: Add loading of systemd credentials even when running borgmatic outside of a systemd
+   service.
+ * #1134: Add a "steps" option to run command hooks around particular sub-action steps like
+   individual checks.
+ * #1149: Add support for Python 3.14.
+ * #1149: Include automated tests in the source dist tarball uploaded to PyPI.
+ * #1151: Fix snapshotting in the ZFS, Btrfs, and LVM hooks to play nicely with the Borg 1.4+
+   "slashdot" hack within source directory paths.
+
+2.0.8
+ * #1114: Document systemd configuration changes for the ZFS filesystem hook.
+ * #1116: Add dumping of database containers via their container names, handy for backing up
+   database containers from the host. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-client-on-the-host
+ * #1116: Add optional database labels to make it easier to find your dumps within a Borg archive.
+ * #1118: Fix a bug in which Borg hangs during database backup when different filesystems are in
+   use.
+ * #1122: To prevent the user from inadvertently excluding the "bootstrap" action's manifest, always
+   error and exit when the borgmatic runtime directory overlaps with the configured excludes.
+ * #1125: Clarify documentation about ZFS, Btrfs, and LVM snapshotting when a separate
+   filesystem is mounted in the source directory. (Spoiler: The separate filesystem doesn't get
+   included in the snapshot.)
+ * #1126: Create LVM snapshots as read-write to avoid an error when snapshotting ext4 filesystems
+   with orphaned files that need recovery.
+ * #1133: Fix the "spot" check to include borgmatic configuration files that were backed up to
+   support the "bootstrap" action.
+ * #1136: For all database hooks, record metadata about the dumps contained within an archive.
+ * #1139: Set "borgmatic" as the user agent when connecting to monitoring services.
+ * #1146: Fix a broken "create" action and "--archive latest" flag when multiple archives share the
+   same name with Borg 2.
+ * Treat configuration file permissions issues as errors instead of warnings.
+ * When running tests, use Ruff for faster and more comprehensive code linting and formatting,
+   replacing Flake8, Black, isort, etc.
+ * Switch from pipx to uv for installing development tools, and added tox-uv for speeding up test
+   environment creation. See the developer documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/
+
+2.0.7
+ * #1032: Fix a bug in which a Borg archive gets created even when a database hook fails.
+ * #1043: Support Btrfs subvolume paths in "source_directories" even when the subvolume is mounted
+   elsewhere.
+ * #1048: Ignore Btrfs subvolumes whose read-only status can't be determined.
+ * #1083: Add "debug_passphrase"/"display_passphrase" options and a "{unixtime}" placeholder in
+   support of Borg 2 features.
+ * #1099: Clarify documentation on command hooks order of execution.
+ * #1100: Fix a bug in which "borg --version" failing isn't considered a "fail" state in a command
+   hook.
+ * #1108: Fix a bug in which quoted "extra_borg_options" values containing spaces are passed to Borg
+   incorrectly.
+ * #1108: Add a "--comment" flag to the "create" action for creating an archive with a comment.
+ * Use the Bandit security analysis tool when running tests.
+ * SECURITY: Add timeouts to all monitoring hooks to prevent hangs on network requests, e.g. due to
+   a compromised monitoring server holding requests open.
+ * SECURITY: For the "spot" check, use a more secure source of randomness when selecting paths to
+   check.
+
+2.0.6
+ * #1068: Fix a warning from LVM about leaked file descriptors.
+ * #1086: Fix for the "spot" check breaking when the "--progress" flag is used.
+ * #1089: Fix for the "spot" check erroring when a checksum command errors.
+ * #1091: Fix for the "config generate" action generating invalid configuration when upgrading
+   deprecated command hooks.
+ * #1093: Fix for the LVM hook erroring when the "--dry-run" flag is used.
+ * #1094: Fix incorrect documentation about customizing Borg exit codes:
+   https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
+ * #1095: Fix for the "spot" check's "xxh64sum_command" option erroring on commands containing
+   spaces.
+ * Add support for Borg 2's "s3:" and "b2:" repository URLs, so you can backup to S3 or B2 cloud
+   storage services even without using Rclone.
+ * During the "spot" check, truncate log messages containing many file paths.
+
+2.0.5
+ * #1033: Add a "password_transport" option to the MariaDB and MySQL database hooks for customizing
+   how borgmatic transmits passwords to the database client.
+ * #1078: Add "keep_3monthly" and "keep_13weekly" options for customizing "prune" action archive
+   retention.
+ * #1078: Add a "use_chunks_archive" option for controlling whether Borg uses its chunks cache
+   directory.
+ * #1078: For the "compact" action, pass "--dry-run" through to Borg.
+ * #1085: Fix a regression in which the default monitoring verbosity is 0 (warnings only) instead of
+   1 (info about steps borgmatic is taking). This prevented logs from showing up in monitoring
+   services like Healthchecks unless you had an explicit monitoring verbosity set.
+ * Move Mastodon social hosting from Fosstodon to FLOSS.social: https://floss.social/@borgmatic
+ * The borgmatic project no longer accepts pull requests on GitHub. But see
+   https://torsion.org/borgmatic/#contributing for how you can still submit pull requests. You can
+   even use your GitHub account to login.
+
+2.0.4
+ * #1072: Fix path rewriting for non-root patterns in the ZFS, Btrfs, and LVM hooks.
+ * #1073: Clarify the documentation about when an "after: error" command hook runs and how it
+   differs from other hooks:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #1075: Fix an incorrect warning about Borg placeholders being unsupported in a command hook.
+ * #1080: If the exact same "everything" command hook is present in multiple configuration files,
+   only run it once.
+
+2.0.3
+ * #1065: Fix a regression in monitoring hooks in which an error pinged the finish state instead of
+   the fail state.
+ * #1066: Add a "states" option to command hooks, so you can optionally skip an "after" hook if
+   borgmatic encounters an error.
+ * #1071: Fix an error in the LVM hook when removing a snapshot directory.
+
+2.0.2
+ * #1035: Document potential performance issues and workarounds with the ZFS, Btrfs, and LVM hooks:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #1053: Display a nicer error message when the "recreate" action encounters an archive that
+   already exists.
+ * #1059: Fix a regression in which soft failure exit codes in command hooks were not respected.
+ * #1060: Fix action command hooks getting run too many times when multiple borgmatic actions are
+   executed (implicitly or explicitly).
+ * #1060: Don't run action command hooks for actions listed in the "skip_actions" option.
+ * #1062: Fix a regression that broke environment variable interpolation.
+ * #1063: List the configured "when" action names in the log entries for command hooks.
+
+2.0.1
+ * #1057: Fix argument parsing to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
+2.0.0
+ * TL;DR: More flexible, completely revamped command hooks. All configuration options settable on
+   the command-line. New configuration options for many command-line flags (including verbosity!).
+   New "key import" and "recreate" actions. Almost everything is backwards compatible—but mind those
+   deprecation warnings!
+ * #262: Add a "default_actions" option that supports disabling default actions when borgmatic is
+   run without any command-line arguments.
+ * #303: Deprecate the "--override" flag in favor of direct command-line flags for every borgmatic
+   configuration option. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides
+ * #303: Add configuration options that serve as defaults for some (but not all) command-line
+   action flags. For example, each entry in "repositories:" now has an "encryption" option that
+   applies to the "repo-create" action, serving as a default for the "--encryption" flag. See the
+   documentation for more information: https://torsion.org/borgmatic/docs/reference/configuration/
+ * #345: Add a "key import" action to import a repository key from backup.
+ * #422: Add home directory expansion to file-based and KeePassXC credential hooks.
+ * #610: Add a "recreate" action for recreating archives, for instance for retroactively excluding
+   particular files from existing archives.
+ * #790, #821: Deprecate all "before_*", "after_*" and "on_error" command hooks in favor of more
+   flexible "commands:". See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
+ * #790: BREAKING: For both new and deprecated command hooks, run a configured "after" hook even if
+   an error occurs first. This allows you to perform cleanup steps that correspond to "before"
+   preparation commands—even when something goes wrong.
+ * #790: BREAKING: Run all command hooks (both new and deprecated) respecting the
+   "working_directory" option if configured, meaning that hook commands are run in that directory.
+ * #793: Add configuration options for all verbosity and logging flags, so you don't have to set
+   them on the command-line.
+ * #836: Add a custom command option for the SQLite hook.
+ * #837: Add custom command options for the MongoDB hook.
+ * #1010: When using Borg 2, don't pass the "--stats" flag to "borg prune".
+ * #1020: Document a database use case involving a temporary database client container:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
+ * #1037: Fix an error with the "extract" action when both a remote repository and a
+   "working_directory" are used.
+ * #1044: Fix an error in the systemd credential hook when the credential name contains a "."
+   character.
+ * #1047: Add "key-file" and "yubikey" options to the KeePassXC credential hook.
+ * #1048: Fix a "no such file or directory" error in ZFS, Btrfs, and LVM hooks with nested
+   directories that reside on separate devices/filesystems.
+ * #1050: Fix a failure in the "spot" check when the archive contains a symlink.
+ * #1051: Add configuration filename to the "Successfully ran configuration file" log message.
+
+1.9.14
+ * #409: With the PagerDuty monitoring hook, send borgmatic logs to PagerDuty so they show up in the
+   incident UI. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
+ * #936: Clarify Zabbix monitoring hook documentation about creating items:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#zabbix-hook
+ * #1017: Fix a regression in which some MariaDB/MySQL passwords were not escaped correctly.
+ * #1021: Fix a regression in which the "exclude_patterns" option didn't expand "~" (the user's
+   home directory). This fix means that all "patterns" and "patterns_from" also now expand "~".
+ * #1023: Fix an error in the Btrfs hook when attempting to snapshot a read-only subvolume. Now,
+   read-only subvolumes are ignored since Btrfs can't actually snapshot them.
+
+1.9.13
+ * #975: Add a "compression" option to the PostgreSQL database hook.
+ * #1001: Fix a ZFS error during snapshot cleanup.
+ * #1003: In the Zabbix monitoring hook, support Zabbix 7.2's authentication changes.
+ * #1009: Send database passwords to MariaDB and MySQL via anonymous pipe, which is more secure than
+   using an environment variable.
+ * #1013: Send database passwords to MongoDB via anonymous pipe, which is more secure than using
+   "--password" on the command-line!
+ * #1015: When ctrl-C is pressed, more strongly encourage Borg to actually exit.
+ * Add a "verify_tls" option to the Uptime Kuma monitoring hook for disabling TLS verification.
+ * Add "tls" options to the MariaDB and MySQL database hooks to enable or disable TLS encryption
+   between client and server.
+
+1.9.12
+ * #1005: Fix the credential hooks to avoid using Python 3.12+ string features. Now borgmatic will
+   work with Python 3.9, 3.10, and 3.11 again.
+
+1.9.11
+ * #795: Add credential loading from file, KeePassXC, and Docker/Podman secrets. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #996: Fix the "create" action to omit the repository label prefix from Borg's output when
+   databases are enabled.
+ * #998: Send the "encryption_passphrase" option to Borg via an anonymous pipe, which is more secure
+   than using an environment variable.
+ * #999: Fix a runtime directory error from a conflict between "extra_borg_options" and special file
+   detection.
+ * #1001: For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from
+   a borgmatic configuration option (e.g. "source_directories")—not from other hooks within
+   borgmatic.
+ * #1001: Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical
+   volumes.
+ * #1001: Don't try to snapshot ZFS datasets that have the "canmount=off" property.
+ * Fix another error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+
+1.9.10
+ * #966: Add a "{credential ...}" syntax for loading systemd credentials into borgmatic
+   configuration files. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #987: Fix a "list" action error when the "encryption_passcommand" option is set.
+ * #987: When both "encryption_passcommand" and "encryption_passphrase" are configured, prefer
+   "encryption_passphrase" even if it's an empty value.
+ * #988: With the "max_duration" option or the "--max-duration" flag, run the archives and
+   repository checks separately so they don't interfere with one another. Previously, borgmatic
+   refused to run checks in this situation.
+ * #989: Fix the log message code to avoid using Python 3.10+ logging features. Now borgmatic will
+   work with Python 3.9 again.
+ * Capture and delay any log records produced before logging is fully configured, so early log
+   records don't get lost.
+ * Add support for Python 3.13.
+
+1.9.9
+ * #635: Log the repository path or label on every relevant log message, not just some logs.
+ * #961: When the "encryption_passcommand" option is set, call the command once from borgmatic to
+   collect the encryption passphrase and then pass it to Borg multiple times. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/
+ * #981: Fix a "spot" check file count delta error.
+ * #982: Fix for borgmatic "exclude_patterns" and "exclude_from" recursing into excluded
+   subdirectories.
+ * #983: Fix the Btrfs hook to support subvolumes with names like "@home" different from their
+   mount points.
+ * #985: Change the default value for the "--original-hostname" flag from "localhost" to no host
+   specified. This way, the "restore" action works without a hostname if there's a single matching
+   database dump.
+
+1.9.8
+ * #979: Fix root patterns so they don't have an invalid "sh:" prefix before getting passed to Borg.
+ * Expand the recent contributors documentation section to include ticket submitters—not just code
+   contributors—because there are multiple ways to contribute to the project! See:
+   https://torsion.org/borgmatic/#recent-contributors
+
+1.9.7
+ * #855: Add a Sentry monitoring hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook
+ * #968: Fix for a "spot" check error when a filename in the most recent archive contains a newline.
+ * #970: Fix for an error when there's a blank line in the configured patterns or excludes.
+ * #971: Fix for "exclude_from" files being completely ignored.
+ * #977: Fix for "exclude_patterns" and "exclude_from" not supporting explicit pattern styles (e.g.,
+   "sh:" or "re:").
+
+1.9.6
+ * #959: Fix an error in the Btrfs hook when a subvolume mounted at "/" is configured in borgmatic's
+   source directories.
+ * #960: Fix for archives storing relative source directory paths such that they contain the working
+   directory.
+ * #960: Fix the "spot" check to support relative source directory paths.
+ * #962: For the ZFS, Btrfs, and LVM hooks, perform path rewriting for excludes and patterns in
+   addition to the existing source directories rewriting.
+ * #962: Under the hood, merge all configured source directories, excludes, and patterns into a
+   unified temporary patterns file for passing to Borg. The borgmatic configuration options remain
+   unchanged.
+ * #962: For the LVM hook, add support for nested logical volumes.
+ * #965: Fix a borgmatic runtime directory error when running the "spot" check with a database hook
+   enabled.
+ * #969: Fix the "restore" action to work on database dumps without a port when a default port is
+   present in configuration.
+ * Fix the "spot" check to no longer consider pipe files within an archive for file comparisons.
+ * Fix the "spot" check to have a nicer error when there are no source paths to compare.
+ * Fix auto-excluding of special files (when databases are configured) to support relative source
+   directory paths.
+ * Drop support for Python 3.8, which has been end-of-lifed.
+
+1.9.5
+ * #418: Backup and restore databases that have the same name but with different ports, hostnames,
+   or hooks.
+ * #947: To avoid a hang in the database hooks, error and exit when the borgmatic runtime
+   directory overlaps with the configured excludes.
+ * #954: Fix a findmnt command error in the Btrfs hook by switching to parsing JSON output.
+ * #956: Fix the printing of a color reset code even when color is disabled.
+ * #958: Drop colorama as a library dependency.
+ * When the ZFS, Btrfs, or LVM hooks aren't configured, don't try to cleanup snapshots for them.
+
+1.9.4
+ * #80 (beta): Add an LVM hook for snapshotting and backing up LVM logical volumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #251 (beta): Add a Btrfs hook for snapshotting and backing up Btrfs subvolumes. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * #926: Fix a library error when running within a PyInstaller bundle.
+ * #950: Fix a snapshot unmount error in the ZFS hook when using nested datasets.
+ * Update the ZFS hook to discover and snapshot ZFS datasets even if they are parent/grandparent
+   directories of your source directories.
+ * Reorganize data source and monitoring hooks to make developing new hooks easier.
+
+1.9.3
+ * #261 (beta): Add a ZFS hook for snapshotting and backing up ZFS datasets. See the documentation
+   for more information: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
+ * Remove any temporary copies of the manifest file created in support of the "bootstrap" action.
+ * Deprecate the "store_config_files" option at the global scope and move it under the "bootstrap"
+   hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive
+ * Require the runtime directory to be an absolute path.
+ * Add a "--deleted" flag to the "repo-list" action for listing deleted archives that haven't
+   yet been compacted (Borg 2 only).
+ * Promote the "spot" check from a beta feature to stable.
+
+1.9.2
+ * #441: Apply the "umask" option to all relevant actions, not just some of them.
+ * #722: Remove the restriction that the "extract" and "mount" actions must match a single
+   repository. Now they work more like other actions, where each repository is applied in turn.
+ * #932: Fix the missing build backend setting in pyproject.toml to allow Fedora builds.
+ * #934: Update the logic that probes for the borgmatic streaming database dump, bootstrap
+   metadata, and check state directories to support more platforms and use cases. See the
+   documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory
+ * #934: Add the "RuntimeDirectory" and "StateDirectory" options to the sample systemd service
+   file to support the new runtime and state directory logic.
+ * #939: Fix borgmatic ignoring the "BORG_RELOCATED_REPO_ACCESS_IS_OK" and
+   "BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK" environment variables.
+ * Add a Pushover monitoring hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pushover-hook
+
+1.9.1
+ * #928: Fix the user runtime directory location on macOS (and possibly Cygwin).
+ * #930: Fix an error with the sample systemd service when no credentials are configured.
+ * #931: Fix an error when implicitly upgrading the check state directory from ~/.borgmatic to
+   ~/.local/state/borgmatic across filesystems.
+
+1.9.0
+ * #609: Fix the glob expansion of "source_directories" values to respect the "working_directory"
+   option.
+ * #609: BREAKING: Apply the "working_directory" option to all actions, not just "create". This
+   includes repository paths, destination paths, mount points, etc.
+ * #562: Deprecate the "borgmatic_source_directory" option in favor of "user_runtime_directory"
+   and "user_state_directory".
+ * #562: BREAKING: Move the default borgmatic streaming database dump and bootstrap metadata
+   directory from ~/.borgmatic to /run/user/$UID/borgmatic, which is more XDG-compliant. You can
+   override this location with the new "user_runtime_directory" option. Existing archives with
+   database dumps at the old location are still restorable. 
+ * #562, #638: Move the default check state directory from ~/.borgmatic to 
+   ~/.local/state/borgmatic. This is more XDG-compliant and also prevents these state files from
+   getting backed up (unless you explicitly include them). You can override this location with the
+   new "user_state_directory" option. After the first time you run the "check" action with borgmatic
+   1.9.0, you can safely delete the ~/.borgmatic directory.
+ * #838: BREAKING: With Borg 1.4+, store database dumps and bootstrap metadata in a "/borgmatic"
+   directory within a backup archive, so the path doesn't depend on the current user. This means
+   that you can now backup as one user and restore or bootstrap as another user, among other use
+   cases.
+ * #902: Add loading of encrypted systemd credentials. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/#using-systemd-service-credentials
+ * #911: Add a "key change-passphrase" action to change the passphrase protecting a repository key.
+ * #914: Fix a confusing apparent hang when when the repository location changes, and instead
+   show a helpful error message.
+ * #915: BREAKING: Rename repository actions like "rcreate" to more explicit names like
+   "repo-create" for compatibility with recent changes in Borg 2.0.0b10.
+ * #918: BREAKING: When databases are configured, don't auto-enable the "one_file_system" option,
+   as existing auto-excludes of special files should be sufficient to prevent Borg from hanging on
+   them. But if this change causes problems for you, you can always enable "one_file_system"
+   explicitly.
+ * #919: Clarify the command-line help for the "--config" flag.
+ * #919: Document a policy for versioning and breaking changes:
+   https://torsion.org/borgmatic/docs/how-to/upgrade/#versioning-and-breaking-changes
+ * #921: BREAKING: Change soft failure command hooks to skip only the current repository rather than
+   all repositories in the configuration file.
+ * #922: Replace setup.py (Python packaging metadata) with the more modern pyproject.toml.
+ * When using Borg 2, default the "archive_name_format" option to just "{hostname}", as Borg 2 does
+   not require unique archive names; identical archive names form a common "series" that can be
+   targeted together. See the Borg 2 documentation for more information:
+   https://borgbackup.readthedocs.io/en/2.0.0b13/changes.html#borg-1-2-x-1-4-x-to-borg-2-0
+ * Add support for Borg 2's "rclone:" repository URLs, so you can backup to 70+ cloud storage
+   services whether or not they support Borg explicitly.
+ * Add support for Borg 2's "sftp://" repository URLs.
+ * Update the "--match-archives" and "--archive" flags to support Borg 2 series names or archive
+   hashes.
+ * Add a "--match-archives" flag to the "prune" action.
+ * Add "--local-path" and "--remote-path" flags to the "config bootstrap" action for setting the
+   Borg executable paths used for bootstrapping.
+ * Add a "--user-runtime-directory" flag to the "config bootstrap" action for helping borgmatic
+   locate the bootstrap metadata stored in an archive.
+ * Add a Zabbix monitoring hook. See the documentation for more information:
+   https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#zabbix-hook
+ * Add a tarball of borgmatic's HTML documentation to the packages on the project page.
+
+1.8.14
+ * #896: Fix an error in borgmatic rcreate/init on an empty repository directory with Borg 1.4.
+ * #898: Add glob ("*") support to the "--repository" flag. Just quote any values containing
+   globs so your shell doesn't interpret them.
+ * #899: Fix for a "bad character" Borg error in which the "spot" check fed Borg an invalid pattern.
+ * #900: Fix for a potential traceback (TypeError) during the handling of another error.
+ * #904: Clarify the configuration reference about the "spot" check options:
+   https://torsion.org/borgmatic/docs/reference/configuration/
+ * #905: Fix the "source_directories_must_exist" option to work with relative "source_directories"
+   paths when a "working_directory" is set.
+ * #906: Add documentation details for how to run custom database dump commands using binaries from
+   running containers:
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
+ * Fix a regression in which the "color" option had no effect.
+ * Add a recent contributors section to the documentation, because credit where credit's due! See:
+   https://torsion.org/borgmatic/#recent-contributors
+
 1.8.13
  * #298: Add "delete" and "rdelete" actions to delete archives or entire repositories.
  * #785: Add an "only_run_on" option to consistency checks so you can limit a check to running on
@@ -49,7 +456,7 @@
  * Fix handling of the NO_COLOR environment variable to ignore an empty value.
  * Add documentation about backing up containerized databases by configuring borgmatic to exec into
    a container to run a dump command:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
 
 1.8.9
  * #311: Add custom dump/restore command options for MySQL and MariaDB.
@@ -319,7 +726,7 @@
    at the command-line. See the configuration reference for more information:
    https://torsion.org/borgmatic/docs/reference/configuration/
  * #649: Add documentation on backing up a database running in a container:
-   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#containers
+   https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-containers
  * #655: Fix error when databases are configured and a source directory doesn't exist.
  * Add code style plugins to enforce use of Python f-strings and prevent single-letter variables.
    To join in the pedantry, refresh your test environment with "tox --recreate".

README.md

@@ -40,8 +40,10 @@ checks:
       frequency: 2 weeks
 
 # Custom preparation scripts to run.
-before_backup:
-    - prepare-for-backup.sh
+commands:
+    - before: action
+      when: [create]
+      run: [prepare-for-backup.sh]
 
 # Databases to dump and include in backups.
 postgresql_databases:
@@ -56,20 +58,41 @@ borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
 
 ## Integrations
 
+### Data
+
 <a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.mongodb.com/"><img src="docs/static/mongodb.png" alt="MongoDB" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://sqlite.org/"><img src="docs/static/sqlite.png" alt="SQLite" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://openzfs.org/"><img src="docs/static/openzfs.png" alt="OpenZFS" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://btrfs.readthedocs.io/"><img src="docs/static/btrfs.png" alt="Btrfs" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sourceware.org/lvm2/"><img src="docs/static/lvm.png" alt="LVM" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://rclone.org"><img src="docs/static/rclone.png" alt="rclone" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Monitoring
+
 <a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://uptime.kuma.pet/"><img src="docs/static/uptimekuma.png" alt="Uptime Kuma" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://www.pagerduty.com/"><img src="docs/static/pagerduty.png" alt="PagerDuty" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.pushover.net/"><img src="docs/static/pushover.png" alt="Pushover" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://ntfy.sh/"><img src="docs/static/ntfy.png" alt="ntfy" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://grafana.com/oss/loki/"><img src="docs/static/loki.png" alt="Loki" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
 <a href="https://github.com/caronc/apprise/wiki"><img src="docs/static/apprise.png" alt="Apprise" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
-<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.zabbix.com/"><img src="docs/static/zabbix.png" alt="Zabbix" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://sentry.io/"><img src="docs/static/sentry.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+
+
+### Credentials
+
+<a href="https://systemd.io/"><img src="docs/static/systemd.png" alt="Sentry" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://www.docker.com/"><img src="docs/static/docker.png" alt="Docker" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://podman.io/"><img src="docs/static/podman.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
+<a href="https://keepassxc.org/"><img src="docs/static/keepassxc.png" alt="Podman" height="40px" style="margin-bottom:20px; margin-right:20px;"></a>
 
 
 ## Getting started
@@ -115,7 +138,8 @@ issues.
 
 ### Social
 
-Follow [borgmatic on Mastodon](https://fosstodon.org/@borgmatic).
+Follow <a rel="me" href="https://floss.social/@borgmatic">borgmatic on
+Mastodon</a>.
 
 
 ### Chat
@@ -136,22 +160,30 @@ Other questions or comments? Contact
 ### Contributing
 
 borgmatic [source code is
-available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
-on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
+available](https://projects.torsion.org/borgmatic-collective/borgmatic) and also
+has a read-only mirror on
+[GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
 
 borgmatic is licensed under the GNU General Public License version 3 or any
 later version.
 
-If you'd like to contribute to borgmatic development, please feel free to
-submit a [Pull
-Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
+If you'd like to contribute to borgmatic development, please feel free to submit
+a [pull
+request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) or
 open an
 [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) to
 discuss your idea. Note that you'll need to
 [register](https://projects.torsion.org/user/sign_up?invite_code=borgmatic)
-first. We also accept Pull Requests on GitHub, if that's more your thing. In
-general, contributions are very welcome. We don't bite!
+first. In general, contributions are very welcome. We don't bite!
 
 Also, please check out the [borgmatic development
 how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
 info on cloning source code, running tests, etc.
+
+### Recent contributors
+
+Thanks to all borgmatic contributors! There are multiple ways to contribute to
+this project, so the following includes those who have fixed bugs, contributed
+features, *or* filed tickets.
+
+{% include borgmatic/contributors.html %}

borgmatic/actions/borg.py

@@ -1,7 +1,7 @@
 import logging
 
 import borgmatic.borg.borg
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
@@ -20,12 +20,11 @@ def run_borg(
     Run the "borg" action for the given repository.
     '''
     if borg_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, borg_arguments.repository
+        repository,
+        borg_arguments.repository,
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Running arbitrary Borg command'
-        )
-        archive_name = borgmatic.borg.rlist.resolve_archive_name(
+        logger.info('Running arbitrary Borg command')
+        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             borg_arguments.archive,
             config,

borgmatic/actions/break_lock.py

@@ -19,11 +19,10 @@ def run_break_lock(
     Run the "break-lock" action for the given repository.
     '''
     if break_lock_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, break_lock_arguments.repository
+        repository,
+        break_lock_arguments.repository,
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Breaking repository and cache locks'
-        )
+        logger.info('Breaking repository and cache locks')
         borgmatic.borg.break_lock.break_lock(
             repository['path'],
             config,

borgmatic/actions/change_passphrase.py

@@ -0,0 +1,37 @@
+import logging
+
+import borgmatic.borg.change_passphrase
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_change_passphrase(
+    repository,
+    config,
+    local_borg_version,
+    change_passphrase_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "key change-passphrase" action for the given repository.
+    '''
+    if (
+        change_passphrase_arguments.repository is None
+        or borgmatic.config.validate.repositories_match(
+            repository,
+            change_passphrase_arguments.repository,
+        )
+    ):
+        logger.info('Changing repository passphrase')
+        borgmatic.borg.change_passphrase.change_passphrase(
+            repository['path'],
+            config,
+            local_borg_version,
+            change_passphrase_arguments,
+            global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )

borgmatic/actions/check.py

@@ -1,4 +1,5 @@
 import calendar
+import contextlib
 import datetime
 import hashlib
 import itertools
@@ -6,14 +7,21 @@ import logging
 import os
 import pathlib
 import random
+import shlex
+import shutil
+import textwrap
 
+import borgmatic.actions.config.bootstrap
+import borgmatic.actions.pattern
 import borgmatic.borg.check
 import borgmatic.borg.create
 import borgmatic.borg.environment
 import borgmatic.borg.extract
 import borgmatic.borg.list
-import borgmatic.borg.rlist
+import borgmatic.borg.pattern
+import borgmatic.borg.repo_list
 import borgmatic.borg.state
+import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.execute
 import borgmatic.hooks.command
@@ -50,12 +58,14 @@ def parse_checks(config, only_checks=None):
 
     if 'disabled' in checks:
         logger.warning(
-            'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead'
+            'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead',
         )
+
         if len(checks) > 1:
             logger.warning(
-                'Multiple checks are configured, but one of them is "disabled"; not running any checks'
+                'Multiple checks are configured, but one of them is "disabled"; not running any checks',
             )
+
         return ()
 
     return checks
@@ -159,6 +169,7 @@ def filter_checks_on_frequency(
                     **dict.fromkeys(day for day in days if day != 'weekday'),
                     **dict.fromkeys(WEEKDAY_DAYS),
                 }
+
             if 'weekend' in days:
                 days = {
                     **dict.fromkeys(day for day in days if day != 'weekend'),
@@ -167,7 +178,7 @@ def filter_checks_on_frequency(
 
             if calendar.day_name[datetime_now().weekday()] not in days:
                 logger.info(
-                    f"Skipping {check} check due to day of the week; check only runs on {'/'.join(days)} (use --force to check anyway)"
+                    f"Skipping {check} check due to day of the week; check only runs on {'/'.join(day.title() for day in days)} (use --force to check anyway)",
                 )
                 filtered_checks.remove(check)
                 continue
@@ -185,7 +196,7 @@ def filter_checks_on_frequency(
         if datetime_now() < check_time + frequency_delta:
             remaining = check_time + frequency_delta - datetime_now()
             logger.info(
-                f'Skipping {check} check due to configured frequency; {remaining} until next check (use --force to check anyway)'
+                f'Skipping {check} check due to configured frequency; {remaining} until next check (use --force to check anyway)',
             )
             filtered_checks.remove(check)
 
@@ -209,15 +220,11 @@ def make_check_time_path(config, borg_repository_id, check_type, archives_check_
     "archives", etc.), and a unique hash of the archives filter flags, return a path for recording
     that check's time (the time of that check last occurring).
     '''
-    borgmatic_source_directory = os.path.expanduser(
-        config.get(
-            'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
-        )
-    )
+    borgmatic_state_directory = borgmatic.config.paths.get_borgmatic_state_directory(config)
 
-    if check_type in ('archives', 'data'):
+    if check_type in {'archives', 'data'}:
         return os.path.join(
-            borgmatic_source_directory,
+            borgmatic_state_directory,
             'checks',
             borg_repository_id,
             check_type,
@@ -225,7 +232,7 @@ def make_check_time_path(config, borg_repository_id, check_type, archives_check_
         )
 
     return os.path.join(
-        borgmatic_source_directory,
+        borgmatic_state_directory,
         'checks',
         borg_repository_id,
         check_type,
@@ -250,7 +257,7 @@ def read_check_time(path):
     logger.debug(f'Reading check time from {path}')
 
     try:
-        return datetime.datetime.fromtimestamp(os.stat(path).st_mtime)
+        return datetime.datetime.fromtimestamp(os.stat(path).st_mtime)  # noqa: DTZ006
     except FileNotFoundError:
         return None
 
@@ -258,7 +265,7 @@ def read_check_time(path):
 def probe_for_check_time(config, borg_repository_id, check, archives_check_id):
     '''
     Given a configuration dict, a Borg repository ID, the name of a check type ("repository",
-    "archives", etc.), and a unique hash of the archives filter flags, return a the corresponding
+    "archives", etc.), and a unique hash of the archives filter flags, return the corresponding
     check time or None if such a check time does not exist.
 
     When the check type is "archives" or "data", this function probes two different paths to find
@@ -281,7 +288,7 @@ def probe_for_check_time(config, borg_repository_id, check, archives_check_id):
             (
                 make_check_time_path(config, borg_repository_id, check, archives_check_id),
                 make_check_time_path(config, borg_repository_id, check),
-            )
+            ),
         )
     )
 
@@ -296,14 +303,38 @@ def upgrade_check_times(config, borg_repository_id):
     Given a configuration dict and a Borg repository ID, upgrade any corresponding check times on
     disk from old-style paths to new-style paths.
 
-    Currently, the only upgrade performed is renaming an archive or data check path that looks like:
+    One upgrade performed is moving the checks directory from:
 
-      ~/.borgmatic/checks/1234567890/archives
+      {borgmatic_source_directory}/checks (e.g., ~/.borgmatic/checks)
 
     to:
 
-      ~/.borgmatic/checks/1234567890/archives/all
+      {borgmatic_state_directory}/checks (e.g. ~/.local/state/borgmatic)
+
+    Another upgrade is renaming an archive or data check path that looks like:
+
+      {borgmatic_state_directory}/checks/1234567890/archives
+
+    to:
+
+      {borgmatic_state_directory}/checks/1234567890/archives/all
     '''
+    borgmatic_source_checks_path = os.path.join(
+        borgmatic.config.paths.get_borgmatic_source_directory(config),
+        'checks',
+    )
+    borgmatic_state_path = borgmatic.config.paths.get_borgmatic_state_directory(config)
+    borgmatic_state_checks_path = os.path.join(borgmatic_state_path, 'checks')
+
+    if os.path.exists(borgmatic_source_checks_path) and not os.path.exists(
+        borgmatic_state_checks_path,
+    ):
+        logger.debug(
+            f'Upgrading archives check times directory from {borgmatic_source_checks_path} to {borgmatic_state_checks_path}',
+        )
+        os.makedirs(borgmatic_state_path, mode=0o700, exist_ok=True)
+        shutil.move(borgmatic_source_checks_path, borgmatic_state_checks_path)
+
     for check_type in ('archives', 'data'):
         new_path = make_check_time_path(config, borg_repository_id, check_type, 'all')
         old_path = os.path.dirname(new_path)
@@ -312,91 +343,115 @@ def upgrade_check_times(config, borg_repository_id):
         if not os.path.isfile(old_path) and not os.path.isfile(temporary_path):
             continue
 
-        logger.debug(f'Upgrading archives check time from {old_path} to {new_path}')
+        logger.debug(f'Upgrading archives check time file from {old_path} to {new_path}')
 
-        try:
-            os.rename(old_path, temporary_path)
-        except FileNotFoundError:
-            pass
+        with contextlib.suppress(FileNotFoundError):
+            shutil.move(old_path, temporary_path)
 
         os.mkdir(old_path)
-        os.rename(temporary_path, new_path)
+        shutil.move(temporary_path, new_path)
 
 
 def collect_spot_check_source_paths(
-    repository, config, local_borg_version, global_arguments, local_path, remote_path
+    repository,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path,
+    remote_path,
+    borgmatic_runtime_directory,
+    bootstrap_config_paths,
 ):
     '''
     Given a repository configuration dict, a configuration dict, the local Borg version, global
-    arguments as an argparse.Namespace instance, the local Borg path, and the remote Borg path,
-    collect the source paths that Borg would use in an actual create (but only include files).
+    arguments as an argparse.Namespace instance, the local Borg path, the remote Borg path, and the
+    bootstrap configuration paths as read from an archive's manifest, collect the source paths that
+    Borg would use in an actual create (but only include files). As part of this, include the
+    bootstrap configuration paths, so that any configuration files included in the archive to
+    support bootstrapping are also spot checked.
     '''
     stream_processes = any(
         borgmatic.hooks.dispatch.call_hooks(
             'use_streaming',
             config,
-            repository['path'],
-            borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        ).values()
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+        ).values(),
     )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
-        borgmatic.borg.create.make_base_create_command(
-            dry_run=True,
-            repository_path=repository['path'],
-            config=config,
-            config_paths=(),
-            local_borg_version=local_borg_version,
-            global_arguments=global_arguments,
-            borgmatic_source_directories=(),
-            local_path=local_path,
-            remote_path=remote_path,
-            list_files=True,
-            stream_processes=stream_processes,
-        )
+    (create_flags, create_positional_arguments, _) = borgmatic.borg.create.make_base_create_command(
+        dry_run=True,
+        repository_path=repository['path'],
+        # Omit "progress" because it interferes with "list_details".
+        config=dict(
+            {option: value for option, value in config.items() if option != 'progress'},
+            list_details=True,
+        ),
+        patterns=borgmatic.actions.pattern.process_patterns(
+            borgmatic.actions.pattern.collect_patterns(config)
+            + tuple(
+                borgmatic.borg.pattern.Pattern(
+                    config_path,
+                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                )
+                for config_path in bootstrap_config_paths
+            ),
+            config,
+            working_directory,
+        ),
+        local_borg_version=local_borg_version,
+        global_arguments=global_arguments,
+        borgmatic_runtime_directory=borgmatic_runtime_directory,
+        local_path=local_path,
+        remote_path=remote_path,
+        stream_processes=stream_processes,
     )
-    borg_environment = borgmatic.borg.environment.make_environment(config)
-
-    try:
-        working_directory = os.path.expanduser(config.get('working_directory'))
-    except TypeError:
-        working_directory = None
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     paths_output = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
+        environment=borgmatic.borg.environment.make_environment(config),
         working_directory=working_directory,
-        extra_environment=borg_environment,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
     paths = tuple(
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
-        if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
+        for path_line in paths_output.splitlines()
+        if path_line and path_line.startswith(('- ', '+ '))
     )
 
-    return tuple(path for path in paths if os.path.isfile(path))
+    return tuple(
+        path for path in paths if os.path.isfile(os.path.join(working_directory or '', path))
+    )
 
 
 BORG_DIRECTORY_FILE_TYPE = 'd'
+BORG_PIPE_FILE_TYPE = 'p'
 
 
 def collect_spot_check_archive_paths(
-    repository, archive, config, local_borg_version, global_arguments, local_path, remote_path
+    repository,
+    archive,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path,
+    remote_path,
+    borgmatic_runtime_directory,
 ):
     '''
     Given a repository configuration dict, the name of the latest archive, a configuration dict, the
-    local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, and
-    the remote Borg path, collect the paths from the given archive (but only include files and
-    symlinks).
+    local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
+    remote Borg path, and the borgmatic runtime directory, collect the paths from the given archive
+    (but only include files and symlinks and exclude borgmatic runtime directories).
+
+    These paths do not have a leading slash, as that's how Borg stores them. As a result, we don't
+    know whether they came from absolute or relative source directories.
     '''
-    borgmatic_source_directory = os.path.expanduser(
-        config.get(
-            'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
-        )
-    )
+    borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return tuple(
         path
@@ -406,17 +461,21 @@ def collect_spot_check_archive_paths(
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} /{path}{NL}',  # noqa: FS003
+            path_format='{type} {path}{NUL}',
             local_path=local_path,
             remote_path=remote_path,
         )
         for (file_type, path) in (line.split(' ', 1),)
-        if file_type != BORG_DIRECTORY_FILE_TYPE
-        if pathlib.Path(borgmatic_source_directory) not in pathlib.Path(path).parents
+        if file_type not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
+        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
+        if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
+        not in pathlib.Path(path).parents
     )
 
 
-SAMPLE_PATHS_SUBSET_COUNT = 10000
+SAMPLE_PATHS_SUBSET_COUNT = 5000
 
 
 def compare_spot_check_hashes(
@@ -427,28 +486,33 @@ def compare_spot_check_hashes(
     global_arguments,
     local_path,
     remote_path,
-    log_label,
     source_paths,
 ):
     '''
     Given a repository configuration dict, the name of the latest archive, a configuration dict, the
     local Borg version, global arguments as an argparse.Namespace instance, the local Borg path, the
-    remote Borg path, a log label, and spot check source paths, compare the hashes for a sampling of
-    the source paths with hashes from corresponding paths in the given archive. Return a sequence of
-    the paths that fail that hash comparison.
+    remote Borg path, and spot check source paths, compare the hashes for a sampling of the source
+    paths with hashes from corresponding paths in the given archive. Return a sequence of the paths
+    that fail that hash comparison.
     '''
     # Based on the configured sample percentage, come up with a list of random sample files from the
     # source directories.
     spot_check_config = next(check for check in config['checks'] if check['name'] == 'spot')
     sample_count = max(
-        int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)), 1
+        int(len(source_paths) * (min(spot_check_config['data_sample_percentage'], 100) / 100)),
+        1,
     )
-    source_sample_paths = tuple(random.sample(source_paths, sample_count))
-    existing_source_sample_paths = {
-        source_path for source_path in source_sample_paths if os.path.exists(source_path)
+    source_sample_paths = tuple(random.SystemRandom().sample(source_paths, sample_count))
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+    hashable_source_sample_path = {
+        source_path
+        for source_path in source_sample_paths
+        for full_source_path in (os.path.join(working_directory or '', source_path),)
+        if os.path.exists(full_source_path)
+        if not os.path.islink(full_source_path)
     }
     logger.debug(
-        f'{log_label}: Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check'
+        f'Sampling {sample_count} source paths (~{spot_check_config["data_sample_percentage"]}%) for spot check',
     )
 
     source_sample_paths_iterator = iter(source_sample_paths)
@@ -460,28 +524,34 @@ def compare_spot_check_hashes(
     while True:
         # Hash each file in the sample paths (if it exists).
         source_sample_paths_subset = tuple(
-            itertools.islice(source_sample_paths_iterator, SAMPLE_PATHS_SUBSET_COUNT)
+            itertools.islice(source_sample_paths_iterator, SAMPLE_PATHS_SUBSET_COUNT),
         )
         if not source_sample_paths_subset:
             break
 
         hash_output = borgmatic.execute.execute_command_and_capture_output(
-            (spot_check_config.get('xxh64sum_command', 'xxh64sum'),)
-            + tuple(
-                path for path in source_sample_paths_subset if path in existing_source_sample_paths
+            tuple(
+                shlex.quote(part)
+                for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
             )
+            + tuple(
+                path for path in source_sample_paths_subset if path in hashable_source_sample_path
+            ),
+            working_directory=working_directory,
         )
 
         source_hashes.update(
             **dict(
                 (reversed(line.split('  ', 1)) for line in hash_output.splitlines()),
-                # Represent non-existent files as having empty hashes so the comparison below still works.
+                # Represent non-existent files as having empty hashes so the comparison below still
+                # works. Same thing for filesystem links, since Borg produces empty archive hashes
+                # for them.
                 **{
                     path: ''
                     for path in source_sample_paths_subset
-                    if path not in existing_source_sample_paths
+                    if path not in hashable_source_sample_path
                 },
-            )
+            ),
         )
 
         # Get the hash for each file in the archive.
@@ -495,19 +565,19 @@ def compare_spot_check_hashes(
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} /{path}{NL}',  # noqa: FS003
+                    path_format='{xxh64} {path}{NUL}',
                     local_path=local_path,
                     remote_path=remote_path,
                 )
                 if line
-            )
+            ),
         )
 
     # Compare the source hashes with the archive hashes to see how many match.
     failing_paths = []
 
     for path, source_hash in source_hashes.items():
-        archive_hash = archive_hashes.get(path)
+        archive_hash = archive_hashes.get(path.lstrip(os.path.sep))
 
         if archive_hash is not None and archive_hash == source_hash:
             continue
@@ -517,6 +587,9 @@ def compare_spot_check_hashes(
     return tuple(failing_paths)
 
 
+MAX_SPOT_CHECK_PATHS_LENGTH = 1000
+
+
 def spot_check(
     repository,
     config,
@@ -524,18 +597,18 @@ def spot_check(
     global_arguments,
     local_path,
     remote_path,
+    borgmatic_runtime_directory,
 ):
     '''
     Given a repository dict, a loaded configuration dict, the local Borg version, global arguments
-    as an argparse.Namespace instance, the local Borg path, and the remote Borg path, perform a spot
-    check for the latest archive in the given repository.
+    as an argparse.Namespace instance, the local Borg path, the remote Borg path, and the borgmatic
+    runtime directory, perform a spot check for the latest archive in the given repository.
 
     A spot check compares file counts and also the hashes for a random sampling of source files on
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    log_label = f'{repository.get("label", repository["path"])}'
-    logger.debug(f'{log_label}: Running spot check')
+    logger.debug('Running spot check')
 
     try:
         spot_check_config = next(
@@ -546,20 +619,10 @@ def spot_check(
 
     if spot_check_config['data_tolerance_percentage'] > spot_check_config['data_sample_percentage']:
         raise ValueError(
-            'The data_tolerance_percentage must be less than or equal to the data_sample_percentage'
+            'The data_tolerance_percentage must be less than or equal to the data_sample_percentage',
         )
 
-    source_paths = collect_spot_check_source_paths(
-        repository,
-        config,
-        local_borg_version,
-        global_arguments,
-        local_path,
-        remote_path,
-    )
-    logger.debug(f'{log_label}: {len(source_paths)} total source paths for spot check')
-
-    archive = borgmatic.borg.rlist.resolve_archive_name(
+    archive = borgmatic.borg.repo_list.resolve_archive_name(
         repository['path'],
         'latest',
         config,
@@ -568,7 +631,26 @@ def spot_check(
         local_path,
         remote_path,
     )
-    logger.debug(f'{log_label}: Using archive {archive} for spot check')
+    logger.debug(f'Using archive {archive} for spot check')
+
+    source_paths = collect_spot_check_source_paths(
+        repository,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path,
+        remote_path,
+        borgmatic_runtime_directory,
+        bootstrap_config_paths=borgmatic.actions.config.bootstrap.load_config_paths_from_archive(
+            repository['path'],
+            archive,
+            config,
+            local_borg_version,
+            global_arguments,
+            borgmatic_runtime_directory,
+        ),
+    )
+    logger.debug(f'{len(source_paths)} total source paths for spot check')
 
     archive_paths = collect_spot_check_archive_paths(
         repository,
@@ -578,22 +660,45 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
+        borgmatic_runtime_directory,
     )
-    logger.debug(f'{log_label}: {len(archive_paths)} total archive paths for spot check')
+    logger.debug(f'{len(archive_paths)} total archive paths for spot check')
+
+    if len(source_paths) == 0:
+        truncated_archive_paths = textwrap.shorten(
+            ', '.join(set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.debug(f'Paths in latest archive but not source paths: {truncated_archive_paths}')
+        raise ValueError(
+            'Spot check failed: There are no source paths to compare against the archive',
+        )
 
     # Calculate the percentage delta between the source paths count and the archive paths count, and
     # compare that delta to the configured count tolerance percentage.
     count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100
 
     if count_delta_percentage > spot_check_config['count_tolerance_percentage']:
-        logger.debug(
-            f'{log_label}: Paths in source paths but not latest archive: {", ".join(set(source_paths) - set(archive_paths)) or "none"}'
+        rootless_source_paths = {path.lstrip(os.path.sep) for path in source_paths}
+        truncated_exclusive_source_paths = textwrap.shorten(
+            ', '.join(rootless_source_paths - set(archive_paths)) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
         )
         logger.debug(
-            f'{log_label}: Paths in latest archive but not source paths: {", ".join(set(archive_paths) - set(source_paths)) or "none"}'
+            f'Paths in source paths but not latest archive: {truncated_exclusive_source_paths}',
+        )
+        truncated_exclusive_archive_paths = textwrap.shorten(
+            ', '.join(set(archive_paths) - rootless_source_paths) or 'none',
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.debug(
+            f'Paths in latest archive but not source paths: {truncated_exclusive_archive_paths}',
         )
         raise ValueError(
-            f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)'
+            f'Spot check failed: {count_delta_percentage:.2f}% file count delta between source paths and latest archive (tolerance is {spot_check_config["count_tolerance_percentage"]}%)',
         )
 
     failing_paths = compare_spot_check_hashes(
@@ -604,25 +709,29 @@ def spot_check(
         global_arguments,
         local_path,
         remote_path,
-        log_label,
         source_paths,
     )
 
     # Error if the percentage of failing hashes exceeds the configured tolerance percentage.
-    logger.debug(f'{log_label}: {len(failing_paths)} non-matching spot check hashes')
+    logger.debug(f'{len(failing_paths)} non-matching spot check hashes')
     data_tolerance_percentage = spot_check_config['data_tolerance_percentage']
     failing_percentage = (len(failing_paths) / len(source_paths)) * 100
 
     if failing_percentage > data_tolerance_percentage:
+        truncated_failing_paths = textwrap.shorten(
+            ', '.join(failing_paths),
+            width=MAX_SPOT_CHECK_PATHS_LENGTH,
+            placeholder=' ...',
+        )
         logger.debug(
-            f'{log_label}: Source paths with data not matching the latest archive: {", ".join(failing_paths)}'
+            f'Source paths with data not matching the latest archive: {truncated_failing_paths}',
         )
         raise ValueError(
-            f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)'
+            f'Spot check failed: {failing_percentage:.2f}% of source paths with data not matching the latest archive (tolerance is {data_tolerance_percentage}%)',
         )
 
     logger.info(
-        f'{log_label}: Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta'
+        f'Spot check passed with a {count_delta_percentage:.2f}% file count delta and a {failing_percentage:.2f}% file data delta',
     )
 
 
@@ -630,12 +739,12 @@ def run_check(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     check_arguments,
     global_arguments,
     local_path,
     remote_path,
+    hook_context,
 ):
     '''
     Run the "check" action for the given repository.
@@ -643,20 +752,13 @@ def run_check(
     Raise ValueError if the Borg repository ID cannot be determined.
     '''
     if check_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, check_arguments.repository
+        repository,
+        check_arguments.repository,
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_check'),
-        config.get('umask'),
-        config_filename,
-        'pre-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+    logger.info('Running consistency checks')
 
-    logger.info(f'{repository.get("label", repository["path"])}: Running consistency checks')
     repository_id = borgmatic.borg.check.get_repository_id(
         repository['path'],
         config,
@@ -668,7 +770,10 @@ def run_check(
     upgrade_check_times(config, repository_id)
     configured_checks = parse_checks(config, check_arguments.only_checks)
     archive_filter_flags = borgmatic.borg.check.make_archive_filter_flags(
-        local_borg_version, config, configured_checks, check_arguments
+        local_borg_version,
+        config,
+        configured_checks,
+        check_arguments,
     )
     archives_check_id = make_archives_check_id(archive_filter_flags)
     checks = filter_checks_on_frequency(
@@ -679,50 +784,75 @@ def run_check(
         archives_check_id,
     )
     borg_specific_checks = set(checks).intersection({'repository', 'archives', 'data'})
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     if borg_specific_checks:
-        borgmatic.borg.check.check_archives(
-            repository['path'],
-            config,
-            local_borg_version,
-            check_arguments,
-            global_arguments,
-            borg_specific_checks,
-            archive_filter_flags,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
-        for check in borg_specific_checks:
-            write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))
+        with borgmatic.hooks.command.Before_after_hooks(
+            command_hooks=config.get('commands'),
+            before_after='step',
+            umask=config.get('umask'),
+            working_directory=working_directory,
+            dry_run=global_arguments.dry_run,
+            action_names=('check',),
+            step_names=('archives_repository_data',),
+            **hook_context,
+        ):
+            borgmatic.borg.check.check_archives(
+                repository['path'],
+                config,
+                local_borg_version,
+                check_arguments,
+                global_arguments,
+                borg_specific_checks,
+                archive_filter_flags,
+                local_path=local_path,
+                remote_path=remote_path,
+            )
+            for check in borg_specific_checks:
+                write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))
 
     if 'extract' in checks:
-        borgmatic.borg.extract.extract_last_archive_dry_run(
-            config,
-            local_borg_version,
-            global_arguments,
-            repository['path'],
-            config.get('lock_wait'),
-            local_path,
-            remote_path,
-        )
-        write_check_time(make_check_time_path(config, repository_id, 'extract'))
+        with borgmatic.hooks.command.Before_after_hooks(
+            command_hooks=config.get('commands'),
+            before_after='step',
+            umask=config.get('umask'),
+            working_directory=working_directory,
+            dry_run=global_arguments.dry_run,
+            action_names=('check',),
+            step_names=('extract',),
+            **hook_context,
+        ):
+            borgmatic.borg.extract.extract_last_archive_dry_run(
+                config,
+                local_borg_version,
+                global_arguments,
+                repository['path'],
+                config.get('lock_wait'),
+                local_path,
+                remote_path,
+            )
+            write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
-        spot_check(
-            repository,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-        )
-        write_check_time(make_check_time_path(config, repository_id, 'spot'))
+        with borgmatic.hooks.command.Before_after_hooks(
+            command_hooks=config.get('commands'),
+            before_after='step',
+            umask=config.get('umask'),
+            working_directory=working_directory,
+            dry_run=global_arguments.dry_run,
+            action_names=('check',),
+            step_names=('spot',),
+            **hook_context,
+        ):
+            with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
+                spot_check(
+                    repository,
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                    borgmatic_runtime_directory,
+                )
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_check'),
-        config.get('umask'),
-        config_filename,
-        'post-check',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+            write_check_time(make_check_time_path(config, repository_id, 'spot'))

borgmatic/actions/compact.py

@@ -12,7 +12,6 @@ def run_compact(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     compact_arguments,
     global_arguments,
@@ -24,22 +23,13 @@ def run_compact(
     Run the "compact" action for the given repository.
     '''
     if compact_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, compact_arguments.repository
+        repository,
+        compact_arguments.repository,
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_compact'),
-        config.get('umask'),
-        config_filename,
-        'pre-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if borgmatic.borg.feature.available(borgmatic.borg.feature.Feature.COMPACT, local_borg_version):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Compacting segments{dry_run_label}'
-        )
+        logger.info(f'Compacting segments{dry_run_label}')
         borgmatic.borg.compact.compact_segments(
             global_arguments.dry_run,
             repository['path'],
@@ -48,19 +38,7 @@ def run_compact(
             global_arguments,
             local_path=local_path,
             remote_path=remote_path,
-            progress=compact_arguments.progress,
             cleanup_commits=compact_arguments.cleanup_commits,
-            threshold=compact_arguments.threshold,
         )
     else:  # pragma: nocover
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Skipping compact (only available/needed in Borg 1.2+)'
-        )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_compact'),
-        config.get('umask'),
-        config_filename,
-        'post-compact',
-        global_arguments.dry_run,
-        **hook_context,
-    )
+        logger.info('Skipping compact (only available/needed in Borg 1.2+)')

borgmatic/actions/config/bootstrap.py

@@ -3,68 +3,94 @@ import logging
 import os
 
 import borgmatic.borg.extract
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
+import borgmatic.config.paths
 import borgmatic.config.validate
 import borgmatic.hooks.command
-from borgmatic.borg.state import DEFAULT_BORGMATIC_SOURCE_DIRECTORY
 
 logger = logging.getLogger(__name__)
 
 
-def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
+def make_bootstrap_config(bootstrap_arguments):
     '''
-    Given the bootstrap arguments as an argparse.Namespace (containing the repository and archive
-    name, borgmatic source directory, destination directory, and whether to strip components), the
-    global arguments as an argparse.Namespace (containing the dry run flag and the local borg
-    version), return the config paths from the manifest.json file in the borgmatic source directory
-    after extracting it from the repository.
+    Given the bootstrap arguments as an argparse.Namespace, return a corresponding config dict.
+    '''
+    return {
+        'borgmatic_source_directory': bootstrap_arguments.borgmatic_source_directory,
+        'local_path': bootstrap_arguments.local_path,
+        'remote_path': bootstrap_arguments.remote_path,
+        # In case the repo has been moved or is accessed from a different path at the point of
+        # bootstrapping.
+        'relocated_repo_access_is_ok': True,
+        'ssh_command': bootstrap_arguments.ssh_command,
+        'user_runtime_directory': bootstrap_arguments.user_runtime_directory,
+    }
+
+
+def load_config_paths_from_archive(
+    repository_path,
+    archive_name,
+    config,
+    local_borg_version,
+    global_arguments,
+    borgmatic_runtime_directory,
+):
+    '''
+    Given a repository path, an archive name, a configuration dict, the local Borg version, the
+    global arguments as an argparse.Namespace, and the borgmatic runtime directory, return the
+    config paths from the manifest.json file in the borgmatic source directory or runtime directory
+    within the repository archive.
 
     Raise ValueError if the manifest JSON is missing, can't be decoded, or doesn't contain the
     expected configuration path data.
     '''
-    borgmatic_source_directory = (
-        bootstrap_arguments.borgmatic_source_directory or DEFAULT_BORGMATIC_SOURCE_DIRECTORY
-    )
-    borgmatic_manifest_path = os.path.expanduser(
-        os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
-    )
-    config = {'ssh_command': bootstrap_arguments.ssh_command}
+    # Probe for the manifest file in multiple locations, as the default location has moved to the
+    # borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But we
+    # still want to support reading the manifest from previously created archives as well.
+    for base_directory in (
+        'borgmatic',
+        borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
+        borgmatic.config.paths.get_borgmatic_source_directory(config),
+    ):
+        borgmatic_manifest_path = 'sh:' + os.path.join(
+            base_directory,
+            'bootstrap',
+            'manifest.json',
+        )
 
-    extract_process = borgmatic.borg.extract.extract_archive(
-        global_arguments.dry_run,
-        bootstrap_arguments.repository,
-        borgmatic.borg.rlist.resolve_archive_name(
-            bootstrap_arguments.repository,
-            bootstrap_arguments.archive,
+        extract_process = borgmatic.borg.extract.extract_archive(
+            global_arguments.dry_run,
+            repository_path,
+            archive_name,
+            [borgmatic_manifest_path],
             config,
             local_borg_version,
             global_arguments,
-        ),
-        [borgmatic_manifest_path],
-        config,
-        local_borg_version,
-        global_arguments,
-        extract_to_stdout=True,
-    )
-    manifest_json = extract_process.stdout.read()
+            local_path=config.get('local_path'),
+            remote_path=config.get('remote_path'),
+            extract_to_stdout=True,
+        )
+        manifest_json = extract_process.stdout.read()
 
-    if not manifest_json:
+        if manifest_json:
+            break
+    else:
         raise ValueError(
-            'Cannot read configuration paths from archive due to missing bootstrap manifest'
+            'Cannot read configuration paths from archive due to missing archive or bootstrap manifest',
         )
 
     try:
         manifest_data = json.loads(manifest_json)
     except json.JSONDecodeError as error:
         raise ValueError(
-            f'Cannot read configuration paths from archive due to invalid bootstrap manifest JSON: {error}'
+            f'Cannot read configuration paths from archive due to invalid bootstrap manifest JSON: {error}',
         )
 
     try:
         return manifest_data['config_paths']
     except KeyError:
         raise ValueError(
-            'Cannot read configuration paths from archive due to invalid bootstrap manifest'
+            'Cannot read configuration paths from archive due to invalid bootstrap manifest',
         )
 
 
@@ -75,29 +101,42 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
     Raise ValueError if the bootstrap configuration could not be loaded.
     Raise CalledProcessError or OSError if Borg could not be run.
     '''
-    manifest_config_paths = get_config_paths(
-        bootstrap_arguments, global_arguments, local_borg_version
+    config = make_bootstrap_config(bootstrap_arguments)
+    archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        bootstrap_arguments.repository,
+        bootstrap_arguments.archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=bootstrap_arguments.local_path,
+        remote_path=bootstrap_arguments.remote_path,
     )
-    config = {'ssh_command': bootstrap_arguments.ssh_command}
+
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
+        manifest_config_paths = load_config_paths_from_archive(
+            bootstrap_arguments.repository,
+            archive_name,
+            config,
+            local_borg_version,
+            global_arguments,
+            borgmatic_runtime_directory,
+        )
 
     logger.info(f"Bootstrapping config paths: {', '.join(manifest_config_paths)}")
 
     borgmatic.borg.extract.extract_archive(
         global_arguments.dry_run,
         bootstrap_arguments.repository,
-        borgmatic.borg.rlist.resolve_archive_name(
-            bootstrap_arguments.repository,
-            bootstrap_arguments.archive,
-            config,
-            local_borg_version,
-            global_arguments,
-        ),
+        archive_name,
         [config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
-        config,
+        # Only add progress here and not the extract_archive() call above, because progress
+        # conflicts with extract_to_stdout.
+        dict(config, progress=bootstrap_arguments.progress or False),
         local_borg_version,
         global_arguments,
+        local_path=bootstrap_arguments.local_path,
+        remote_path=bootstrap_arguments.remote_path,
         extract_to_stdout=False,
         destination_path=bootstrap_arguments.destination,
         strip_components=bootstrap_arguments.strip_components,
-        progress=bootstrap_arguments.progress,
     )

borgmatic/actions/config/generate.py

@@ -19,7 +19,7 @@ def run_generate(generate_arguments, global_arguments):
     dry_run_label = ' (dry run; not actually writing anything)' if global_arguments.dry_run else ''
 
     logger.answer(
-        f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}'
+        f'Generating a configuration file at: {generate_arguments.destination_filename}{dry_run_label}',
     )
 
     borgmatic.config.generate.generate_sample_configuration(
@@ -36,7 +36,7 @@ def run_generate(generate_arguments, global_arguments):
 Merged in the contents of configuration file at: {generate_arguments.source_filename}
 To review the changes made, run:
 
-    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}'''
+    diff --unified {generate_arguments.source_filename} {generate_arguments.destination_filename}''',
         )
 
     logger.answer(
@@ -44,5 +44,5 @@ To review the changes made, run:
 This includes all available configuration options with example values, the few
 required options as indicated. Please edit the file to suit your needs.
 
-If you ever need help: https://torsion.org/borgmatic/#issues'''
+If you ever need help: https://torsion.org/borgmatic/#issues''',
     )

borgmatic/actions/config/validate.py

@@ -18,7 +18,7 @@ def run_validate(validate_arguments, configs):
     borgmatic.logger.add_custom_log_levels()
 
     if validate_arguments.show:
-        for config_path, config in configs.items():
+        for config in configs.values():
             if len(configs) > 1:
                 logger.answer('---')
 

borgmatic/actions/create.py

@@ -1,54 +1,23 @@
-import importlib.metadata
-import json
 import logging
-import os
 
 import borgmatic.actions.json
 import borgmatic.borg.create
-import borgmatic.borg.state
+import borgmatic.borg.feature
+import borgmatic.borg.rename
+import borgmatic.borg.repo_list
+import borgmatic.config.paths
 import borgmatic.config.validate
-import borgmatic.hooks.command
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
+from borgmatic.actions import pattern
 
 logger = logging.getLogger(__name__)
 
 
-def create_borgmatic_manifest(config, config_paths, dry_run):
-    '''
-    Create a borgmatic manifest file to store the paths to the configuration files used to create
-    the archive.
-    '''
-    if dry_run:
-        return
-
-    borgmatic_source_directory = config.get(
-        'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
-    )
-
-    borgmatic_manifest_path = os.path.expanduser(
-        os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
-    )
-
-    if not os.path.exists(borgmatic_manifest_path):
-        os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
-
-    with open(borgmatic_manifest_path, 'w') as config_list_file:
-        json.dump(
-            {
-                'borgmatic_version': importlib.metadata.version('borgmatic'),
-                'config_paths': config_paths,
-            },
-            config_list_file,
-        )
-
-
 def run_create(
     config_filename,
     repository,
     config,
     config_paths,
-    hook_context,
     local_borg_version,
     create_arguments,
     global_arguments,
@@ -62,71 +31,169 @@ def run_create(
     If create_arguments.json is True, yield the JSON output from creating the archive.
     '''
     if create_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, create_arguments.repository
+        repository,
+        create_arguments.repository,
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_backup'),
-        config.get('umask'),
-        config_filename,
-        'pre-backup',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-    logger.info(f'{repository.get("label", repository["path"])}: Creating archive{dry_run_label}')
-    borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-        'remove_data_source_dumps',
-        config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        global_arguments.dry_run,
-    )
-    active_dumps = borgmatic.hooks.dispatch.call_hooks(
-        'dump_data_sources',
-        config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        global_arguments.dry_run,
-    )
-    if config.get('store_config_files', True):
-        create_borgmatic_manifest(
+    if config.get('list_details') and config.get('progress'):
+        raise ValueError(
+            'With the create action, only one of --list/--files/list_details and --progress/progress can be used.',
+        )
+
+    if config.get('list_details') and create_arguments.json:
+        raise ValueError(
+            'With the create action, only one of --list/--files/list_details and --json can be used.',
+        )
+
+    logger.info(f'Creating archive{dry_run_label}')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
             config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            borgmatic_runtime_directory,
+            global_arguments.dry_run,
+        )
+        patterns = pattern.process_patterns(
+            pattern.collect_patterns(config),
+            config,
+            working_directory,
+        )
+        active_dumps = borgmatic.hooks.dispatch.call_hooks(
+            'dump_data_sources',
+            config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
             config_paths,
+            borgmatic_runtime_directory,
+            patterns,
             global_arguments.dry_run,
         )
-    stream_processes = [process for processes in active_dumps.values() for process in processes]
 
-    json_output = borgmatic.borg.create.create_archive(
-        global_arguments.dry_run,
-        repository['path'],
+        # Process the patterns again in case any data source hooks updated them. Without this step,
+        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+        # the same named pipe twice.
+        patterns = pattern.process_patterns(
+            patterns,
+            config,
+            working_directory,
+            skip_expand_paths=config_paths,
+        )
+        stream_processes = [process for processes in active_dumps.values() for process in processes]
+
+        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+        # to make sure we only create a real archive if all the streaming processes completed
+        # successfully (create_archive will fail if a streaming process fails, but the archive might
+        # have already been created at this point).
+        use_checkpoint = bool(stream_processes)
+
+        json_output = borgmatic.borg.create.create_archive(
+            global_arguments.dry_run,
+            repository['path'],
+            config,
+            patterns,
+            local_borg_version,
+            global_arguments,
+            borgmatic_runtime_directory,
+            archive_suffix='.checkpoint' if use_checkpoint else '',
+            local_path=local_path,
+            remote_path=remote_path,
+            json=create_arguments.json,
+            comment=create_arguments.comment,
+            stream_processes=stream_processes,
+        )
+
+        if use_checkpoint:
+            rename_checkpoint_archive(
+                repository['path'],
+                global_arguments,
+                config,
+                local_borg_version,
+                local_path,
+                remote_path,
+            )
+
+        if json_output:
+            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+            if use_checkpoint:
+                # Patch archive name and ID
+                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                    repository['path'],
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                )
+
+                output['archive']['name'] = renamed_archive['name']
+                output['archive']['id'] = renamed_archive['id']
+
+            yield output
+
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            borgmatic_runtime_directory,
+            global_arguments.dry_run,
+        )
+
+
+def rename_checkpoint_archive(
+    repository_path,
+    global_arguments,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    '''
+    Renames the latest archive to not have a '.checkpoint' suffix.
+
+    Raises ValueError if
+    - there is not latest archive
+    - the latest archive does not have a '.checkpoint' suffix
+
+    Implementation note: We cannot reliably get the just created archive name.
+    So we resort to listing the archives and picking the last one.
+
+    A similar comment applies to retrieving the ID of the renamed archive.
+    '''
+    archive = borgmatic.borg.repo_list.get_latest_archive(
+        repository_path,
         config,
-        config_paths,
         local_borg_version,
         global_arguments,
-        local_path=local_path,
-        remote_path=remote_path,
-        progress=create_arguments.progress,
-        stats=create_arguments.stats,
-        json=create_arguments.json,
-        list_files=create_arguments.list_files,
-        stream_processes=stream_processes,
+        local_path,
+        remote_path,
+        consider_checkpoints=True,
     )
-    if json_output:
-        yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
 
-    borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-        'remove_data_source_dumps',
+    archive_name = archive['name']
+
+    if not archive_name.endswith('.checkpoint'):
+        raise ValueError(f'Latest archive did not have a .checkpoint suffix. Got: {archive_name}')
+
+    new_archive_name = archive_name.removesuffix('.checkpoint')
+
+    logger.info(f'Renaming archive {archive_name} -> {new_archive_name}')
+
+    borgmatic.borg.rename.rename_archive(
+        repository_path,
+        (
+            archive['id']
+            if borgmatic.borg.feature.available(
+                borgmatic.borg.feature.Feature.ARCHIVE_SERIES, local_borg_version
+            )
+            else archive['name']
+        ),
+        new_archive_name,
+        global_arguments.dry_run,
         config,
-        config_filename,
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        global_arguments.dry_run,
-    )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_backup'),
-        config.get('umask'),
-        config_filename,
-        'post-backup',
-        global_arguments.dry_run,
-        **hook_context,
+        local_borg_version,
+        local_path,
+        remote_path,
     )

borgmatic/actions/delete.py

@@ -2,8 +2,8 @@ import logging
 
 import borgmatic.actions.arguments
 import borgmatic.borg.delete
-import borgmatic.borg.rdelete
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_delete
+import borgmatic.borg.repo_list
 
 logger = logging.getLogger(__name__)
 
@@ -21,12 +21,13 @@ def run_delete(
     Run the "delete" action for the given repository and archive(s).
     '''
     if delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, delete_arguments.repository
+        repository,
+        delete_arguments.repository,
     ):
-        logger.answer(f'{repository.get("label", repository["path"])}: Deleting archives')
+        logger.answer('Deleting archives')
 
         archive_name = (
-            borgmatic.borg.rlist.resolve_archive_name(
+            borgmatic.borg.repo_list.resolve_archive_name(
                 repository['path'],
                 delete_arguments.archive,
                 config,

borgmatic/actions/export_key.py

@@ -19,9 +19,10 @@ def run_export_key(
     Run the "key export" action for the given repository.
     '''
     if export_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, export_arguments.repository
+        repository,
+        export_arguments.repository,
     ):
-        logger.info(f'{repository.get("label", repository["path"])}: Exporting repository key')
+        logger.info('Exporting repository key')
         borgmatic.borg.export_key.export_key(
             repository['path'],
             config,

borgmatic/actions/export_tar.py

@@ -1,7 +1,7 @@
 import logging
 
 import borgmatic.borg.export_tar
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
@@ -20,15 +20,14 @@ def run_export_tar(
     Run the "export-tar" action for the given repository.
     '''
     if export_tar_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, export_tar_arguments.repository
+        repository,
+        export_tar_arguments.repository,
     ):
-        logger.info(
-            f'{repository["path"]}: Exporting archive {export_tar_arguments.archive} as tar file'
-        )
+        logger.info(f'Exporting archive {export_tar_arguments.archive} as tar file')
         borgmatic.borg.export_tar.export_tar_archive(
             global_arguments.dry_run,
             repository['path'],
-            borgmatic.borg.rlist.resolve_archive_name(
+            borgmatic.borg.repo_list.resolve_archive_name(
                 repository['path'],
                 export_tar_arguments.archive,
                 config,
@@ -45,6 +44,5 @@ def run_export_tar(
             local_path=local_path,
             remote_path=remote_path,
             tar_filter=export_tar_arguments.tar_filter,
-            list_files=export_tar_arguments.list_files,
             strip_components=export_tar_arguments.strip_components,
         )

borgmatic/actions/extract.py

@@ -1,7 +1,7 @@
 import logging
 
 import borgmatic.borg.extract
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
 import borgmatic.hooks.command
 
@@ -12,7 +12,6 @@ def run_extract(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     extract_arguments,
     global_arguments,
@@ -22,24 +21,15 @@ def run_extract(
     '''
     Run the "extract" action for the given repository.
     '''
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_extract'),
-        config.get('umask'),
-        config_filename,
-        'pre-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )
     if extract_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, extract_arguments.repository
+        repository,
+        extract_arguments.repository,
     ):
-        logger.info(
-            f'{repository.get("label", repository["path"])}: Extracting archive {extract_arguments.archive}'
-        )
+        logger.info(f'Extracting archive {extract_arguments.archive}')
         borgmatic.borg.extract.extract_archive(
             global_arguments.dry_run,
             repository['path'],
-            borgmatic.borg.rlist.resolve_archive_name(
+            borgmatic.borg.repo_list.resolve_archive_name(
                 repository['path'],
                 extract_arguments.archive,
                 config,
@@ -56,13 +46,4 @@ def run_extract(
             remote_path=remote_path,
             destination_path=extract_arguments.destination,
             strip_components=extract_arguments.strip_components,
-            progress=extract_arguments.progress,
         )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_extract'),
-        config.get('umask'),
-        config_filename,
-        'post-extract',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/import_key.py

@@ -0,0 +1,34 @@
+import logging
+
+import borgmatic.borg.import_key
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_import_key(
+    repository,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "key import" action for the given repository.
+    '''
+    if import_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository,
+        import_arguments.repository,
+    ):
+        logger.info('Importing repository key')
+        borgmatic.borg.import_key.import_key(
+            repository['path'],
+            config,
+            local_borg_version,
+            import_arguments,
+            global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )

borgmatic/actions/info.py

@@ -3,7 +3,7 @@ import logging
 import borgmatic.actions.arguments
 import borgmatic.actions.json
 import borgmatic.borg.info
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
@@ -24,13 +24,13 @@ def run_info(
     If info_arguments.json is True, yield the JSON output from the info for the archive.
     '''
     if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, info_arguments.repository
+        repository,
+        info_arguments.repository,
     ):
         if not info_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying archive summary information'
-            )
-        archive_name = borgmatic.borg.rlist.resolve_archive_name(
+            logger.answer('Displaying archive summary information')
+
+        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             info_arguments.archive,
             config,

borgmatic/actions/list.py

@@ -23,15 +23,16 @@ def run_list(
     If list_arguments.json is True, yield the JSON output from listing the archive.
     '''
     if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, list_arguments.repository
+        repository,
+        list_arguments.repository,
     ):
         if not list_arguments.json:
             if list_arguments.find_paths:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
+                logger.answer('Searching archives')
             elif not list_arguments.archive:  # pragma: no cover
-                logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
+                logger.answer('Listing archives')
 
-        archive_name = borgmatic.borg.rlist.resolve_archive_name(
+        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
             repository['path'],
             list_arguments.archive,
             config,

borgmatic/actions/mount.py

@@ -1,7 +1,7 @@
 import logging
 
 import borgmatic.borg.mount
-import borgmatic.borg.rlist
+import borgmatic.borg.repo_list
 import borgmatic.config.validate
 
 logger = logging.getLogger(__name__)
@@ -20,18 +20,17 @@ def run_mount(
     Run the "mount" action for the given repository.
     '''
     if mount_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, mount_arguments.repository
+        repository,
+        mount_arguments.repository,
     ):
         if mount_arguments.archive:
-            logger.info(
-                f'{repository.get("label", repository["path"])}: Mounting archive {mount_arguments.archive}'
-            )
+            logger.info(f'Mounting archive {mount_arguments.archive}')
         else:  # pragma: nocover
-            logger.info(f'{repository.get("label", repository["path"])}: Mounting repository')
+            logger.info('Mounting repository')
 
         borgmatic.borg.mount.mount_archive(
             repository['path'],
-            borgmatic.borg.rlist.resolve_archive_name(
+            borgmatic.borg.repo_list.resolve_archive_name(
                 repository['path'],
                 mount_arguments.archive,
                 config,

borgmatic/actions/pattern.py

@@ -0,0 +1,298 @@
+import glob
+import itertools
+import logging
+import os
+import pathlib
+
+import borgmatic.borg.pattern
+
+logger = logging.getLogger(__name__)
+
+
+def parse_pattern(pattern_line, default_style=borgmatic.borg.pattern.Pattern_style.NONE):
+    '''
+    Given a Borg pattern as a string, parse it into a borgmatic.borg.pattern.Pattern instance and
+    return it.
+    '''
+    try:
+        (pattern_type, remainder) = pattern_line.split(' ', maxsplit=1)
+    except ValueError:
+        raise ValueError(f'Invalid pattern: {pattern_line}')
+
+    try:
+        (parsed_pattern_style, path) = remainder.split(':', maxsplit=1)
+        pattern_style = borgmatic.borg.pattern.Pattern_style(parsed_pattern_style)
+    except ValueError:
+        pattern_style = default_style
+        path = remainder
+
+    return borgmatic.borg.pattern.Pattern(
+        path,
+        borgmatic.borg.pattern.Pattern_type(pattern_type),
+        borgmatic.borg.pattern.Pattern_style(pattern_style),
+        source=borgmatic.borg.pattern.Pattern_source.CONFIG,
+    )
+
+
+def collect_patterns(config):
+    '''
+    Given a configuration dict, produce a single sequence of patterns comprised of the configured
+    source directories, patterns, excludes, pattern files, and exclude files.
+
+    The idea is that Borg has all these different ways of specifying includes, excludes, source
+    directories, etc., but we'd like to collapse them all down to one common format (patterns) for
+    ease of manipulation within borgmatic.
+    '''
+    try:
+        return (
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    source_directory,
+                    source=borgmatic.borg.pattern.Pattern_source.CONFIG,
+                )
+                for source_directory in config.get('source_directories', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for pattern_line in config.get('patterns', ())
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for exclude_line in config.get('exclude_patterns', ())
+            )
+            + tuple(
+                parse_pattern(pattern_line.strip())
+                for filename in config.get('patterns_from', ())
+                for pattern_line in open(filename, encoding='utf-8').readlines()
+                if not pattern_line.lstrip().startswith('#')
+                if pattern_line.strip()
+            )
+            + tuple(
+                parse_pattern(
+                    f'{borgmatic.borg.pattern.Pattern_type.NO_RECURSE.value} {exclude_line.strip()}',
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                )
+                for filename in config.get('exclude_from', ())
+                for exclude_line in open(filename, encoding='utf-8').readlines()
+                if not exclude_line.lstrip().startswith('#')
+                if exclude_line.strip()
+            )
+        )
+    except (FileNotFoundError, OSError) as error:
+        logger.debug(error)
+
+        raise ValueError(f'Cannot read patterns_from/exclude_from file: {error.filename}')
+
+
+def expand_directory(directory, working_directory):
+    '''
+    Given a directory path, expand any tilde (representing a user's home directory) and any globs
+    therein. Return a list of one or more resulting paths.
+
+    Take into account the given working directory so that relative paths are supported.
+    '''
+    expanded_directory = os.path.expanduser(directory)
+
+    # This would be a lot easier to do with glob(..., root_dir=working_directory), but root_dir is
+    # only available in Python 3.10+.
+    normalized_directory = os.path.join(working_directory or '', expanded_directory)
+    glob_paths = glob.glob(normalized_directory)
+
+    if not glob_paths:
+        return [expanded_directory]
+
+    working_directory_prefix = os.path.join(working_directory or '', '')
+
+    return [
+        (
+            glob_path
+            # If these are equal, that means we didn't add any working directory prefix above.
+            if normalized_directory == expanded_directory
+            # Remove the working directory prefix added above in order to make glob() work. We
+            # can't use os.path.relpath() here because it collapses any use of Borg's slashdot hack.
+            else glob_path.removeprefix(working_directory_prefix)
+        )
+        for glob_path in glob_paths
+    ]
+
+
+def expand_patterns(patterns, working_directory=None, skip_paths=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    expand tildes and globs in each root pattern and expand just tildes in each non-root pattern.
+    The idea is that non-root patterns may be regular expressions or other pattern styles containing
+    "*" that borgmatic should not expand as a shell glob.
+
+    Return all the resulting patterns as a tuple.
+
+    If a set of paths are given to skip, then don't expand any patterns matching them.
+    '''
+    if patterns is None:
+        return ()
+
+    return tuple(
+        itertools.chain.from_iterable(
+            (
+                (
+                    borgmatic.borg.pattern.Pattern(
+                        expanded_path,
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    )
+                    for expanded_path in expand_directory(pattern.path, working_directory)
+                )
+                if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.path not in (skip_paths or ())
+                else (
+                    borgmatic.borg.pattern.Pattern(
+                        os.path.expanduser(pattern.path),
+                        pattern.type,
+                        pattern.style,
+                        pattern.device,
+                        pattern.source,
+                    ),
+                )
+            )
+            for pattern in patterns
+        ),
+    )
+
+
+def get_existent_path_or_parent(path):
+    '''
+    Given a path, return it if it exists. Otherwise, return the longest parent directory of the path
+    that exists. Return None if none of these paths exist.
+
+    This is used below for finding an existent path prefix of pattern's path, which is necessary if
+    the path contain globs or other special characters that we don't want to try to interpret
+    (because we want to leave that responsibility to Borg).
+    '''
+    if path.startswith('/e2e/'):
+        return None
+
+    try:
+        return next(
+            candidate_path
+            for candidate_path in (
+                path,
+                *tuple(str(parent) for parent in pathlib.PurePath(path).parents),
+            )
+            if os.path.exists(candidate_path)
+        )
+    except StopIteration:
+        return None
+
+
+def device_map_patterns(patterns, working_directory=None):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and an optional working directory,
+    determine the identifier for the device on which the pattern's path resides—or None if the path
+    doesn't exist or is from a non-root pattern. Return an updated sequence of patterns with the
+    device field populated. But if the device field is already set, don't bother setting it again.
+
+    This is handy for determining whether two different pattern paths are on the same filesystem
+    (have the same device identifier).
+
+    This function only considers the start of a pattern's path—from the start of the path up until
+    there's a path component with a glob or other non-literal character. If there are no such
+    characters, the whole path is considered. The rationale is that it's not feasible for borgmatic
+    to interpret Borg's patterns to see which actual files (and therefore devices) they map to. So
+    for instance, a pattern with a path of "/var/log/*/data" would end up with its device set to the
+    device of "/var/log"—ignoring the "/*/data" part due to that glob.
+
+    The one exception is that if a regular expression pattern path starts with "^", that will get
+    stripped off for purposes of determining its device.
+    '''
+    return tuple(
+        borgmatic.borg.pattern.Pattern(
+            pattern.path,
+            pattern.type,
+            pattern.style,
+            device=pattern.device or (os.stat(existent_path).st_dev if existent_path else None),
+            source=pattern.source,
+        )
+        for pattern in patterns
+        for existent_path in (
+            get_existent_path_or_parent(
+                os.path.join(working_directory or '', pattern.path.lstrip('^')),
+            ),
+        )
+    )
+
+
+def deduplicate_patterns(patterns, config):
+    '''
+    Given a sequence of borgmatic.borg.pattern.Pattern instances and a configuration dict, return
+    them with all duplicate root child patterns removed. For instance, if two root patterns are
+    given with paths "/foo" and "/foo/bar", return just the one with "/foo". Non-root patterns are
+    passed through without modification.
+
+    The one exception to deduplication is if two paths are on different filesystems (devices) and
+    "one_file_system" is True in the given configuration. In that case, the paths won't get
+    deduplicated, because Borg won't cross filesystem boundaries when "one_file_system" is True.
+
+    The idea is that if Borg is given a root parent pattern, then it doesn't also need to be given
+    child patterns, because it will naturally spider the contents of the parent pattern's path. And
+    there are cases where Borg coming across the same file twice will result in duplicate reads and
+    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
+    Borg.
+    '''
+    deduplicated = {}  # Use just the keys as an ordered set.
+
+    for pattern in patterns:
+        if pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+            deduplicated[pattern] = True
+            continue
+
+        parents = pathlib.PurePath(pattern.path).parents
+
+        # If another directory in the given list is a parent of current directory (even n levels up)
+        # and both are on the same filesystem (or one_file_system is not set), then the current
+        # directory is a duplicate.
+        for other_pattern in patterns:
+            if other_pattern.type != borgmatic.borg.pattern.Pattern_type.ROOT:
+                continue
+
+            if any(
+                pathlib.PurePath(other_pattern.path) == parent
+                and pattern.device is not None
+                and (
+                    other_pattern.device == pattern.device
+                    or config.get('one_file_system') is not True
+                )
+                for parent in parents
+            ):
+                break
+        else:
+            deduplicated[pattern] = True
+
+    return tuple(deduplicated.keys())
+
+
+def process_patterns(patterns, config, working_directory, skip_expand_paths=None):
+    '''
+    Given a sequence of Borg patterns, a configuration dict, a configured working directory, and a
+    sequence of paths to skip path expansion for, expand and deduplicate any "root" patterns,
+    returning the resulting root and non-root patterns as a list.
+    '''
+    skip_paths = set(skip_expand_paths or ())
+
+    return list(
+        deduplicate_patterns(
+            device_map_patterns(
+                expand_patterns(
+                    patterns,
+                    working_directory=working_directory,
+                    skip_paths=skip_paths,
+                ),
+            ),
+            config,
+        ),
+    )

borgmatic/actions/prune.py

@@ -11,7 +11,6 @@ def run_prune(
     config_filename,
     repository,
     config,
-    hook_context,
     local_borg_version,
     prune_arguments,
     global_arguments,
@@ -23,19 +22,12 @@ def run_prune(
     Run the "prune" action for the given repository.
     '''
     if prune_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, prune_arguments.repository
+        repository,
+        prune_arguments.repository,
     ):
         return
 
-    borgmatic.hooks.command.execute_hook(
-        config.get('before_prune'),
-        config.get('umask'),
-        config_filename,
-        'pre-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )
-    logger.info(f'{repository.get("label", repository["path"])}: Pruning archives{dry_run_label}')
+    logger.info(f'Pruning archives{dry_run_label}')
     borgmatic.borg.prune.prune_archives(
         global_arguments.dry_run,
         repository['path'],
@@ -46,11 +38,3 @@ def run_prune(
         local_path=local_path,
         remote_path=remote_path,
     )
-    borgmatic.hooks.command.execute_hook(
-        config.get('after_prune'),
-        config.get('umask'),
-        config_filename,
-        'post-prune',
-        global_arguments.dry_run,
-        **hook_context,
-    )

borgmatic/actions/rcreate.py

@@ -1,41 +0,0 @@
-import logging
-
-import borgmatic.borg.rcreate
-import borgmatic.config.validate
-
-logger = logging.getLogger(__name__)
-
-
-def run_rcreate(
-    repository,
-    config,
-    local_borg_version,
-    rcreate_arguments,
-    global_arguments,
-    local_path,
-    remote_path,
-):
-    '''
-    Run the "rcreate" action for the given repository.
-    '''
-    if rcreate_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, rcreate_arguments.repository
-    ):
-        return
-
-    logger.info(f'{repository.get("label", repository["path"])}: Creating repository')
-    borgmatic.borg.rcreate.create_repository(
-        global_arguments.dry_run,
-        repository['path'],
-        config,
-        local_borg_version,
-        global_arguments,
-        rcreate_arguments.encryption_mode,
-        rcreate_arguments.source_repository,
-        rcreate_arguments.copy_crypt_key,
-        rcreate_arguments.append_only,
-        rcreate_arguments.storage_quota,
-        rcreate_arguments.make_parent_dirs,
-        local_path=local_path,
-        remote_path=remote_path,
-    )

borgmatic/actions/rdelete.py

@@ -1,36 +0,0 @@
-import logging
-
-import borgmatic.borg.rdelete
-
-logger = logging.getLogger(__name__)
-
-
-def run_rdelete(
-    repository,
-    config,
-    local_borg_version,
-    rdelete_arguments,
-    global_arguments,
-    local_path,
-    remote_path,
-):
-    '''
-    Run the "rdelete" action for the given repository.
-    '''
-    if rdelete_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, rdelete_arguments.repository
-    ):
-        logger.answer(
-            f'{repository.get("label", repository["path"])}: Deleting repository'
-            + (' cache' if rdelete_arguments.cache_only else '')
-        )
-
-        borgmatic.borg.rdelete.delete_repository(
-            repository,
-            config,
-            local_borg_version,
-            rdelete_arguments,
-            global_arguments,
-            local_path,
-            remote_path,
-        )

borgmatic/actions/recreate.py

@@ -0,0 +1,88 @@
+import logging
+import subprocess
+
+import borgmatic.borg.info
+import borgmatic.borg.recreate
+import borgmatic.borg.repo_list
+import borgmatic.config.validate
+from borgmatic.actions.pattern import collect_patterns, process_patterns
+
+logger = logging.getLogger(__name__)
+
+
+BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS = 30
+
+
+def run_recreate(
+    repository,
+    config,
+    local_borg_version,
+    recreate_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "recreate" action for the given repository.
+    '''
+    if recreate_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository,
+        recreate_arguments.repository,
+    ):
+        if recreate_arguments.archive:
+            logger.answer(f'Recreating archive {recreate_arguments.archive}')
+        else:
+            logger.answer('Recreating repository')
+
+        # Collect and process patterns.
+        processed_patterns = process_patterns(
+            collect_patterns(config),
+            config,
+            borgmatic.config.paths.get_working_directory(config),
+        )
+
+        archive = borgmatic.borg.repo_list.resolve_archive_name(
+            repository['path'],
+            recreate_arguments.archive,
+            config,
+            local_borg_version,
+            global_arguments,
+            local_path,
+            remote_path,
+        )
+
+        if archive and archive.endswith('.recreate'):
+            if recreate_arguments.archive == 'latest':
+                raise ValueError(
+                    f'The latest archive "{archive}" is leftover from a prior recreate. Delete it first or select a different archive.',
+                )
+
+            raise ValueError(
+                f'The archive "{recreate_arguments.archive}" is leftover from a prior recreate. Select a different archive.',
+            )
+
+        try:
+            borgmatic.borg.recreate.recreate_archive(
+                repository['path'],
+                archive,
+                config,
+                local_borg_version,
+                recreate_arguments,
+                global_arguments,
+                local_path=local_path,
+                remote_path=remote_path,
+                patterns=processed_patterns,
+            )
+        except subprocess.CalledProcessError as error:
+            if error.returncode == BORG_EXIT_CODE_ARCHIVE_ALREADY_EXISTS:
+                if recreate_arguments.target:
+                    raise ValueError(
+                        f'The archive "{recreate_arguments.target}" already exists. Delete it first or set a different target archive name.',
+                    )
+
+                if archive:
+                    raise ValueError(
+                        f'The archive "{archive}.recreate" is leftover from a prior recreate. Delete it first or select a different archive.',
+                    )
+
+            raise

borgmatic/actions/repo_create.py

@@ -0,0 +1,62 @@
+import logging
+
+import borgmatic.borg.repo_create
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_repo_create(
+    repository,
+    config,
+    local_borg_version,
+    repo_create_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "repo-create" action for the given repository.
+    '''
+    if repo_create_arguments.repository and not borgmatic.config.validate.repositories_match(
+        repository,
+        repo_create_arguments.repository,
+    ):
+        return
+
+    logger.info('Creating repository')
+
+    encryption_mode = repo_create_arguments.encryption_mode or repository.get('encryption')
+
+    if not encryption_mode:
+        raise ValueError(
+            'With the repo-create action, either the --encryption flag or the repository encryption option is required.',
+        )
+
+    borgmatic.borg.repo_create.create_repository(
+        global_arguments.dry_run,
+        repository['path'],
+        config,
+        local_borg_version,
+        global_arguments,
+        encryption_mode,
+        repo_create_arguments.source_repository,
+        repo_create_arguments.copy_crypt_key,
+        (
+            repository.get('append_only')
+            if repo_create_arguments.append_only is None
+            else repo_create_arguments.append_only
+        ),
+        (
+            repository.get('storage_quota')
+            if repo_create_arguments.storage_quota is None
+            else repo_create_arguments.storage_quota
+        ),
+        (
+            repository.get('make_parent_directories')
+            if repo_create_arguments.make_parent_directories is None
+            else repo_create_arguments.make_parent_directories
+        ),
+        local_path=local_path,
+        remote_path=remote_path,
+    )

borgmatic/actions/repo_delete.py

@@ -0,0 +1,36 @@
+import logging
+
+import borgmatic.borg.repo_delete
+
+logger = logging.getLogger(__name__)
+
+
+def run_repo_delete(
+    repository,
+    config,
+    local_borg_version,
+    repo_delete_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "repo-delete" action for the given repository.
+    '''
+    if repo_delete_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository,
+        repo_delete_arguments.repository,
+    ):
+        logger.answer(
+            'Deleting repository' + (' cache' if repo_delete_arguments.cache_only else ''),
+        )
+
+        borgmatic.borg.repo_delete.delete_repository(
+            repository,
+            config,
+            local_borg_version,
+            repo_delete_arguments,
+            global_arguments,
+            local_path,
+            remote_path,
+        )

borgmatic/actions/repo_info.py

@@ -0,0 +1,41 @@
+import logging
+
+import borgmatic.actions.json
+import borgmatic.borg.repo_info
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_repo_info(
+    repository,
+    config,
+    local_borg_version,
+    repo_info_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "repo-info" action for the given repository.
+
+    If repo_info_arguments.json is True, yield the JSON output from the info for the repository.
+    '''
+    if repo_info_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository,
+        repo_info_arguments.repository,
+    ):
+        if not repo_info_arguments.json:
+            logger.answer('Displaying repository summary information')
+
+        json_output = borgmatic.borg.repo_info.display_repository_info(
+            repository['path'],
+            config,
+            local_borg_version,
+            repo_info_arguments=repo_info_arguments,
+            global_arguments=global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )
+        if json_output:
+            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/repo_list.py

@@ -0,0 +1,41 @@
+import logging
+
+import borgmatic.actions.json
+import borgmatic.borg.repo_list
+import borgmatic.config.validate
+
+logger = logging.getLogger(__name__)
+
+
+def run_repo_list(
+    repository,
+    config,
+    local_borg_version,
+    repo_list_arguments,
+    global_arguments,
+    local_path,
+    remote_path,
+):
+    '''
+    Run the "repo-list" action for the given repository.
+
+    If repo_list_arguments.json is True, yield the JSON output from listing the repository.
+    '''
+    if repo_list_arguments.repository is None or borgmatic.config.validate.repositories_match(
+        repository,
+        repo_list_arguments.repository,
+    ):
+        if not repo_list_arguments.json:
+            logger.answer('Listing repository')
+
+        json_output = borgmatic.borg.repo_list.list_repository(
+            repository['path'],
+            config,
+            local_borg_version,
+            repo_list_arguments=repo_list_arguments,
+            global_arguments=global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+        )
+        if json_output:
+            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/restore.py

@@ -1,67 +1,170 @@
-import copy
+import collections
 import logging
 import os
+import pathlib
+import shutil
+import tempfile
 
 import borgmatic.borg.extract
 import borgmatic.borg.list
 import borgmatic.borg.mount
-import borgmatic.borg.rlist
-import borgmatic.borg.state
+import borgmatic.borg.repo_list
+import borgmatic.config.paths
 import borgmatic.config.validate
+import borgmatic.hooks.data_source.dump
 import borgmatic.hooks.dispatch
-import borgmatic.hooks.dump
 
 logger = logging.getLogger(__name__)
 
 
-UNSPECIFIED_HOOK = object()
+UNSPECIFIED = object()
 
 
-def get_configured_data_source(
-    config,
-    archive_data_source_names,
-    hook_name,
-    data_source_name,
-    configuration_data_source_name=None,
-):
+Dump = collections.namedtuple(
+    'Dump',
+    ('hook_name', 'data_source_name', 'hostname', 'port', 'label', 'container'),
+    defaults=(None, None, None, None),
+)
+
+
+def dumps_match(first, second, default_port=None):
     '''
-    Find the first data source with the given hook name and data source name in the configuration
-    dict and the given archive data source names dict (from hook name to data source names contained
-    in a particular backup archive). If UNSPECIFIED_HOOK is given as the hook name, search all data
-    source hooks for the named data source. If a configuration data source name is given, use that
-    instead of the data source name to lookup the data source in the given hooks configuration.
-
-    Return the found data source as a tuple of (found hook name, data source configuration dict) or
-    (None, None) if not found.
+    Compare two Dump instances for equality while supporting a field value of UNSPECIFIED, which
+    indicates that the field should match any value. If a default port is given, then consider any
+    dump having that port to match with a dump having a None port.
     '''
-    if not configuration_data_source_name:
-        configuration_data_source_name = data_source_name
-
-    if hook_name == UNSPECIFIED_HOOK:
-        hooks_to_search = {
-            hook_name: value
-            for (hook_name, value) in config.items()
-            if hook_name in borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES
-        }
+    # label kinda counts as an unique id, if they match ignore host/container/port
+    if first.label not in {None, UNSPECIFIED} and first.label == second.label:
+        field_list = ('hook_name', 'data_source_name')
     else:
-        try:
-            hooks_to_search = {hook_name: config[hook_name]}
-        except KeyError:
-            return (None, None)
+        field_list = Dump._fields
 
-    return next(
-        (
-            (name, hook_data_source)
-            for (name, hook) in hooks_to_search.items()
-            for hook_data_source in hook
-            if hook_data_source['name'] == configuration_data_source_name
-            and data_source_name in archive_data_source_names.get(name, [])
-        ),
-        (None, None),
+    for field_name in field_list:
+        first_value = getattr(first, field_name)
+        second_value = getattr(second, field_name)
+
+        if default_port is not None and field_name == 'port':
+            if first_value == default_port and second_value is None:
+                continue
+
+            if second_value == default_port and first_value is None:
+                continue
+
+        if first_value == UNSPECIFIED or second_value == UNSPECIFIED:  # noqa: PLR1714
+            continue
+
+        if first_value != second_value:
+            return False
+
+    return True
+
+
+def render_dump_metadata(dump):
+    '''
+    Given a Dump instance, make a display string describing it for use in log messages.
+    '''
+    label = dump.label or UNSPECIFIED
+    name = 'unspecified' if dump.data_source_name is UNSPECIFIED else dump.data_source_name
+    host = dump.container or dump.hostname or UNSPECIFIED
+    port = None if dump.port is UNSPECIFIED else dump.port
+
+    if label is not UNSPECIFIED:
+        metadata = f'{name}@{label}'
+    elif port:
+        metadata = f'{name}@:{port}' if host is UNSPECIFIED else f'{name}@{host}:{port}'
+    else:
+        metadata = f'{name}' if host is UNSPECIFIED else f'{name}@{host}'
+
+    if dump.hook_name not in {None, UNSPECIFIED}:
+        return f'{metadata} ({dump.hook_name})'
+
+    return metadata
+
+
+def get_configured_data_source(config, restore_dump):
+    '''
+    Search in the given configuration dict for dumps corresponding to the given dump to restore. If
+    there are multiple matches, error.
+
+    Return the found data source as a data source configuration dict or None if not found.
+    '''
+    try:
+        hooks_to_search = {restore_dump.hook_name: config[restore_dump.hook_name]}
+    except KeyError:
+        return None
+
+    matching_dumps = tuple(
+        hook_data_source
+        for (hook_name, hook_config) in hooks_to_search.items()
+        for hook_data_source in hook_config
+        for default_port in (
+            borgmatic.hooks.dispatch.call_hook(
+                function_name='get_default_port',
+                config=config,
+                hook_name=hook_name,
+            ),
+        )
+        if dumps_match(
+            Dump(
+                hook_name,
+                hook_data_source.get('name'),
+                hook_data_source.get('hostname'),
+                hook_data_source.get('port'),
+                hook_data_source.get('label') or UNSPECIFIED,
+                hook_data_source.get('container'),
+            ),
+            restore_dump,
+            default_port,
+        )
     )
 
+    if not matching_dumps:
+        return None
 
-def restore_single_data_source(
+    if len(matching_dumps) > 1:
+        raise ValueError(
+            f'Cannot restore data source {render_dump_metadata(restore_dump)} because there are multiple matching data sources configured',
+        )
+
+    return matching_dumps[0]
+
+
+def strip_path_prefix_from_extracted_dump_destination(
+    destination_path,
+    borgmatic_runtime_directory,
+):
+    '''
+    Directory-format dump files get extracted into a temporary directory containing a path prefix
+    that depends how the files were stored in the archive. So, given the destination path where the
+    dump was extracted and the borgmatic runtime directory, move the dump files such that the
+    restore doesn't have to deal with that varying path prefix.
+
+    For instance, if the dump was extracted to:
+
+      /run/user/0/borgmatic/tmp1234/borgmatic/postgresql_databases/test/...
+
+    or:
+
+      /run/user/0/borgmatic/tmp1234/root/.borgmatic/postgresql_databases/test/...
+
+    then this function moves it to:
+
+      /run/user/0/borgmatic/postgresql_databases/test/...
+    '''
+    for subdirectory_path, _, _ in os.walk(destination_path):
+        databases_directory = os.path.basename(subdirectory_path)
+
+        if not databases_directory.endswith('_databases'):
+            continue
+
+        shutil.move(
+            subdirectory_path,
+            os.path.join(borgmatic_runtime_directory, databases_directory),
+        )
+        break
+
+
+def restore_single_dump(
     repository,
     config,
     local_borg_version,
@@ -72,55 +175,86 @@ def restore_single_data_source(
     hook_name,
     data_source,
     connection_params,
-):  # pragma: no cover
+    borgmatic_runtime_directory,
+):
     '''
     Given (among other things) an archive name, a data source hook name, the hostname, port,
     username/password as connection params, and a configured data source configuration dict, restore
     that data source from the archive.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Restoring data source {data_source["name"]}'
+    dump_metadata = render_dump_metadata(
+        Dump(
+            hook_name,
+            data_source['name'],
+            data_source.get('hostname'),
+            data_source.get('port'),
+            data_source.get('label') or UNSPECIFIED,
+            data_source.get('container'),
+        ),
     )
 
-    dump_pattern = borgmatic.hooks.dispatch.call_hooks(
-        'make_data_source_dump_pattern',
+    logger.info(f'Restoring data source {dump_metadata}')
+
+    dump_patterns = borgmatic.hooks.dispatch.call_hooks(
+        'make_data_source_dump_patterns',
         config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
+        borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+        borgmatic_runtime_directory,
         data_source['name'],
-    )[hook_name]
+    )[hook_name.split('_databases', 1)[0]]
 
-    # Kick off a single data source extract to stdout.
-    extract_process = borgmatic.borg.extract.extract_archive(
-        dry_run=global_arguments.dry_run,
-        repository=repository['path'],
-        archive=archive_name,
-        paths=borgmatic.hooks.dump.convert_glob_patterns_to_borg_patterns([dump_pattern]),
-        config=config,
-        local_borg_version=local_borg_version,
-        global_arguments=global_arguments,
-        local_path=local_path,
-        remote_path=remote_path,
-        destination_path='/',
-        # A directory format dump isn't a single file, and therefore can't extract
-        # to stdout. In this case, the extract_process return value is None.
-        extract_to_stdout=bool(data_source.get('format') != 'directory'),
+    destination_path = (
+        tempfile.mkdtemp(dir=borgmatic_runtime_directory)
+        if data_source.get('format') == 'directory'
+        else None
     )
 
+    try:
+        # Kick off a single data source extract. If using a directory format, extract to a temporary
+        # directory. Otherwise extract the single dump file to stdout.
+        extract_process = borgmatic.borg.extract.extract_archive(
+            dry_run=global_arguments.dry_run,
+            repository=repository['path'],
+            archive=archive_name,
+            paths=[
+                borgmatic.hooks.data_source.dump.convert_glob_patterns_to_borg_pattern(
+                    dump_patterns,
+                ),
+            ],
+            config=config,
+            local_borg_version=local_borg_version,
+            global_arguments=global_arguments,
+            local_path=local_path,
+            remote_path=remote_path,
+            destination_path=destination_path,
+            # A directory format dump isn't a single file, and therefore can't extract
+            # to stdout. In this case, the extract_process return value is None.
+            extract_to_stdout=bool(data_source.get('format') != 'directory'),
+        )
+
+        if destination_path and not global_arguments.dry_run:
+            strip_path_prefix_from_extracted_dump_destination(
+                destination_path,
+                borgmatic_runtime_directory,
+            )
+    finally:
+        if destination_path and not global_arguments.dry_run:
+            shutil.rmtree(destination_path, ignore_errors=True)
+
     # Run a single data source restore, consuming the extract stdout (if any).
-    borgmatic.hooks.dispatch.call_hooks(
+    borgmatic.hooks.dispatch.call_hook(
         function_name='restore_data_source_dump',
         config=config,
-        log_prefix=repository['path'],
-        hook_names=[hook_name],
+        hook_name=hook_name,
         data_source=data_source,
         dry_run=global_arguments.dry_run,
         extract_process=extract_process,
         connection_params=connection_params,
+        borgmatic_runtime_directory=borgmatic_runtime_directory,
     )
 
 
-def collect_archive_data_source_names(
+def collect_dumps_from_archive(
     repository,
     archive,
     config,
@@ -128,18 +262,81 @@ def collect_archive_data_source_names(
     global_arguments,
     local_path,
     remote_path,
+    borgmatic_runtime_directory,
 ):
     '''
     Given a local or remote repository path, a resolved archive name, a configuration dict, the
-    local Borg version, global_arguments an argparse.Namespace, and local and remote Borg paths,
-    query the archive for the names of data sources it contains as dumps and return them as a dict
-    from hook name to a sequence of data source names.
+    local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
+    borgmatic runtime directory, query the archive for the names of data sources dumps it contains
+    and return them as a set of Dump instances.
     '''
-    borgmatic_source_directory = os.path.expanduser(
-        config.get(
-            'borgmatic_source_directory', borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
+    dumps_from_archive = set()
+
+    # There is (at most) one dump metadata file per data source hook. Load each.
+    for dumps_metadata_path in borgmatic.borg.list.capture_archive_listing(
+        repository,
+        archive,
+        config,
+        local_borg_version,
+        global_arguments,
+        list_paths=[
+            'sh:'
+            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
+                base_directory,
+                '*_databases/dumps.json',
+            )
+            # Probe for dump metadata files in multiple locations, as the default location is
+            # "/borgmatic/*_databases/dumps.json" with Borg 1.4+, but instead begins with the
+            # borgmatic runtime directory for older versions of Borg.
+            for base_directory in (
+                'borgmatic',
+                borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
+            )
+        ],
+        local_path=local_path,
+        remote_path=remote_path,
+    ):
+        if not dumps_metadata_path:
+            continue
+
+        dumps_from_archive.update(
+            set(
+                borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
+                    borgmatic.borg.extract.extract_archive(
+                        global_arguments.dry_run,
+                        repository,
+                        archive,
+                        [dumps_metadata_path],
+                        config,
+                        local_borg_version,
+                        global_arguments,
+                        local_path=local_path,
+                        remote_path=remote_path,
+                        extract_to_stdout=True,
+                    )
+                    .stdout.read()
+                    .decode(),
+                    dumps_metadata_path,
+                )
+            )
         )
-    ).lstrip('/')
+
+    # If we've successfully loaded any dumps metadata, we're done.
+    if dumps_from_archive:
+        logger.debug('Collecting database dumps from archive data source dumps metadata files')
+
+        return dumps_from_archive
+
+    # No dumps metadata files were found, so for backwards compatibility, fall back to parsing the
+    # paths of dumps found in the archive to get their respective dump metadata.
+    logger.debug('Collecting database dumps from archive data source dump paths (fallback)')
+    borgmatic_source_directory = str(
+        pathlib.Path(borgmatic.config.paths.get_borgmatic_source_directory(config)),
+    )
+
+    # Probe for the data source dumps in multiple locations, as the default location has moved to
+    # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
+    # we still want to support reading dumps from previously created archives as well.
     dump_paths = borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
@@ -147,108 +344,172 @@ def collect_archive_data_source_names(
         local_borg_version,
         global_arguments,
         list_paths=[
-            os.path.expanduser(
-                borgmatic.hooks.dump.make_data_source_dump_path(borgmatic_source_directory, pattern)
+            'sh:'
+            + borgmatic.hooks.data_source.dump.make_data_source_dump_path(
+                base_directory,
+                '*_databases/*/*',
+            )
+            for base_directory in (
+                'borgmatic',
+                borgmatic.config.paths.make_runtime_directory_glob(borgmatic_runtime_directory),
+                borgmatic_source_directory.lstrip('/'),
             )
-            for pattern in ('*_databases/*/*',)
         ],
         local_path=local_path,
         remote_path=remote_path,
     )
 
-    # Determine the data source names corresponding to the dumps found in the archive and
-    # add them to restore_names.
-    archive_data_source_names = {}
-
     for dump_path in dump_paths:
-        try:
-            (hook_name, _, data_source_name) = dump_path.split(
-                borgmatic_source_directory + os.path.sep, 1
-            )[1].split(os.path.sep)[0:3]
-        except (ValueError, IndexError):
-            logger.warning(
-                f'{repository}: Ignoring invalid data source dump path "{dump_path}" in archive {archive}'
+        if not dump_path:
+            continue
+
+        # Probe to find the base directory that's at the start of the dump path.
+        for base_directory in (
+            'borgmatic',
+            borgmatic_runtime_directory,
+            borgmatic_source_directory,
+        ):
+            try:
+                (hook_name, host_and_port, data_source_name) = dump_path.split(
+                    base_directory + os.path.sep,
+                    1,
+                )[1].split(os.path.sep)[0:3]
+            except (ValueError, IndexError):
+                continue
+
+            parts = host_and_port.split(':', 1)
+
+            if len(parts) == 1:
+                parts += (None,)
+
+            (hostname, port) = parts
+
+            try:
+                port = int(port)
+            except (ValueError, TypeError):
+                port = None
+
+            dumps_from_archive.add(
+                Dump(
+                    hook_name, data_source_name, None if hostname == 'localhost' else hostname, port
+                )
             )
+
+            # We've successfully parsed the dump path, so need to probe any further.
+            break
         else:
-            if data_source_name not in archive_data_source_names.get(hook_name, []):
-                archive_data_source_names.setdefault(hook_name, []).extend([data_source_name])
+            logger.warning(
+                f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}',
+            )
 
-    return archive_data_source_names
+    return dumps_from_archive
 
 
-def find_data_sources_to_restore(requested_data_source_names, archive_data_source_names):
+def get_dumps_to_restore(restore_arguments, dumps_from_archive):
     '''
-    Given a sequence of requested data source names to restore and a dict of hook name to the names
-    of data sources found in an archive, return an expanded sequence of data source names to
-    restore, replacing "all" with actual data source names as appropriate.
+    Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
+    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    Dump instances from the archive to restore. As part of this, replace any Dump having a data
+    source name of "all" with multiple named Dump instances as appropriate.
 
-    Raise ValueError if any of the requested data source names cannot be found in the archive.
+    Raise ValueError if any of the requested data source names cannot be found in the archive or if
+    there are multiple archive dump matches for a given requested dump.
     '''
-    # A map from data source hook name to the data source names to restore for that hook.
-    restore_names = (
-        {UNSPECIFIED_HOOK: requested_data_source_names}
-        if requested_data_source_names
-        else {UNSPECIFIED_HOOK: ['all']}
+    requested_dumps = (
+        {
+            Dump(
+                hook_name=(
+                    (
+                        restore_arguments.hook
+                        if restore_arguments.hook.endswith('_databases')
+                        else f'{restore_arguments.hook}_databases'
+                    )
+                    if restore_arguments.hook
+                    else UNSPECIFIED
+                ),
+                data_source_name=name,
+                hostname=restore_arguments.original_hostname or UNSPECIFIED,
+                port=restore_arguments.original_port,
+                label=restore_arguments.original_label or UNSPECIFIED,
+                container=restore_arguments.original_container or UNSPECIFIED,
+            )
+            for name in restore_arguments.data_sources or (UNSPECIFIED,)
+        }
+        if restore_arguments.hook
+        or restore_arguments.data_sources
+        or restore_arguments.original_hostname
+        or restore_arguments.original_port
+        or restore_arguments.original_label
+        or restore_arguments.original_container
+        else {
+            Dump(
+                hook_name=UNSPECIFIED,
+                data_source_name='all',
+                hostname=UNSPECIFIED,
+                port=UNSPECIFIED,
+                label=UNSPECIFIED,
+                container=UNSPECIFIED,
+            ),
+        }
     )
+    missing_dumps = set()
+    dumps_to_restore = set()
 
-    # If "all" is in restore_names, then replace it with the names of dumps found within the
-    # archive.
-    if 'all' in restore_names[UNSPECIFIED_HOOK]:
-        restore_names[UNSPECIFIED_HOOK].remove('all')
+    # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
+    if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
+        dumps_to_restore.update(dumps_from_archive)
 
-        for hook_name, data_source_names in archive_data_source_names.items():
-            restore_names.setdefault(hook_name, []).extend(data_source_names)
+    # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
+    for requested_dump in requested_dumps:
+        if requested_dump.data_source_name == 'all':
+            continue
 
-            # If a data source is to be restored as part of "all", then remove it from restore names
-            # so it doesn't get restored twice.
-            for data_source_name in data_source_names:
-                if data_source_name in restore_names[UNSPECIFIED_HOOK]:
-                    restore_names[UNSPECIFIED_HOOK].remove(data_source_name)
-
-    if not restore_names[UNSPECIFIED_HOOK]:
-        restore_names.pop(UNSPECIFIED_HOOK)
-
-    combined_restore_names = set(
-        name for data_source_names in restore_names.values() for name in data_source_names
-    )
-    combined_archive_data_source_names = set(
-        name
-        for data_source_names in archive_data_source_names.values()
-        for name in data_source_names
-    )
-
-    missing_names = sorted(set(combined_restore_names) - combined_archive_data_source_names)
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
-        raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from archive"
+        matching_dumps = tuple(
+            archive_dump
+            for archive_dump in dumps_from_archive
+            if dumps_match(requested_dump, archive_dump)
         )
 
-    return restore_names
+        if len(matching_dumps) == 0:
+            missing_dumps.add(requested_dump)
+        elif len(matching_dumps) == 1:
+            dumps_to_restore.add(matching_dumps[0])
+        else:
+            raise ValueError(
+                f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.',
+            )
+
+    if missing_dumps:
+        rendered_dumps = ', '.join(
+            f'{render_dump_metadata(dump)}' for dump in sorted(missing_dumps)
+        )
+
+        raise ValueError(
+            f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive",
+        )
+
+    return dumps_to_restore
 
 
-def ensure_data_sources_found(restore_names, remaining_restore_names, found_names):
+def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
     '''
-    Given a dict from hook name to data source names to restore, a dict from hook name to remaining
-    data source names to restore, and a sequence of found (actually restored) data source names,
-    raise ValueError if requested data source to restore were missing from the archive and/or
+    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    if any requested dumps to restore weren't restored, indicating that they were missing from the
     configuration.
     '''
-    combined_restore_names = set(
-        name
-        for data_source_names in tuple(restore_names.values())
-        + tuple(remaining_restore_names.values())
-        for name in data_source_names
+    if not dumps_actually_restored:
+        raise ValueError('No data source dumps were found to restore')
+
+    missing_dumps = sorted(
+        dumps_to_restore - dumps_actually_restored,
+        key=lambda dump: dump.data_source_name,
     )
 
-    if not combined_restore_names and not found_names:
-        raise ValueError('No data sources were found to restore')
+    if missing_dumps:
+        rendered_dumps = ', '.join(f'{render_dump_metadata(dump)}' for dump in missing_dumps)
 
-    missing_names = sorted(set(combined_restore_names) - set(found_names))
-    if missing_names:
-        joined_names = ', '.join(f'"{name}"' for name in missing_names)
         raise ValueError(
-            f"Cannot restore data source{'s' if len(missing_names) > 1 else ''} {joined_names} missing from borgmatic's configuration"
+            f"Cannot restore data source{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from borgmatic's configuration",
         )
 
 
@@ -265,70 +526,87 @@ def run_restore(
     Run the "restore" action for the given repository, but only if the repository matches the
     requested repository in restore arguments.
 
-    Raise ValueError if a configured data source could not be found to restore.
+    Raise ValueError if a configured data source could not be found to restore or there's no
+    matching dump in the archive.
     '''
     if restore_arguments.repository and not borgmatic.config.validate.repositories_match(
-        repository, restore_arguments.repository
+        repository,
+        restore_arguments.repository,
     ):
         return
 
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Restoring data sources from archive {restore_arguments.archive}'
-    )
+    logger.info(f'Restoring data sources from archive {restore_arguments.archive}')
 
-    borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-        'remove_data_source_dumps',
-        config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        global_arguments.dry_run,
-    )
+    with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            borgmatic_runtime_directory,
+            global_arguments.dry_run,
+        )
 
-    archive_name = borgmatic.borg.rlist.resolve_archive_name(
-        repository['path'],
-        restore_arguments.archive,
-        config,
-        local_borg_version,
-        global_arguments,
-        local_path,
-        remote_path,
-    )
-    archive_data_source_names = collect_archive_data_source_names(
-        repository['path'],
-        archive_name,
-        config,
-        local_borg_version,
-        global_arguments,
-        local_path,
-        remote_path,
-    )
-    restore_names = find_data_sources_to_restore(
-        restore_arguments.data_sources, archive_data_source_names
-    )
-    found_names = set()
-    remaining_restore_names = {}
-    connection_params = {
-        'hostname': restore_arguments.hostname,
-        'port': restore_arguments.port,
-        'username': restore_arguments.username,
-        'password': restore_arguments.password,
-        'restore_path': restore_arguments.restore_path,
-    }
+        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+            repository['path'],
+            restore_arguments.archive,
+            config,
+            local_borg_version,
+            global_arguments,
+            local_path,
+            remote_path,
+        )
+        dumps_from_archive = collect_dumps_from_archive(
+            repository['path'],
+            archive_name,
+            config,
+            local_borg_version,
+            global_arguments,
+            local_path,
+            remote_path,
+            borgmatic_runtime_directory,
+        )
+        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
 
-    for hook_name, data_source_names in restore_names.items():
-        for data_source_name in data_source_names:
-            found_hook_name, found_data_source = get_configured_data_source(
-                config, archive_data_source_names, hook_name, data_source_name
+        dumps_actually_restored = set()
+        connection_params = {
+            'container': restore_arguments.container,
+            'hostname': restore_arguments.hostname,
+            'port': restore_arguments.port,
+            'username': restore_arguments.username,
+            'password': restore_arguments.password,
+            'restore_path': restore_arguments.restore_path,
+        }
+
+        # Restore each dump.
+        for restore_dump in dumps_to_restore:
+            found_data_source = get_configured_data_source(
+                config,
+                restore_dump,
             )
-
+            # For a dump that wasn't found via an exact match in the configuration, try to fallback
+            # to an "all" data source.
             if not found_data_source:
-                remaining_restore_names.setdefault(found_hook_name or hook_name, []).append(
-                    data_source_name
+                found_data_source = get_configured_data_source(
+                    config,
+                    Dump(
+                        restore_dump.hook_name,
+                        'all',
+                        restore_dump.hostname,
+                        restore_dump.port,
+                        restore_dump.label,
+                        restore_dump.container,
+                    ),
                 )
-                continue
 
-            found_names.add(data_source_name)
-            restore_single_data_source(
+                if not found_data_source:
+                    continue
+
+                found_data_source = dict(found_data_source)
+                found_data_source['name'] = restore_dump.data_source_name
+
+            dumps_actually_restored.add(restore_dump)
+
+            restore_single_dump(
                 repository,
                 config,
                 local_borg_version,
@@ -336,45 +614,18 @@ def run_restore(
                 local_path,
                 remote_path,
                 archive_name,
-                found_hook_name or hook_name,
-                dict(found_data_source, **{'schemas': restore_arguments.schemas}),
+                restore_dump.hook_name,
+                dict(found_data_source, schemas=restore_arguments.schemas),
                 connection_params,
+                borgmatic_runtime_directory,
             )
 
-    # For any data sources that weren't found via exact matches in the configuration, try to
-    # fallback to "all" entries.
-    for hook_name, data_source_names in remaining_restore_names.items():
-        for data_source_name in data_source_names:
-            found_hook_name, found_data_source = get_configured_data_source(
-                config, archive_data_source_names, hook_name, data_source_name, 'all'
-            )
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            borgmatic_runtime_directory,
+            global_arguments.dry_run,
+        )
 
-            if not found_data_source:
-                continue
-
-            found_names.add(data_source_name)
-            data_source = copy.copy(found_data_source)
-            data_source['name'] = data_source_name
-
-            restore_single_data_source(
-                repository,
-                config,
-                local_borg_version,
-                global_arguments,
-                local_path,
-                remote_path,
-                archive_name,
-                found_hook_name or hook_name,
-                dict(data_source, **{'schemas': restore_arguments.schemas}),
-                connection_params,
-            )
-
-    borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-        'remove_data_source_dumps',
-        config,
-        repository['path'],
-        borgmatic.hooks.dump.DATA_SOURCE_HOOK_NAMES,
-        global_arguments.dry_run,
-    )
-
-    ensure_data_sources_found(restore_names, remaining_restore_names, found_names)
+    ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)

borgmatic/actions/rinfo.py

@@ -1,42 +0,0 @@
-import logging
-
-import borgmatic.actions.json
-import borgmatic.borg.rinfo
-import borgmatic.config.validate
-
-logger = logging.getLogger(__name__)
-
-
-def run_rinfo(
-    repository,
-    config,
-    local_borg_version,
-    rinfo_arguments,
-    global_arguments,
-    local_path,
-    remote_path,
-):
-    '''
-    Run the "rinfo" action for the given repository.
-
-    If rinfo_arguments.json is True, yield the JSON output from the info for the repository.
-    '''
-    if rinfo_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, rinfo_arguments.repository
-    ):
-        if not rinfo_arguments.json:
-            logger.answer(
-                f'{repository.get("label", repository["path"])}: Displaying repository summary information'
-            )
-
-        json_output = borgmatic.borg.rinfo.display_repository_info(
-            repository['path'],
-            config,
-            local_borg_version,
-            rinfo_arguments=rinfo_arguments,
-            global_arguments=global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/rlist.py

@@ -1,40 +0,0 @@
-import logging
-
-import borgmatic.actions.json
-import borgmatic.borg.rlist
-import borgmatic.config.validate
-
-logger = logging.getLogger(__name__)
-
-
-def run_rlist(
-    repository,
-    config,
-    local_borg_version,
-    rlist_arguments,
-    global_arguments,
-    local_path,
-    remote_path,
-):
-    '''
-    Run the "rlist" action for the given repository.
-
-    If rlist_arguments.json is True, yield the JSON output from listing the repository.
-    '''
-    if rlist_arguments.repository is None or borgmatic.config.validate.repositories_match(
-        repository, rlist_arguments.repository
-    ):
-        if not rlist_arguments.json:
-            logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
-
-        json_output = borgmatic.borg.rlist.list_repository(
-            repository['path'],
-            config,
-            local_borg_version,
-            rlist_arguments=rlist_arguments,
-            global_arguments=global_arguments,
-            local_path=local_path,
-            remote_path=remote_path,
-        )
-        if json_output:
-            yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))

borgmatic/actions/transfer.py

@@ -17,9 +17,13 @@ def run_transfer(
     '''
     Run the "transfer" action for the given repository.
     '''
-    logger.info(
-        f'{repository.get("label", repository["path"])}: Transferring archives to repository'
-    )
+    if transfer_arguments.archive and config.get('match_archives'):
+        raise ValueError(
+            'With the transfer action, only one of --archive and --match-archives/match_archives can be used.',
+        )
+
+    logger.info('Transferring archives to repository')
+
     borgmatic.borg.transfer.transfer_archives(
         global_arguments.dry_run,
         repository['path'],

borgmatic/borg/borg.py

@@ -2,6 +2,7 @@ import logging
 import shlex
 
 import borgmatic.commands.arguments
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, flags
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
@@ -38,9 +39,9 @@ def run_arbitrary_borg(
         borg_command = tuple(options[:command_options_start_index])
         command_options = tuple(options[command_options_start_index:])
 
-        if borg_command and borg_command[0] in borgmatic.commands.arguments.ACTION_ALIASES.keys():
+        if borg_command and borg_command[0] in borgmatic.commands.arguments.ACTION_ALIASES:
             logger.warning(
-                f"Borg's {borg_command[0]} subcommand is supported natively by borgmatic. Try this instead: borgmatic {borg_command[0]}"
+                f"Borg's {borg_command[0]} subcommand is supported natively by borgmatic. Try this instead: borgmatic {borg_command[0]}",
             )
     except IndexError:
         borg_command = ()
@@ -56,17 +57,16 @@ def run_arbitrary_borg(
         + command_options
     )
 
-    return execute_command(
+    return execute_command(  # noqa: S604
         tuple(shlex.quote(part) for part in full_command),
         output_file=DO_NOT_CAPTURE,
         shell=True,
-        extra_environment=dict(
+        environment=dict(
             (environment.make_environment(config) or {}),
-            **{
-                'BORG_REPO': repository_path,
-                'ARCHIVE': archive if archive else '',
-            },
+            BORG_REPO=repository_path,
+            ARCHIVE=archive if archive else '',
         ),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/break_lock.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 from borgmatic.borg import environment, flags
 from borgmatic.execute import execute_command
 
@@ -26,17 +27,17 @@ def break_lock(
         (local_path, 'break-lock')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    borg_environment = environment.make_environment(config)
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/change_passphrase.py

@@ -0,0 +1,67 @@
+import logging
+
+import borgmatic.config.paths
+import borgmatic.execute
+import borgmatic.logger
+from borgmatic.borg import environment, flags
+
+logger = logging.getLogger(__name__)
+
+
+def change_passphrase(
+    repository_path,
+    config,
+    local_borg_version,
+    change_passphrase_arguments,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+):
+    '''
+    Given a local or remote repository path, a configuration dict, the local Borg version, change
+    passphrase arguments, and optional local and remote Borg paths, change the repository passphrase
+    based on an interactive prompt.
+    '''
+    borgmatic.logger.add_custom_log_levels()
+    umask = config.get('umask', None)
+    lock_wait = config.get('lock_wait', None)
+
+    full_command = (
+        (local_path, 'key', 'change-passphrase')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (('--log-json',) if config.get('log_json') else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + flags.make_repository_flags(
+            repository_path,
+            local_borg_version,
+        )
+    )
+
+    if global_arguments.dry_run:
+        logger.info('Skipping change password (dry run)')
+        return
+
+    # If the original passphrase is set programmatically, then Borg won't prompt for a new one! So
+    # don't give Borg any passphrase, and it'll ask the user for both old and new ones.
+    config_without_passphrase = {
+        option_name: value
+        for (option_name, value) in config.items()
+        if option_name not in {'encryption_passphrase', 'encryption_passcommand'}
+    }
+
+    borgmatic.execute.execute_command(
+        full_command,
+        output_file=borgmatic.execute.DO_NOT_CAPTURE,
+        output_log_level=logging.ANSWER,
+        environment=environment.make_environment(config_without_passphrase),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )
+
+    logger.answer(
+        f"{repository_path}: Don't forget to update your encryption_passphrase option (if needed)",
+    )

borgmatic/borg/check.py

@@ -1,8 +1,10 @@
 import argparse
 import json
 import logging
+import shlex
 
-from borgmatic.borg import environment, feature, flags, rinfo
+import borgmatic.config.paths
+from borgmatic.borg import environment, feature, flags, repo_info
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
 
 logger = logging.getLogger(__name__)
@@ -31,7 +33,7 @@ def make_archive_filter_flags(local_borg_version, config, checks, check_argument
             if prefix
             else (
                 flags.make_match_archives_flags(
-                    check_arguments.match_archives or config.get('match_archives'),
+                    config.get('match_archives'),
                     config.get('archive_name_format'),
                     local_borg_version,
                 )
@@ -40,11 +42,12 @@ def make_archive_filter_flags(local_borg_version, config, checks, check_argument
 
     if check_last:
         logger.warning(
-            'Ignoring check_last option, as "archives" or "data" are not in consistency checks'
+            'Ignoring check_last option, as "archives" or "data" are not in consistency checks',
         )
+
     if prefix:
         logger.warning(
-            'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks'
+            'Ignoring consistency prefix option, as "archives" or "data" are not in consistency checks',
         )
 
     return ()
@@ -63,28 +66,29 @@ def make_check_name_flags(checks, archive_filter_flags):
 
         ('--repository-only',)
 
-    However, if both "repository" and "archives" are in checks, then omit them from the returned
-    flags because Borg does both checks by default. If "data" is in checks, that implies "archives".
+    However, if both "repository" and "archives" are in checks, then omit the "only" flags from the
+    returned flags because Borg does both checks by default. Note that a "data" check only works
+    along with an "archives" check.
     '''
-    if 'data' in checks:
-        data_flags = ('--verify-data',)
-        checks.update({'archives'})
-    else:
-        data_flags = ()
-
+    data_flags = ('--verify-data',) if 'data' in checks else ()
     common_flags = (archive_filter_flags if 'archives' in checks else ()) + data_flags
 
     if {'repository', 'archives'}.issubset(checks):
         return common_flags
 
     return (
-        tuple(f'--{check}-only' for check in checks if check in ('repository', 'archives'))
+        tuple(f'--{check}-only' for check in checks if check in {'repository', 'archives'})
         + common_flags
     )
 
 
 def get_repository_id(
-    repository_path, config, local_borg_version, global_arguments, local_path, remote_path
+    repository_path,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path,
+    remote_path,
 ):
     '''
     Given a local or remote repository path, a configuration dict, the local Borg version, global
@@ -94,7 +98,7 @@ def get_repository_id(
     '''
     try:
         return json.loads(
-            rinfo.display_repository_info(
+            repo_info.display_repository_info(
                 repository_path,
                 config,
                 local_borg_version,
@@ -102,7 +106,7 @@ def get_repository_id(
                 global_arguments,
                 local_path,
                 remote_path,
-            )
+            ),
         )['repository']['id']
     except (json.JSONDecodeError, KeyError):
         raise ValueError(f'Cannot determine Borg repository ID for {repository_path}')
@@ -129,8 +133,10 @@ def check_archives(
     extra_borg_options = config.get('extra_borg_options', {}).get('check', '')
 
     verbosity_flags = ()
+
     if logger.isEnabledFor(logging.INFO):
         verbosity_flags = ('--info',)
+
     if logger.isEnabledFor(logging.DEBUG):
         verbosity_flags = ('--debug', '--show-rc')
 
@@ -141,46 +147,52 @@ def check_archives(
     except StopIteration:
         repository_check_config = {}
 
-    if check_arguments.max_duration and 'archives' in checks:
-        raise ValueError('The archives check cannot run when the --max-duration flag is used')
-    if repository_check_config.get('max_duration') and 'archives' in checks:
-        raise ValueError(
-            'The archives check cannot run when the repository check has the max_duration option set'
-        )
-
     max_duration = check_arguments.max_duration or repository_check_config.get('max_duration')
 
-    borg_environment = environment.make_environment(config)
+    umask = config.get('umask')
     borg_exit_codes = config.get('borg_exit_codes')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    full_command = (
-        (local_path, 'check')
-        + (('--repair',) if check_arguments.repair else ())
-        + (('--max-duration', str(max_duration)) if max_duration else ())
-        + make_check_name_flags(checks, archive_filter_flags)
-        + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if global_arguments.log_json else ())
-        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + verbosity_flags
-        + (('--progress',) if check_arguments.progress else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
-        + flags.make_repository_flags(repository_path, local_borg_version)
-    )
+    if 'data' in checks:
+        checks.add('archives')
+
+    grouped_checks = (checks,)
+
+    # If max_duration is set, then archives and repository checks need to be run separately, as Borg
+    # doesn't support --max-duration along with an archives checks.
+    if max_duration and 'archives' in checks and 'repository' in checks:
+        checks.remove('repository')
+        grouped_checks = (checks, {'repository'})
+
+    for checks_subset in grouped_checks:
+        full_command = (
+            (local_path, 'check')
+            + (('--repair',) if check_arguments.repair else ())
+            + (
+                ('--max-duration', str(max_duration))
+                if max_duration and 'repository' in checks_subset
+                else ()
+            )
+            + make_check_name_flags(checks_subset, archive_filter_flags)
+            + (('--remote-path', remote_path) if remote_path else ())
+            + (('--umask', str(umask)) if umask else ())
+            + (('--log-json',) if config.get('log_json') else ())
+            + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+            + verbosity_flags
+            + (('--progress',) if config.get('progress') else ())
+            + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
+            + flags.make_repository_flags(repository_path, local_borg_version)
+        )
 
-    # The Borg repair option triggers an interactive prompt, which won't work when output is
-    # captured. And progress messes with the terminal directly.
-    if check_arguments.repair or check_arguments.progress:
         execute_command(
             full_command,
-            output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-    else:
-        execute_command(
-            full_command,
-            extra_environment=borg_environment,
+            # The Borg repair option triggers an interactive prompt, which won't work when output is
+            # captured. And progress messes with the terminal directly.
+            output_file=(
+                DO_NOT_CAPTURE if check_arguments.repair or config.get('progress') else None
+            ),
+            environment=environment.make_environment(config),
+            working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )

borgmatic/borg/compact.py

@@ -1,6 +1,8 @@
 import logging
+import shlex
 
-from borgmatic.borg import environment, flags
+import borgmatic.config.paths
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
@@ -14,9 +16,7 @@ def compact_segments(
     global_arguments,
     local_path='borg',
     remote_path=None,
-    progress=False,
     cleanup_commits=False,
-    threshold=None,
 ):
     '''
     Given dry-run flag, a local or remote repository path, a configuration dict, and the local Borg
@@ -25,30 +25,37 @@ def compact_segments(
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
     extra_borg_options = config.get('extra_borg_options', {}).get('compact', '')
+    threshold = config.get('compact_threshold')
 
     full_command = (
         (local_path, 'compact')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + (('--progress',) if progress else ())
+        + (('--progress',) if config.get('progress') else ())
         + (('--cleanup-commits',) if cleanup_commits else ())
         + (('--threshold', str(threshold)) if threshold else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (
+            ('--dry-run',)
+            if dry_run and feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version)
+            else ()
+        )
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    if dry_run:
-        logging.info(f'{repository_path}: Skipping compact (dry run)')
+    if dry_run and not feature.available(feature.Feature.DRY_RUN_COMPACT, local_borg_version):
+        logger.info('Skipping compact (dry run)')
         return
 
     execute_command(
         full_command,
         output_log_level=logging.INFO,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/create.py

@@ -1,13 +1,14 @@
-import glob
-import itertools
 import logging
 import os
 import pathlib
+import shlex
 import stat
-import tempfile
+import textwrap
 
+import borgmatic.borg.pattern
+import borgmatic.config.paths
 import borgmatic.logger
-from borgmatic.borg import environment, feature, flags, state
+from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import (
     DO_NOT_CAPTURE,
     execute_command,
@@ -18,241 +19,14 @@ from borgmatic.execute import (
 logger = logging.getLogger(__name__)
 
 
-def expand_directory(directory):
-    '''
-    Given a directory path, expand any tilde (representing a user's home directory) and any globs
-    therein. Return a list of one or more resulting paths.
-    '''
-    expanded_directory = os.path.expanduser(directory)
-
-    return glob.glob(expanded_directory) or [expanded_directory]
-
-
-def expand_directories(directories):
-    '''
-    Given a sequence of directory paths, expand tildes and globs in each one. Return all the
-    resulting directories as a single flattened tuple.
-    '''
-    if directories is None:
-        return ()
-
-    return tuple(
-        itertools.chain.from_iterable(expand_directory(directory) for directory in directories)
-    )
-
-
-def expand_home_directories(directories):
-    '''
-    Given a sequence of directory paths, expand tildes in each one. Do not perform any globbing.
-    Return the results as a tuple.
-    '''
-    if directories is None:
-        return ()
-
-    return tuple(os.path.expanduser(directory) for directory in directories)
-
-
-def map_directories_to_devices(directories):
-    '''
-    Given a sequence of directories, return a map from directory to an identifier for the device on
-    which that directory resides or None if the path doesn't exist.
-
-    This is handy for determining whether two different directories are on the same filesystem (have
-    the same device identifier).
-    '''
-    return {
-        directory: os.stat(directory).st_dev if os.path.exists(directory) else None
-        for directory in directories
-    }
-
-
-def deduplicate_directories(directory_devices, additional_directory_devices):
-    '''
-    Given a map from directory to the identifier for the device on which that directory resides,
-    return the directories as a sorted tuple with all duplicate child directories removed. For
-    instance, if paths is ('/foo', '/foo/bar'), return just: ('/foo',)
-
-    The one exception to this rule is if two paths are on different filesystems (devices). In that
-    case, they won't get de-duplicated in case they both need to be passed to Borg (e.g. the
-    location.one_file_system option is true).
-
-    The idea is that if Borg is given a parent directory, then it doesn't also need to be given
-    child directories, because it will naturally spider the contents of the parent directory. And
-    there are cases where Borg coming across the same file twice will result in duplicate reads and
-    even hangs, e.g. when a database hook is using a named pipe for streaming database dumps to
-    Borg.
-
-    If any additional directory devices are given, also deduplicate against them, but don't include
-    them in the returned directories.
-    '''
-    deduplicated = set()
-    directories = sorted(directory_devices.keys())
-    additional_directories = sorted(additional_directory_devices.keys())
-    all_devices = {**directory_devices, **additional_directory_devices}
-
-    for directory in directories:
-        deduplicated.add(directory)
-        parents = pathlib.PurePath(directory).parents
-
-        # If another directory in the given list (or the additional list) is a parent of current
-        # directory (even n levels up) and both are on the same filesystem, then the current
-        # directory is a duplicate.
-        for other_directory in directories + additional_directories:
-            for parent in parents:
-                if (
-                    pathlib.PurePath(other_directory) == parent
-                    and all_devices[directory] is not None
-                    and all_devices[other_directory] == all_devices[directory]
-                ):
-                    if directory in deduplicated:
-                        deduplicated.remove(directory)
-                    break
-
-    return tuple(sorted(deduplicated))
-
-
-def write_pattern_file(patterns=None, sources=None, pattern_file=None):
-    '''
-    Given a sequence of patterns and an optional sequence of source directories, write them to a
-    named temporary file (with the source directories as additional roots) and return the file.
-    If an optional open pattern file is given, overwrite it instead of making a new temporary file.
-    Return None if no patterns are provided.
-    '''
-    if not patterns and not sources:
-        return None
-
-    if pattern_file is None:
-        pattern_file = tempfile.NamedTemporaryFile('w')
-    else:
-        pattern_file.seek(0)
-
-    pattern_file.write(
-        '\n'.join(tuple(patterns or ()) + tuple(f'R {source}' for source in (sources or [])))
-    )
-    pattern_file.flush()
-
-    return pattern_file
-
-
-def ensure_files_readable(*filename_lists):
-    '''
-    Given a sequence of filename sequences, ensure that each filename is openable. This prevents
-    unreadable files from being passed to Borg, which in certain situations only warns instead of
-    erroring.
-    '''
-    for file_object in itertools.chain.from_iterable(
-        filename_list for filename_list in filename_lists if filename_list
-    ):
-        open(file_object).close()
-
-
-def make_pattern_flags(config, pattern_filename=None):
-    '''
-    Given a configuration dict with a potential patterns_from option, and a filename containing any
-    additional patterns, return the corresponding Borg flags for those files as a tuple.
-    '''
-    pattern_filenames = tuple(config.get('patterns_from') or ()) + (
-        (pattern_filename,) if pattern_filename else ()
-    )
-
-    return tuple(
-        itertools.chain.from_iterable(
-            ('--patterns-from', pattern_filename) for pattern_filename in pattern_filenames
-        )
-    )
-
-
-def make_exclude_flags(config, exclude_filename=None):
-    '''
-    Given a configuration dict with various exclude options, and a filename containing any exclude
-    patterns, return the corresponding Borg flags as a tuple.
-    '''
-    exclude_filenames = tuple(config.get('exclude_from') or ()) + (
-        (exclude_filename,) if exclude_filename else ()
-    )
-    exclude_from_flags = tuple(
-        itertools.chain.from_iterable(
-            ('--exclude-from', exclude_filename) for exclude_filename in exclude_filenames
-        )
-    )
-    caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
-    if_present_flags = tuple(
-        itertools.chain.from_iterable(
-            ('--exclude-if-present', if_present)
-            for if_present in config.get('exclude_if_present', ())
-        )
-    )
-    keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
-    exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
-
-    return (
-        exclude_from_flags
-        + caches_flag
-        + if_present_flags
-        + keep_exclude_tags_flags
-        + exclude_nodump_flags
-    )
-
-
-def make_list_filter_flags(local_borg_version, dry_run):
-    '''
-    Given the local Borg version and whether this is a dry run, return the corresponding flags for
-    passing to "--list --filter". The general idea is that excludes are shown for a dry run or when
-    the verbosity is debug.
-    '''
-    base_flags = 'AME'
-    show_excludes = logger.isEnabledFor(logging.DEBUG)
-
-    if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
-        if show_excludes or dry_run:
-            return f'{base_flags}+-'
-        else:
-            return base_flags
-
-    if show_excludes:
-        return f'{base_flags}x-'
-    else:
-        return f'{base_flags}-'
-
-
-def collect_borgmatic_source_directories(borgmatic_source_directory):
-    '''
-    Return a list of borgmatic-specific source directories used for state like database backups.
-    '''
-    if not borgmatic_source_directory:
-        borgmatic_source_directory = state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY
-
-    return (
-        [borgmatic_source_directory]
-        if os.path.exists(os.path.expanduser(borgmatic_source_directory))
-        else []
-    )
-
-
-ROOT_PATTERN_PREFIX = 'R '
-
-
-def pattern_root_directories(patterns=None):
-    '''
-    Given a sequence of patterns, parse out and return just the root directories.
-    '''
-    if not patterns:
-        return []
-
-    return [
-        pattern.split(ROOT_PATTERN_PREFIX, maxsplit=1)[1]
-        for pattern in patterns
-        if pattern.startswith(ROOT_PATTERN_PREFIX)
-    ]
-
-
-def special_file(path):
+def special_file(path, working_directory=None):
     '''
     Return whether the given path is a special file (character device, block device, or named pipe
-    / FIFO).
+    / FIFO). If a working directory is given, take it into account when making the full path to
+    check.
     '''
     try:
-        mode = os.stat(path).st_mode
+        mode = os.stat(os.path.join(working_directory or '', path)).st_mode
     except (FileNotFoundError, OSError):
         return False
 
@@ -271,101 +45,115 @@ def any_parent_directories(path, candidate_parents):
     return False
 
 
-def collect_special_file_paths(
-    create_command, config, local_path, working_directory, borg_environment, skip_directories
+def validate_planned_backup_paths(
+    dry_run,
+    create_command,
+    config,
+    patterns,
+    local_path,
+    working_directory,
+    borgmatic_runtime_directory,
 ):
     '''
-    Given a Borg create command as a tuple, a configuration dict, a local Borg path, a working
-    directory, a dict of environment variables to pass to Borg, and a sequence of parent directories
-    to skip, collect the paths for any special files (character devices, block devices, and named
-    pipes / FIFOs) that Borg would encounter during a create. These are all paths that could cause
-    Borg to hang if its --read-special flag is used.
+    Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
+    a working directory, and the borgmatic runtime directory, perform a "borg create --dry-run" to
+    determine whether Borg's planned paths to include in a backup look good. Specifically, if the
+    given runtime directory exists, validate that it will be included in a backup and hasn't been
+    excluded.
+
+    Raise ValueError if the runtime directory has been excluded via "exclude_patterns" or similar,
+    because any features that rely on the runtime directory getting backed up will break.  For
+    instance, without the runtime directory, Borg can't consume any database dumps and borgmatic may
+    hang waiting for them to be consumed.
     '''
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
-    # files including any named pipe we've created.
+    # files including any named pipe we've created. And omit "--filter" because that can break the
+    # paths output parsing below such that path lines no longer start with the expected "- ".
     paths_output = execute_command_and_capture_output(
-        tuple(argument for argument in create_command if argument != '--exclude-nodump')
-        + ('--dry-run', '--list'),
+        (
+            *flags.omit_flag_and_value(
+                flags.omit_flag(create_command, '--exclude-nodump'), '--filter'
+            ),
+            '--dry-run',
+            '--list',
+        ),
         capture_stderr=True,
         working_directory=working_directory,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
+    # These are all the individual files that Borg is planning to backup as determined by the Borg
+    # create dry run above.
     paths = tuple(
         path_line.split(' ', 1)[1]
         for path_line in paths_output.split('\n')
-        if path_line and path_line.startswith('- ') or path_line.startswith('+ ')
+        if path_line and path_line.startswith(('- ', '+ '))
     )
 
-    return tuple(
-        path
-        for path in paths
-        if special_file(path) and not any_parent_directories(path, skip_directories)
+    # These are the subset of output paths contained within the borgmatic runtime directory.
+    paths_inside_runtime_directory = {
+        path for path in paths if any_parent_directories(path, (borgmatic_runtime_directory,))
+    }
+
+    # If the runtime directory isn't present in the source patterns, then we shouldn't expect it to
+    # be in the paths output from the Borg dry run.
+    runtime_directory_present_in_patterns = any(
+        pattern
+        for pattern in patterns
+        if any_parent_directories(pattern.path, (borgmatic_runtime_directory,))
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
     )
 
+    # If no paths to backup are inside the runtime directory, it must've been excluded.
+    if (
+        not paths_inside_runtime_directory
+        and runtime_directory_present_in_patterns
+        and not dry_run
+        and os.path.exists(borgmatic_runtime_directory)
+    ):
+        raise ValueError(
+            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.',
+        )
 
-def check_all_source_directories_exist(source_directories):
-    '''
-    Given a sequence of source directories, check that they all exist. If any do not, raise an
-    exception.
-    '''
-    missing_directories = [
-        source_directory
-        for source_directory in source_directories
-        if not all([os.path.exists(directory) for directory in expand_directory(source_directory)])
-    ]
-    if missing_directories:
-        raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}")
+    return tuple(path for path in paths if path not in paths_inside_runtime_directory)
+
+
+MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
 
 
 def make_base_create_command(
     dry_run,
     repository_path,
     config,
-    config_paths,
+    patterns,
     local_borg_version,
     global_arguments,
-    borgmatic_source_directories,
+    borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
-    progress=False,
     json=False,
-    list_files=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, global arguments as an
-    argparse.Namespace instance, and a sequence of borgmatic source directories, return a tuple of
-    (base Borg create command flags, Borg create command positional arguments, open pattern file
-    handle, open exclude file handle).
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    sequence of patterns as borgmatic.borg.pattern.Pattern instances, the local Borg version, global
+    arguments as an argparse.Namespace instance, the borgmatic runtime directory, a string suffix to
+    add to the archive name, the local Borg path, the remote Borg path, whether to output JSON,
+    comment text to add to the created archive, and a sequence of processes streaming data to Borg,
+    return a tuple of (base Borg create command flags, Borg create command positional arguments,
+    open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
-        check_all_source_directories_exist(config.get('source_directories'))
+        borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
 
-    sources = deduplicate_directories(
-        map_directories_to_devices(
-            expand_directories(
-                tuple(config.get('source_directories', ()))
-                + borgmatic_source_directories
-                + tuple(config_paths if config.get('store_config_files', True) else ())
-            )
-        ),
-        additional_directory_devices=map_directories_to_devices(
-            expand_directories(pattern_root_directories(config.get('patterns')))
-        ),
+    patterns_file = borgmatic.borg.pattern.write_patterns_file(
+        patterns,
+        borgmatic_runtime_directory,
     )
-
-    ensure_files_readable(config.get('patterns_from'), config.get('exclude_from'))
-
-    pattern_file = (
-        write_pattern_file(config.get('patterns'), sources)
-        if config.get('patterns') or config.get('patterns_from')
-        else None
-    )
-    exclude_file = write_pattern_file(expand_home_directories(config.get('exclude_patterns')))
     checkpoint_interval = config.get('checkpoint_interval', None)
     checkpoint_volume = config.get('checkpoint_volume', None)
     chunker_params = config.get('chunker_params', None)
@@ -374,9 +162,12 @@ def make_base_create_command(
     upload_buffer_size = config.get('upload_buffer_size', None)
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
-    list_filter_flags = make_list_filter_flags(local_borg_version, dry_run)
+    list_filter_flags = flags.make_list_filter_flags(local_borg_version, dry_run)
     files_cache = config.get('files_cache')
-    archive_name_format = config.get('archive_name_format', flags.DEFAULT_ARCHIVE_NAME_FORMAT)
+    archive_name_format = (
+        config.get('archive_name_format', flags.get_default_archive_name_format(local_borg_version))
+        + archive_suffix
+    )
     extra_borg_options = config.get('extra_borg_options', {}).get('create', '')
 
     if feature.available(feature.Feature.ATIME, local_borg_version):
@@ -406,15 +197,16 @@ def make_base_create_command(
     create_flags = (
         tuple(local_path.split(' '))
         + ('create',)
-        + make_pattern_flags(config, pattern_file.name if pattern_file else None)
-        + make_exclude_flags(config, exclude_file.name if exclude_file else None)
+        + (('--patterns-from', patterns_file.name) if patterns_file else ())
+        + flags.make_exclude_flags(config)
+        + (('--comment', comment) if comment else ())
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
         + (('--checkpoint-volume', str(checkpoint_volume)) if checkpoint_volume else ())
         + (('--chunker-params', chunker_params) if chunker_params else ())
         + (('--compression', compression) if compression else ())
         + upload_ratelimit_flags
         + (('--upload-buffer', str(upload_buffer_size)) if upload_buffer_size else ())
-        + (('--one-file-system',) if config.get('one_file_system') or stream_processes else ())
+        + (('--one-file-system',) if config.get('one_file_system') else ())
         + numeric_ids_flags
         + atime_flags
         + (('--noctime',) if config.get('ctime') is False else ())
@@ -424,128 +216,137 @@ def make_base_create_command(
         + (('--files-cache', files_cache) if files_cache else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (
             ('--list', '--filter', list_filter_flags)
-            if list_files and not json and not progress
+            if config.get('list_details') and not json and not config.get('progress')
             else ()
         )
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
     )
 
     create_positional_arguments = flags.make_repository_archive_flags(
-        repository_path, archive_name_format, local_borg_version
-    ) + (sources if not pattern_file else ())
+        repository_path,
+        archive_name_format,
+        local_borg_version,
+    )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    logger.debug('Checking file paths Borg plans to include')
+    planned_backup_paths = validate_planned_backup_paths(
+        dry_run,
+        create_flags + create_positional_arguments,
+        config,
+        patterns,
+        local_path,
+        working_directory,
+        borgmatic_runtime_directory=borgmatic_runtime_directory,
+    )
 
     # If database hooks are enabled (as indicated by streaming processes), exclude files that might
     # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
     if stream_processes and not config.get('read_special'):
         logger.warning(
-            f'{repository_path}: Ignoring configured "read_special" value of false, as true is needed for database hooks.'
+            'Ignoring configured "read_special" value of false, as true is needed for database hooks.',
         )
-        try:
-            working_directory = os.path.expanduser(config.get('working_directory'))
-        except TypeError:
-            working_directory = None
 
-        borg_environment = environment.make_environment(config)
-
-        logger.debug(f'{repository_path}: Collecting special file paths')
-        special_file_paths = collect_special_file_paths(
-            create_flags + create_positional_arguments,
-            config,
-            local_path,
-            working_directory,
-            borg_environment,
-            skip_directories=borgmatic_source_directories,
+        special_file_paths = tuple(
+            path for path in planned_backup_paths if special_file(path, working_directory)
         )
 
         if special_file_paths:
+            truncated_special_file_paths = textwrap.shorten(
+                ', '.join(special_file_paths),
+                width=MAX_SPECIAL_FILE_PATHS_LENGTH,
+                placeholder=' ...',
+            )
             logger.warning(
-                f'{repository_path}: Excluding special files to prevent Borg from hanging: {", ".join(special_file_paths)}'
+                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
             )
-            exclude_file = write_pattern_file(
-                expand_home_directories(
-                    tuple(config.get('exclude_patterns') or ()) + special_file_paths
+            patterns_file = borgmatic.borg.pattern.write_patterns_file(
+                tuple(
+                    borgmatic.borg.pattern.Pattern(
+                        special_file_path,
+                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                    )
+                    for special_file_path in special_file_paths
                 ),
-                pattern_file=exclude_file,
+                borgmatic_runtime_directory,
+                patterns_file=patterns_file,
             )
-            create_flags += make_exclude_flags(config, exclude_file.name)
 
-    return (create_flags, create_positional_arguments, pattern_file, exclude_file)
+            if '--patterns-from' not in create_flags:
+                create_flags += ('--patterns-from', patterns_file.name)
+
+    return (create_flags, create_positional_arguments, patterns_file)
 
 
 def create_archive(
     dry_run,
     repository_path,
     config,
-    config_paths,
+    patterns,
     local_borg_version,
     global_arguments,
+    borgmatic_runtime_directory,
+    archive_suffix='',
     local_path='borg',
     remote_path=None,
-    progress=False,
-    stats=False,
     json=False,
-    list_files=False,
+    comment=None,
     stream_processes=None,
 ):
     '''
-    Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
-    sequence of loaded configuration paths, the local Borg version, and global arguments as an
-    argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
+    Given verbosity/dry-run flags, a local or remote repository path, a configuration dict, a
+    sequence of loaded configuration paths, the local Borg version, global arguments as an
+    argparse.Namespace instance, the borgmatic runtime directory, a string suffix to add to the
+    archive name, the local Borg path, the remote Borg path, whether to output JSON, and comment
+    text to add to the created archive, and a sequence of processes streaming data to Borg, create a
+    Borg archive and return Borg's JSON output (if any).
 
     If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
     create command while also triggering the given processes to produce output.
     '''
     borgmatic.logger.add_custom_log_levels()
-    borgmatic_source_directories = expand_directories(
-        collect_borgmatic_source_directories(config.get('borgmatic_source_directory'))
-    )
 
-    (create_flags, create_positional_arguments, pattern_file, exclude_file) = (
-        make_base_create_command(
-            dry_run,
-            repository_path,
-            config,
-            config_paths,
-            local_borg_version,
-            global_arguments,
-            borgmatic_source_directories,
-            local_path,
-            remote_path,
-            progress,
-            json,
-            list_files,
-            stream_processes,
-        )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    (create_flags, create_positional_arguments, _) = make_base_create_command(
+        dry_run,
+        repository_path,
+        config,
+        patterns,
+        local_borg_version,
+        global_arguments,
+        borgmatic_runtime_directory,
+        archive_suffix,
+        local_path,
+        remote_path,
+        json,
+        comment,
+        stream_processes,
     )
 
     if json:
         output_log_level = None
-    elif list_files or (stats and not dry_run):
+    elif config.get('list_details') or (config.get('statistics') and not dry_run):
         output_log_level = logging.ANSWER
     else:
         output_log_level = logging.INFO
 
     # The progress output isn't compatible with captured and logged output, as progress messes with
     # the terminal directly.
-    output_file = DO_NOT_CAPTURE if progress else None
-
-    try:
-        working_directory = os.path.expanduser(config.get('working_directory'))
-    except TypeError:
-        working_directory = None
-
-    borg_environment = environment.make_environment(config)
+    output_file = DO_NOT_CAPTURE if config.get('progress') else None
 
     create_flags += (
         (('--info',) if logger.getEffectiveLevel() == logging.INFO and not json else ())
-        + (('--stats',) if stats and not json and not dry_run else ())
+        + (('--stats',) if config.get('statistics') and not json and not dry_run else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) and not json else ())
-        + (('--progress',) if progress else ())
+        + (('--progress',) if config.get('progress') else ())
         + (('--json',) if json else ())
     )
     borg_exit_codes = config.get('borg_exit_codes')
@@ -557,25 +358,28 @@ def create_archive(
             output_log_level,
             output_file,
             working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
-    elif output_log_level is None:
+
+    if output_log_level is None:
         return execute_command_and_capture_output(
             create_flags + create_positional_arguments,
             working_directory=working_directory,
-            extra_environment=borg_environment,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-    else:
-        execute_command(
-            create_flags + create_positional_arguments,
-            output_log_level,
-            output_file,
-            working_directory=working_directory,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
+
+    execute_command(
+        create_flags + create_positional_arguments,
+        output_log_level,
+        output_file,
+        working_directory=working_directory,
+        environment=environment.make_environment(config),
+        borg_local_path=local_path,
+        borg_exit_codes=borg_exit_codes,
+    )
+
+    return None

borgmatic/borg/delete.py

@@ -4,12 +4,16 @@ import logging
 import borgmatic.borg.environment
 import borgmatic.borg.feature
 import borgmatic.borg.flags
-import borgmatic.borg.rdelete
+import borgmatic.borg.repo_delete
+import borgmatic.config.paths
 import borgmatic.execute
 
 logger = logging.getLogger(__name__)
 
 
+FORCE_HARDER_FLAG_COUNT = 2
+
+
 def make_delete_command(
     repository,
     config,
@@ -30,11 +34,15 @@ def make_delete_command(
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
-        + borgmatic.borg.flags.make_flags('log-json', global_arguments.log_json)
+        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
+        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
-        + borgmatic.borg.flags.make_flags('list', delete_arguments.list_archives)
+        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (
-            (('--force',) + (('--force',) if delete_arguments.force >= 2 else ()))
+            (
+                ('--force',)
+                + (('--force',) if delete_arguments.force >= FORCE_HARDER_FLAG_COUNT else ())
+            )
             if delete_arguments.force
             else ()
         )
@@ -46,9 +54,17 @@ def make_delete_command(
             local_borg_version=local_borg_version,
             default_archive_name_format='*',
         )
+        + (('--stats',) if config.get('statistics') else ())
         + borgmatic.borg.flags.make_flags_from_arguments(
             delete_arguments,
-            excludes=('list_archives', 'force', 'match_archives', 'archive', 'repository'),
+            excludes=(
+                'list_details',
+                'statistics',
+                'force',
+                'match_archives',
+                'archive',
+                'repository',
+            ),
         )
         + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
     )
@@ -88,24 +104,25 @@ def delete_archives(
         for argument_name in ARCHIVE_RELATED_ARGUMENT_NAMES
     ):
         if borgmatic.borg.feature.available(
-            borgmatic.borg.feature.Feature.RDELETE, local_borg_version
+            borgmatic.borg.feature.Feature.REPO_DELETE,
+            local_borg_version,
         ):
             logger.warning(
-                'Deleting an entire repository with the delete action is deprecated when using Borg 2.x+. Use the rdelete action instead.'
+                'Deleting an entire repository with the delete action is deprecated when using Borg 2.x+. Use the repo-delete action instead.',
             )
 
-        rdelete_arguments = argparse.Namespace(
+        repo_delete_arguments = argparse.Namespace(
             repository=repository['path'],
-            list_archives=delete_arguments.list_archives,
+            list_details=delete_arguments.list_details,
             force=delete_arguments.force,
             cache_only=delete_arguments.cache_only,
             keep_security_info=delete_arguments.keep_security_info,
         )
-        borgmatic.borg.rdelete.delete_repository(
+        borgmatic.borg.repo_delete.delete_repository(
             repository,
             config,
             local_borg_version,
-            rdelete_arguments,
+            repo_delete_arguments,
             global_arguments,
             local_path,
             remote_path,
@@ -126,7 +143,8 @@ def delete_archives(
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/environment.py

@@ -1,3 +1,8 @@
+import os
+
+import borgmatic.borg.passcommand
+import borgmatic.hooks.credential.parse
+
 OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_base_directory': 'BORG_BASE_DIR',
     'borg_config_directory': 'BORG_CONFIG_DIR',
@@ -5,47 +10,91 @@ OPTION_TO_ENVIRONMENT_VARIABLE = {
     'borg_files_cache_ttl': 'BORG_FILES_CACHE_TTL',
     'borg_security_directory': 'BORG_SECURITY_DIR',
     'borg_keys_directory': 'BORG_KEYS_DIR',
-    'encryption_passcommand': 'BORG_PASSCOMMAND',
-    'encryption_passphrase': 'BORG_PASSPHRASE',
     'ssh_command': 'BORG_RSH',
     'temporary_directory': 'TMPDIR',
 }
 
-DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE = {
-    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
-    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE = {
+    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
 }
 
-DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE = {
-    'check_i_know_what_i_am_doing': 'BORG_CHECK_I_KNOW_WHAT_I_AM_DOING',
+DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE = {
+    'debug_passphrase': 'BORG_DEBUG_PASSPHRASE',
+    'display_passphrase': 'BORG_DISPLAY_PASSPHRASE',
+    'relocated_repo_access_is_ok': 'BORG_RELOCATED_REPO_ACCESS_IS_OK',
+    'unknown_unencrypted_repo_access_is_ok': 'BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK',
+    'use_chunks_archive': 'BORG_USE_CHUNKS_ARCHIVE',
 }
 
 
 def make_environment(config):
     '''
-    Given a borgmatic configuration dict, return its options converted to a Borg environment
-    variable dict.
+    Given a borgmatic configuration dict, convert it to a Borg environment variable dict, merge it
+    with a copy of the current environment variables, and return the result.
+
+    Do not reuse this environment across multiple Borg invocations, because it can include
+    references to resources like anonymous pipes for passphrases—which can only be consumed once.
+
+    Here's how native Borg precedence works for a few of the environment variables:
+
+      1. BORG_PASSPHRASE, if set, is used first.
+      2. BORG_PASSCOMMAND is used only if BORG_PASSPHRASE isn't set.
+      3. BORG_PASSPHRASE_FD is used only if neither of the above are set.
+
+    In borgmatic, we want to simulate this precedence order, but there are some additional
+    complications. First, values can come from either configuration or from environment variables
+    set outside borgmatic; configured options should take precedence. Second, when borgmatic gets a
+    passphrase—directly from configuration or indirectly via a credential hook or a passcommand—we
+    want to pass that passphrase to Borg via an anonymous pipe (+ BORG_PASSPHRASE_FD), since that's
+    more secure than using an environment variable (BORG_PASSPHRASE).
     '''
-    environment = {}
+    environment = dict(os.environ)
 
     for option_name, environment_variable_name in OPTION_TO_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
 
-        if value:
+        if value is not None:
             environment[environment_variable_name] = str(value)
 
-    for (
-        option_name,
-        environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_DOWNCASE_ENVIRONMENT_VARIABLE.items():
-        value = config.get(option_name)
-        if value is not None:
-            environment[environment_variable_name] = 'yes' if value else 'no'
+    if 'encryption_passphrase' in config:
+        environment.pop('BORG_PASSPHRASE', None)
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    if 'encryption_passcommand' in config:
+        environment.pop('BORG_PASSCOMMAND', None)
+
+    passphrase = borgmatic.hooks.credential.parse.resolve_credential(
+        config.get('encryption_passphrase'),
+        config,
+    )
+
+    if passphrase is None:
+        passphrase = borgmatic.borg.passcommand.get_passphrase_from_passcommand(config)
+
+    # If there's a passphrase (from configuration, from a configured credential, or from a
+    # configured passcommand), send it to Borg via an anonymous pipe.
+    if passphrase is not None:
+        read_file_descriptor, write_file_descriptor = os.pipe()
+        os.write(write_file_descriptor, passphrase.encode('utf-8'))
+        os.close(write_file_descriptor)
+
+        # This plus subprocess.Popen(..., close_fds=False) in execute.py is necessary for the Borg
+        # child process to inherit the file descriptor.
+        os.set_inheritable(read_file_descriptor, True)
+        environment['BORG_PASSPHRASE_FD'] = str(read_file_descriptor)
 
     for (
         option_name,
         environment_variable_name,
-    ) in DEFAULT_BOOL_OPTION_TO_UPPERCASE_ENVIRONMENT_VARIABLE.items():
+    ) in DEFAULT_BOOL_OPTION_TO_ENVIRONMENT_VARIABLE.items():
+        if os.environ.get(environment_variable_name) is None:
+            value = config.get(option_name)
+            environment[environment_variable_name] = 'YES' if value else 'NO'
+
+    for (
+        option_name,
+        environment_variable_name,
+    ) in DEFAULT_BOOL_OPTION_TO_UNCONDITIONAL_ENVIRONMENT_VARIABLE.items():
         value = config.get(option_name)
         if value is not None:
             environment[environment_variable_name] = 'YES' if value else 'NO'

borgmatic/borg/export_key.py

@@ -1,6 +1,7 @@
 import logging
 import os
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, flags
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
@@ -18,9 +19,9 @@ def export_key(
     remote_path=None,
 ):
     '''
-    Given a local or remote repository path, a configuration dict, the local Borg version, and
-    optional local and remote Borg paths, export the repository key to the destination path
-    indicated in the export arguments.
+    Given a local or remote repository path, a configuration dict, the local Borg version, export
+    arguments, and optional local and remote Borg paths, export the repository key to the
+    destination path indicated in the export arguments.
 
     If the destination path is empty or "-", then print the key to stdout instead of to a file.
 
@@ -29,11 +30,12 @@ def export_key(
     borgmatic.logger.add_custom_log_levels()
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     if export_arguments.path and export_arguments.path != '-':
-        if os.path.exists(export_arguments.path):
+        if os.path.exists(os.path.join(working_directory or '', export_arguments.path)):
             raise FileExistsError(
-                f'Destination path {export_arguments.path} already exists. Aborting.'
+                f'Destination path {export_arguments.path} already exists. Aborting.',
             )
 
         output_file = None
@@ -44,7 +46,7 @@ def export_key(
         (local_path, 'key', 'export')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -58,14 +60,15 @@ def export_key(
     )
 
     if global_arguments.dry_run:
-        logging.info(f'{repository_path}: Skipping key export (dry run)')
+        logger.info('Skipping key export (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=output_file,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/export_tar.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, flags
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
@@ -19,7 +20,6 @@ def export_tar_archive(
     local_path='borg',
     remote_path=None,
     tar_filter=None,
-    list_files=False,
     strip_components=None,
 ):
     '''
@@ -39,10 +39,10 @@ def export_tar_archive(
         (local_path, 'export-tar')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
-        + (('--list',) if list_files else ())
+        + (('--list',) if config.get('list_details') else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
         + (('--tar-filter', tar_filter) if tar_filter else ())
@@ -56,20 +56,18 @@ def export_tar_archive(
         + (tuple(paths) if paths else ())
     )
 
-    if list_files:
-        output_log_level = logging.ANSWER
-    else:
-        output_log_level = logging.INFO
+    output_log_level = logging.ANSWER if config.get('list_details') else logging.INFO
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping export to tar file (dry run)')
+        logger.info('Skipping export to tar file (dry run)')
         return
 
     execute_command(
         full_command,
         output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/extract.py

@@ -2,8 +2,9 @@ import logging
 import os
 import subprocess
 
+import borgmatic.config.paths
 import borgmatic.config.validate
-from borgmatic.borg import environment, feature, flags, rlist
+from borgmatic.borg import environment, feature, flags, repo_list
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
 
 logger = logging.getLogger(__name__)
@@ -29,7 +30,7 @@ def extract_last_archive_dry_run(
         verbosity_flags = ('--info',)
 
     try:
-        last_archive_name = rlist.resolve_archive_name(
+        last_archive_name = repo_list.resolve_archive_name(
             repository_path,
             'latest',
             config,
@@ -43,23 +44,24 @@ def extract_last_archive_dry_run(
         return
 
     list_flag = ('--list',) if logger.isEnabledFor(logging.DEBUG) else ()
-    borg_environment = environment.make_environment(config)
     full_extract_command = (
         (local_path, 'extract', '--dry-run')
         + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + verbosity_flags
         + list_flag
         + flags.make_repository_archive_flags(
-            repository_path, last_archive_name, local_borg_version
+            repository_path,
+            last_archive_name,
+            local_borg_version,
         )
     )
 
     execute_command(
         full_extract_command,
-        working_directory=None,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
@@ -77,7 +79,6 @@ def extract_archive(
     remote_path=None,
     destination_path=None,
     strip_components=None,
-    progress=False,
     extract_to_stdout=False,
 ):
     '''
@@ -92,8 +93,8 @@ def extract_archive(
     umask = config.get('umask', None)
     lock_wait = config.get('lock_wait', None)
 
-    if progress and extract_to_stdout:
-        raise ValueError('progress and extract_to_stdout cannot both be set')
+    if config.get('progress') and extract_to_stdout:
+        raise ValueError('progress and extract to stdout cannot both be set')
 
     if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
         numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
@@ -109,43 +110,51 @@ def extract_archive(
         # is so this doesn't count the final path component, e.g. the filename itself.
         strip_components = max(
             0,
-            *(len(tuple(piece for piece in path.split(os.path.sep) if piece)) - 1 for path in paths)
+            *(
+                len(tuple(piece for piece in path.split(os.path.sep) if piece)) - 1
+                for path in paths
+            ),
         )
 
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
     full_command = (
         (local_path, 'extract')
         + (('--remote-path', remote_path) if remote_path else ())
         + numeric_ids_flags
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
         + (('--strip-components', str(strip_components)) if strip_components else ())
-        + (('--progress',) if progress else ())
+        + (('--progress',) if config.get('progress') else ())
         + (('--stdout',) if extract_to_stdout else ())
         + flags.make_repository_archive_flags(
-            # Make the repository path absolute so the working directory changes below don't
-            # prevent Borg from finding the repo.
-            borgmatic.config.validate.normalize_repository_path(repository),
+            # Make the repository path absolute so the destination directory used below via changing
+            # the working directory doesn't prevent Borg from finding the repo. But also apply the
+            # user's configured working directory (if any) to the repo path.
+            borgmatic.config.validate.normalize_repository_path(repository, working_directory),
             archive,
             local_borg_version,
         )
         + (tuple(paths) if paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
+    full_destination_path = (
+        os.path.join(working_directory or '', destination_path) if destination_path else None
+    )
 
     # The progress output isn't compatible with captured and logged output, as progress messes with
     # the terminal directly.
-    if progress:
+    if config.get('progress'):
         return execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            working_directory=destination_path,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
+            working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -155,9 +164,9 @@ def extract_archive(
         return execute_command(
             full_command,
             output_file=subprocess.PIPE,
-            working_directory=destination_path,
             run_to_completion=False,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
+            working_directory=full_destination_path,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
@@ -166,8 +175,10 @@ def extract_archive(
     # if the restore paths don't exist in the archive.
     execute_command(
         full_command,
-        working_directory=destination_path,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=full_destination_path,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
     )
+
+    return None

borgmatic/borg/feature.py

@@ -10,12 +10,15 @@ class Feature(Enum):
     NUMERIC_IDS = 4
     UPLOAD_RATELIMIT = 5
     SEPARATE_REPOSITORY_ARCHIVE = 6
-    RCREATE = 7
-    RLIST = 8
-    RINFO = 9
-    RDELETE = 10
+    REPO_CREATE = 7
+    REPO_LIST = 8
+    REPO_INFO = 9
+    REPO_DELETE = 10
     MATCH_ARCHIVES = 11
     EXCLUDED_FILES_MINUS = 12
+    ARCHIVE_SERIES = 13
+    NO_PRUNE_STATS = 14
+    DRY_RUN_COMPACT = 15
 
 
 FEATURE_TO_MINIMUM_BORG_VERSION = {
@@ -25,12 +28,15 @@ FEATURE_TO_MINIMUM_BORG_VERSION = {
     Feature.NUMERIC_IDS: parse('1.2.0b3'),  # borg create/extract/mount --numeric-ids
     Feature.UPLOAD_RATELIMIT: parse('1.2.0b3'),  # borg create --upload-ratelimit
     Feature.SEPARATE_REPOSITORY_ARCHIVE: parse('2.0.0a2'),  # --repo with separate archive
-    Feature.RCREATE: parse('2.0.0a2'),  # borg rcreate
-    Feature.RLIST: parse('2.0.0a2'),  # borg rlist
-    Feature.RINFO: parse('2.0.0a2'),  # borg rinfo
-    Feature.RDELETE: parse('2.0.0a2'),  # borg rdelete
+    Feature.REPO_CREATE: parse('2.0.0a2'),  # borg repo-create
+    Feature.REPO_LIST: parse('2.0.0a2'),  # borg repo-list
+    Feature.REPO_INFO: parse('2.0.0a2'),  # borg repo-info
+    Feature.REPO_DELETE: parse('2.0.0a2'),  # borg repo-delete
     Feature.MATCH_ARCHIVES: parse('2.0.0b3'),  # borg --match-archives
     Feature.EXCLUDED_FILES_MINUS: parse('2.0.0b5'),  # --list --filter uses "-" for excludes
+    Feature.ARCHIVE_SERIES: parse('2.0.0b11'),  # identically named archives form a series
+    Feature.NO_PRUNE_STATS: parse('2.0.0b10'),  # prune --stats is not available
+    Feature.DRY_RUN_COMPACT: parse('1.2.9'),  # borg compact --dry-run support
 }
 
 

borgmatic/borg/flags.py

@@ -34,7 +34,7 @@ def make_flags_from_arguments(arguments, excludes=()):
             make_flags(name, value=getattr(arguments, name))
             for name in sorted(vars(arguments))
             if name not in excludes and not name.startswith('_')
-        )
+        ),
     )
 
 
@@ -50,6 +50,9 @@ def make_repository_flags(repository_path, local_borg_version):
     ) + (repository_path,)
 
 
+ARCHIVE_HASH_PATTERN = re.compile(r'[0-9a-fA-F]{8,}$')
+
+
 def make_repository_archive_flags(repository_path, archive, local_borg_version):
     '''
     Given the path of a Borg repository, an archive name or pattern, and the local Borg version,
@@ -57,20 +60,41 @@ def make_repository_archive_flags(repository_path, archive, local_borg_version):
     and archive.
     '''
     return (
-        ('--repo', repository_path, archive)
+        (
+            '--repo',
+            repository_path,
+            (
+                f'aid:{archive}'
+                if feature.available(feature.Feature.ARCHIVE_SERIES, local_borg_version)
+                and ARCHIVE_HASH_PATTERN.match(archive)
+                and not archive.startswith('aid:')
+                else archive
+            ),
+        )
         if feature.available(feature.Feature.SEPARATE_REPOSITORY_ARCHIVE, local_borg_version)
         else (f'{repository_path}::{archive}',)
     )
 
 
-DEFAULT_ARCHIVE_NAME_FORMAT = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'  # noqa: FS003
+DEFAULT_ARCHIVE_NAME_FORMAT_WITHOUT_SERIES = '{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'
+DEFAULT_ARCHIVE_NAME_FORMAT_WITH_SERIES = '{hostname}'
 
 
-def make_match_archives_flags(
+def get_default_archive_name_format(local_borg_version):
+    '''
+    Given the local Borg version, return the corresponding default archive name format.
+    '''
+    if feature.available(feature.Feature.ARCHIVE_SERIES, local_borg_version):
+        return DEFAULT_ARCHIVE_NAME_FORMAT_WITH_SERIES
+
+    return DEFAULT_ARCHIVE_NAME_FORMAT_WITHOUT_SERIES
+
+
+def make_match_archives_flags(  # noqa: PLR0911
     match_archives,
     archive_name_format,
     local_borg_version,
-    default_archive_name_format=DEFAULT_ARCHIVE_NAME_FORMAT,
+    default_archive_name_format=None,
 ):
     '''
     Return match archives flags based on the given match archives value, if any. If it isn't set,
@@ -83,12 +107,23 @@ def make_match_archives_flags(
             return ()
 
         if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
+            if (
+                feature.available(feature.Feature.ARCHIVE_SERIES, local_borg_version)
+                and ARCHIVE_HASH_PATTERN.match(match_archives)
+                and not match_archives.startswith('aid:')
+            ):
+                return ('--match-archives', f'aid:{match_archives}')
+
             return ('--match-archives', match_archives)
-        else:
-            return ('--glob-archives', re.sub(r'^sh:', '', match_archives))
+
+        return ('--glob-archives', re.sub(r'^sh:', '', match_archives))
 
     derived_match_archives = re.sub(
-        r'\{(now|utcnow|pid)([:%\w\.-]*)\}', '*', archive_name_format or default_archive_name_format
+        r'\{(now|utcnow|pid)([:%\w\.-]*)\}',
+        '*',
+        archive_name_format
+        or default_archive_name_format
+        or get_default_archive_name_format(local_borg_version),
     )
 
     if derived_match_archives == '*':
@@ -96,8 +131,8 @@ def make_match_archives_flags(
 
     if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version):
         return ('--match-archives', f'sh:{derived_match_archives}')
-    else:
-        return ('--glob-archives', f'{derived_match_archives}')
+
+    return ('--glob-archives', f'{derived_match_archives}')
 
 
 def warn_for_aggressive_archive_flags(json_command, json_output):
@@ -115,9 +150,89 @@ def warn_for_aggressive_archive_flags(json_command, json_output):
         if len(json.loads(json_output)['archives']) == 0:
             logger.warning('An archive filter was applied, but no matching archives were found.')
             logger.warning(
-                'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.'
+                'Try adding --match-archives "*" or adjusting archive_name_format/match_archives in configuration.',
             )
     except json.JSONDecodeError as error:
         logger.debug(f'Cannot parse JSON output from archive command: {error}')
     except (TypeError, KeyError):
         logger.debug('Cannot parse JSON output from archive command: No "archives" key found')
+
+
+def omit_flag(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given (valueless) flag
+    omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    return tuple(argument for argument in arguments if argument != flag)
+
+
+def omit_flag_and_value(arguments, flag):
+    '''
+    Given a sequence of Borg command-line arguments, return them with the given flag and its
+    corresponding value omitted. For instance, if the flag is "--flag" and arguments is:
+
+        ('borg', 'create', '--flag', 'value', '--other-flag')
+
+    ... or:
+
+        ('borg', 'create', '--flag=value', '--other-flag')
+
+    ... then return:
+
+        ('borg', 'create', '--other-flag')
+    '''
+    # This works by zipping together a list of overlapping pairwise arguments. E.g., ('one', 'two',
+    # 'three', 'four') becomes ((None, 'one'), ('one, 'two'), ('two', 'three'), ('three', 'four')).
+    # This makes it easy to "look back" at the previous arguments so we can exclude both a flag and
+    # its value.
+    return tuple(
+        argument
+        for (previous_argument, argument) in zip((None, *arguments), arguments)
+        if flag not in {previous_argument, argument}
+        if not argument.startswith(f'{flag}=')
+    )
+
+
+def make_exclude_flags(config):
+    '''
+    Given a configuration dict with various exclude options, return the corresponding Borg flags as
+    a tuple.
+    '''
+    caches_flag = ('--exclude-caches',) if config.get('exclude_caches') else ()
+    if_present_flags = tuple(
+        itertools.chain.from_iterable(
+            ('--exclude-if-present', if_present)
+            for if_present in config.get('exclude_if_present', ())
+        ),
+    )
+    keep_exclude_tags_flags = ('--keep-exclude-tags',) if config.get('keep_exclude_tags') else ()
+    exclude_nodump_flags = ('--exclude-nodump',) if config.get('exclude_nodump') else ()
+
+    return caches_flag + if_present_flags + keep_exclude_tags_flags + exclude_nodump_flags
+
+
+def make_list_filter_flags(local_borg_version, dry_run):
+    '''
+    Given the local Borg version and whether this is a dry run, return the corresponding flags for
+    passing to "--list --filter". The general idea is that excludes are shown for a dry run or when
+    the verbosity is debug.
+    '''
+    base_flags = 'AME'
+    show_excludes = logger.isEnabledFor(logging.DEBUG)
+
+    if feature.available(feature.Feature.EXCLUDED_FILES_MINUS, local_borg_version):
+        if show_excludes or dry_run:
+            return f'{base_flags}+-'
+
+        return base_flags
+
+    if show_excludes:
+        return f'{base_flags}x-'
+
+    return f'{base_flags}-'

borgmatic/borg/import_key.py

@@ -0,0 +1,70 @@
+import logging
+import os
+
+import borgmatic.config.paths
+import borgmatic.logger
+from borgmatic.borg import environment, flags
+from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+
+logger = logging.getLogger(__name__)
+
+
+def import_key(
+    repository_path,
+    config,
+    local_borg_version,
+    import_arguments,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+):
+    '''
+    Given a local or remote repository path, a configuration dict, the local Borg version, import
+    arguments, and optional local and remote Borg paths, import the repository key from the
+    path indicated in the import arguments.
+
+    If the path is empty or "-", then read the key from stdin.
+
+    Raise ValueError if the path is given and it does not exist.
+    '''
+    umask = config.get('umask', None)
+    lock_wait = config.get('lock_wait', None)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
+
+    if import_arguments.path and import_arguments.path != '-':
+        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+
+        input_file = None
+    else:
+        input_file = DO_NOT_CAPTURE
+
+    full_command = (
+        (local_path, 'key', 'import')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (('--log-json',) if config.get('log_json') else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + flags.make_flags('paper', import_arguments.paper)
+        + flags.make_repository_flags(
+            repository_path,
+            local_borg_version,
+        )
+        + ((import_arguments.path,) if input_file is None else ())
+    )
+
+    if global_arguments.dry_run:
+        logger.info('Skipping key import (dry run)')
+        return
+
+    execute_command(
+        full_command,
+        input_file=input_file,
+        output_log_level=logging.INFO,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/info.py

@@ -1,6 +1,7 @@
 import argparse
 import logging
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command, execute_command_and_capture_output
@@ -35,7 +36,8 @@ def make_info_command(
             else ()
         )
         + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
+        + flags.make_flags('umask', config.get('umask'))
+        + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
@@ -46,16 +48,15 @@ def make_info_command(
             if info_arguments.prefix
             else (
                 flags.make_match_archives_flags(
-                    info_arguments.match_archives
-                    or info_arguments.archive
-                    or config.get('match_archives'),
+                    info_arguments.archive or config.get('match_archives'),
                     config.get('archive_name_format'),
                     local_borg_version,
                 )
             )
         )
         + flags.make_flags_from_arguments(
-            info_arguments, excludes=('repository', 'archive', 'prefix', 'match_archives')
+            info_arguments,
+            excludes=('repository', 'archive', 'prefix', 'match_archives'),
         )
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
@@ -96,10 +97,12 @@ def display_archives_info(
         remote_path,
     )
     borg_exit_codes = config.get('borg_exit_codes')
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     json_info = execute_command_and_capture_output(
         json_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
     )
@@ -112,7 +115,10 @@ def display_archives_info(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
     )
+
+    return None

borgmatic/borg/list.py

@@ -3,20 +3,22 @@ import copy
 import logging
 import re
 
+import borgmatic.config.paths
 import borgmatic.logger
-from borgmatic.borg import environment, feature, flags, rlist
+from borgmatic.borg import environment, feature, flags, repo_list
 from borgmatic.execute import execute_command, execute_command_and_capture_output
 
 logger = logging.getLogger(__name__)
 
 
-ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST = ('prefix', 'match_archives', 'sort_by', 'first', 'last')
+ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST = ('prefix', 'match_archives', 'sort_by', 'first', 'last')
 MAKE_FLAGS_EXCLUDES = (
     'repository',
     'archive',
     'paths',
     'find_paths',
-) + ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST
+    *ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST,
+)
 
 
 def make_list_command(
@@ -33,8 +35,6 @@ def make_list_command(
     and local and remote Borg paths, return a command as a tuple to list archives or paths within an
     archive.
     '''
-    lock_wait = config.get('lock_wait', None)
-
     return (
         (local_path, 'list')
         + (
@@ -48,12 +48,15 @@ def make_list_command(
             else ()
         )
         + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
-        + flags.make_flags('lock-wait', lock_wait)
+        + flags.make_flags('umask', config.get('umask'))
+        + flags.make_flags('log-json', config.get('log_json'))
+        + flags.make_flags('lock-wait', config.get('lock_wait'))
         + flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
         + (
             flags.make_repository_archive_flags(
-                repository_path, list_arguments.archive, local_borg_version
+                repository_path,
+                list_arguments.archive,
+                local_borg_version,
             )
             if list_arguments.archive
             else flags.make_repository_flags(repository_path, local_borg_version)
@@ -100,13 +103,12 @@ def capture_archive_listing(
     remote_path=None,
 ):
     '''
-    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
-    version, global arguments as an argparse.Namespace, the archive paths in which to list files,
-    the Borg path format to use for the output, and local and remote Borg paths, capture the output
-    of listing that archive and return it as a list of file paths.
+    Given a local or remote repository path, an archive name, a configuration
+    dict, the local Borg version, global arguments as an argparse.Namespace,
+    the archive paths (or Borg patterns) in which to list files, the Borg path
+    format to use for the output, and local and remote Borg paths, capture the
+    output of listing that archive and return it as a list of file paths.
     '''
-    borg_environment = environment.make_environment(config)
-
     return tuple(
         execute_command_and_capture_output(
             make_list_command(
@@ -116,21 +118,22 @@ def capture_archive_listing(
                 argparse.Namespace(
                     repository=repository_path,
                     archive=archive,
-                    paths=[f'sh:{path}' for path in list_paths] if list_paths else None,
+                    paths=list(list_paths) if list_paths else None,
                     find_paths=None,
                     json=None,
-                    format=path_format or '{path}{NL}',  # noqa: FS003
+                    format=path_format or '{path}{NUL}',
                 ),
                 global_arguments,
                 local_path,
                 remote_path,
             ),
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\n')
-        .split('\n')
+        .strip('\0')
+        .split('\0'),
     )
 
 
@@ -154,12 +157,12 @@ def list_archive(
     borgmatic.logger.add_custom_log_levels()
 
     if not list_arguments.archive and not list_arguments.find_paths:
-        if feature.available(feature.Feature.RLIST, local_borg_version):
+        if feature.available(feature.Feature.REPO_LIST, local_borg_version):
             logger.warning(
-                'Omitting the --archive flag on the list action is deprecated when using Borg 2.x+. Use the rlist action instead.'
+                'Omitting the --archive flag on the list action is deprecated when using Borg 2.x+. Use the repo-list action instead.',
             )
 
-        rlist_arguments = argparse.Namespace(
+        repo_list_arguments = argparse.Namespace(
             repository=repository_path,
             short=list_arguments.short,
             format=list_arguments.format,
@@ -170,35 +173,34 @@ def list_archive(
             first=list_arguments.first,
             last=list_arguments.last,
         )
-        return rlist.list_repository(
+        return repo_list.list_repository(
             repository_path,
             config,
             local_borg_version,
-            rlist_arguments,
+            repo_list_arguments,
             global_arguments,
             local_path,
             remote_path,
         )
 
     if list_arguments.archive:
-        for name in ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST:
+        for name in ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST:
             if getattr(list_arguments, name, None):
                 logger.warning(
-                    f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag."
+                    f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag.",
                 )
 
     if list_arguments.json:
         raise ValueError(
-            'The --json flag on the list action is not supported when using the --archive/--find flags.'
+            'The --json flag on the list action is not supported when using the --archive/--find flags.',
         )
 
-    borg_environment = environment.make_environment(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     # If there are any paths to find (and there's not a single archive already selected), start by
     # getting a list of archives to search.
     if list_arguments.find_paths and not list_arguments.archive:
-        rlist_arguments = argparse.Namespace(
+        repo_list_arguments = argparse.Namespace(
             repository=repository_path,
             short=True,
             format=None,
@@ -213,35 +215,36 @@ def list_archive(
         # Ask Borg to list archives. Capture its output for use below.
         archive_lines = tuple(
             execute_command_and_capture_output(
-                rlist.make_rlist_command(
+                repo_list.make_repo_list_command(
                     repository_path,
                     config,
                     local_borg_version,
-                    rlist_arguments,
+                    repo_list_arguments,
                     global_arguments,
                     local_path,
                     remote_path,
                 ),
-                extra_environment=borg_environment,
+                environment=environment.make_environment(config),
+                working_directory=borgmatic.config.paths.get_working_directory(config),
                 borg_local_path=local_path,
                 borg_exit_codes=borg_exit_codes,
             )
             .strip('\n')
-            .split('\n')
+            .splitlines(),
         )
     else:
         archive_lines = (list_arguments.archive,)
 
     # For each archive listed by Borg, run list on the contents of that archive.
     for archive in archive_lines:
-        logger.answer(f'{repository_path}: Listing archive {archive}')
+        logger.answer(f'Listing archive {archive}')
 
         archive_arguments = copy.copy(list_arguments)
         archive_arguments.archive = archive
 
         # This list call is to show the files in a single archive, not list multiple archives. So
         # blank out any archive filtering flags. They'll break anyway in Borg 2.
-        for name in ARCHIVE_FILTER_FLAGS_MOVED_TO_RLIST:
+        for name in ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST:
             setattr(archive_arguments, name, None)
 
         main_command = make_list_command(
@@ -257,7 +260,10 @@ def list_archive(
         execute_command(
             main_command,
             output_log_level=logging.ANSWER,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
+
+    return None

borgmatic/borg/mount.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
 
@@ -29,7 +30,7 @@ def mount_archive(
         (local_path, 'mount')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -58,14 +59,15 @@ def mount_archive(
         + (tuple(mount_arguments.paths) if mount_arguments.paths else ())
     )
 
-    borg_environment = environment.make_environment(config)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
 
     # Don't capture the output when foreground mode is used so that ctrl-C can work properly.
     if mount_arguments.foreground:
         execute_command(
             full_command,
             output_file=DO_NOT_CAPTURE,
-            extra_environment=borg_environment,
+            environment=environment.make_environment(config),
+            working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
@@ -73,7 +75,8 @@ def mount_archive(
 
     execute_command(
         full_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/passcommand.py

@@ -0,0 +1,40 @@
+import functools
+import logging
+import shlex
+
+import borgmatic.config.paths
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+@functools.cache
+def run_passcommand(passcommand, working_directory):
+    '''
+    Run the given passcommand using the given working directory and return the passphrase produced
+    by the command.
+
+    Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
+    per borgmatic invocation.
+    '''
+    return borgmatic.execute.execute_command_and_capture_output(
+        shlex.split(passcommand),
+        working_directory=working_directory,
+    )
+
+
+def get_passphrase_from_passcommand(config):
+    '''
+    Given the configuration dict, call the configured passcommand to produce and return an
+    encryption passphrase. In effect, we're doing an end-run around Borg by invoking its passcommand
+    ourselves. This allows us to pass the resulting passphrase to multiple different Borg
+    invocations without the user having to be prompted multiple times.
+
+    If no passcommand is configured, then return None.
+    '''
+    passcommand = config.get('encryption_passcommand')
+
+    if not passcommand:
+        return None
+
+    return run_passcommand(passcommand, borgmatic.config.paths.get_working_directory(config))

borgmatic/borg/pattern.py

@@ -0,0 +1,106 @@
+import collections
+import enum
+import logging
+import os
+import tempfile
+
+logger = logging.getLogger(__name__)
+
+
+# See https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
+class Pattern_type(enum.Enum):
+    ROOT = 'R'  # A ROOT pattern always has a NONE pattern style.
+    PATTERN_STYLE = 'P'
+    EXCLUDE = '-'
+    NO_RECURSE = '!'
+    INCLUDE = '+'
+
+
+class Pattern_style(enum.Enum):
+    NONE = ''
+    FNMATCH = 'fm'
+    SHELL = 'sh'
+    REGULAR_EXPRESSION = 're'
+    PATH_PREFIX = 'pp'
+    PATH_FULL_MATCH = 'pf'
+
+
+class Pattern_source(enum.Enum):
+    '''
+    Where the pattern came from within borgmatic. This is important because certain use cases (like
+    filesystem snapshotting) only want to consider patterns that the user actually put in a
+    configuration file and not patterns from other sources.
+    '''
+
+    # The pattern is from a borgmatic configuration option, e.g. listed in "source_directories".
+    CONFIG = 'config'
+
+    # The pattern is generated internally within borgmatic, e.g. for special file excludes.
+    INTERNAL = 'internal'
+
+    # The pattern originates from within a borgmatic hook, e.g. a database hook that adds its dump
+    # directory.
+    HOOK = 'hook'
+
+
+Pattern = collections.namedtuple(
+    'Pattern',
+    ('path', 'type', 'style', 'device', 'source'),
+    defaults=(
+        Pattern_type.ROOT,
+        Pattern_style.NONE,
+        None,
+        Pattern_source.HOOK,
+    ),
+)
+
+
+def write_patterns_file(patterns, borgmatic_runtime_directory, patterns_file=None):
+    '''
+    Given a sequence of patterns as Pattern instances, write them to a named temporary file in the
+    given borgmatic runtime directory and return the file object so it can continue to exist on disk
+    as long as the caller needs it.
+
+    If an optional open pattern file is given, append to it instead of making a new temporary file.
+    Return None if no patterns are provided.
+    '''
+    if not patterns:
+        return None
+
+    if patterns_file is None:
+        patterns_file = tempfile.NamedTemporaryFile(
+            'w', dir=borgmatic_runtime_directory, encoding='utf-8'
+        )
+        operation_name = 'Writing'
+    else:
+        patterns_file.write('\n')
+        operation_name = 'Appending'
+
+    patterns_output = '\n'.join(
+        f'{pattern.type.value} {pattern.style.value}{":" if pattern.style.value else ""}{pattern.path}'
+        for pattern in patterns
+    )
+    logger.debug(f'{operation_name} patterns to {patterns_file.name}:\n{patterns_output}')
+
+    patterns_file.write(patterns_output)
+    patterns_file.flush()
+
+    return patterns_file
+
+
+def check_all_root_patterns_exist(patterns):
+    '''
+    Given a sequence of Pattern instances, check that all root pattern paths exist. If any don't,
+    raise an exception.
+    '''
+    missing_paths = [
+        pattern.path
+        for pattern in patterns
+        if pattern.type == Pattern_type.ROOT
+        if not os.path.exists(pattern.path)
+    ]
+
+    if missing_paths:
+        raise ValueError(
+            f"Source directories or root pattern paths do not exist: {', '.join(missing_paths)}",
+        )

borgmatic/borg/prune.py

@@ -1,5 +1,7 @@
 import logging
+import shlex
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command
@@ -7,9 +9,10 @@ from borgmatic.execute import execute_command
 logger = logging.getLogger(__name__)
 
 
-def make_prune_flags(config, local_borg_version):
+def make_prune_flags(config, prune_arguments, local_borg_version):
     '''
-    Given a configuration dict mapping from option name to value, transform it into an sequence of
+    Given a configuration dict mapping from option name to value, prune arguments as an
+    argparse.Namespace instance, and the local Borg version, produce a corresponding sequence of
     command-line flags.
 
     For example, given a retention config of:
@@ -68,25 +71,31 @@ def prune_archives(
 
     full_command = (
         (local_path, 'prune')
-        + make_prune_flags(config, local_borg_version)
+        + make_prune_flags(config, prune_arguments, local_borg_version)
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + (('--stats',) if prune_arguments.stats and not dry_run else ())
+        + (
+            ('--stats',)
+            if config.get('statistics')
+            and not dry_run
+            and not feature.available(feature.Feature.NO_PRUNE_STATS, local_borg_version)
+            else ()
+        )
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + flags.make_flags_from_arguments(
             prune_arguments,
-            excludes=('repository', 'stats', 'list_archives'),
+            excludes=('repository', 'match_archives', 'statistics', 'list_details'),
         )
-        + (('--list',) if prune_arguments.list_archives else ())
+        + (('--list',) if config.get('list_details') else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    if prune_arguments.stats or prune_arguments.list_archives:
+    if config.get('statistics') or config.get('list_details'):
         output_log_level = logging.ANSWER
     else:
         output_log_level = logging.INFO
@@ -94,7 +103,8 @@ def prune_archives(
     execute_command(
         full_command,
         output_log_level=output_log_level,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/recreate.py

@@ -0,0 +1,106 @@
+import logging
+import shlex
+
+import borgmatic.borg.environment
+import borgmatic.borg.feature
+import borgmatic.config.paths
+import borgmatic.execute
+from borgmatic.borg import flags
+from borgmatic.borg.pattern import write_patterns_file
+
+logger = logging.getLogger(__name__)
+
+
+def recreate_archive(
+    repository,
+    archive,
+    config,
+    local_borg_version,
+    recreate_arguments,
+    global_arguments,
+    local_path,
+    remote_path=None,
+    patterns=None,
+):
+    '''
+    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
+    version string, an argparse.Namespace of recreate arguments, an argparse.Namespace of global
+    arguments, optional local and remote Borg paths, executes the recreate command with the given
+    arguments.
+    '''
+    lock_wait = config.get('lock_wait', None)
+    exclude_flags = flags.make_exclude_flags(config)
+    compression = config.get('compression', None)
+    chunker_params = config.get('chunker_params', None)
+
+    # Available recompress MODES: "if-different", "always", "never" (default)
+    recompress = config.get('recompress', None)
+
+    # Write patterns to a temporary file and use that file with --patterns-from.
+    patterns_file = write_patterns_file(
+        patterns,
+        borgmatic.config.paths.get_working_directory(config),
+    )
+
+    recreate_command = (
+        (local_path, 'recreate')
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--log-json',) if config.get('log_json') else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (('--patterns-from', patterns_file.name) if patterns_file else ())
+        + (
+            (
+                '--list',
+                '--filter',
+                flags.make_list_filter_flags(local_borg_version, global_arguments.dry_run),
+            )
+            if config.get('list_details')
+            else ()
+        )
+        # Flag --target works only for a single archive.
+        + (('--target', recreate_arguments.target) if recreate_arguments.target and archive else ())
+        + (
+            ('--comment', shlex.quote(recreate_arguments.comment))
+            if recreate_arguments.comment
+            else ()
+        )
+        + (('--timestamp', recreate_arguments.timestamp) if recreate_arguments.timestamp else ())
+        + (('--compression', compression) if compression else ())
+        + (('--chunker-params', chunker_params) if chunker_params else ())
+        + (('--recompress', recompress) if recompress else ())
+        + exclude_flags
+        + (
+            (
+                flags.make_repository_flags(repository, local_borg_version)
+                + flags.make_match_archives_flags(
+                    archive or config.get('match_archives'),
+                    config.get('archive_name_format'),
+                    local_borg_version,
+                )
+            )
+            if borgmatic.borg.feature.available(
+                borgmatic.borg.feature.Feature.SEPARATE_REPOSITORY_ARCHIVE,
+                local_borg_version,
+            )
+            else (
+                flags.make_repository_archive_flags(repository, archive, local_borg_version)
+                if archive
+                else flags.make_repository_flags(repository, local_borg_version)
+            )
+        )
+    )
+
+    if global_arguments.dry_run:
+        logger.info('Skipping the archive recreation (dry run)')
+        return
+
+    borgmatic.execute.execute_command(
+        full_command=recreate_command,
+        output_log_level=logging.INFO,
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/rename.py

@@ -0,0 +1,64 @@
+import logging
+
+import borgmatic.borg.flags
+
+logger = logging.getLogger(__name__)
+
+
+def make_rename_command(
+    dry_run,
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    return (
+        (local_path, 'rename')
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + borgmatic.borg.flags.make_flags('dry-run', dry_run)
+        + borgmatic.borg.flags.make_flags('remote-path', remote_path)
+        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
+        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
+        + borgmatic.borg.flags.make_repository_archive_flags(
+            repository_name,
+            old_archive_name,
+            local_borg_version,
+        )
+        + (new_archive_name,)
+    )
+
+
+def rename_archive(
+    repository_name,
+    old_archive_name,
+    new_archive_name,
+    dry_run,
+    config,
+    local_borg_version,
+    local_path,
+    remote_path,
+):
+    command = make_rename_command(
+        dry_run,
+        repository_name,
+        old_archive_name,
+        new_archive_name,
+        config,
+        local_borg_version,
+        local_path,
+        remote_path,
+    )
+
+    borgmatic.execute.execute_command(
+        command,
+        output_log_level=logging.INFO,
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
+    )

borgmatic/borg/repo_create.py

@@ -1,15 +1,17 @@
 import argparse
 import json
 import logging
+import shlex
 import subprocess
 
-from borgmatic.borg import environment, feature, flags, rinfo
+import borgmatic.config.paths
+from borgmatic.borg import environment, feature, flags, repo_info
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
 
 logger = logging.getLogger(__name__)
 
 
-RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES = {2, 13}
+REPO_INFO_REPOSITORY_NOT_FOUND_EXIT_CODES = {2, 13, 15}
 
 
 def create_repository(
@@ -23,7 +25,7 @@ def create_repository(
     copy_crypt_key=False,
     append_only=None,
     storage_quota=None,
-    make_parent_dirs=False,
+    make_parent_directories=False,
     local_path='borg',
     remote_path=None,
 ):
@@ -39,7 +41,7 @@ def create_repository(
     '''
     try:
         info_data = json.loads(
-            rinfo.display_repository_info(
+            repo_info.display_repository_info(
                 repository_path,
                 config,
                 local_borg_version,
@@ -47,29 +49,30 @@ def create_repository(
                 global_arguments,
                 local_path,
                 remote_path,
-            )
+            ),
         )
         repository_encryption_mode = info_data.get('encryption', {}).get('mode')
 
         if repository_encryption_mode != encryption_mode:
             raise ValueError(
-                f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
+                f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"',
             )
 
-        logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
+        logger.info('Repository already exists. Skipping creation.')
         return
     except subprocess.CalledProcessError as error:
-        if error.returncode not in RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
+        if error.returncode not in REPO_INFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
             raise
 
     lock_wait = config.get('lock_wait')
-    extra_borg_options = config.get('extra_borg_options', {}).get('rcreate', '')
+    umask = config.get('umask')
+    extra_borg_options = config.get('extra_borg_options', {}).get('repo-create', '')
 
-    rcreate_command = (
+    repo_create_command = (
         (local_path,)
         + (
-            ('rcreate',)
-            if feature.available(feature.Feature.RCREATE, local_borg_version)
+            ('repo-create',)
+            if feature.available(feature.Feature.REPO_CREATE, local_borg_version)
             else ('init',)
         )
         + (('--encryption', encryption_mode) if encryption_mode else ())
@@ -77,25 +80,27 @@ def create_repository(
         + (('--copy-crypt-key',) if copy_crypt_key else ())
         + (('--append-only',) if append_only else ())
         + (('--storage-quota', storage_quota) if storage_quota else ())
-        + (('--make-parent-dirs',) if make_parent_dirs else ())
+        + (('--make-parent-dirs',) if make_parent_directories else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
-        + (('--log-json',) if global_arguments.log_json else ())
+        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--remote-path', remote_path) if remote_path else ())
-        + (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
     if dry_run:
-        logging.info(f'{repository_path}: Skipping repository creation (dry run)')
+        logger.info('Skipping repository creation (dry run)')
         return
 
     # Do not capture output here, so as to support interactive prompts.
     execute_command(
-        rcreate_command,
+        repo_create_command,
         output_file=DO_NOT_CAPTURE,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/repo_delete.py

@@ -3,31 +3,36 @@ import logging
 import borgmatic.borg.environment
 import borgmatic.borg.feature
 import borgmatic.borg.flags
+import borgmatic.config.paths
 import borgmatic.execute
 
 logger = logging.getLogger(__name__)
 
 
-def make_rdelete_command(
+FORCE_HARDER_FLAG_COUNT = 2
+
+
+def make_repo_delete_command(
     repository,
     config,
     local_borg_version,
-    rdelete_arguments,
+    repo_delete_arguments,
     global_arguments,
     local_path,
     remote_path,
 ):
     '''
     Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the rdelete action as an argparse.Namespace, and global arguments, return a command
-    as a tuple to rdelete the entire repository.
+    arguments to the repo_delete action as an argparse.Namespace, and global arguments, return a command
+    as a tuple to repo_delete the entire repository.
     '''
     return (
         (local_path,)
         + (
-            ('rdelete',)
+            ('repo-delete',)
             if borgmatic.borg.feature.available(
-                borgmatic.borg.feature.Feature.RDELETE, local_borg_version
+                borgmatic.borg.feature.Feature.REPO_DELETE,
+                local_borg_version,
             )
             else ('delete',)
         )
@@ -35,16 +40,21 @@ def make_rdelete_command(
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
-        + borgmatic.borg.flags.make_flags('log-json', global_arguments.log_json)
+        + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
+        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
-        + borgmatic.borg.flags.make_flags('list', rdelete_arguments.list_archives)
+        + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (
-            (('--force',) + (('--force',) if rdelete_arguments.force >= 2 else ()))
-            if rdelete_arguments.force
+            (
+                ('--force',)
+                + (('--force',) if repo_delete_arguments.force >= FORCE_HARDER_FLAG_COUNT else ())
+            )
+            if repo_delete_arguments.force
             else ()
         )
         + borgmatic.borg.flags.make_flags_from_arguments(
-            rdelete_arguments, excludes=('list_archives', 'force', 'repository')
+            repo_delete_arguments,
+            excludes=('list_details', 'force', 'repository'),
         )
         + borgmatic.borg.flags.make_repository_flags(repository['path'], local_borg_version)
     )
@@ -54,23 +64,23 @@ def delete_repository(
     repository,
     config,
     local_borg_version,
-    rdelete_arguments,
+    repo_delete_arguments,
     global_arguments,
     local_path='borg',
     remote_path=None,
 ):
     '''
     Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the rdelete action as an argparse.Namespace, global arguments as an
-    argparse.Namespace, and local and remote Borg paths, rdelete the entire repository.
+    arguments to the repo_delete action as an argparse.Namespace, global arguments as an
+    argparse.Namespace, and local and remote Borg paths, repo_delete the entire repository.
     '''
     borgmatic.logger.add_custom_log_levels()
 
-    command = make_rdelete_command(
+    command = make_repo_delete_command(
         repository,
         config,
         local_borg_version,
-        rdelete_arguments,
+        repo_delete_arguments,
         global_arguments,
         local_path,
         remote_path,
@@ -83,10 +93,11 @@ def delete_repository(
         # prompt won't work.
         output_file=(
             None
-            if rdelete_arguments.force or rdelete_arguments.cache_only
+            if repo_delete_arguments.force or repo_delete_arguments.cache_only
             else borgmatic.execute.DO_NOT_CAPTURE
         ),
-        extra_environment=borgmatic.borg.environment.make_environment(config),
+        environment=borgmatic.borg.environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/repo_info.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command, execute_command_and_capture_output
@@ -11,14 +12,14 @@ def display_repository_info(
     repository_path,
     config,
     local_borg_version,
-    rinfo_arguments,
+    repo_info_arguments,
     global_arguments,
     local_path='borg',
     remote_path=None,
 ):
     '''
     Given a local or remote repository path, a configuration dict, the local Borg version, the
-    arguments to the rinfo action, and global arguments as an argparse.Namespace, display summary
+    arguments to the repo_info action, and global arguments as an argparse.Namespace, display summary
     information for the Borg repository or return JSON summary information.
     '''
     borgmatic.logger.add_custom_log_levels()
@@ -27,42 +28,47 @@ def display_repository_info(
     full_command = (
         (local_path,)
         + (
-            ('rinfo',)
-            if feature.available(feature.Feature.RINFO, local_borg_version)
+            ('repo-info',)
+            if feature.available(feature.Feature.REPO_INFO, local_borg_version)
             else ('info',)
         )
         + (
             ('--info',)
-            if logger.getEffectiveLevel() == logging.INFO and not rinfo_arguments.json
+            if logger.getEffectiveLevel() == logging.INFO and not repo_info_arguments.json
             else ()
         )
         + (
             ('--debug', '--show-rc')
-            if logger.isEnabledFor(logging.DEBUG) and not rinfo_arguments.json
+            if logger.isEnabledFor(logging.DEBUG) and not repo_info_arguments.json
             else ()
         )
         + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
+        + flags.make_flags('umask', config.get('umask'))
+        + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', lock_wait)
-        + (('--json',) if rinfo_arguments.json else ())
+        + (('--json',) if repo_info_arguments.json else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
-    extra_environment = environment.make_environment(config)
+    working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
-    if rinfo_arguments.json:
+    if repo_info_arguments.json:
         return execute_command_and_capture_output(
             full_command,
-            extra_environment=extra_environment,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-    else:
-        execute_command(
-            full_command,
-            output_log_level=logging.ANSWER,
-            extra_environment=extra_environment,
+            environment=environment.make_environment(config),
+            working_directory=working_directory,
             borg_local_path=local_path,
             borg_exit_codes=borg_exit_codes,
         )
+
+    execute_command(
+        full_command,
+        output_log_level=logging.ANSWER,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
+        borg_local_path=local_path,
+        borg_exit_codes=borg_exit_codes,
+    )
+
+    return None

borgmatic/borg/repo_list.py

@@ -1,6 +1,8 @@
 import argparse
+import json
 import logging
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, feature, flags
 from borgmatic.execute import execute_command, execute_command_and_capture_output
@@ -21,38 +23,83 @@ def resolve_archive_name(
     Given a local or remote repository path, an archive name, a configuration dict, the local Borg
     version, global arguments as an argparse.Namespace, a local Borg path, and a remote Borg path,
     return the archive name. But if the archive name is "latest", then instead introspect the
-    repository for the latest archive and return its name.
+    repository for the latest archive and return its name or ID, depending on whether the version of
+    Borg in use supports archive series—different archives that share the same name but have unique
+    IDs.
 
     Raise ValueError if "latest" is given but there are no archives in the repository.
     '''
     if archive != 'latest':
         return archive
 
-    full_command = (
-        (
-            local_path,
-            'rlist' if feature.available(feature.Feature.RLIST, local_borg_version) else 'list',
-        )
-        + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
-        + flags.make_flags('lock-wait', config.get('lock_wait'))
-        + flags.make_flags('last', 1)
-        + ('--short',)
-        + flags.make_repository_flags(repository_path, local_borg_version)
+    latest_archive = get_latest_archive(
+        repository_path,
+        config,
+        local_borg_version,
+        global_arguments,
+        local_path=local_path,
+        remote_path=remote_path,
     )
 
-    output = execute_command_and_capture_output(
+    return (
+        latest_archive['id']
+        if feature.available(feature.Feature.ARCHIVE_SERIES, local_borg_version)
+        else latest_archive['name']
+    )
+
+
+def get_latest_archive(
+    repository_path,
+    config,
+    local_borg_version,
+    global_arguments,
+    local_path='borg',
+    remote_path=None,
+    consider_checkpoints=False,
+):
+    '''
+    Returns a dict with information about the latest archive of a repository.
+
+    Raises ValueError if there are no archives in the repository.
+    '''
+
+    full_command = (
+        local_path,
+        (
+            'repo-list'
+            if feature.available(feature.Feature.REPO_LIST, local_borg_version)
+            else 'list'
+        ),
+        *flags.make_flags('remote-path', remote_path),
+        *flags.make_flags('umask', config.get('umask')),
+        *flags.make_flags('log-json', config.get('log_json')),
+        *flags.make_flags('lock-wait', config.get('lock_wait')),
+        *(
+            flags.make_flags('consider-checkpoints', consider_checkpoints)
+            if not feature.available(feature.Feature.REPO_LIST, local_borg_version)
+            else ()
+        ),
+        *flags.make_flags('last', 1),
+        '--json',
+        *flags.make_repository_flags(repository_path, local_borg_version),
+    )
+
+    json_output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )
+
+    archives = json.loads(json_output)['archives']
+
     try:
-        latest_archive = output.strip().splitlines()[-1]
+        latest_archive = archives[-1]
     except IndexError:
         raise ValueError('No archives found in the repository')
 
-    logger.debug(f'{repository_path}: Latest archive is {latest_archive}')
+    logger.debug(f'Latest archive is {latest_archive["name"]} ({latest_archive["id"]})')
 
     return latest_archive
 
@@ -60,54 +107,59 @@ def resolve_archive_name(
 MAKE_FLAGS_EXCLUDES = ('repository', 'prefix', 'match_archives')
 
 
-def make_rlist_command(
+def make_repo_list_command(
     repository_path,
     config,
     local_borg_version,
-    rlist_arguments,
+    repo_list_arguments,
     global_arguments,
     local_path='borg',
     remote_path=None,
 ):
     '''
     Given a local or remote repository path, a configuration dict, the local Borg version, the
-    arguments to the rlist action, global arguments as an argparse.Namespace instance, and local and
+    arguments to the repo_list action, global arguments as an argparse.Namespace instance, and local and
     remote Borg paths, return a command as a tuple to list archives with a repository.
     '''
     return (
         (
             local_path,
-            'rlist' if feature.available(feature.Feature.RLIST, local_borg_version) else 'list',
+            (
+                'repo-list'
+                if feature.available(feature.Feature.REPO_LIST, local_borg_version)
+                else 'list'
+            ),
         )
         + (
             ('--info',)
-            if logger.getEffectiveLevel() == logging.INFO and not rlist_arguments.json
+            if logger.getEffectiveLevel() == logging.INFO and not repo_list_arguments.json
             else ()
         )
         + (
             ('--debug', '--show-rc')
-            if logger.isEnabledFor(logging.DEBUG) and not rlist_arguments.json
+            if logger.isEnabledFor(logging.DEBUG) and not repo_list_arguments.json
             else ()
         )
         + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
+        + flags.make_flags('umask', config.get('umask'))
+        + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
-                flags.make_flags('match-archives', f'sh:{rlist_arguments.prefix}*')
+                flags.make_flags('match-archives', f'sh:{repo_list_arguments.prefix}*')
                 if feature.available(feature.Feature.MATCH_ARCHIVES, local_borg_version)
-                else flags.make_flags('glob-archives', f'{rlist_arguments.prefix}*')
+                else flags.make_flags('glob-archives', f'{repo_list_arguments.prefix}*')
             )
-            if rlist_arguments.prefix
+            if repo_list_arguments.prefix
             else (
                 flags.make_match_archives_flags(
-                    rlist_arguments.match_archives or config.get('match_archives'),
+                    config.get('match_archives'),
                     config.get('archive_name_format'),
                     local_borg_version,
                 )
             )
         )
-        + flags.make_flags_from_arguments(rlist_arguments, excludes=MAKE_FLAGS_EXCLUDES)
+        + flags.make_flags_from_arguments(repo_list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
         + flags.make_repository_flags(repository_path, local_borg_version)
     )
 
@@ -116,7 +168,7 @@ def list_repository(
     repository_path,
     config,
     local_borg_version,
-    rlist_arguments,
+    repo_list_arguments,
     global_arguments,
     local_path='borg',
     remote_path=None,
@@ -128,36 +180,37 @@ def list_repository(
     return JSON output).
     '''
     borgmatic.logger.add_custom_log_levels()
-    borg_environment = environment.make_environment(config)
 
-    main_command = make_rlist_command(
+    main_command = make_repo_list_command(
         repository_path,
         config,
         local_borg_version,
-        rlist_arguments,
+        repo_list_arguments,
         global_arguments,
         local_path,
         remote_path,
     )
-    json_command = make_rlist_command(
+    json_command = make_repo_list_command(
         repository_path,
         config,
         local_borg_version,
-        argparse.Namespace(**dict(rlist_arguments.__dict__, json=True)),
+        argparse.Namespace(**dict(repo_list_arguments.__dict__, json=True)),
         global_arguments,
         local_path,
         remote_path,
     )
+    working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
     json_listing = execute_command_and_capture_output(
         json_command,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
     )
 
-    if rlist_arguments.json:
+    if repo_list_arguments.json:
         return json_listing
 
     flags.warn_for_aggressive_archive_flags(json_command, json_listing)
@@ -165,7 +218,10 @@ def list_repository(
     execute_command(
         main_command,
         output_log_level=logging.ANSWER,
-        extra_environment=borg_environment,
+        environment=environment.make_environment(config),
+        working_directory=working_directory,
         borg_local_path=local_path,
         borg_exit_codes=borg_exit_codes,
     )
+
+    return None

borgmatic/borg/transfer.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 import borgmatic.logger
 from borgmatic.borg import environment, flags
 from borgmatic.execute import DO_NOT_CAPTURE, execute_command
@@ -29,18 +30,24 @@ def transfer_archives(
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_flags('remote-path', remote_path)
-        + flags.make_flags('log-json', global_arguments.log_json)
-        + flags.make_flags('lock-wait', config.get('lock_wait', None))
+        + flags.make_flags('umask', config.get('umask'))
+        + flags.make_flags('log-json', config.get('log_json'))
+        + flags.make_flags('lock-wait', config.get('lock_wait'))
+        + flags.make_flags('progress', config.get('progress'))
         + (
             flags.make_flags_from_arguments(
                 transfer_arguments,
-                excludes=('repository', 'source_repository', 'archive', 'match_archives'),
+                excludes=(
+                    'repository',
+                    'source_repository',
+                    'archive',
+                    'match_archives',
+                    'progress',
+                ),
             )
             or (
                 flags.make_match_archives_flags(
-                    transfer_arguments.match_archives
-                    or transfer_arguments.archive
-                    or config.get('match_archives'),
+                    transfer_arguments.archive or config.get('match_archives'),
                     config.get('archive_name_format'),
                     local_borg_version,
                 )
@@ -54,8 +61,9 @@ def transfer_archives(
     return execute_command(
         full_command,
         output_log_level=logging.ANSWER,
-        output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
+        output_file=DO_NOT_CAPTURE if config.get('progress') else None,
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
-        extra_environment=environment.make_environment(config),
     )

borgmatic/borg/umount.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
@@ -18,5 +19,8 @@ def unmount_archive(config, mount_point, local_path='borg'):
     )
 
     execute_command(
-        full_command, borg_local_path=local_path, borg_exit_codes=config.get('borg_exit_codes')
+        full_command,
+        working_directory=borgmatic.config.paths.get_working_directory(config),
+        borg_local_path=local_path,
+        borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/borg/version.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.config.paths
 from borgmatic.borg import environment
 from borgmatic.execute import execute_command_and_capture_output
 
@@ -8,7 +9,7 @@ logger = logging.getLogger(__name__)
 
 def local_borg_version(config, local_path='borg'):
     '''
-    Given a configuration dict and a local Borg binary path, return a version string for it.
+    Given a configuration dict and a local Borg executable path, return a version string for it.
 
     Raise OSError or CalledProcessError if there is a problem running Borg.
     Raise ValueError if the version cannot be parsed.
@@ -20,7 +21,8 @@ def local_borg_version(config, local_path='borg'):
     )
     output = execute_command_and_capture_output(
         full_command,
-        extra_environment=environment.make_environment(config),
+        environment=environment.make_environment(config),
+        working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,
         borg_exit_codes=config.get('borg_exit_codes'),
     )

borgmatic/commands/completion/actions.py

@@ -22,15 +22,15 @@ def available_actions(subparsers, current_action=None):
     action of "config" but not "list".
     '''
     action_to_subactions = borgmatic.commands.arguments.get_subactions_for_actions(
-        subparsers.choices
+        subparsers.choices,
     )
     current_subactions = action_to_subactions.get(current_action)
 
     if current_subactions:
         return current_subactions
 
-    all_subactions = set(
+    all_subactions = {
         subaction for subactions in action_to_subactions.values() for subaction in subactions
-    )
+    }
 
-    return tuple(action for action in subparsers.choices.keys() if action not in all_subactions)
+    return tuple(action for action in subparsers.choices if action not in all_subactions)

borgmatic/commands/completion/bash.py

@@ -1,5 +1,7 @@
 import borgmatic.commands.arguments
 import borgmatic.commands.completion.actions
+import borgmatic.commands.completion.flag
+import borgmatic.config.validate
 
 
 def parser_flags(parser):
@@ -7,7 +9,12 @@ def parser_flags(parser):
     Given an argparse.ArgumentParser instance, return its argument flags in a space-separated
     string.
     '''
-    return ' '.join(option for action in parser._actions for option in action.option_strings)
+    return ' '.join(
+        flag_variant
+        for action in parser._actions
+        for flag_name in action.option_strings
+        for flag_variant in borgmatic.commands.completion.flag.variants(flag_name)
+    )
 
 
 def bash_completion():
@@ -16,13 +23,17 @@ def bash_completion():
     borgmatic's command-line argument parsers.
     '''
     (
-        unused_global_parser,
+        _,
         action_parsers,
         global_plus_action_parser,
-    ) = borgmatic.commands.arguments.make_parsers()
+    ) = borgmatic.commands.arguments.make_parsers(
+        schema=borgmatic.config.validate.load_schema(borgmatic.config.validate.schema_filename()),
+        unparsed_arguments=(),
+    )
     global_flags = parser_flags(global_plus_action_parser)
 
     # Avert your eyes.
+    # fmt: off
     return '\n'.join(
         (
             'check_version() {',
@@ -37,24 +48,22 @@ def bash_completion():
             '    fi',
             '}',
             'complete_borgmatic() {',
-        )
-        + tuple(
+            *tuple(
             '''    if [[ " ${COMP_WORDS[*]} " =~ " %s " ]]; then
         COMPREPLY=($(compgen -W "%s %s %s" -- "${COMP_WORDS[COMP_CWORD]}"))
         return 0
-    fi'''
+    fi'''  # noqa: UP031
             % (
                 action,
                 parser_flags(action_parser),
                 ' '.join(
-                    borgmatic.commands.completion.actions.available_actions(action_parsers, action)
+                    borgmatic.commands.completion.actions.available_actions(action_parsers, action),
                 ),
                 global_flags,
             )
             for action, action_parser in reversed(action_parsers.choices.items())
-        )
-        + (
-            '    COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'  # noqa: FS003
+        ),
+            '    COMPREPLY=($(compgen -W "%s %s" -- "${COMP_WORDS[COMP_CWORD]}"))'  # noqa: UP031
             % (
                 ' '.join(borgmatic.commands.completion.actions.available_actions(action_parsers)),
                 global_flags,
@@ -62,5 +71,5 @@ def bash_completion():
             '    (check_version &)',
             '}',
             '\ncomplete -o bashdefault -o default -F complete_borgmatic borgmatic',
-        )
+        ),
     )

borgmatic/commands/completion/fish.py

@@ -4,16 +4,17 @@ from textwrap import dedent
 
 import borgmatic.commands.arguments
 import borgmatic.commands.completion.actions
+import borgmatic.config.validate
 
 
 def has_file_options(action: Action):
     '''
     Given an argparse.Action instance, return True if it takes a file argument.
     '''
-    return action.metavar in (
+    return action.metavar in {
         'FILENAME',
         'PATH',
-    ) or action.dest in ('config_paths',)
+    } or action.dest in {'config_paths'}
 
 
 def has_choice_options(action: Action):
@@ -26,18 +27,20 @@ def has_choice_options(action: Action):
 def has_unknown_required_param_options(action: Action):
     '''
     A catch-all for options that take a required parameter, but we don't know what the parameter is.
-    This should be used last. These are actions that take something like a glob, a list of numbers, or a string.
+    This should be used last. These are actions that take something like a glob, a list of numbers,
+    or a string.
 
-    Actions that match this pattern should not show the normal arguments, because those are unlikely to be valid.
+    Actions that match this pattern should not show the normal arguments, because those are unlikely
+    to be valid.
     '''
     return (
         action.required is True
         or action.nargs
-        in (
+        in {
             '+',
             '*',
-        )
-        or action.metavar in ('PATTERN', 'KEYS', 'N')
+        }
+        or action.metavar in {'PATTERN', 'KEYS', 'N'}
         or (action.type is not None and action.default is None)
     )
 
@@ -52,9 +55,9 @@ def has_exact_options(action: Action):
 
 def exact_options_completion(action: Action):
     '''
-    Given an argparse.Action instance, return a completion invocation that forces file completions, options completion,
-    or just that some value follow the action, if the action takes such an argument and was the last action on the
-    command line prior to the cursor.
+    Given an argparse.Action instance, return a completion invocation that forces file completions,
+    options completion, or just that some value follow the action, if the action takes such an
+    argument and was the last action on the command line prior to the cursor.
 
     Otherwise, return an empty string.
     '''
@@ -74,14 +77,15 @@ def exact_options_completion(action: Action):
         return f'''\ncomplete -c borgmatic -x -n "__borgmatic_current_arg {args}"'''
 
     raise ValueError(
-        f'Unexpected action: {action} passes has_exact_options but has no choices produced'
+        f'Unexpected action: {action} passes has_exact_options but has no choices produced',
     )
 
 
 def dedent_strip_as_tuple(string: str):
     '''
-    Dedent a string, then strip it to avoid requiring your first line to have content, then return a tuple of the string.
-    Makes it easier to write multiline strings for completions when you join them with a tuple.
+    Dedent a string, then strip it to avoid requiring your first line to have content, then return a
+    tuple of the string. Makes it easier to write multiline strings for completions when you join
+    them with a tuple.
     '''
     return (dedent(string).strip('\n'),)
 
@@ -92,12 +96,15 @@ def fish_completion():
     borgmatic's command-line argument parsers.
     '''
     (
-        unused_global_parser,
+        _,
         action_parsers,
         global_plus_action_parser,
-    ) = borgmatic.commands.arguments.make_parsers()
+    ) = borgmatic.commands.arguments.make_parsers(
+        schema=borgmatic.config.validate.load_schema(borgmatic.config.validate.schema_filename()),
+        unparsed_arguments=(),
+    )
 
-    all_action_parsers = ' '.join(action for action in action_parsers.choices.keys())
+    all_action_parsers = ' '.join(action for action in action_parsers.choices)
 
     exact_option_args = tuple(
         ' '.join(action.option_strings)
@@ -112,8 +119,9 @@ def fish_completion():
     )
 
     # Avert your eyes.
-    return '\n'.join(
-        dedent_strip_as_tuple(
+    # fmt: off
+    return '\n'.join((
+        *dedent_strip_as_tuple(
             f'''
             function __borgmatic_check_version
                 set -fx this_filename (status current-filename)
@@ -150,27 +158,27 @@ def fish_completion():
 
             set --local action_parser_condition "not __fish_seen_subcommand_from {all_action_parsers}"
             set --local exact_option_condition "not __borgmatic_current_arg {' '.join(exact_option_args)}"
-            '''
-        )
-        + ('\n# action_parser completions',)
-        + tuple(
+            ''',
+        ),
+        '\n# action_parser completions',
+        *tuple(
             f'''complete -c borgmatic -f -n "$action_parser_condition" -n "$exact_option_condition" -a '{action_name}' -d {shlex.quote(action_parser.description)}'''
             for action_name, action_parser in action_parsers.choices.items()
-        )
-        + ('\n# global flags',)
-        + tuple(
+        ),
+        '\n# global flags',
+        *tuple(
             # -n is checked in order, so put faster / more likely to be true checks first
             f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)}{exact_options_completion(action)}'''
             for action in global_plus_action_parser._actions
             # ignore the noargs action, as this is an impossible completion for fish
             if len(action.option_strings) > 0
             if 'Deprecated' not in action.help
-        )
-        + ('\n# action_parser flags',)
-        + tuple(
+        ),
+        '\n# action_parser flags',
+        *tuple(
             f'''complete -c borgmatic -f -n "$exact_option_condition" -a '{' '.join(action.option_strings)}' -d {shlex.quote(action.help)} -n "__fish_seen_subcommand_from {action_name}"{exact_options_completion(action)}'''
             for action_name, action_parser in action_parsers.choices.items()
             for action in action_parser._actions
             if 'Deprecated' not in (action.help or ())
-        )
-    )
+        ),
+    ))

borgmatic/commands/completion/flag.py

@@ -0,0 +1,13 @@
+def variants(flag_name):
+    '''
+    Given a flag name as a string, yield it and any variations that should be complete-able as well.
+    For instance, for a string like "--foo[0].bar", yield "--foo[0].bar", "--foo[1].bar", ...,
+    "--foo[9].bar".
+    '''
+    if '[0]' in flag_name:
+        for index in range(10):
+            yield flag_name.replace('[0]', f'[{index}]')
+
+        return
+
+    yield flag_name

borgmatic/commands/generate_config.py

@@ -10,8 +10,8 @@ def main():
             levelno=logging.WARNING,
             levelname='WARNING',
             msg='generate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config generate" instead.',
-        )
+        ),
     )
 
-    sys.argv = ['borgmatic', 'config', 'generate'] + sys.argv[1:]
+    sys.argv = ['borgmatic', 'config', 'generate', *sys.argv[1:]]
     borgmatic.commands.borgmatic.main([warning_log])

borgmatic/commands/validate_config.py

@@ -10,8 +10,8 @@ def main():
             levelno=logging.WARNING,
             levelname='WARNING',
             msg='validate-borgmatic-config is deprecated and will be removed from a future release. Please use "borgmatic config validate" instead.',
-        )
+        ),
     )
 
-    sys.argv = ['borgmatic', 'config', 'validate'] + sys.argv[1:]
+    sys.argv = ['borgmatic', 'config', 'validate', *sys.argv[1:]]
     borgmatic.commands.borgmatic.main([warning_log])

borgmatic/config/arguments.py

@@ -0,0 +1,176 @@
+import io
+import re
+
+import ruamel.yaml
+
+import borgmatic.config.schema
+
+LIST_INDEX_KEY_PATTERN = re.compile(r'^(?P<list_name>[a-zA-z-]+)\[(?P<index>\d+)\]$')
+
+
+def set_values(config, keys, value):
+    '''
+    Given a configuration dict, a sequence of parsed key strings, and a string value, descend into
+    the configuration hierarchy based on the given keys and set the value into the right place.
+    For example, consider these keys:
+
+        ('foo', 'bar', 'baz')
+
+    This looks up "foo" in the given configuration dict. And within that, it looks up "bar". And
+    then within that, it looks up "baz" and sets it to the given value. Another example:
+
+        ('mylist[0]', 'foo')
+
+    This looks for the zeroth element of "mylist" in the given configuration. And within that, it
+    looks up "foo" and sets it to the given value.
+    '''
+    if not keys:
+        return
+
+    first_key = keys[0]
+
+    # Support "mylist[0]" list index syntax.
+    match = LIST_INDEX_KEY_PATTERN.match(first_key)
+
+    if match:
+        list_key = match.group('list_name')
+        list_index = int(match.group('index'))
+
+        try:
+            if len(keys) == 1:
+                config[list_key][list_index] = value
+
+                return
+
+            if list_key not in config:
+                config[list_key] = []
+
+            set_values(config[list_key][list_index], keys[1:], value)
+        except (IndexError, KeyError):
+            raise ValueError(f'Argument list index {first_key} is out of range')
+
+        return
+
+    if len(keys) == 1:
+        config[first_key] = value
+
+        return
+
+    if first_key not in config:
+        config[first_key] = {}
+
+    set_values(config[first_key], keys[1:], value)
+
+
+def type_for_option(schema, option_keys):
+    '''
+    Given a configuration schema dict and a sequence of keys identifying a potentially nested
+    option, e.g. ('extra_borg_options', 'create'), return the schema type of that option as a
+    string.
+
+    Return None if the option or its type cannot be found in the schema.
+    '''
+    option_schema = schema
+
+    for key in option_keys:
+        # Support "name[0]"-style list index syntax.
+        match = LIST_INDEX_KEY_PATTERN.match(key)
+        properties = borgmatic.config.schema.get_properties(option_schema)
+
+        try:
+            if match:
+                option_schema = properties[match.group('list_name')]['items']
+            else:
+                option_schema = properties[key]
+        except KeyError:
+            return None
+
+    try:
+        return option_schema['type']
+    except KeyError:
+        return None
+
+
+def convert_value_type(value, option_type):
+    '''
+    Given a string value and its schema type as a string, determine its logical type (string,
+    boolean, integer, etc.), and return it converted to that type.
+
+    If the destination option type is a string, then leave the value as-is so that special
+    characters in it don't get interpreted as YAML during conversion.
+
+    And if the source value isn't a string, return it as-is.
+
+    Raise ruamel.yaml.error.YAMLError if there's a parse issue with the YAML.
+    Raise ValueError if the parsed value doesn't match the option type.
+    '''
+    if not isinstance(value, str):
+        return value
+
+    if option_type == 'string':
+        return value
+
+    try:
+        parsed_value = ruamel.yaml.YAML(typ='safe').load(io.StringIO(value))
+    except ruamel.yaml.error.YAMLError as error:
+        raise ValueError(f'Argument value "{value}" is invalid: {error.problem}')
+
+    if not isinstance(parsed_value, borgmatic.config.schema.parse_type(option_type)):
+        raise ValueError(f'Argument value "{value}" is not of the expected type: {option_type}')
+
+    return parsed_value
+
+
+def prepare_arguments_for_config(global_arguments, schema):
+    '''
+    Given global arguments as an argparse.Namespace and a configuration schema dict, parse each
+    argument that corresponds to an option in the schema and return a sequence of tuples (keys,
+    values) for that option, where keys is a sequence of strings. For instance, given the following
+    arguments:
+
+        argparse.Namespace(**{'my_option.sub_option': 'value1', 'other_option': 'value2'})
+
+    ... return this:
+
+        (
+            (('my_option', 'sub_option'), 'value1'),
+            (('other_option',), 'value2'),
+        )
+    '''
+    prepared_values = []
+
+    for argument_name, value in global_arguments.__dict__.items():
+        if value is None:
+            continue
+
+        keys = tuple(argument_name.split('.'))
+        option_type = type_for_option(schema, keys)
+
+        # The argument doesn't correspond to any option in the schema, or it is a complex argument, so ignore it.
+        # It's probably a flag that borgmatic has on the command-line but not in configuration.
+        if option_type in {'object', None}:
+            continue
+
+        prepared_values.append(
+            (
+                keys,
+                convert_value_type(value, option_type),
+            ),
+        )
+
+    return tuple(prepared_values)
+
+
+def apply_arguments_to_config(config, schema, arguments):
+    '''
+    Given a configuration dict, a corresponding configuration schema dict, and arguments as a dict
+    from action name to argparse.Namespace, set those given argument values into their corresponding
+    configuration options in the configuration dict.
+
+    This supports argument flags of the from "--foo.bar.baz" where each dotted component is a nested
+    configuration object. Additionally, flags like "--foo.bar[0].baz" are supported to update a list
+    element in the configuration.
+    '''
+    for action_arguments in arguments.values():
+        for keys, value in prepare_arguments_for_config(action_arguments, schema):
+            set_values(config, keys, value)

borgmatic/config/collect.py

@@ -49,6 +49,6 @@ def collect_config_filenames(config_paths):
 
         for filename in sorted(os.listdir(path)):
             full_filename = os.path.join(path, filename)
-            matching_filetype = full_filename.endswith('.yaml') or full_filename.endswith('.yml')
+            matching_filetype = full_filename.endswith(('.yaml', '.yml'))
             if matching_filetype and not os.path.isdir(full_filename):
                 yield os.path.abspath(full_filename)

borgmatic/config/constants.py

@@ -1,3 +1,4 @@
+import contextlib
 import shlex
 
 
@@ -6,17 +7,18 @@ def coerce_scalar(value):
     Given a configuration value, coerce it to an integer or a boolean as appropriate and return the
     result.
     '''
-    try:
+    with contextlib.suppress(TypeError, ValueError):
         return int(value)
-    except (TypeError, ValueError):
-        pass
 
-    if value == 'true' or value == 'True':
-        return True
-    if value == 'false' or value == 'False':
-        return False
-
-    return value
+    try:
+        return {
+            'true': True,
+            'True': True,
+            'false': False,
+            'False': False,
+        }.get(value, value)
+    except TypeError:  # e.g. for an unhashable type
+        return value
 
 
 def apply_constants(value, constants, shell_escape=False):
@@ -55,8 +57,7 @@ def apply_constants(value, constants, shell_escape=False):
                 constants,
                 shell_escape=(
                     shell_escape
-                    or option_name.startswith('before_')
-                    or option_name.startswith('after_')
+                    or option_name.startswith(('before_', 'after_'))
                     or option_name == 'on_error'
                 ),
             )

borgmatic/config/environment.py

@@ -2,7 +2,7 @@ import os
 import re
 
 VARIABLE_PATTERN = re.compile(
-    r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})'
+    r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})',
 )
 
 

borgmatic/config/generate.py

@@ -1,10 +1,12 @@
 import collections
+import contextlib
 import io
 import os
 import re
 
 import ruamel.yaml
 
+import borgmatic.config.schema
 from borgmatic.config import load, normalize
 
 INDENT = 4
@@ -17,49 +19,74 @@ def insert_newline_before_comment(config, field_name):
     field and its comments.
     '''
     config.ca.items[field_name][1].insert(
-        0, ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None)
+        0,
+        ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None),
     )
 
 
-def get_properties(schema):
-    '''
-    Given a schema dict, return its properties. But if it's got sub-schemas with multiple different
-    potential properties, returned their merged properties instead.
-    '''
-    if 'oneOf' in schema:
-        return dict(
-            collections.ChainMap(*[sub_schema['properties'] for sub_schema in schema['oneOf']])
-        )
-
-    return schema['properties']
+SCALAR_SCHEMA_TYPES = {'string', 'boolean', 'integer', 'number'}
 
 
-def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
+def schema_to_sample_configuration(schema, source_config=None, level=0, parent_is_sequence=False):
     '''
-    Given a loaded configuration schema, generate and return sample config for it. Include comments
-    for each option based on the schema "description".
+    Given a loaded configuration schema and a source configuration, generate and return sample
+    config for the schema. Include comments for each option based on the schema "description".
+
+    If a source config is given, walk it alongside the given schema so that both can be taken into
+    account when commenting out particular options in add_comments_to_configuration_object().
     '''
     schema_type = schema.get('type')
     example = schema.get('example')
-    if example is not None:
-        return example
 
-    if schema_type == 'array':
+    if borgmatic.config.schema.compare_types(schema_type, {'array'}):
         config = ruamel.yaml.comments.CommentedSeq(
-            [schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
+            example
+            if borgmatic.config.schema.compare_types(
+                schema['items'].get('type'),
+                SCALAR_SCHEMA_TYPES,
+            )
+            else [
+                schema_to_sample_configuration(
+                    schema['items'],
+                    source_config,
+                    level,
+                    parent_is_sequence=True,
+                ),
+            ],
         )
         add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
-    elif schema_type == 'object':
-        config = ruamel.yaml.comments.CommentedMap(
-            [
-                (field_name, schema_to_sample_configuration(sub_schema, level + 1))
-                for field_name, sub_schema in get_properties(schema).items()
-            ]
+    elif borgmatic.config.schema.compare_types(schema_type, {'object'}):
+        if source_config and isinstance(source_config, list) and isinstance(source_config[0], dict):
+            source_config = source_config[0]
+
+        config = (
+            ruamel.yaml.comments.CommentedMap(
+                [
+                    (
+                        field_name,
+                        schema_to_sample_configuration(
+                            sub_schema,
+                            (source_config or {}).get(field_name, {}),
+                            level + 1,
+                        ),
+                    )
+                    for field_name, sub_schema in borgmatic.config.schema.get_properties(
+                        schema,
+                    ).items()
+                ],
+            )
+            or example
         )
         indent = (level * INDENT) + (SEQUENCE_INDENT if parent_is_sequence else 0)
         add_comments_to_configuration_object(
-            config, schema, indent=indent, skip_first=parent_is_sequence
+            config,
+            schema,
+            source_config,
+            indent=indent,
+            skip_first_field=parent_is_sequence,
         )
+    elif borgmatic.config.schema.compare_types(schema_type, SCALAR_SCHEMA_TYPES, match=all):
+        return example
     else:
         raise ValueError(f'Schema at level {level} is unsupported: {schema}')
 
@@ -93,17 +120,26 @@ def comment_out_optional_configuration(rendered_config):
     '''
     lines = []
     optional = False
+    indent_characters = None
+    indent_characters_at_sentinel = None
 
     for line in rendered_config.split('\n'):
+        indent_characters = len(line) - len(line.lstrip())
+
         # Upon encountering an optional configuration option, comment out lines until the next blank
         # line.
         if line.strip().startswith(f'# {COMMENTED_OUT_SENTINEL}'):
             optional = True
+            indent_characters_at_sentinel = indent_characters
             continue
 
-        # Hit a blank line, so reset commenting.
-        if not line.strip():
+        # Hit a blank line or dedented, so reset commenting.
+        if not line.strip() or (
+            indent_characters_at_sentinel is not None
+            and indent_characters < indent_characters_at_sentinel
+        ):
             optional = False
+            indent_characters_at_sentinel = None
 
         lines.append(comment_out_line(line) if optional else line)
 
@@ -130,15 +166,13 @@ def write_configuration(config_filename, rendered_config, mode=0o600, overwrite=
     '''
     if not overwrite and os.path.exists(config_filename):
         raise FileExistsError(
-            f'{config_filename} already exists. Aborting. Use --overwrite to replace the file.'
+            f'{config_filename} already exists. Aborting. Use --overwrite to replace the file.',
         )
 
-    try:
+    with contextlib.suppress(FileExistsError, FileNotFoundError):
         os.makedirs(os.path.dirname(config_filename), mode=0o700)
-    except (FileExistsError, FileNotFoundError):
-        pass
 
-    with open(config_filename, 'w') as config_file:
+    with open(config_filename, 'w', encoding='utf-8') as config_file:
         config_file.write(rendered_config)
 
     os.chmod(config_filename, mode)
@@ -163,8 +197,8 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
     if schema['items'].get('type') != 'object':
         return
 
-    for field_name in config[0].keys():
-        field_schema = get_properties(schema['items']).get(field_name, {})
+    for field_name in config[0]:
+        field_schema = borgmatic.config.schema.get_properties(schema['items']).get(field_name, {})
         description = field_schema.get('description')
 
         # No description to use? Skip it.
@@ -178,28 +212,44 @@ def add_comments_to_configuration_sequence(config, schema, indent=0):
         return
 
 
-REQUIRED_KEYS = {'source_directories', 'repositories', 'keep_daily'}
+DEFAULT_KEYS = {'source_directories', 'repositories', 'keep_daily'}
 COMMENTED_OUT_SENTINEL = 'COMMENT_OUT'
 
 
-def add_comments_to_configuration_object(config, schema, indent=0, skip_first=False):
+def add_comments_to_configuration_object(
+    config,
+    schema,
+    source_config=None,
+    indent=0,
+    skip_first_field=False,
+):
     '''
     Using descriptions from a schema as a source, add those descriptions as comments to the given
-    config mapping, before each field. Indent the comment the given number of characters.
+    configuration dict, putting them before each field. Indent the comment the given number of
+    characters.
+
+    If skip_first_field is True, omit the comment for the initial field. This is useful for
+    sequences, where the comment for the first field goes before the sequence itself.
+
+    And a sentinel for commenting out options that are neither in DEFAULT_KEYS nor the the given
+    source configuration dict. The idea is that any options used in the source configuration should
+    stay active in the generated configuration.
     '''
     for index, field_name in enumerate(config.keys()):
-        if skip_first and index == 0:
+        if skip_first_field and index == 0:
             continue
 
-        field_schema = get_properties(schema).get(field_name, {})
+        field_schema = borgmatic.config.schema.get_properties(schema).get(field_name, {})
         description = field_schema.get('description', '').strip()
 
-        # If this is an optional key, add an indicator to the comment flagging it to be commented
+        # If this isn't a default key, add an indicator to the comment, flagging it to be commented
         # out from the sample configuration. This sentinel is consumed by downstream processing that
         # does the actual commenting out.
-        if field_name not in REQUIRED_KEYS:
+        if field_name not in DEFAULT_KEYS and (
+            source_config is None or field_name not in source_config
+        ):
             description = (
-                '\n'.join((description, COMMENTED_OUT_SENTINEL))
+                f'{description}\n{COMMENTED_OUT_SENTINEL}'
                 if description
                 else COMMENTED_OUT_SENTINEL
             )
@@ -217,21 +267,6 @@ def add_comments_to_configuration_object(config, schema, indent=0, skip_first=Fa
 RUAMEL_YAML_COMMENTS_INDEX = 1
 
 
-def remove_commented_out_sentinel(config, field_name):
-    '''
-    Given a configuration CommentedMap and a top-level field name in it, remove any "commented out"
-    sentinel found at the end of its YAML comments. This prevents the given field name from getting
-    commented out by downstream processing that consumes the sentinel.
-    '''
-    try:
-        last_comment_value = config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX][-1].value
-    except KeyError:
-        return
-
-    if last_comment_value == f'# {COMMENTED_OUT_SENTINEL}\n':
-        config.ca.items[field_name][RUAMEL_YAML_COMMENTS_INDEX].pop()
-
-
 def merge_source_configuration_into_destination(destination_config, source_config):
     '''
     Deep merge the given source configuration dict into the destination configuration CommentedMap,
@@ -242,20 +277,16 @@ def merge_source_configuration_into_destination(destination_config, source_confi
     '''
     if not source_config:
         return destination_config
+
     if not destination_config or not isinstance(source_config, collections.abc.Mapping):
         return source_config
 
     for field_name, source_value in source_config.items():
-        # Since this key/value is from the source configuration, leave it uncommented and remove any
-        # sentinel that would cause it to get commented out.
-        remove_commented_out_sentinel(
-            ruamel.yaml.comments.CommentedMap(destination_config), field_name
-        )
-
         # This is a mapping. Recurse for this key/value.
         if isinstance(source_value, collections.abc.Mapping):
             destination_config[field_name] = merge_source_configuration_into_destination(
-                destination_config[field_name], source_value
+                destination_config[field_name],
+                source_value,
             )
             continue
 
@@ -269,18 +300,22 @@ def merge_source_configuration_into_destination(destination_config, source_confi
                         source_item,
                     )
                     for index, source_item in enumerate(source_value)
-                ]
+                ],
             )
             continue
 
         # This is some sort of scalar. Set it into the destination.
-        destination_config[field_name] = source_config[field_name]
+        destination_config[field_name] = source_value
 
     return destination_config
 
 
 def generate_sample_configuration(
-    dry_run, source_filename, destination_filename, schema_filename, overwrite=False
+    dry_run,
+    source_filename,
+    destination_filename,
+    schema_filename,
+    overwrite=False,
 ):
     '''
     Given an optional source configuration filename, and a required destination configuration
@@ -289,15 +324,22 @@ def generate_sample_configuration(
     schema. If a source filename is provided, merge the parsed contents of that configuration into
     the generated configuration.
     '''
-    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
+    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename, encoding='utf-8'))
     source_config = None
 
     if source_filename:
         source_config = load.load_configuration(source_filename)
         normalize.normalize(source_filename, source_config)
 
+        # The borgmatic.config.normalize.normalize() function tacks on an empty "bootstrap" if
+        # needed, so the hook gets used by default. But we don't want it to end up in the generated
+        # config unless the user has set it explicitly, as an empty "bootstrap:" won't validate.
+        if source_config and source_config.get('bootstrap') == {}:
+            del source_config['bootstrap']
+
     destination_config = merge_source_configuration_into_destination(
-        schema_to_sample_configuration(schema), source_config
+        schema_to_sample_configuration(schema, source_config),
+        source_config,
     )
 
     if dry_run:

borgmatic/config/load.py

@@ -31,7 +31,7 @@ def probe_and_include_file(filename, include_directories, config_paths):
             return load_configuration(candidate_filename, config_paths)
 
     raise FileNotFoundError(
-        f'Could not find include {filename} at {" or ".join(candidate_filenames)}'
+        f'Could not find include {filename} at {" or ".join(candidate_filenames)}',
     )
 
 
@@ -69,7 +69,7 @@ def include_configuration(loader, filename_node, include_directory, config_paths
         ]
 
     raise ValueError(
-        '!include value is not supported; use a single filename or a list of filenames'
+        'The value given for the !include tag is invalid; use a single filename or a list of filenames instead',
     )
 
 
@@ -85,7 +85,7 @@ def raise_retain_node_error(loader, node):
     '''
     if isinstance(node, (ruamel.yaml.nodes.MappingNode, ruamel.yaml.nodes.SequenceNode)):
         raise ValueError(
-            'The !retain tag may only be used within a configuration file containing a merged !include tag.'
+            'The !retain tag may only be used within a configuration file containing a merged !include tag.',
         )
 
     raise ValueError('The !retain tag may only be used on a mapping or list.')
@@ -100,7 +100,7 @@ def raise_omit_node_error(loader, node):
     tags are handled by deep_merge_nodes() below.
     '''
     raise ValueError(
-        'The !omit tag may only be used on a scalar (e.g., string) or list element within a configuration file containing a merged !include tag.'
+        'The !omit tag may only be used on a scalar (e.g., string) or list element within a configuration file containing a merged !include tag.',
     )
 
 
@@ -111,9 +111,13 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
     '''
 
     def __init__(
-        self, preserve_quotes=None, loader=None, include_directory=None, config_paths=None
+        self,
+        preserve_quotes=None,
+        loader=None,
+        include_directory=None,
+        config_paths=None,
     ):
-        super(Include_constructor, self).__init__(preserve_quotes, loader)
+        super().__init__(preserve_quotes, loader)
         self.add_constructor(
             '!include',
             functools.partial(
@@ -147,7 +151,7 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
         representer = ruamel.yaml.representer.SafeRepresenter()
 
         for index, (key_node, value_node) in enumerate(node.value):
-            if key_node.tag == u'tag:yaml.org,2002:merge' and value_node.tag == '!include':
+            if key_node.tag == 'tag:yaml.org,2002:merge' and value_node.tag == '!include':
                 # Replace the merge include with a sequence of included configuration nodes ready
                 # for merging. The construct_object() call here triggers include_configuration()
                 # among other constructors.
@@ -157,7 +161,7 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
                 )
 
         # This super().flatten_mapping() call actually performs "<<" merges.
-        super(Include_constructor, self).flatten_mapping(node)
+        super().flatten_mapping(node)
 
         node.value = deep_merge_nodes(node.value)
 
@@ -179,7 +183,7 @@ def load_configuration(filename, config_paths=None):
     # because yaml.Constructor has to be an actual class.)
     class Include_constructor_with_extras(Include_constructor):
         def __init__(self, preserve_quotes=None, loader=None):
-            super(Include_constructor_with_extras, self).__init__(
+            super().__init__(
                 preserve_quotes,
                 loader,
                 include_directory=os.path.dirname(filename),
@@ -190,7 +194,7 @@ def load_configuration(filename, config_paths=None):
     yaml.Constructor = Include_constructor_with_extras
     config_paths.add(filename)
 
-    with open(filename) as file:
+    with open(filename, encoding='utf-8') as file:
         return yaml.load(file.read())
 
 
@@ -318,17 +322,18 @@ def deep_merge_nodes(nodes):
 
     # Bucket the nodes by their keys. Then merge all of the values sharing the same key.
     for key_name, grouped_nodes in itertools.groupby(
-        sorted(nodes, key=get_node_key_name), get_node_key_name
+        sorted(nodes, key=get_node_key_name),
+        get_node_key_name,
     ):
-        grouped_nodes = list(grouped_nodes)
+        grouped_nodes = list(grouped_nodes)  # noqa: PLW2901
 
         # The merged node inherits its attributes from the final node in the group.
         (last_node_key, last_node_value) = grouped_nodes[-1]
-        value_types = set(type(value) for (_, value) in grouped_nodes)
+        value_types = {type(value) for (_, value) in grouped_nodes}
 
         if len(value_types) > 1:
             raise ValueError(
-                f'Incompatible types found when trying to merge "{key_name}:" values across configuration files: {", ".join(value_type.id for value_type in value_types)}'
+                f'Incompatible types found when trying to merge "{key_name}:" values across configuration files: {", ".join(value_type.id for value_type in value_types)}',
             )
 
         # If we're dealing with MappingNodes, recurse and merge its values as well.
@@ -351,7 +356,7 @@ def deep_merge_nodes(nodes):
                             comment=last_node_value.comment,
                             anchor=last_node_value.anchor,
                         ),
-                    )
+                    ),
                 )
 
             continue
@@ -374,7 +379,7 @@ def deep_merge_nodes(nodes):
                             comment=last_node_value.comment,
                             anchor=last_node_value.anchor,
                         ),
-                    )
+                    ),
                 )
 
             continue

borgmatic/config/normalize.py

@@ -25,12 +25,12 @@ def normalize_sections(config_filename, config):
         and location.get('prefix') != consistency.get('prefix')
     ):
         raise ValueError(
-            'The retention prefix and the consistency prefix cannot have different values (unless one is not set).'
+            'The retention prefix and the consistency prefix cannot have different values (unless one is not set).',
         )
 
     if storage.get('umask') and hooks.get('umask') and storage.get('umask') != hooks.get('umask'):
         raise ValueError(
-            'The storage umask and the hooks umask cannot have different values (unless one is not set).'
+            'The storage umask and the hooks umask cannot have different values (unless one is not set).',
         )
 
     any_section_upgraded = False
@@ -51,14 +51,98 @@ def normalize_sections(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: Configuration sections (like location:, storage:, retention:, consistency:, and hooks:) are deprecated and support will be removed from a future release. To prepare for this, move your options out of sections to the global scope.',
-                )
-            )
+                ),
+            ),
         ]
 
     return []
 
 
-def normalize(config_filename, config):
+def make_command_hook_deprecation_log(config_filename, option_name):  # pragma: no cover
+    '''
+    Given a configuration filename and the name of a configuration option, return a deprecation
+    warning log for it.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            levelno=logging.WARNING,
+            levelname='WARNING',
+            msg=f'{config_filename}: {option_name} is deprecated and support will be removed from a future release. Use commands: instead.',
+        ),
+    )
+
+
+def normalize_commands(config_filename, config):
+    '''
+    Given a configuration filename and a configuration dict, transform any "before_*"- and
+    "after_*"-style command hooks into "commands:".
+    '''
+    logs = []
+
+    # Normalize "before_actions" and "after_actions".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_actions'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'repository',
+                    'run': commands,
+                },
+            )
+
+    # Normalize "before_backup", "before_prune", "after_backup", "after_prune", etc.
+    for action_name in ('create', 'prune', 'compact', 'check', 'extract'):
+        for preposition in ('before', 'after'):
+            option_name = f'{preposition}_{"backup" if action_name == "create" else action_name}'
+            commands = config.pop(option_name, None)
+
+            if not commands:
+                continue
+
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'action',
+                    'when': [action_name],
+                    'run': commands,
+                },
+            )
+
+    # Normalize "on_error".
+    commands = config.pop('on_error', None)
+
+    if commands:
+        logs.append(make_command_hook_deprecation_log(config_filename, 'on_error'))
+        config.setdefault('commands', []).append(
+            {
+                'after': 'error',
+                'when': ['create', 'prune', 'compact', 'check'],
+                'run': commands,
+            },
+        )
+
+    # Normalize "before_everything" and "after_everything".
+    for preposition in ('before', 'after'):
+        option_name = f'{preposition}_everything'
+        commands = config.pop(option_name, None)
+
+        if commands:
+            logs.append(make_command_hook_deprecation_log(config_filename, option_name))
+            config.setdefault('commands', []).append(
+                {
+                    preposition: 'everything',
+                    'when': ['create'],
+                    'run': commands,
+                },
+            )
+
+    return logs
+
+
+def normalize(config_filename, config):  # noqa: PLR0912, PLR0915
     '''
     Given a configuration filename and a configuration dict of its loaded contents, apply particular
     hard-coded rules to normalize the configuration to adhere to the current schema. Return any log
@@ -67,6 +151,18 @@ def normalize(config_filename, config):
     Raise ValueError the configuration cannot be normalized.
     '''
     logs = normalize_sections(config_filename, config)
+    logs += normalize_commands(config_filename, config)
+
+    if config.get('borgmatic_source_directory'):
+        logs.append(
+            logging.makeLogRecord(
+                dict(
+                    levelno=logging.WARNING,
+                    levelname='WARNING',
+                    msg=f'{config_filename}: The borgmatic_source_directory option is deprecated and will be removed from a future release. Use user_runtime_directory and user_state_directory instead.',
+                ),
+            ),
+        )
 
     # Upgrade exclude_if_present from a string to a list.
     exclude_if_present = config.get('exclude_if_present')
@@ -77,11 +173,30 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The exclude_if_present option now expects a list value. String values for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['exclude_if_present'] = [exclude_if_present]
 
+    # Unconditionally set the bootstrap hook so that it's enabled by default and config files get
+    # stored in each Borg archive.
+    config.setdefault('bootstrap', {})
+
+    # Move store_config_files from the global scope to the bootstrap hook.
+    store_config_files = config.get('store_config_files')
+    if store_config_files is not None:
+        logs.append(
+            logging.makeLogRecord(
+                dict(
+                    levelno=logging.WARNING,
+                    levelname='WARNING',
+                    msg=f'{config_filename}: The store_config_files option has moved under the bootstrap hook. Specifying store_config_files at the global scope is deprecated and support will be removed from a future release.',
+                ),
+            ),
+        )
+        del config['store_config_files']
+        config['bootstrap']['store_config_files'] = store_config_files
+
     # Upgrade various monitoring hooks from a string to a dict.
     healthchecks = config.get('healthchecks')
     if isinstance(healthchecks, str):
@@ -91,8 +206,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The healthchecks hook now expects a key/value pair with "ping_url" as a key. String values for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['healthchecks'] = {'ping_url': healthchecks}
 
@@ -104,8 +219,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['cronitor'] = {'ping_url': cronitor}
 
@@ -117,8 +232,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['pagerduty'] = {'integration_key': pagerduty}
 
@@ -130,8 +245,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The healthchecks hook now expects key/value pairs. String values for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['cronhub'] = {'ping_url': cronhub}
 
@@ -144,8 +259,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The checks option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['checks'] = [{'name': check_type} for check_type in checks]
 
@@ -158,8 +273,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The numeric_owner option has been renamed to numeric_ids. numeric_owner is deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['numeric_ids'] = numeric_owner
 
@@ -171,8 +286,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The bsd_flags option has been renamed to flags. bsd_flags is deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['flags'] = bsd_flags
 
@@ -184,8 +299,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The remote_rate_limit option has been renamed to upload_rate_limit. remote_rate_limit is deprecated and support will be removed from a future release.',
-                )
-            )
+                ),
+            ),
         )
         config['upload_rate_limit'] = remote_rate_limit
 
@@ -199,8 +314,8 @@ def normalize(config_filename, config):
                         levelno=logging.WARNING,
                         levelname='WARNING',
                         msg=f'{config_filename}: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.',
-                    )
-                )
+                    ),
+                ),
             )
             config['repositories'] = [
                 {'path': repository} if isinstance(repository, str) else repository
@@ -211,7 +326,11 @@ def normalize(config_filename, config):
         config['repositories'] = []
 
         for repository_dict in repositories:
-            repository_path = repository_dict['path']
+            repository_path = repository_dict.get('path')
+
+            if repository_path is None:
+                continue
+
             if '~' in repository_path:
                 logs.append(
                     logging.makeLogRecord(
@@ -219,21 +338,22 @@ def normalize(config_filename, config):
                             levelno=logging.WARNING,
                             levelname='WARNING',
                             msg=f'{config_filename}: Repository paths containing "~" are deprecated in borgmatic and support will be removed from a future release.',
-                        )
-                    )
+                        ),
+                    ),
                 )
+
             if ':' in repository_path:
                 if repository_path.startswith('file://'):
                     updated_repository_path = os.path.abspath(
-                        repository_path.partition('file://')[-1]
+                        repository_path.partition('file://')[-1],
                     )
                     config['repositories'].append(
                         dict(
                             repository_dict,
                             path=updated_repository_path,
-                        )
+                        ),
                     )
-                elif repository_path.startswith('ssh://'):
+                elif repository_path.startswith(('ssh://', 'sftp://', 'rclone:', 's3:', 'b2:')):
                     config['repositories'].append(repository_dict)
                 else:
                     rewritten_repository_path = f"ssh://{repository_path.replace(':~', '/~').replace(':/', '/').replace(':', '/./')}"
@@ -242,15 +362,15 @@ def normalize(config_filename, config):
                             dict(
                                 levelno=logging.WARNING,
                                 levelname='WARNING',
-                                msg=f'{config_filename}: Remote repository paths without ssh:// syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
-                            )
-                        )
+                                msg=f'{config_filename}: Remote repository paths without ssh://, sftp://, rclone:, s3:, or b2:, syntax are deprecated and support will be removed from a future release. Interpreting "{repository_path}" as "{rewritten_repository_path}"',
+                            ),
+                        ),
                     )
                     config['repositories'].append(
                         dict(
                             repository_dict,
                             path=rewritten_repository_path,
-                        )
+                        ),
                     )
             else:
                 config['repositories'].append(repository_dict)
@@ -262,8 +382,8 @@ def normalize(config_filename, config):
                     levelno=logging.WARNING,
                     levelname='WARNING',
                     msg=f'{config_filename}: The prefix option is deprecated and support will be removed from a future release. Use archive_name_format or match_archives instead.',
-                )
-            )
+                ),
+            ),
         )
 
     return logs

borgmatic/config/override.py

@@ -1,7 +1,10 @@
 import io
+import logging
 
 import ruamel.yaml
 
+logger = logging.getLogger(__name__)
+
 
 def set_values(config, keys, value):
     '''
@@ -15,7 +18,7 @@ def set_values(config, keys, value):
     if len(keys) == 1:
         if isinstance(config, list):
             raise ValueError(
-                'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)'
+                'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)',
             )
 
         config[first_key] = value
@@ -66,11 +69,11 @@ def type_for_option(schema, option_keys):
     '''
     option_schema = schema
 
-    for key in option_keys:
-        try:
+    try:
+        for key in option_keys:
             option_schema = option_schema['properties'][key]
-        except KeyError:
-            return None
+    except KeyError:
+        return None
 
     try:
         return option_schema['type']
@@ -100,8 +103,8 @@ def parse_overrides(raw_overrides, schema):
 
     parsed_overrides = []
 
-    for raw_override in raw_overrides:
-        try:
+    try:
+        for raw_override in raw_overrides:
             raw_keys, value = raw_override.split('=', 1)
             keys = tuple(raw_keys.split('.'))
             option_type = type_for_option(schema, keys)
@@ -110,14 +113,14 @@ def parse_overrides(raw_overrides, schema):
                 (
                     keys,
                     convert_value_type(value, option_type),
-                )
+                ),
             )
-        except ValueError:
-            raise ValueError(
-                f"Invalid override '{raw_override}'. Make sure you use the form: OPTION=VALUE or OPTION.SUBOPTION=VALUE"
-            )
-        except ruamel.yaml.error.YAMLError as error:
-            raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
+    except ValueError:
+        raise ValueError(
+            f"Invalid override '{raw_override}'. Make sure you use the form: OPTION=VALUE or OPTION.SUBOPTION=VALUE",
+        )
+    except ruamel.yaml.error.YAMLError as error:
+        raise ValueError(f"Invalid override '{raw_override}': {error.problem}")
 
     return tuple(parsed_overrides)
 
@@ -134,6 +137,11 @@ def apply_overrides(config, schema, raw_overrides):
     '''
     overrides = parse_overrides(raw_overrides, schema)
 
+    if overrides:
+        logger.warning(
+            "The --override flag is deprecated and will be removed from a future release. Instead, use a command-line flag corresponding to the configuration option you'd like to set.",
+        )
+
     for keys, value in overrides:
         set_values(config, keys, value)
         set_values(config, strip_section_names(keys), value)

borgmatic/config/paths.py

@@ -0,0 +1,181 @@
+import contextlib
+import logging
+import os
+import tempfile
+
+logger = logging.getLogger(__name__)
+
+
+def expand_user_in_path(path):
+    '''
+    Given a directory path, expand any tildes in it.
+    '''
+    try:
+        return os.path.expanduser(path or '') or None
+    except TypeError:
+        return None
+
+
+def get_working_directory(config):  # pragma: no cover
+    '''
+    Given a configuration dict, get the working directory from it, expanding any tildes.
+    '''
+    return expand_user_in_path(config.get('working_directory'))
+
+
+def get_borgmatic_source_directory(config):
+    '''
+    Given a configuration dict, get the (deprecated) borgmatic source directory, expanding any
+    tildes. Defaults to ~/.borgmatic.
+    '''
+    return expand_user_in_path(config.get('borgmatic_source_directory') or '~/.borgmatic')
+
+
+TEMPORARY_DIRECTORY_PREFIX = 'borgmatic-'
+
+
+def replace_temporary_subdirectory_with_glob(
+    path,
+    temporary_directory_prefix=TEMPORARY_DIRECTORY_PREFIX,
+):
+    '''
+    Given an absolute temporary directory path and an optional temporary directory prefix, look for
+    a subdirectory within it starting with the temporary directory prefix (or a default) and replace
+    it with an appropriate glob. For instance, given:
+
+        /tmp/borgmatic-aet8kn93/borgmatic
+
+    ... replace it with:
+
+        /tmp/borgmatic-*/borgmatic
+
+    This is useful for finding previous temporary directories from prior borgmatic runs.
+    '''
+    return os.path.join(
+        '/',
+        *(
+            (
+                f'{temporary_directory_prefix}*'
+                if subdirectory.startswith(temporary_directory_prefix)
+                else subdirectory
+            )
+            for subdirectory in path.split(os.path.sep)
+        ),
+    )
+
+
+class Runtime_directory:
+    '''
+    A Python context manager for creating and cleaning up the borgmatic runtime directory used for
+    storing temporary runtime data like streaming database dumps and bootstrap metadata.
+
+    Example use as a context manager:
+
+        with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
+            do_something_with(borgmatic_runtime_directory)
+
+    For the scope of that "with" statement, the runtime directory is available. Afterwards, it
+    automatically gets cleaned up as necessary.
+    '''
+
+    def __init__(self, config):
+        '''
+        Given a configuration dict determine the borgmatic runtime directory, creating a secure,
+        temporary directory within it if necessary. Defaults to $XDG_RUNTIME_DIR/./borgmatic or
+        $RUNTIME_DIRECTORY/./borgmatic or $TMPDIR/borgmatic-[random]/./borgmatic or
+        $TEMP/borgmatic-[random]/./borgmatic or /tmp/borgmatic-[random]/./borgmatic where "[random]"
+        is a randomly generated string intended to avoid path collisions.
+
+        If XDG_RUNTIME_DIR or RUNTIME_DIRECTORY is set and already ends in "/borgmatic", then don't
+        tack on a second "/borgmatic" path component.
+
+        The "/./" is taking advantage of a Borg feature such that the part of the path before the "/./"
+        does not get stored in the file path within an archive. That way, the path of the runtime
+        directory can change without leaving database dumps within an archive inaccessible.
+        '''
+        runtime_directory = (
+            config.get('user_runtime_directory')
+            or os.environ.get('XDG_RUNTIME_DIR')  # Set by PAM on Linux.
+            or os.environ.get('RUNTIME_DIRECTORY')  # Set by systemd if configured.
+        )
+
+        if runtime_directory:
+            if not runtime_directory.startswith(os.path.sep):
+                raise ValueError('The runtime directory must be an absolute path')
+
+            self.temporary_directory = None
+        else:
+            base_directory = (
+                os.environ.get('TMPDIR') or os.environ.get('TEMP') or '/tmp'  # noqa: S108
+            )
+
+            if not base_directory.startswith(os.path.sep):
+                raise ValueError('The temporary directory must be an absolute path')
+
+            os.makedirs(base_directory, mode=0o700, exist_ok=True)
+            self.temporary_directory = tempfile.TemporaryDirectory(
+                prefix=TEMPORARY_DIRECTORY_PREFIX,
+                dir=base_directory,
+            )
+            runtime_directory = self.temporary_directory.name
+
+        (base_path, final_directory) = os.path.split(runtime_directory.rstrip(os.path.sep))
+
+        self.runtime_path = expand_user_in_path(
+            os.path.join(
+                base_path if final_directory == 'borgmatic' else runtime_directory,
+                '.',  # Borg 1.4+ "slashdot" hack.
+                'borgmatic',
+            ),
+        )
+        os.makedirs(self.runtime_path, mode=0o700, exist_ok=True)
+
+        logger.debug(f'Using runtime directory {os.path.normpath(self.runtime_path)}')
+
+    def __enter__(self):
+        '''
+        Return the borgmatic runtime path as a string.
+        '''
+        return self.runtime_path
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Delete any temporary directory that was created as part of initialization.
+        '''
+        if self.temporary_directory:
+            # The cleanup() call errors if, for instance, there's still a
+            # mounted filesystem within the temporary directory. There's
+            # nothing we can do about that here, so swallow the error.
+            with contextlib.suppress(OSError):
+                self.temporary_directory.cleanup()
+
+
+def make_runtime_directory_glob(borgmatic_runtime_directory):
+    '''
+    Given a borgmatic runtime directory path, make a glob that would match that path, specifically
+    replacing any randomly generated temporary subdirectory with "*" since such a directory's name
+    changes on every borgmatic run.
+    '''
+    return os.path.join(
+        *(
+            '*' if subdirectory.startswith(TEMPORARY_DIRECTORY_PREFIX) else subdirectory
+            for subdirectory in os.path.normpath(borgmatic_runtime_directory).split(os.path.sep)
+        ),
+    )
+
+
+def get_borgmatic_state_directory(config):
+    '''
+    Given a configuration dict, get the borgmatic state directory used for storing borgmatic state
+    files like records of when checks last ran. Defaults to $XDG_STATE_HOME/borgmatic or
+    ~/.local/state/./borgmatic.
+    '''
+    return expand_user_in_path(
+        os.path.join(
+            config.get('user_state_directory')
+            or os.environ.get('XDG_STATE_HOME')
+            or os.environ.get('STATE_DIRECTORY')  # Set by systemd if configured.
+            or '~/.local/state',
+            'borgmatic',
+        ),
+    )

borgmatic/config/schema.py

@@ -0,0 +1,66 @@
+import decimal
+import itertools
+
+
+def get_properties(schema):
+    '''
+    Given a schema dict, return its properties. But if it's got sub-schemas with multiple different
+    potential properties, return their merged properties instead (interleaved so the first
+    properties of each sub-schema come first). The idea is that the user should see all possible
+    options even if they're not all possible together.
+    '''
+    if 'oneOf' in schema:
+        return dict(
+            item
+            for item in itertools.chain(
+                *itertools.zip_longest(
+                    *[sub_schema['properties'].items() for sub_schema in schema['oneOf']],
+                ),
+            )
+            if item is not None
+        )
+
+    return schema.get('properties', {})
+
+
+SCHEMA_TYPE_TO_PYTHON_TYPE = {
+    'array': list,
+    'boolean': bool,
+    'integer': int,
+    'number': decimal.Decimal,
+    'object': dict,
+    'string': str,
+}
+
+
+def parse_type(schema_type, **overrides):
+    '''
+    Given a schema type as a string, return the corresponding Python type.
+
+    If any overrides are given in the from of a schema type string to a Python type, then override
+    the default type mapping with them.
+
+    Raise ValueError if the schema type is unknown.
+    '''
+    try:
+        return dict(
+            SCHEMA_TYPE_TO_PYTHON_TYPE,
+            **overrides,
+        )[schema_type]
+    except KeyError:
+        raise ValueError(f'Unknown type in configuration schema: {schema_type}')
+
+
+def compare_types(schema_type, target_types, match=any):
+    '''
+    Given a schema type as a string or a list of strings (representing multiple types) and a set of
+    target type strings, return whether every schema type is in the set of target types.
+
+    If the schema type is a list of strings, use the given match function (such as any or all) to
+    compare elements. For instance, if match is given as all, then every element of the schema_type
+    list must be in the target types.
+    '''
+    if isinstance(schema_type, list):
+        return match(element_schema_type in target_types for element_schema_type in schema_type)
+
+    return schema_type in target_types

borgmatic/config/validate.py

@@ -1,9 +1,10 @@
+import fnmatch
 import os
 
 import jsonschema
 import ruamel.yaml
 
-import borgmatic.config
+import borgmatic.config.arguments
 from borgmatic.config import constants, environment, load, normalize, override
 
 
@@ -16,10 +17,22 @@ def schema_filename():
     '''
     schema_path = os.path.join(os.path.dirname(borgmatic.config.__file__), 'schema.yaml')
 
-    with open(schema_path):
+    with open(schema_path, encoding='utf-8'):
         return schema_path
 
 
+def load_schema(schema_path):  # pragma: no cover
+    '''
+    Given a schema filename path, load the schema and return it as a dict.
+
+    Raise Validation_error if the schema could not be parsed.
+    '''
+    try:
+        return load.load_configuration(schema_path)
+    except (ruamel.yaml.error.YAMLError, RecursionError) as error:
+        raise Validation_error(schema_path, (str(error),))
+
+
 def format_json_error_path_element(path_element):
     '''
     Given a path element into a JSON data structure, format it for display as a string.
@@ -83,12 +96,21 @@ def apply_logical_validation(config_filename, parsed_configuration):
             )
 
 
-def parse_configuration(config_filename, schema_filename, overrides=None, resolve_env=True):
+def parse_configuration(
+    config_filename,
+    schema_filename,
+    arguments,
+    overrides=None,
+    resolve_env=True,
+):
     '''
     Given the path to a config filename in YAML format, the path to a schema filename in a YAML
-    rendition of JSON Schema format, a sequence of configuration file override strings in the form
-    of "option.suboption=value", return the parsed configuration as a data structure of nested dicts
-    and lists corresponding to the schema. Example return value:
+    rendition of JSON Schema format, arguments as dict from action name to argparse.Namespace, a
+    sequence of configuration file override strings in the form of "option.suboption=value", and
+    whether to resolve environment variables, return the parsed configuration as a data structure of
+    nested dicts and lists corresponding to the schema. Example return value.
+
+    Example return value:
 
         {
             'source_directories': ['/home', '/etc'],
@@ -111,6 +133,7 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     except (ruamel.yaml.error.YAMLError, RecursionError) as error:
         raise Validation_error(config_filename, (str(error),))
 
+    borgmatic.config.arguments.apply_arguments_to_config(config, schema, arguments)
     override.apply_overrides(config, schema, overrides)
     constants.apply_constants(config, config.get('constants') if config else {})
 
@@ -123,11 +146,13 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
         validator = jsonschema.Draft7Validator(schema)
     except AttributeError:  # pragma: no cover
         validator = jsonschema.Draft4Validator(schema)
+
     validation_errors = tuple(validator.iter_errors(config))
 
     if validation_errors:
         raise Validation_error(
-            config_filename, tuple(format_json_error(error) for error in validation_errors)
+            config_filename,
+            tuple(format_json_error(error) for error in validation_errors),
         )
 
     apply_logical_validation(config_filename, config)
@@ -135,32 +160,53 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
     return config, config_paths, logs
 
 
-def normalize_repository_path(repository):
+def normalize_repository_path(repository, base=None):
     '''
     Given a repository path, return the absolute path of it (for local repositories).
+    Optionally, use a base path for resolving relative paths, e.g. to the configured working directory.
     '''
     # A colon in the repository could mean that it's either a file:// URL or a remote repository.
     # If it's a remote repository, we don't want to normalize it. If it's a file:// URL, we do.
     if ':' not in repository:
-        return os.path.abspath(repository)
-    elif repository.startswith('file://'):
-        return os.path.abspath(repository.partition('file://')[-1])
-    else:
-        return repository
+        return (
+            os.path.abspath(os.path.join(base, repository)) if base else os.path.abspath(repository)
+        )
+
+    if repository.startswith('file://'):
+        local_path = repository.partition('file://')[-1]
+        return (
+            os.path.abspath(os.path.join(base, local_path)) if base else os.path.abspath(local_path)
+        )
+
+    return repository
+
+
+def glob_match(first, second):
+    '''
+    Given two strings, return whether the first matches the second. Globs are
+    supported.
+    '''
+    if first is None or second is None:
+        return False
+
+    return fnmatch.fnmatch(first, second) or fnmatch.fnmatch(second, first)
 
 
 def repositories_match(first, second):
     '''
-    Given two repository dicts with keys 'path' (relative and/or absolute),
-    and 'label', or two repository paths, return whether they match.
+    Given two repository dicts with keys "path" (relative and/or absolute),
+    and "label", two repository paths as strings, or a mix of the two formats,
+    return whether they match. Globs are supported.
     '''
     if isinstance(first, str):
         first = {'path': first, 'label': first}
+
     if isinstance(second, str):
         second = {'path': second, 'label': second}
-    return (first.get('label') == second.get('label')) or (
-        normalize_repository_path(first.get('path'))
-        == normalize_repository_path(second.get('path'))
+
+    return glob_match(first.get('label'), second.get('label')) or glob_match(
+        normalize_repository_path(first.get('path')),
+        normalize_repository_path(second.get('path')),
     )
 
 
@@ -181,31 +227,8 @@ def guard_configuration_contains_repository(repository, configurations):
             for config in configurations.values()
             for config_repository in config['repositories']
             if repositories_match(config_repository, repository)
-        )
+        ),
     )
 
     if count == 0:
         raise ValueError(f'Repository "{repository}" not found in configuration files')
-
-
-def guard_single_repository_selected(repository, configurations):
-    '''
-    Given a repository path and a dict mapping from config filename to corresponding parsed config
-    dict, ensure either a single repository exists across all configuration files or a repository
-    path was given.
-    '''
-    if repository:
-        return
-
-    count = len(
-        tuple(
-            config_repository
-            for config in configurations.values()
-            for config_repository in config['repositories']
-        )
-    )
-
-    if count != 1:
-        raise ValueError(
-            "Can't determine which repository to use. Use --repository to disambiguate"
-        )

borgmatic/execute.py

@@ -1,11 +1,12 @@
 import collections
 import enum
 import logging
-import os
 import select
 import subprocess
 import textwrap
 
+import borgmatic.logger
+
 logger = logging.getLogger(__name__)
 
 
@@ -30,6 +31,7 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
     '''
     if exit_code is None:
         return Exit_status.STILL_RUNNING
+
     if exit_code == 0:
         return Exit_status.SUCCESS
 
@@ -41,12 +43,13 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
 
                 if treat_as == 'error':
                     logger.error(
-                        f'Treating exit code {exit_code} as an error, as per configuration'
+                        f'Treating exit code {exit_code} as an error, as per configuration',
                     )
                     return Exit_status.ERROR
-                elif treat_as == 'warning':
+
+                if treat_as == 'warning':
                     logger.warning(
-                        f'Treating exit code {exit_code} as a warning, as per configuration'
+                        f'Treating exit code {exit_code} as a warning, as per configuration',
                     )
                     return Exit_status.WARNING
 
@@ -101,7 +104,7 @@ def append_last_lines(last_lines, captured_output, line, output_log_level):
         logger.log(output_log_level, line)
 
 
-def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):
+def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):  # noqa: PLR0912
     '''
     Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
     process with the requested log level. Additionally, raise a CalledProcessError if a process
@@ -130,7 +133,7 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
     still_running = True
 
     # Log output for each process until they all exit.
-    while True:
+    while True:  # noqa: PLR1702
         if output_buffers:
             (ready_buffers, _, _) = select.select(output_buffers, [], [])
 
@@ -180,7 +183,7 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
             command = process.args.split(' ') if isinstance(process.args, str) else process.args
             exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
 
-            if exit_status in (Exit_status.ERROR, Exit_status.WARNING):
+            if exit_status in {Exit_status.ERROR, Exit_status.WARNING}:
                 # If an error occurs, include its output in the raised exception so that we don't
                 # inadvertently hide error output.
                 output_buffer = output_buffer_for_process(process, exclude_stdouts)
@@ -193,7 +196,10 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
                         break
 
                     append_last_lines(
-                        last_lines, captured_outputs[process], line, output_log_level=logging.ERROR
+                        last_lines,
+                        captured_outputs[process],
+                        line,
+                        output_log_level=logging.ERROR,
                     )
 
                 if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
@@ -208,7 +214,9 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
 
                 if exit_status == Exit_status.ERROR:
                     raise subprocess.CalledProcessError(
-                        exit_code, command_for_process(process), '\n'.join(last_lines)
+                        exit_code,
+                        command_for_process(process),
+                        '\n'.join(last_lines),
                     )
 
                 still_running = False
@@ -219,6 +227,8 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
             process: '\n'.join(output_lines) for process, output_lines in captured_outputs.items()
         }
 
+    return None
+
 
 SECRET_COMMAND_FLAG_NAMES = {'--password'}
 
@@ -241,6 +251,9 @@ def mask_command_secrets(full_command):
 MAX_LOGGED_COMMAND_LENGTH = 1000
 
 
+PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG = ('BORG_', 'PG', 'MARIADB_', 'MYSQL_')
+
+
 def log_command(full_command, input_file=None, output_file=None, environment=None):
     '''
     Log the given command (a sequence of command/argument strings), along with its input/output file
@@ -249,14 +262,21 @@ def log_command(full_command, input_file=None, output_file=None, environment=Non
     logger.debug(
         textwrap.shorten(
             ' '.join(
-                tuple(f'{key}=***' for key in (environment or {}).keys())
-                + mask_command_secrets(full_command)
+                tuple(
+                    f'{key}=***'
+                    for key in (environment or {})
+                    if any(
+                        key.startswith(prefix)
+                        for prefix in PREFIXES_OF_ENVIRONMENT_VARIABLES_TO_LOG
+                    )
+                )
+                + mask_command_secrets(full_command),
             ),
             width=MAX_LOGGED_COMMAND_LENGTH,
             placeholder=' ...',
         )
-        + (f" < {getattr(input_file, 'name', '')}" if input_file else '')
-        + (f" > {getattr(output_file, 'name', '')}" if output_file else '')
+        + (f" < {getattr(input_file, 'name', input_file)}" if input_file else '')
+        + (f" > {getattr(output_file, 'name', output_file)}" if output_file else ''),
     )
 
 
@@ -272,32 +292,32 @@ def execute_command(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
     run_to_completion=True,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
     given log level. If an open output file object is given, then write stdout to the file and only
     log stderr. If an open input file object is given, then read stdin from the file. If shell is
-    True, execute the command within a shell. If an extra environment dict is given, then use it to
-    augment the current environment, and pass the result into the command. If a working directory is
-    given, use that as the present working directory when running the command. If a Borg local path
-    is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
-    instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
-    dicts, then use that configuration to decide what's an error and what's a warning. If run to
-    completion is False, then return the process for the command without executing it to completion.
+    True, execute the command within a shell. If an environment variables dict is given, then pass
+    it into the command. If a working directory is given, use that as the present working directory
+    when running the command. If a Borg local path is given, and the command matches it (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning. If run to completion is False, then return the process for
+    the command without executing it to completion.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
-    process = subprocess.Popen(
+    process = subprocess.Popen(  # noqa: S603
         command,
         stdin=input_file,
         stdout=None if do_not_capture else (output_file or subprocess.PIPE),
@@ -305,52 +325,59 @@ def execute_command(
         shell=shell,
         env=environment,
         cwd=working_directory,
+        close_fds=close_fds,
     )
     if not run_to_completion:
         return process
 
-    log_outputs(
-        (process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        log_outputs(
+            (process,),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
+
+    return None
 
 
 def execute_command_and_capture_output(
     full_command,
+    input_file=None,
     capture_stderr=False,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If capture stderr is True, then capture and return stderr in addition to
-    stdout. If shell is True, execute the command within a shell. If an extra environment dict is
-    given, then use it to augment the current environment, and pass the result into the command. If
-    a working directory is given, use that as the present working directory when running the
-    command. If a Borg local path is given, and the command matches it (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    execute the command within a shell. If an environment variables dict is given, then pass it into
+    the command. If a working directory is given, use that as the present working directory when
+    running the command. If a Borg local path is given, and the command matches it (regardless of
+    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
-    log_command(full_command, environment=extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, environment=environment)
     command = ' '.join(full_command) if shell else full_command
 
     try:
-        output = subprocess.check_output(
+        output = subprocess.check_output(  # noqa: S603
             command,
+            stdin=input_file,
             stderr=subprocess.STDOUT if capture_stderr else None,
             shell=shell,
             env=environment,
             cwd=working_directory,
+            close_fds=close_fds,
         )
     except subprocess.CalledProcessError as error:
         if (
@@ -358,6 +385,7 @@ def execute_command_and_capture_output(
             == Exit_status.ERROR
         ):
             raise
+
         output = error.output
 
     return output.decode() if output is not None else None
@@ -370,10 +398,11 @@ def execute_command_with_processes(
     output_file=None,
     input_file=None,
     shell=False,
-    extra_environment=None,
+    environment=None,
     working_directory=None,
     borg_local_path=None,
     borg_exit_codes=None,
+    close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
     Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -384,24 +413,22 @@ def execute_command_with_processes(
     If an open output file object is given, then write stdout to the file and only log stderr. But
     if output log level is None, instead suppress logging and return the captured output for (only)
     the given command. If an open input file object is given, then read stdin from the file. If
-    shell is True, execute the command within a shell. If an extra environment dict is given, then
-    use it to augment the current environment, and pass the result into the command. If a working
-    directory is given, use that as the present working directory when running the command. If a
-    Borg local path is given, then for any matching command or process (regardless of arguments),
-    treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
-    sequence of exit code configuration dicts, then use that configuration to decide what's an error
-    and what's a warning.
+    shell is True, execute the command within a shell. If an environment variables dict is given,
+    then pass it into the command. If a working directory is given, use that as the present working
+    directory when running the command. If a Borg local path is given, then for any matching command
+    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    configuration to decide what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
     upstream process.
     '''
-    log_command(full_command, input_file, output_file, extra_environment)
-    environment = {**os.environ, **extra_environment} if extra_environment else None
+    log_command(full_command, input_file, output_file, environment)
     do_not_capture = bool(output_file is DO_NOT_CAPTURE)
     command = ' '.join(full_command) if shell else full_command
 
     try:
-        command_process = subprocess.Popen(
+        command_process = subprocess.Popen(  # noqa: S603
             command,
             stdin=input_file,
             stdout=None if do_not_capture else (output_file or subprocess.PIPE),
@@ -411,6 +438,7 @@ def execute_command_with_processes(
             shell=shell,
             env=environment,
             cwd=working_directory,
+            close_fds=close_fds,
         )
     except (subprocess.CalledProcessError, OSError):
         # Something has gone wrong. So vent each process' output buffer to prevent it from hanging.
@@ -419,15 +447,19 @@ def execute_command_with_processes(
             if process.poll() is None:
                 process.stdout.read(0)
                 process.kill()
+
         raise
 
-    captured_outputs = log_outputs(
-        tuple(processes) + (command_process,),
-        (input_file, output_file),
-        output_log_level,
-        borg_local_path,
-        borg_exit_codes,
-    )
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        captured_outputs = log_outputs(
+            (*processes, command_process),
+            (input_file, output_file),
+            output_log_level,
+            borg_local_path,
+            borg_exit_codes,
+        )
 
     if output_log_level is None:
         return captured_outputs.get(command_process)
+
+    return None

borgmatic/hooks/command.py

@@ -1,17 +1,35 @@
+import functools
 import logging
 import os
 import re
 import shlex
+import subprocess
+import sys
 
-from borgmatic import execute
+import borgmatic.execute
+import borgmatic.logger
 
 logger = logging.getLogger(__name__)
 
 
 SOFT_FAIL_EXIT_CODE = 75
+BORG_PLACEHOLDERS = {
+    '{hostname}',
+    '{fqdn}',
+    '{reverse-fqdn}',
+    '{now}',
+    '{utcnow}',
+    '{unixtime}',
+    '{user}',
+    '{pid}',
+    '{borgversion}',
+    '{borgmajor}',
+    '{borgminor}',
+    '{borgpatch}',
+}
 
 
-def interpolate_context(config_filename, hook_description, command, context):
+def interpolate_context(hook_description, command, context):
     '''
     Given a config filename, a hook description, a single hook command, and a dict of context
     names/values, interpolate the values by "{name}" into the command and return the result.
@@ -19,70 +37,236 @@ def interpolate_context(config_filename, hook_description, command, context):
     for name, value in context.items():
         command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
 
-    for unsupported_variable in re.findall(r'{\w+}', command):
-        logger.warning(
-            f"{config_filename}: Variable '{unsupported_variable}' is not supported in {hook_description} hook"
-        )
+    for unsupported_variable in re.findall(r'\{\w+\}', command):
+        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
+        # be a Borg placeholder, as Borg should hopefully consume it.
+        if unsupported_variable not in BORG_PLACEHOLDERS:
+            logger.warning(
+                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook',
+            )
 
     return command
 
 
-def execute_hook(commands, umask, config_filename, description, dry_run, **context):
+def make_environment(current_environment, sys_module=sys):
     '''
-    Given a list of hook commands to execute, a umask to execute with (or None), a config filename,
-    a hook description, and whether this is a dry run, run the given commands. Or, don't run them
-    if this is a dry run.
+    Given the existing system environment as a map from environment variable name to value, return a
+    copy of it, augmented with any extra environment variables that should be used when running
+    command hooks.
+    '''
+    environment = dict(current_environment)
+
+    # Detect whether we're running within a PyInstaller bundle. If so, set or clear LD_LIBRARY_PATH
+    # based on the value of LD_LIBRARY_PATH_ORIG. This prevents library version information errors.
+    if getattr(sys_module, 'frozen', False) and hasattr(sys_module, '_MEIPASS'):
+        environment['LD_LIBRARY_PATH'] = environment.get('LD_LIBRARY_PATH_ORIG', '')
+
+    return environment
+
+
+def filter_hooks(command_hooks, before=None, after=None, action_names=None, step_names=None, state_names=None):
+    '''
+    Given a sequence of command hook dicts from configuration and one or more filters (before name,
+    after name, a sequence of action names, a sequence of sub-action steps, and/or a sequence of
+    execution result state names), filter down the command hooks to just the ones that match the
+    given filters.
+    '''
+    return tuple(
+        hook_config
+        for hook_config in command_hooks or ()
+        for config_action_names in (hook_config.get('when'),)
+        for config_step_names in (hook_config.get('steps'),)
+        for config_state_names in (hook_config.get('states'),)
+        if before is None or hook_config.get('before') == before
+        if after is None or hook_config.get('after') == after
+        if action_names is None
+        or config_action_names is None
+        or set(config_action_names or ()).intersection(set(action_names))
+        if step_names is None
+        or config_step_names is None
+        or set(config_step_names or ()).intersection(set(step_names))
+        if state_names is None
+        or config_state_names is None
+        or set(config_state_names or ()).intersection(set(state_names))
+    )
+
+
+def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):  # noqa: PLR0912
+    '''
+    Given a sequence of command hook dicts from configuration, a umask to execute with (or None), a
+    working directory to execute with, and whether this is a dry run, run the commands for each
+    hook. Or don't run them if this is a dry run.
 
     The context contains optional values interpolated by name into the hook commands.
 
-    Raise ValueError if the umask cannot be parsed.
+    Raise ValueError if the umask cannot be parsed or a hook is invalid.
     Raise subprocesses.CalledProcessError if an error occurs in a hook.
     '''
-    if not commands:
-        logger.debug(f'{config_filename}: No commands to run for {description} hook')
-        return
+    borgmatic.logger.add_custom_log_levels()
 
     dry_run_label = ' (dry run; not actually running hooks)' if dry_run else ''
 
-    context['configuration_filename'] = config_filename
-    commands = [
-        interpolate_context(config_filename, description, command, context) for command in commands
-    ]
-
-    if len(commands) == 1:
-        logger.info(f'{config_filename}: Running command for {description} hook{dry_run_label}')
-    else:
-        logger.info(
-            f'{config_filename}: Running {len(commands)} commands for {description} hook{dry_run_label}',
+    for hook_config in command_hooks:
+        commands = hook_config.get('run')
+        when_description = (
+            f"{'/'.join(hook_config.get('when'))} " if hook_config.get('when') else ''
         )
 
-    if umask:
-        parsed_umask = int(str(umask), 8)
-        logger.debug(f'{config_filename}: Set hook umask to {oct(parsed_umask)}')
-        original_umask = os.umask(parsed_umask)
-    else:
-        original_umask = None
+        if 'before' in hook_config:
+            description = f'before {when_description}{hook_config.get("before")}'
+        elif 'after' in hook_config:
+            description = f'after {when_description}{hook_config.get("after")}'
+        else:
+            raise ValueError(f'Invalid hook configuration: {hook_config}')
 
-    try:
-        for command in commands:
-            if not dry_run:
-                execute.execute_command(
+        if not commands:
+            logger.debug(f'No commands to run for {description} hook')
+            continue
+
+        commands = [interpolate_context(description, command, context) for command in commands]
+
+        if len(commands) == 1:
+            logger.info(f'Running {description} command hook{dry_run_label}')
+        else:
+            logger.info(
+                f'Running {len(commands)} commands for {description} hook{dry_run_label}',
+            )
+
+        if umask:
+            parsed_umask = int(str(umask), 8)
+            logger.debug(f'Setting hook umask to {oct(parsed_umask)}')
+            original_umask = os.umask(parsed_umask)
+        else:
+            original_umask = None
+
+        try:
+            for command in commands:
+                if dry_run:
+                    continue
+
+                borgmatic.execute.execute_command(  # noqa: S604
                     [command],
                     output_log_level=(
-                        logging.ERROR if description == 'on-error' else logging.WARNING
+                        logging.ERROR if hook_config.get('after') == 'error' else logging.ANSWER
                     ),
                     shell=True,
+                    environment=make_environment(os.environ),
+                    working_directory=working_directory,
                 )
-    finally:
-        if original_umask:
-            os.umask(original_umask)
+        finally:
+            if original_umask:
+                os.umask(original_umask)
 
 
-def considered_soft_failure(config_filename, error):
+class Before_after_hooks:
+    '''
+    A Python context manager for executing command hooks both before and after the wrapped code.
+
+    Example use as a context manager:
+
+       with borgmatic.hooks.command.Before_after_hooks(
+           command_hooks=config.get('commands'),
+           before_after='do_stuff',
+           umask=config.get('umask'),
+           dry_run=dry_run,
+           action_names=['check'],
+           step_names=['spot'],
+       ):
+            do()
+            some()
+            stuff()
+
+    With that context manager in place, "before" command hooks execute before the wrapped code runs,
+    and "after" command hooks execute after the wrapped code completes.
+    '''
+
+    def __init__(
+        self,
+        command_hooks,
+        before_after,
+        umask,
+        working_directory,
+        dry_run,
+        action_names=None,
+        step_names=None,
+        **context,
+    ):
+        '''
+        Given a sequence of command hook configuration dicts, the before/after name, a umask to run
+        commands with, a working directory to run commands with, a dry run flag, a sequence of
+        action names, a sequence of sub-action step names, and any context for the executed
+        commands, save those data points for use below.
+        '''
+        self.command_hooks = command_hooks
+        self.before_after = before_after
+        self.umask = umask
+        self.working_directory = working_directory
+        self.dry_run = dry_run
+        self.action_names = action_names
+        self.step_names = step_names
+        self.context = context
+
+    def __enter__(self):
+        '''
+        Run the configured "before" command hooks that match the initialized data points.
+        '''
+        try:
+            execute_hooks(
+                borgmatic.hooks.command.filter_hooks(
+                    self.command_hooks,
+                    before=self.before_after,
+                    action_names=self.action_names,
+                    step_names=self.step_names,
+                ),
+                self.umask,
+                self.working_directory,
+                self.dry_run,
+                **self.context,
+            )
+        except (OSError, subprocess.CalledProcessError) as error:
+            if considered_soft_failure(error):
+                raise
+
+            # Trigger the after hook manually, since raising here will prevent it from being run
+            # otherwise.
+            self.__exit__(exception_type=type(error), exception=error, traceback=None)
+
+            raise ValueError(f'Error running before {self.before_after} hook: {error}')
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Run the configured "after" command hooks that match the initialized data points.
+        '''
+        try:
+            execute_hooks(
+                borgmatic.hooks.command.filter_hooks(
+                    self.command_hooks,
+                    after=self.before_after,
+                    action_names=self.action_names,
+                    step_names=self.step_names,
+                    state_names=['fail' if exception_type else 'finish'],
+                ),
+                self.umask,
+                self.working_directory,
+                self.dry_run,
+                **self.context,
+            )
+        except (OSError, subprocess.CalledProcessError) as error:
+            if considered_soft_failure(error):
+                raise
+
+            raise ValueError(f'Error running after {self.before_after} hook: {error}')
+
+
+@functools.cache
+def considered_soft_failure(error):
     '''
     Given a configuration filename and an exception object, return whether the exception object
     represents a subprocess.CalledProcessError with a return code of SOFT_FAIL_EXIT_CODE. If so,
     that indicates that the error is a "soft failure", and should not result in an error.
+
+    The results of this function are cached so that it can be called multiple times without logging
+    multiple times.
     '''
     exit_code = getattr(error, 'returncode', None)
     if exit_code is None:
@@ -90,7 +274,7 @@ def considered_soft_failure(config_filename, error):
 
     if exit_code == SOFT_FAIL_EXIT_CODE:
         logger.info(
-            f'{config_filename}: Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining actions',
+            f'Command hook exited with soft failure exit code ({SOFT_FAIL_EXIT_CODE}); skipping remaining repository actions',
         )
         return True
 

borgmatic/hooks/credential/container.py

@@ -0,0 +1,44 @@
+import logging
+import os
+import re
+
+logger = logging.getLogger(__name__)
+
+
+SECRET_NAME_PATTERN = re.compile(r'^\w+$')
+DEFAULT_SECRETS_DIRECTORY = '/run/secrets'
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a secret name to load, read the secret from the corresponding container secrets file
+    and return it.
+
+    Raise ValueError if the credential parameters is not one element, the secret name is invalid, or
+    the secret file cannot be read.
+    '''
+    try:
+        (secret_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid secret name: "{name}"')
+
+    if not SECRET_NAME_PATTERN.match(secret_name):
+        raise ValueError(f'Cannot load invalid secret name: "{secret_name}"')
+
+    try:
+        with open(
+            os.path.join(
+                config.get('working_directory', ''),
+                (hook_config or {}).get('secrets_directory', DEFAULT_SECRETS_DIRECTORY),
+                secret_name,
+            ),
+            encoding='utf-8',
+        ) as secret_file:
+            return secret_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load secret "{secret_name}" from file: {error.filename}')

borgmatic/hooks/credential/file.py

@@ -0,0 +1,33 @@
+import logging
+import os
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential path to load, load the credential from file and return it.
+
+    Raise ValueError if the credential parameters is not one element or the secret file cannot be
+    read.
+    '''
+    try:
+        (credential_path,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential: "{name}"')
+
+    expanded_credential_path = os.path.expanduser(credential_path)
+
+    try:
+        with open(
+            os.path.join(config.get('working_directory', ''), expanded_credential_path),
+            encoding='utf-8',
+        ) as credential_file:
+            return credential_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load credential file: {error.filename}')

borgmatic/hooks/credential/keepassxc.py

@@ -0,0 +1,45 @@
+import logging
+import os
+import shlex
+
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a KeePassXC database path and an attribute name to load, run keepassxc-cli to fetch
+    the corresponding KeePassXC credential and return it.
+
+    Raise ValueError if keepassxc-cli can't retrieve the credential.
+    '''
+    try:
+        (database_path, attribute_name) = credential_parameters
+    except ValueError:
+        raise ValueError(f'Invalid KeePassXC credential parameters: {credential_parameters}')
+
+    expanded_database_path = os.path.expanduser(database_path)
+
+    if not os.path.exists(expanded_database_path):
+        raise ValueError(f'KeePassXC database path does not exist: {database_path}')
+
+    # Build the keepassxc-cli command.
+    command = (
+        tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
+        + ('show', '--show-protected', '--attributes', 'Password')
+        + (
+            ('--key-file', hook_config['key_file'])
+            if hook_config and hook_config.get('key_file')
+            else ()
+        )
+        + (
+            ('--yubikey', hook_config['yubikey'])
+            if hook_config and hook_config.get('yubikey')
+            else ()
+        )
+        + (expanded_database_path, attribute_name)  # Ensure database and entry are last.
+    )
+
+    return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)

borgmatic/hooks/credential/parse.py

@@ -0,0 +1,127 @@
+import functools
+import re
+import shlex
+
+import borgmatic.hooks.dispatch
+
+IS_A_HOOK = False
+
+
+class Hash_adapter:
+    '''
+    A Hash_adapter instance wraps an unhashable object and pretends it's hashable. This is intended
+    for passing to a @functools.cache-decorated function to prevent it from complaining that an
+    argument is unhashable. It should only be used for arguments that you don't want to actually
+    impact the cache hashing, because Hash_adapter doesn't actually hash the object's contents.
+
+    Example usage:
+
+        @functools.cache
+        def func(a, b):
+            print(a, b.actual_value)
+            return a
+
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Calls func(), prints, and returns.
+        func(5, Hash_adapter({1: 2, 3: 4}))  # Hits the cache and just returns the value.
+        func(5, Hash_adapter({5: 6, 7: 8}))  # Also uses cache, since the Hash_adapter is ignored.
+
+    In the above function, the "b" value is one that has been wrapped with Hash_adappter, and
+    therefore "b.actual_value" is necessary to access the original value.
+    '''
+
+    def __init__(self, actual_value):
+        self.actual_value = actual_value
+
+    def __eq__(self, other):
+        return True
+
+    def __hash__(self):
+        return 0
+
+
+UNHASHABLE_TYPES = (dict, list, set)
+
+
+def cache_ignoring_unhashable_arguments(function):
+    '''
+    A function decorator that caches calls to the decorated function but ignores any unhashable
+    arguments when performing cache lookups. This is intended to be a drop-in replacement for
+    functools.cache.
+
+    Example usage:
+
+        @cache_ignoring_unhashable_arguments
+        def func(a, b):
+            print(a, b)
+            return a
+
+        func(5, {1: 2, 3: 4})  # Calls func(), prints, and returns.
+        func(5, {1: 2, 3: 4})  # Hits the cache and just returns the value.
+        func(5, {5: 6, 7: 8})  # Also uses cache, since the unhashable value (the dict) is ignored.
+    '''
+
+    @functools.cache
+    def cached_function(*args, **kwargs):
+        return function(
+            *(arg.actual_value if isinstance(arg, Hash_adapter) else arg for arg in args),
+            **{
+                key: value.actual_value if isinstance(value, Hash_adapter) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    @functools.wraps(function)
+    def wrapper_function(*args, **kwargs):
+        return cached_function(
+            *(Hash_adapter(arg) if isinstance(arg, UNHASHABLE_TYPES) else arg for arg in args),
+            **{
+                key: Hash_adapter(value) if isinstance(value, UNHASHABLE_TYPES) else value
+                for (key, value) in kwargs.items()
+            },
+        )
+
+    wrapper_function.cache_clear = cached_function.cache_clear
+
+    return wrapper_function
+
+
+CREDENTIAL_PATTERN = re.compile(r'\{credential( +(?P<hook_and_parameters>.*))?\}')
+
+
+@cache_ignoring_unhashable_arguments
+def resolve_credential(value, config):
+    '''
+    Given a configuration value containing a string like "{credential hookname credentialname}" and
+    a configuration dict, resolve the credential by calling the relevant hook to get the actual
+    credential value. If the given value does not actually contain a credential tag, then return it
+    unchanged.
+
+    Cache the value (ignoring the config for purposes of caching), so repeated calls to this
+    function don't need to load the credential repeatedly.
+
+    Raise ValueError if the config could not be parsed or the credential could not be loaded.
+    '''
+    if value is None:
+        return value
+
+    matcher = CREDENTIAL_PATTERN.match(value)
+
+    if not matcher:
+        return value
+
+    hook_and_parameters = matcher.group('hook_and_parameters')
+
+    if not hook_and_parameters:
+        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
+
+    (hook_name, *credential_parameters) = shlex.split(hook_and_parameters)
+
+    if not credential_parameters:
+        raise ValueError(f'Cannot load credential with invalid syntax "{value}"')
+
+    return borgmatic.hooks.dispatch.call_hook(
+        'load_credential',
+        config,
+        hook_name,
+        tuple(credential_parameters),
+    )

borgmatic/hooks/credential/systemd.py

@@ -0,0 +1,61 @@
+import logging
+import os
+import re
+import shlex
+
+import borgmatic.execute
+
+logger = logging.getLogger(__name__)
+
+
+CREDENTIAL_NAME_PATTERN = re.compile(r'^[\w.-]+$')
+
+
+def load_credential(hook_config, config, credential_parameters):
+    '''
+    Given the hook configuration dict, the configuration dict, and a credential parameters tuple
+    containing a credential name to load, read the credential from the corresponding systemd
+    credential file and return it.
+
+    Raise ValueError if the systemd CREDENTIALS_DIRECTORY environment variable is not set, the
+    credential name is invalid, or the credential file cannot be read.
+    '''
+    try:
+        (credential_name,) = credential_parameters
+    except ValueError:
+        name = ' '.join(credential_parameters)
+
+        raise ValueError(f'Cannot load invalid credential name: "{name}"')
+
+    if not CREDENTIAL_NAME_PATTERN.match(credential_name):
+        raise ValueError(f'Cannot load invalid credential name "{credential_name}"')
+
+    credentials_directory = os.environ.get('CREDENTIALS_DIRECTORY')
+
+    if not credentials_directory:
+        logger.debug(
+            f'Falling back to loading credential "{credential_name}" via systemd-creds because the systemd CREDENTIALS_DIRECTORY environment variable is not set'
+        )
+
+        command = (
+            *shlex.split((hook_config or {}).get('systemd_creds_command', 'systemd-creds')),
+            'decrypt',
+            os.path.join(
+                (hook_config or {}).get(
+                    'encrypted_credentials_directory', '/etc/credstore.encrypted'
+                ),
+                credential_name,
+            ),
+        )
+
+        return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+
+    try:
+        with open(
+            os.path.join(credentials_directory, credential_name), encoding='utf-8'
+        ) as credential_file:
+            return credential_file.read().rstrip(os.linesep)
+    except (FileNotFoundError, OSError) as error:
+        logger.warning(error)
+
+        raise ValueError(f'Cannot load credential "{credential_name}" from file: {error.filename}')

borgmatic/hooks/data_source/bootstrap.py

@@ -0,0 +1,136 @@
+import contextlib
+import glob
+import importlib
+import json
+import logging
+import os
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic configuration file
+    paths, the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    create a borgmatic manifest file to store the paths of the configuration files used to create
+    the archive. But skip this if the bootstrap store_config_files option is False or if this is a
+    dry run.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+    '''
+    if hook_config and hook_config.get('store_config_files') is False:
+        return []
+
+    borgmatic_manifest_path = os.path.join(
+        borgmatic_runtime_directory,
+        'bootstrap',
+        'manifest.json',
+    )
+
+    if dry_run:
+        return []
+
+    os.makedirs(os.path.dirname(borgmatic_manifest_path), exist_ok=True)
+
+    with open(borgmatic_manifest_path, 'w', encoding='utf-8') as manifest_file:
+        json.dump(
+            {
+                'borgmatic_version': importlib.metadata.version('borgmatic'),
+                'config_paths': config_paths,
+            },
+            manifest_file,
+        )
+
+    patterns.extend(
+        borgmatic.borg.pattern.Pattern(
+            config_path,
+            source=borgmatic.borg.pattern.Pattern_source.HOOK,
+        )
+        for config_path in config_paths
+    )
+    patterns.append(
+        borgmatic.borg.pattern.Pattern(
+            os.path.join(borgmatic_runtime_directory, 'bootstrap'),
+            source=borgmatic.borg.pattern.Pattern_source.HOOK,
+        ),
+    )
+
+    return []
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given a bootstrap configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, then remove the manifest file created above. If this is a dry run,
+    then don't actually remove anything.
+    '''
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    manifest_glob = os.path.join(
+        borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(borgmatic_runtime_directory),
+        ),
+        'bootstrap',
+    )
+    logger.debug(
+        f'Looking for bootstrap manifest files to remove in {manifest_glob}{dry_run_label}',
+    )
+
+    for manifest_directory in glob.glob(manifest_glob):
+        manifest_file_path = os.path.join(manifest_directory, 'manifest.json')
+        logger.debug(f'Removing bootstrap manifest at {manifest_file_path}{dry_run_label}')
+
+        if dry_run:
+            continue
+
+        with contextlib.suppress(FileNotFoundError):
+            os.remove(manifest_file_path)
+
+        with contextlib.suppress(FileNotFoundError):
+            os.rmdir(manifest_directory)
+
+
+def make_data_source_dump_patterns(
+    hook_config,
+    config,
+    borgmatic_runtime_directory,
+    name=None,
+):  # pragma: no cover
+    '''
+    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
+    generic "restore".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the
+    generic "restore".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/data_source/btrfs.py

@@ -0,0 +1,463 @@
+import collections
+import glob
+import itertools
+import json
+import logging
+import os
+import shutil
+import subprocess
+
+import borgmatic.borg.pattern
+import borgmatic.config.paths
+import borgmatic.execute
+import borgmatic.hooks.data_source.snapshot
+
+logger = logging.getLogger(__name__)
+
+
+def use_streaming(hook_config, config):  # pragma: no cover
+    '''
+    Return whether dump streaming is used for this hook. (Spoiler: It isn't.)
+    '''
+    return False
+
+
+def get_contained_subvolume_paths(btrfs_command, subvolume_path):
+    '''
+    Given the path of a Btrfs subvolume, return it in a sequence along with the paths of its
+    contained subvolumes.
+
+    If the btrfs command errors, log that error and return an empty sequence.
+    '''
+    try:
+        btrfs_output = borgmatic.execute.execute_command_and_capture_output(
+            (
+                *btrfs_command.split(' '),
+                'subvolume',
+                'list',
+                subvolume_path,
+            ),
+            close_fds=True,
+        )
+    except subprocess.CalledProcessError as error:
+        logger.debug(
+            f'Ignoring Btrfs subvolume {subvolume_path} because of error listing its subvolumes: {error}',
+        )
+
+        return ()
+
+    return (
+        subvolume_path,
+        *tuple(
+            os.path.join(subvolume_path, line.split(' ')[-1])
+            for line in btrfs_output.splitlines()
+            if line.strip()
+        ),
+    )
+
+
+FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION = 'subvolid=5'
+
+
+def get_all_subvolume_paths(btrfs_command, findmnt_command):
+    '''
+    Given btrfs and findmnt commands to run, get the sorted paths for all Btrfs subvolumes on the
+    system.
+    '''
+    findmnt_output = borgmatic.execute.execute_command_and_capture_output(
+        (
+            *findmnt_command.split(' '),
+            '-t',  # Filesystem type.
+            'btrfs',
+            '--json',
+            '--list',  # Request a flat list instead of a nested subvolume hierarchy.
+        ),
+        close_fds=True,
+    )
+
+    try:
+        return tuple(
+            sorted(
+                itertools.chain.from_iterable(
+                    # If findmnt gave us a Btrfs root filesystem, list the subvolumes within it.
+                    # This is necessary because findmnt only returns a subvolume's mount point
+                    # rather than its original subvolume path (which can differ). For instance,
+                    # a subvolume might exist at /mnt/subvolume but be mounted at /home/myuser.
+                    # findmnt is still useful though because it's a global way to discover all
+                    # Btrfs subvolumes—even if we have to do some additional legwork ourselves.
+                    (
+                        get_contained_subvolume_paths(btrfs_command, filesystem['target'])
+                        if FINDMNT_BTRFS_ROOT_SUBVOLUME_OPTION in filesystem['options'].split(',')
+                        else (filesystem['target'],)
+                    )
+                    for filesystem in json.loads(findmnt_output)['filesystems']
+                ),
+            ),
+        )
+    except json.JSONDecodeError as error:
+        raise ValueError(f'Invalid {findmnt_command} JSON output: {error}')
+    except KeyError as error:
+        raise ValueError(f'Invalid {findmnt_command} output: Missing key "{error}"')
+
+
+Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'), defaults=((),))
+
+
+def get_subvolume_property(btrfs_command, subvolume_path, property_name):
+    '''
+    Given a btrfs command, a subvolume path, and a property name to lookup, return the value of the
+    corresponding property.
+
+    Raise subprocess.CalledProcessError if the btrfs command errors.
+    '''
+    output = borgmatic.execute.execute_command_and_capture_output(
+        (
+            *btrfs_command.split(' '),
+            'property',
+            'get',
+            '-t',  # Type.
+            'subvol',
+            subvolume_path,
+            property_name,
+        ),
+        close_fds=True,
+    )
+
+    try:
+        value = output.strip().split('=')[1]
+    except IndexError:
+        raise ValueError(f'Invalid {btrfs_command} property output')
+
+    return {
+        'true': True,
+        'false': False,
+    }.get(value, value)
+
+
+def omit_read_only_subvolume_paths(btrfs_command, subvolume_paths):
+    '''
+    Given a Btrfs command to run and a sequence of Btrfs subvolume paths, filter them down to just
+    those that are read-write. The idea is that Btrfs can't actually snapshot a read-only subvolume,
+    so we should just ignore them.
+    '''
+    retained_subvolume_paths = []
+
+    for subvolume_path in subvolume_paths:
+        try:
+            if get_subvolume_property(btrfs_command, subvolume_path, 'ro'):
+                logger.debug(f'Ignoring Btrfs subvolume {subvolume_path} because it is read-only')
+            else:
+                retained_subvolume_paths.append(subvolume_path)
+        except subprocess.CalledProcessError as error:  # noqa: PERF203
+            logger.debug(
+                f'Error determining read-only status of Btrfs subvolume {subvolume_path}: {error}',
+            )
+
+    return tuple(retained_subvolume_paths)
+
+
+def get_subvolumes(btrfs_command, findmnt_command, patterns=None):
+    '''
+    Given a Btrfs command to run and a sequence of configured patterns, find the intersection
+    between the current Btrfs filesystem and subvolume paths and the paths of any patterns.  The
+    idea is that these pattern paths represent the requested subvolumes to snapshot.
+
+    Only include subvolumes that contain at least one root pattern sourced from borgmatic
+    configuration (as opposed to generated elsewhere in borgmatic). But if patterns is None, then
+    return all subvolumes instead, sorted by path.
+
+    Return the result as a sequence of matching Subvolume instances.
+    '''
+    candidate_patterns = set(patterns or ())
+    subvolumes = []
+
+    # For each subvolume path, match it against the given patterns to find the subvolumes to
+    # backup. Sort the subvolumes from longest to shortest mount points, so longer mount points get
+    # a whack at the candidate pattern piñata before their parents do. (Patterns are consumed during
+    # this process, so no two subvolumes end up with the same contained patterns.)
+    for subvolume_path in reversed(
+        omit_read_only_subvolume_paths(
+            btrfs_command,
+            get_all_subvolume_paths(btrfs_command, findmnt_command),
+        ),
+    ):
+        subvolumes.extend(
+            Subvolume(subvolume_path, contained_patterns)
+            for contained_patterns in (
+                borgmatic.hooks.data_source.snapshot.get_contained_patterns(
+                    subvolume_path,
+                    candidate_patterns,
+                ),
+            )
+            if patterns is None
+            or any(
+                pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+                and pattern.source == borgmatic.borg.pattern.Pattern_source.CONFIG
+                for pattern in contained_patterns
+            )
+        )
+
+    return tuple(sorted(subvolumes, key=lambda subvolume: subvolume.path))
+
+
+BORGMATIC_SNAPSHOT_PREFIX = '.borgmatic-snapshot-'
+
+
+def make_snapshot_path(subvolume_path):
+    '''
+    Given the path to a subvolume, make a corresponding snapshot path for it.
+    '''
+    return os.path.join(
+        subvolume_path,
+        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        # Included so that the snapshot ends up in the Borg archive at the "original" subvolume path.
+    ) + subvolume_path.rstrip(os.path.sep)
+
+
+def make_snapshot_exclude_pattern(subvolume_path):  # pragma: no cover
+    '''
+    Given the path to a subvolume, make a corresponding exclude pattern for its embedded snapshot
+    path. This is to work around a quirk of Btrfs: If you make a snapshot path as a child directory
+    of a subvolume, then the snapshot's own initial directory component shows up as an empty
+    directory within the snapshot itself. For instance, if you have a Btrfs subvolume at /mnt and
+    make a snapshot of it at:
+
+        /mnt/.borgmatic-snapshot-1234/mnt
+
+    ... then the snapshot itself will have an empty directory at:
+
+        /mnt/.borgmatic-snapshot-1234/mnt/.borgmatic-snapshot-1234
+
+    So to prevent that from ending up in the Borg archive, this function produces an exclude pattern
+    to exclude that path.
+    '''
+    snapshot_directory = f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}'
+
+    return borgmatic.borg.pattern.Pattern(
+        os.path.join(
+            subvolume_path,
+            snapshot_directory,
+            subvolume_path.lstrip(os.path.sep),
+            snapshot_directory,
+        ),
+        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+        borgmatic.borg.pattern.Pattern_style.FNMATCH,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
+    )
+
+
+def make_borg_snapshot_pattern(subvolume_path, pattern):
+    '''
+    Given the path to a subvolume and a pattern as a borgmatic.borg.pattern.Pattern instance whose
+    path is inside the subvolume, return a new Pattern with its path rewritten to be in a snapshot
+    path intended for giving to Borg.
+
+    Move any initial caret in a regular expression pattern path to the beginning, so as not to break
+    the regular expression.
+    '''
+    initial_caret = (
+        '^'
+        if pattern.style == borgmatic.borg.pattern.Pattern_style.REGULAR_EXPRESSION
+        and pattern.path.startswith('^')
+        else ''
+    )
+
+    rewritten_path = initial_caret + os.path.join(
+        subvolume_path,
+        f'{BORGMATIC_SNAPSHOT_PREFIX}{os.getpid()}',
+        # Use the Borg 1.4+ "slashdot" hack to prevent the snapshot path prefix from getting
+        # included in the archive—but only if there's not already a slashdot hack present in the
+        # pattern.
+        ('' if f'{os.path.sep}.{os.path.sep}' in pattern.path else '.'),
+        # Included so that the source directory ends up in the Borg archive at its "original" path.
+        pattern.path.lstrip('^').lstrip(os.path.sep),
+    )
+
+    return borgmatic.borg.pattern.Pattern(
+        rewritten_path,
+        pattern.type,
+        pattern.style,
+        pattern.device,
+        source=borgmatic.borg.pattern.Pattern_source.HOOK,
+    )
+
+
+def snapshot_subvolume(btrfs_command, subvolume_path, snapshot_path):  # pragma: no cover
+    '''
+    Given a Btrfs command to run, the path to a subvolume, and the path for a snapshot, create a new
+    Btrfs snapshot of the subvolume.
+    '''
+    os.makedirs(os.path.dirname(snapshot_path), mode=0o700, exist_ok=True)
+
+    borgmatic.execute.execute_command(
+        (
+            *btrfs_command.split(' '),
+            'subvolume',
+            'snapshot',
+            '-r',  # Read-only.
+            subvolume_path,
+            snapshot_path,
+        ),
+        output_log_level=logging.DEBUG,
+        close_fds=True,
+    )
+
+
+def dump_data_sources(
+    hook_config,
+    config,
+    config_paths,
+    borgmatic_runtime_directory,
+    patterns,
+    dry_run,
+):
+    '''
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic configuration file paths,
+    the borgmatic runtime directory, the configured patterns, and whether this is a dry run,
+    auto-detect and snapshot any Btrfs subvolume paths listed in the given patterns. Also update
+    those patterns, replacing subvolume paths with corresponding snapshot directories so they get
+    stored in the Borg archive instead.
+
+    Return an empty sequence, since there are no ongoing dump processes from this hook.
+
+    If this is a dry run, then don't actually snapshot anything.
+    '''
+    dry_run_label = ' (dry run; not actually snapshotting anything)' if dry_run else ''
+    logger.info(f'Snapshotting Btrfs subvolumes{dry_run_label}')
+
+    # Based on the configured patterns, determine Btrfs subvolumes to backup. Only consider those
+    # patterns that came from actual user configuration (as opposed to, say, other hooks).
+    btrfs_command = hook_config.get('btrfs_command', 'btrfs')
+    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
+    subvolumes = get_subvolumes(btrfs_command, findmnt_command, patterns)
+
+    if not subvolumes:
+        logger.warning(f'No Btrfs subvolumes found to snapshot{dry_run_label}')
+
+    # Snapshot each subvolume, rewriting patterns to use their snapshot paths.
+    for subvolume in subvolumes:
+        logger.debug(f'Creating Btrfs snapshot for {subvolume.path} subvolume')
+
+        snapshot_path = make_snapshot_path(subvolume.path)
+
+        if dry_run:
+            continue
+
+        snapshot_subvolume(btrfs_command, subvolume.path, snapshot_path)
+
+        for pattern in subvolume.contained_patterns:
+            snapshot_pattern = make_borg_snapshot_pattern(subvolume.path, pattern)
+
+            # Attempt to update the pattern in place, since pattern order matters to Borg.
+            try:
+                patterns[patterns.index(pattern)] = snapshot_pattern
+            except ValueError:
+                patterns.append(snapshot_pattern)
+
+        patterns.append(make_snapshot_exclude_pattern(subvolume.path))
+
+    return []
+
+
+def delete_snapshot(btrfs_command, snapshot_path):  # pragma: no cover
+    '''
+    Given a Btrfs command to run and the name of a snapshot path, delete it.
+    '''
+    borgmatic.execute.execute_command(
+        (
+            *btrfs_command.split(' '),
+            'subvolume',
+            'delete',
+            snapshot_path,
+        ),
+        output_log_level=logging.DEBUG,
+        close_fds=True,
+    )
+
+
+def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, dry_run):
+    '''
+    Given a Btrfs configuration dict, a configuration dict, the borgmatic runtime directory, and
+    whether this is a dry run, delete any Btrfs snapshots created by borgmatic. If this is a dry run
+    or Btrfs isn't configured in borgmatic's configuration, then don't actually remove anything.
+    '''
+    if hook_config is None:
+        return
+
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    btrfs_command = hook_config.get('btrfs_command', 'btrfs')
+    findmnt_command = hook_config.get('findmnt_command', 'findmnt')
+
+    try:
+        all_subvolumes = get_subvolumes(btrfs_command, findmnt_command)
+    except FileNotFoundError as error:
+        logger.debug(f'Could not find "{error.filename}" command')
+        return
+    except subprocess.CalledProcessError as error:
+        logger.debug(error)
+        return
+
+    # Reversing the sorted subvolumes ensures that we remove longer paths of child subvolumes before
+    # the shorter paths of parent subvolumes.
+    for subvolume in reversed(all_subvolumes):
+        subvolume_snapshots_glob = borgmatic.config.paths.replace_temporary_subdirectory_with_glob(
+            os.path.normpath(make_snapshot_path(subvolume.path)),
+            temporary_directory_prefix=BORGMATIC_SNAPSHOT_PREFIX,
+        )
+
+        logger.debug(
+            f'Looking for snapshots to remove in {subvolume_snapshots_glob}{dry_run_label}',
+        )
+
+        for snapshot_path in glob.glob(subvolume_snapshots_glob):
+            if not os.path.isdir(snapshot_path):
+                continue
+
+            logger.debug(f'Deleting Btrfs snapshot {snapshot_path}{dry_run_label}')
+
+            if dry_run:
+                continue
+
+            try:
+                delete_snapshot(btrfs_command, snapshot_path)
+            except FileNotFoundError:
+                logger.debug(f'Could not find "{btrfs_command}" command')
+                return
+            except subprocess.CalledProcessError as error:
+                logger.debug(error)
+                return
+
+            # Remove the snapshot parent directory if it still exists. (It might not exist if the
+            # snapshot was for "/".)
+            snapshot_parent_dir = snapshot_path.rsplit(subvolume.path, 1)[0]
+
+            if os.path.isdir(snapshot_parent_dir):
+                shutil.rmtree(snapshot_parent_dir)
+
+
+def make_data_source_dump_patterns(
+    hook_config,
+    config,
+    borgmatic_runtime_directory,
+    name=None,
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    return ()
+
+
+def restore_data_source_dump(
+    hook_config,
+    config,
+    data_source,
+    dry_run,
+    extract_process,
+    connection_params,
+    borgmatic_runtime_directory,
+):  # pragma: no cover
+    '''
+    Restores aren't implemented, because stored files can be extracted directly with "extract".
+    '''
+    raise NotImplementedError()

borgmatic/hooks/data_source/config.py

@@ -0,0 +1,104 @@
+import json
+import logging
+import shutil
+import subprocess
+
+from borgmatic.execute import execute_command_and_capture_output
+
+IS_A_HOOK = False
+
+logger = logging.getLogger(__name__)
+
+
+def resolve_database_option(option, data_source, connection_params=None, restore=False):
+    '''
+    Resolves a database option from the given data source configuration dict and
+    connection parameters dict. If restore is set to True it will consider the
+    `restore_<option>` instead.
+
+    Returns the resolved option or None. Can raise a ValueError if the hostname lookup
+    results in a container IP check.
+    '''
+    # Special case `hostname` since it overlaps with `container`
+    if option == 'hostname':
+        return get_hostname_from_config(data_source, connection_params, restore)
+    if connection_params and (value := connection_params.get(option)):
+        return value
+    if restore and f'restore_{option}' in data_source:
+        return data_source[f'restore_{option}']
+
+    return data_source.get(option)
+
+
+def get_hostname_from_config(data_source, connection_params=None, restore=False):
+    '''
+    Specialisation of `resolve_database_option` to handle the extra complexity of
+    the hostname option to also handle containers.
+
+    Returns a hostname/IP or raises an ValueError if a container IP lookup fails.
+    '''
+    # connection params win, full stop
+    if connection_params:
+        if container := connection_params.get('container'):
+            return get_ip_from_container(container)
+        if hostname := connection_params.get('hostname'):
+            return hostname
+    # ... then try the restore config
+    if restore:
+        if 'restore_container' in data_source:
+            return get_ip_from_container(data_source['restore_container'])
+        if 'restore_hostname' in data_source:
+            return data_source['restore_hostname']
+    # ... and finally fall back to the normal options
+    if 'container' in data_source:
+        return get_ip_from_container(data_source['container'])
+
+    return data_source.get('hostname')
+
+
+def get_ip_from_container(container):
+    '''
+    Determine the IP for a given container name via podman and docker.
+
+    Returns an IP or raises a ValueError if the lookup fails.
+    '''
+    engines = (shutil.which(engine) for engine in ('docker', 'podman'))
+    engines = [engine for engine in engines if engine]
+
+    if not engines:
+        raise ValueError("Neither 'docker' nor 'podman' could be found on the system")
+
+    last_error = None
+    for engine in engines:
+        try:
+            output = execute_command_and_capture_output(
+                (
+                    engine,
+                    'container',
+                    'inspect',
+                    '--format={{json .NetworkSettings}}',
+                    container,
+                )
+            )
+        except subprocess.CalledProcessError as error:
+            last_error = error
+            logger.debug(f"Could not find container '{container}' with engine '{engine}'")
+            continue  # Container does not exist
+
+        try:
+            network_data = json.loads(output.strip())
+        except json.JSONDecodeError as e:
+            raise ValueError(f'Could not decode JSON output from {engine}') from e
+        if main_ip := network_data.get('IPAddress'):
+            return main_ip
+        # No main IP found, try the networks
+        for network in network_data.get('Networks', {}).values():
+            if ip := network.get('IPAddress'):
+                return ip
+
+    if last_error:
+        raise last_error
+
+    raise ValueError(
+        f"Could not determine ip address for container '{container}'; running in host mode or userspace networking?"
+    )

borgmatic/hooks/data_source/dump.py

@@ -0,0 +1,129 @@
+import fnmatch
+import json
+import logging
+import os
+import shutil
+
+import borgmatic.actions.restore
+
+logger = logging.getLogger(__name__)
+
+IS_A_HOOK = False
+
+
+def make_data_source_dump_path(borgmatic_runtime_directory, data_source_hook_name):
+    '''
+    Given a borgmatic runtime directory and a data source hook name, construct a data source dump
+    path.
+    '''
+    return os.path.join(borgmatic_runtime_directory, data_source_hook_name)
+
+
+def make_data_source_dump_filename(
+    dump_path, name, hostname=None, port=None, container=None, label=None
+):
+    '''
+    Based on the given dump directory path, data source name, hostname, and port, return a filename
+    to use for the data source dump. The hostname defaults to localhost.
+
+    Raise ValueError if the data source name is invalid.
+    '''
+    if os.path.sep in name:
+        raise ValueError(f'Invalid data source name {name}')
+
+    if container:
+        hostname = container
+    elif not hostname:
+        hostname = 'localhost'
+
+    identifier = label if label else hostname + ('' if port is None else f':{port}')
+
+    return os.path.join(dump_path, identifier, name)
+
+
+def write_data_source_dumps_metadata(borgmatic_runtime_directory, hook_name, dumps_metadata):
+    '''
+    Given the borgmatic runtime directory, a data source hook name, and a sequence of
+    borgmatic.actions.restore.Dump instances of dump metadata, write a metadata file describing all
+    of those dumps. This metadata is being dumped so that it's available upon restore, e.g. to
+    support the user selecting which data source(s) should be restored.
+
+    Raise ValueError if writing to the file results in an operating system error.
+    '''
+    dumps_metadata_path = os.path.join(borgmatic_runtime_directory, hook_name, 'dumps.json')
+
+    try:
+        with open(dumps_metadata_path, 'w', encoding='utf-8') as metadata_file:
+            json.dump(
+                {
+                    'dumps': [dump._asdict() for dump in dumps_metadata],
+                },
+                metadata_file,
+                sort_keys=True,
+            )
+    except OSError as error:
+        raise ValueError(f'Error writing to dumps metadata at {dumps_metadata_path}: {error}')
+
+
+def parse_data_source_dumps_metadata(dumps_json, dumps_metadata_path):
+    '''
+    Given a dumps metadata JSON string as extracted from an archive and its path within the archive,
+    parse it into a tuple of borgmatic.actions.restore.Dump instances and return them.
+
+    Raise ValueError if parsing the JSON results in a JSON decode error or the data does not have
+    the expected keys.
+    '''
+    try:
+        return tuple(
+            borgmatic.actions.restore.Dump(**dump) for dump in json.loads(dumps_json)['dumps']
+        )
+    except (json.JSONDecodeError, TypeError, KeyError) as error:
+        raise ValueError(
+            f'Cannot read archive data source dumps metadata at {dumps_metadata_path} due to invalid JSON: {error}',
+        )
+
+
+def create_parent_directory_for_dump(dump_path):
+    '''
+    Create a directory to contain the given dump path.
+    '''
+    os.makedirs(os.path.dirname(dump_path), mode=0o700, exist_ok=True)
+
+
+def create_named_pipe_for_dump(dump_path):
+    '''
+    Create a named pipe at the given dump path.
+    '''
+    create_parent_directory_for_dump(dump_path)
+    os.mkfifo(dump_path, mode=0o600)
+
+
+def remove_data_source_dumps(dump_path, data_source_type_name, dry_run):
+    '''
+    Remove all data source dumps in the given dump directory path (including the directory itself).
+    If this is a dry run, then don't actually remove anything.
+    '''
+    dry_run_label = ' (dry run; not actually removing anything)' if dry_run else ''
+
+    logger.debug(f'Removing {data_source_type_name} data source dumps{dry_run_label}')
+
+    if dry_run:
+        return
+
+    if os.path.exists(dump_path):
+        shutil.rmtree(dump_path)
+
+
+def convert_glob_patterns_to_borg_pattern(patterns):
+    '''
+    Convert a sequence of shell glob patterns like "/etc/*", "/tmp/*" to the corresponding Borg
+    regular expression archive pattern as a single string like "re:etc/.*|tmp/.*".
+    '''
+    # Remove the "\z" or "\Z" generated by fnmatch.translate() because we don't want the pattern to
+    # match only at the end of a path, as directory format dumps require extracting files with paths
+    # longer than the pattern. E.g., a pattern of "borgmatic/*/foo_databases/test" should also match
+    # paths like "borgmatic/*/foo_databases/test/toc.dat"
+    return 're:' + '|'.join(
+        fnmatch.translate(pattern.lstrip('/')).replace('\\z', '').replace('\\Z', '')
+        for pattern in patterns
+    )