Bump version for release.

Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1235

NEWS

@@ -1,9 +1,56 @@
+2.1.0
+ * TL;DR: Many logging, memory, and performance improvements. Mind those breaking changes!
+ * #485: When running commands (database clients, command hooks, etc.), elevate stderr output to
+   show up as borgmatic error logs.
+ * #858: With the "--log-json" flag, log borgmatic's own logs as JSON, not just Borg's.
+ * #1092: BREAKING: Treat most Borg warnings as errors by default, so for instance backups now fail
+   when source directories are missing. You can still override this behavior with the
+   "borg_exit_codes" option. See the documentation for more information:
+   https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
+ * #1092: Deprecate the "source_directories_must_exist" option, as borgmatic now treats "backup file
+   not found" warnings from Borg as errors, which accomplishes the same thing.
+ * #1132: BREAKING/SECURITY: For the Healthchecks, Apprise, Pagerduty, and Loki monitoring hooks,
+   disable log sending when not explicitly enabled. This avoids revealing private log information to
+   third-party services. To send logs anyway, set the monitoring hook's "send_logs" option to
+   "true".
+ * #1204: When verbosity levels differ between console/monitoring/syslog/file, log Borg's output to
+   each one at a different level. Previously, it was logged at the maximum level of all the
+   verbosities.
+ * #1208: Fix for the "restore" action incorrectly extracting more database dumps than the
+   "--database" flag specifies.
+ * #1210: Fix an error when running the "spot" check or "extract" action with the "progress" option
+   or "--progress" flag.
+ * #1211: Fix an error about the runtime directory getting excluded by tweaking its logic and
+   lowering the error to a warning.
+ * #1212: Fix an error when restoring multiple directory-format database dumps at once.
+ * #1213: BREAKING: Support disabling both constant and variable interpolation by escaping with
+   backslashes. For instance, interpret "\{name\}" as literally "{name}" instead of trying to
+   resolve it as a constant/variable.
+ * #1220: Cleanup snapshots immediately after ZFS, LVM, or Btrfs hooks error—rather than waiting
+   until the next time borgmatic runs.
+ * #1221: Add an "unsafe_skip_path_validation_before_create" option to skip pre-backup safety
+   validation so as to reduce backup times on large filesystems.
+ * #1224: When running an "extract" check with the "--progress" flag, show file extraction progress.
+ * #1225: Improve performance and greatly reduce memory usage during pre-backup safety validation on
+   large filesystems.
+ * #1230: Fix another warning from LVM about leaked file descriptors, this time when calling
+   "lvcreate" from a command hook.
+ * #1234: Fix for the ntfy monitoring hook erroring on emojis in the "title" option.
+ * When syslog verbosity is enabled, log to systemd's journal (if present) with
+   structured data. See the documentation for more information:
+   https://torsion.org/borgmatic/reference/command-line/logging/#systemd-journal
+ * SECURITY: Prevent shell injection attacks via constant interpolation in command hooks. (This was
+   already implemented for deprecated "before_*"/"after_*" command hooks.)
+ * Fix for an error in the "key import" action when importing a key from stdin.
+ * Fix the "recreate" action to include borgmatic-specific paths (database dumps, etc.) in recreated
+   archives.
+ * Update the "list" action to support the "--json" flag when the "--archive" flag is also used.
+ * Promote the ZFS, LVM, and Btrfs hooks from beta features to stable.
+
 2.0.13
  * #1054: Allow the Btrfs hook to create and delete snapshots even when running
    as a non-root user. See the documentation for more information:
    https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#non-root-user
- * #1122: To prevent the user from inadvertently excluding the "bootstrap" action's manifest, always
-   error and exit when the borgmatic runtime directory overlaps with the configured excludes.
  * #1179: Add a "file_list_format" option for setting the "list" action's output format and an
    "archive_list_format" option for setting the "repo-list" action's format.
  * #1192: Fix for over-aggressive deduplication of source directories that contain the borgmatic

borgmatic/actions/check.py

@@ -380,10 +380,7 @@ def collect_spot_check_source_paths(
         dry_run=True,
         repository_path=repository['path'],
         # Omit "progress" because it interferes with "list_details".
-        config=dict(
+        config=dict(config, progress=False, list_details=True),
-            {option: value for option, value in config.items() if option != 'progress'},
-            list_details=True,
-        ),
         patterns=borgmatic.actions.pattern.process_patterns(
             borgmatic.actions.pattern.collect_patterns(config)
             + tuple(
@@ -405,7 +402,7 @@ def collect_spot_check_source_paths(
     )
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    paths_output = borgmatic.execute.execute_command_and_capture_output(
+    path_lines = borgmatic.execute.execute_command_and_capture_output(
         create_flags + create_positional_arguments,
         capture_stderr=True,
         environment=borgmatic.borg.environment.make_environment(config),
@@ -414,9 +411,9 @@ def collect_spot_check_source_paths(
         borg_exit_codes=config.get('borg_exit_codes'),
     )
 
-    paths = tuple(
+    paths = (
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.splitlines()
+        for path_line in path_lines
         if path_line and path_line.startswith(('- ', '+ '))
     )
 
@@ -451,24 +448,22 @@ def collect_spot_check_archive_paths(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return tuple(
-        path
+        entry['path']
-        for line in borgmatic.borg.list.capture_archive_listing(
+        for entry in borgmatic.borg.list.capture_archive_listing(
             repository['path'],
             archive,
             config,
             local_borg_version,
             global_arguments,
-            path_format='{type} {path}{NUL}',
             local_path=local_path,
             remote_path=remote_path,
         )
-        for (file_type, path) in (line.split(' ', 1),)
+        if entry['type'] not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
-        if file_type not in {BORG_DIRECTORY_FILE_TYPE, BORG_PIPE_FILE_TYPE}
+        if pathlib.Path('borgmatic') not in pathlib.Path(entry['path']).parents
-        if pathlib.Path('borgmatic') not in pathlib.Path(path).parents
         if pathlib.Path(borgmatic_source_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
         if pathlib.Path(borgmatic_runtime_directory.lstrip(os.path.sep))
-        not in pathlib.Path(path).parents
+        not in pathlib.Path(entry['path']).parents
     )
 
 
@@ -526,20 +521,34 @@ def compare_spot_check_hashes(
         if not source_sample_paths_subset:
             break
 
-        hash_output = borgmatic.execute.execute_command_and_capture_output(
+        hash_paths = tuple(
+            path for path in source_sample_paths_subset if path in hashable_source_sample_path
+        )
+        hash_lines = borgmatic.execute.execute_command_and_capture_output(
             tuple(
                 shlex.quote(part)
                 for part in shlex.split(spot_check_config.get('xxh64sum_command', 'xxh64sum'))
             )
-            + tuple(
+            + hash_paths,
-                path for path in source_sample_paths_subset if path in hashable_source_sample_path
-            ),
             working_directory=working_directory,
         )
 
         source_hashes.update(
             **dict(
-                (reversed(line.split('  ', 1)) for line in hash_output.splitlines()),
+                zip(
+                    # xxh64sum rewrites/escapes the paths that it returns alongside its hashes, for
+                    # instance if they contain special characters. When that happens, they don't
+                    # match the original source paths and therefore hash lookups fail. So when
+                    # building this lookup dict, use the original unaltered paths we provided as
+                    # input to xxh64sum.
+                    hash_paths,
+                    (
+                        # For some reason, xxh64sum prefixes the hash with a backslash if the path
+                        # contains a newline. Work around that.
+                        line.split('  ', 1)[0].lstrip('\\')
+                        for line in hash_lines
+                    ),
+                ),
                 # Represent non-existent files as having empty hashes so the comparison below still
                 # works. Same thing for filesystem links, since Borg produces empty archive hashes
                 # for them.
@@ -553,21 +562,21 @@ def compare_spot_check_hashes(
 
         # Get the hash for each file in the archive.
         archive_hashes.update(
-            **dict(
+            **{
-                reversed(line.split(' ', 1))
+                entry['path']: entry['xxh64']
-                for line in borgmatic.borg.list.capture_archive_listing(
+                for entry in borgmatic.borg.list.capture_archive_listing(
                     repository['path'],
                     archive,
                     config,
                     local_borg_version,
                     global_arguments,
                     list_paths=source_sample_paths_subset,
-                    path_format='{xxh64} {path}{NUL}',
+                    path_format='{xxh64}{path}',
                     local_path=local_path,
                     remote_path=remote_path,
                 )
-                if line
+                if entry
-            ),
+            },
         )
 
     # Compare the source hashes with the archive hashes to see how many match.
@@ -605,8 +614,6 @@ def spot_check(
     disk to those stored in the latest archive. If any differences are beyond configured tolerances,
     then the check fails.
     '''
-    logger.debug('Running spot check')
-
     try:
         spot_check_config = next(
             check for check in config.get('checks', ()) if check.get('name') == 'spot'
@@ -791,6 +798,7 @@ def run_check(
             write_check_time(make_check_time_path(config, repository_id, check, archives_check_id))
 
     if 'extract' in checks:
+        logger.info('Running extract check')
         borgmatic.borg.extract.extract_last_archive_dry_run(
             config,
             local_borg_version,
@@ -803,6 +811,7 @@ def run_check(
         write_check_time(make_check_time_path(config, repository_id, 'extract'))
 
     if 'spot' in checks:
+        logger.info('Running spot check')
         with borgmatic.config.paths.Runtime_directory(config) as borgmatic_runtime_directory:
             spot_check(
                 repository,

borgmatic/actions/config/bootstrap.py

@@ -127,8 +127,6 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
         bootstrap_arguments.repository,
         archive_name,
         [config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
-        # Only add progress here and not the extract_archive() call above, because progress
-        # conflicts with extract_to_stdout.
         dict(config, progress=bootstrap_arguments.progress or False),
         local_borg_version,
         global_arguments,

borgmatic/actions/create.py

@@ -1,5 +1,6 @@
 import logging
 
+import borgmatic.actions.dump
 import borgmatic.actions.json
 import borgmatic.borg.create
 import borgmatic.borg.feature
@@ -51,96 +52,85 @@ def run_create(
         )
 
         original_patterns = list(patterns)
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            original_patterns,
-            global_arguments.dry_run,
-        )
-        active_dumps = borgmatic.hooks.dispatch.call_hooks(
-            'dump_data_sources',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            config_paths,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )
 
-        # Process the patterns again in case any data source hooks updated them. Without this step,
+        # Use the original patterns so as to disregard any modifications made by any data source
-        # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+        # hooks, e.g. via dump_data_sources() below.
-        # the same named pipe twice.
+        with borgmatic.actions.dump.Dump_cleanup(
-        patterns = pattern.process_patterns(
+            config, borgmatic_runtime_directory, original_patterns, global_arguments.dry_run
-            patterns,
+        ):
-            config,
+            active_dumps = borgmatic.hooks.dispatch.call_hooks(
-            working_directory,
+                'dump_data_sources',
-            borgmatic_runtime_directory,
-            skip_expand_paths=config_paths,
-        )
-        stream_processes = [process for processes in active_dumps.values() for process in processes]
-
-        # If we have stream processes, we first create an archive with .checkpoint suffix. This is
-        # to make sure we only create a real archive if all the streaming processes completed
-        # successfully (create_archive will fail if a streaming process fails, but the archive might
-        # have already been created at this point).
-        use_checkpoint = bool(stream_processes)
-
-        json_output = borgmatic.borg.create.create_archive(
-            global_arguments.dry_run,
-            repository['path'],
-            config,
-            patterns,
-            local_borg_version,
-            global_arguments,
-            borgmatic_runtime_directory,
-            archive_suffix='.checkpoint' if use_checkpoint else '',
-            local_path=local_path,
-            remote_path=remote_path,
-            json=create_arguments.json,
-            comment=create_arguments.comment,
-            stream_processes=stream_processes,
-        )
-
-        if use_checkpoint:
-            rename_checkpoint_archive(
-                repository['path'],
-                global_arguments,
                 config,
+                borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                config_paths,
+                borgmatic_runtime_directory,
+                patterns,
+                global_arguments.dry_run,
+            )
+
+            # Process the patterns again in case any data source hooks updated them. Without this step,
+            # we could end up with duplicate paths that cause Borg to hang when it tries to read from
+            # the same named pipe twice.
+            patterns = pattern.process_patterns(
+                patterns,
+                config,
+                working_directory,
+                borgmatic_runtime_directory,
+                skip_expand_paths=config_paths,
+            )
+            stream_processes = [
+                process for processes in active_dumps.values() for process in processes
+            ]
+
+            # If we have stream processes, we first create an archive with .checkpoint suffix. This is
+            # to make sure we only create a real archive if all the streaming processes completed
+            # successfully (create_archive will fail if a streaming process fails, but the archive might
+            # have already been created at this point).
+            use_checkpoint = bool(stream_processes)
+
+            json_output = borgmatic.borg.create.create_archive(
+                global_arguments.dry_run,
+                repository['path'],
+                config,
+                patterns,
                 local_borg_version,
-                local_path,
+                global_arguments,
-                remote_path,
+                borgmatic_runtime_directory,
+                archive_suffix='.checkpoint' if use_checkpoint else '',
+                local_path=local_path,
+                remote_path=remote_path,
+                json=create_arguments.json,
+                comment=create_arguments.comment,
+                stream_processes=stream_processes,
             )
 
-        if json_output:
-            output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
             if use_checkpoint:
-                # Patch archive name and ID
+                rename_checkpoint_archive(
-                renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
                     repository['path'],
+                    global_arguments,
                     config,
                     local_borg_version,
-                    global_arguments,
                     local_path,
                     remote_path,
                 )
 
-                output['archive']['name'] = renamed_archive['name']
+            if json_output:
-                output['archive']['id'] = renamed_archive['id']
+                output = borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+                if use_checkpoint:
+                    # Patch archive name and ID
+                    renamed_archive = borgmatic.borg.repo_list.get_latest_archive(
+                        repository['path'],
+                        config,
+                        local_borg_version,
+                        global_arguments,
+                        local_path,
+                        remote_path,
+                    )
 
-            yield output
+                    output['archive']['name'] = renamed_archive['name']
+                    output['archive']['id'] = renamed_archive['id']
 
-        # Use the original patterns so as to disregard any modifications made by any data source
+                yield output
-        # hooks, e.g. via dump_data_sources() above.
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            original_patterns,
-            global_arguments.dry_run,
-        )
 
 
 def rename_checkpoint_archive(

borgmatic/actions/dump.py

@@ -0,0 +1,53 @@
+import borgmatic.hooks.dispatch
+
+
+class Dump_cleanup:
+    '''
+    A Python context manager for removing data source dumps before and after wrapped code. Besides
+    doing this for general filesystem cleanliness reasons, leaving old dumps lying around can
+    prevent future dumping from working (in the case of filesystem snapshots) or cause Borg hangs
+    (in the case of database dump named pipes).
+
+    Example use as a context manager:
+
+        with borgmatic.actions.dump.Dump_cleanup(
+            config, borgmatic_runtime_directory, patterns, dry_run,
+        ):
+            do_something_like_perform_a_dump_or_restore()
+    '''
+
+    def __init__(self, config, borgmatic_runtime_directory, patterns, dry_run):
+        '''
+        Given a configuration dict, the borgmatic runtime directory, the configured patterns, and
+        whether this is a dry-run, store these values for use below.
+        '''
+        self.config = config
+        self.borgmatic_runtime_directory = borgmatic_runtime_directory
+        self.patterns = patterns
+        self.dry_run = dry_run
+
+    def __enter__(self):
+        '''
+        Remove all data source dumps that exist prior to the wrapped code running.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )
+
+    def __exit__(self, exception_type, exception, traceback):
+        '''
+        Remove all data source dumps, including any created by the wrapped code.
+        '''
+        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            'remove_data_source_dumps',
+            self.config,
+            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+            self.borgmatic_runtime_directory,
+            self.patterns,
+            self.dry_run,
+        )

borgmatic/actions/recreate.py

@@ -1,10 +1,10 @@
 import logging
 import subprocess
 
-import borgmatic.borg.info
+import borgmatic.actions.pattern
+import borgmatic.borg.pattern
 import borgmatic.borg.recreate
 import borgmatic.borg.repo_list
-from borgmatic.actions.pattern import collect_patterns, process_patterns
 
 logger = logging.getLogger(__name__)
 
@@ -30,8 +30,16 @@ def run_recreate(
         logger.answer('Recreating repository')
 
     # Collect and process patterns.
-    processed_patterns = process_patterns(
+    processed_patterns = borgmatic.actions.pattern.process_patterns(
-        collect_patterns(config),
+        (
+            *borgmatic.actions.pattern.collect_patterns(config),
+            # Also add borgmatic-specific paths, so they don't get excluded from the recreated
+            # archive. Note that this doesn't currently work for archives created with Borg 1.2 or
+            # below.
+            borgmatic.borg.pattern.Pattern(
+                '/borgmatic', source=borgmatic.borg.pattern.Pattern_source.INTERNAL
+            ),
+        ),
         config,
         borgmatic.config.paths.get_working_directory(config),
     )

borgmatic/actions/restore.py

@@ -5,6 +5,7 @@ import pathlib
 import shutil
 import tempfile
 
+import borgmatic.actions.dump
 import borgmatic.actions.pattern
 import borgmatic.borg.extract
 import borgmatic.borg.list
@@ -129,14 +130,14 @@ def get_configured_data_source(config, restore_dump):
 
 
 def strip_path_prefix_from_extracted_dump_destination(
-    destination_path,
+    extract_path,
     borgmatic_runtime_directory,
 ):
     '''
     Directory-format dump files get extracted into a temporary directory containing a path prefix
-    that depends how the files were stored in the archive. So, given the destination path where the
+    that depends how the files were stored in the archive. So, given the path where the dump was
-    dump was extracted and the borgmatic runtime directory, move the dump files such that the
+    extracted and the borgmatic runtime directory, move the dump files such that the restore doesn't
-    restore doesn't have to deal with that varying path prefix.
+    have to deal with that varying path prefix.
 
     For instance, if the dump was extracted to:
 
@@ -150,16 +151,16 @@ def strip_path_prefix_from_extracted_dump_destination(
 
       /run/user/0/borgmatic/postgresql_databases/test/...
     '''
-    for subdirectory_path, _, _ in os.walk(destination_path):
+    for subdirectory_path, _, _ in os.walk(extract_path):
         databases_directory = os.path.basename(subdirectory_path)
 
         if not databases_directory.endswith('_databases'):
             continue
 
-        shutil.move(
+        destination_path = os.path.join(borgmatic_runtime_directory, databases_directory)
-            subdirectory_path,
+        shutil.rmtree(destination_path, ignore_errors=True)
-            os.path.join(borgmatic_runtime_directory, databases_directory),
+        shutil.move(subdirectory_path, destination_path)
-        )
+
         break
 
 
@@ -200,6 +201,10 @@ def restore_single_dump(
         borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
         borgmatic_runtime_directory,
         data_source['name'],
+        data_source.get('hostname'),
+        data_source.get('port'),
+        data_source.get('container'),
+        data_source.get('label'),
     )[hook_name.split('_databases', 1)[0]]
 
     destination_path = (
@@ -267,12 +272,12 @@ def collect_dumps_from_archive(
     Given a local or remote repository path, a resolved archive name, a configuration dict, the
     local Borg version, global arguments an argparse.Namespace, local and remote Borg paths, and the
     borgmatic runtime directory, query the archive for the names of data sources dumps it contains
-    and return them as a set of Dump instances.
+    and return them as a tuple of Dump instances.
     '''
-    dumps_from_archive = set()
+    dumps_from_archive = {}  # Use a dict as an ordered set.
 
     # There is (at most) one dump metadata file per data source hook. Load each.
-    for dumps_metadata_path in borgmatic.borg.list.capture_archive_listing(
+    for dumps_metadata_entry in borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
         config,
@@ -295,36 +300,33 @@ def collect_dumps_from_archive(
         local_path=local_path,
         remote_path=remote_path,
     ):
-        if not dumps_metadata_path:
+        if not dumps_metadata_entry.get('path'):
             continue
 
-        dumps_from_archive.update(
+        for dump in borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
-            set(
+            borgmatic.borg.extract.extract_archive(
-                borgmatic.hooks.data_source.dump.parse_data_source_dumps_metadata(
+                global_arguments.dry_run,
-                    borgmatic.borg.extract.extract_archive(
+                repository,
-                        global_arguments.dry_run,
+                archive,
-                        repository,
+                [dumps_metadata_entry['path']],
-                        archive,
+                config,
-                        [dumps_metadata_path],
+                local_borg_version,
-                        config,
+                global_arguments,
-                        local_borg_version,
+                local_path=local_path,
-                        global_arguments,
+                remote_path=remote_path,
-                        local_path=local_path,
+                extract_to_stdout=True,
-                        remote_path=remote_path,
-                        extract_to_stdout=True,
-                    )
-                    .stdout.read()
-                    .decode(),
-                    dumps_metadata_path,
-                )
             )
-        )
+            .stdout.read()
+            .decode(),
+            dumps_metadata_entry['path'],
+        ):
+            dumps_from_archive[dump] = None
 
     # If we've successfully loaded any dumps metadata, we're done.
     if dumps_from_archive:
         logger.debug('Collecting database dumps from archive data source dumps metadata files')
 
-        return dumps_from_archive
+        return tuple(dumps_from_archive.keys())
 
     # No dumps metadata files were found, so for backwards compatibility, fall back to parsing the
     # paths of dumps found in the archive to get their respective dump metadata.
@@ -336,7 +338,7 @@ def collect_dumps_from_archive(
     # Probe for the data source dumps in multiple locations, as the default location has moved to
     # the borgmatic runtime directory (which gets stored as just "/borgmatic" with Borg 1.4+). But
     # we still want to support reading dumps from previously created archives as well.
-    dump_paths = borgmatic.borg.list.capture_archive_listing(
+    dump_entries = borgmatic.borg.list.capture_archive_listing(
         repository,
         archive,
         config,
@@ -358,7 +360,9 @@ def collect_dumps_from_archive(
         remote_path=remote_path,
     )
 
-    for dump_path in dump_paths:
+    for dump_entry in dump_entries:
+        dump_path = dump_entry.get('path')
+
         if not dump_path:
             continue
 
@@ -388,11 +392,11 @@ def collect_dumps_from_archive(
             except (ValueError, TypeError):
                 port = None
 
-            dumps_from_archive.add(
+            dumps_from_archive[
                 Dump(
                     hook_name, data_source_name, None if hostname == 'localhost' else hostname, port
                 )
-            )
+            ] = None
 
             # We've successfully parsed the dump path, so need to probe any further.
             break
@@ -401,20 +405,21 @@ def collect_dumps_from_archive(
                 f'Ignoring invalid data source dump path "{dump_path}" in archive {archive}',
             )
 
-    return dumps_from_archive
+    return tuple(dumps_from_archive.keys())
 
 
 def get_dumps_to_restore(restore_arguments, dumps_from_archive):
     '''
     Given restore arguments as an argparse.Namespace instance indicating which dumps to restore and
-    a set of Dump instances representing the dumps found in an archive, return a set of specific
+    a tuple of Dump instances representing the dumps found in an archive, return a tuple of specific
     Dump instances from the archive to restore. As part of this, replace any Dump having a data
     source name of "all" with multiple named Dump instances as appropriate.
 
     Raise ValueError if any of the requested data source names cannot be found in the archive or if
     there are multiple archive dump matches for a given requested dump.
     '''
-    requested_dumps = (
+    requested_dumps = tuple(
+        # Use a dict comprehension as an ordered set.
         {
             Dump(
                 hook_name=(
@@ -431,16 +436,16 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                 port=restore_arguments.original_port,
                 label=restore_arguments.original_label or UNSPECIFIED,
                 container=restore_arguments.original_container or UNSPECIFIED,
-            )
+            ): None
             for name in restore_arguments.data_sources or (UNSPECIFIED,)
-        }
+        }.keys()
         if restore_arguments.hook
         or restore_arguments.data_sources
         or restore_arguments.original_hostname
         or restore_arguments.original_port
         or restore_arguments.original_label
         or restore_arguments.original_container
-        else {
+        else (
             Dump(
                 hook_name=UNSPECIFIED,
                 data_source_name='all',
@@ -449,14 +454,15 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
                 label=UNSPECIFIED,
                 container=UNSPECIFIED,
             ),
-        }
+        )
     )
     missing_dumps = set()
-    dumps_to_restore = set()
+    dumps_to_restore = {}  # Use a dict as an ordered set.
 
     # If there's a requested "all" dump, add every dump from the archive to the dumps to restore.
     if any(dump for dump in requested_dumps if dump.data_source_name == 'all'):
-        dumps_to_restore.update(dumps_from_archive)
+        for dump in dumps_from_archive:
+            dumps_to_restore[dump] = None
 
     # If any archive dump matches a requested dump, add the archive dump to the dumps to restore.
     for requested_dump in requested_dumps:
@@ -472,7 +478,7 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
         if len(matching_dumps) == 0:
             missing_dumps.add(requested_dump)
         elif len(matching_dumps) == 1:
-            dumps_to_restore.add(matching_dumps[0])
+            dumps_to_restore[matching_dumps[0]] = None
         else:
             raise ValueError(
                 f'Cannot restore data source {render_dump_metadata(requested_dump)} because there are multiple matching dumps in the archive. Try adding flags to disambiguate.',
@@ -487,20 +493,20 @@ def get_dumps_to_restore(restore_arguments, dumps_from_archive):
             f"Cannot restore data source dump{'s' if len(missing_dumps) > 1 else ''} {rendered_dumps} missing from archive",
         )
 
-    return dumps_to_restore
+    return tuple(dumps_to_restore.keys())
 
 
 def ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored):
     '''
-    Given a set of requested dumps to restore and a set of dumps actually restored, raise ValueError
+    Given a tuple of requested dumps to restore and a set of dumps actually restored, raise
-    if any requested dumps to restore weren't restored, indicating that they were missing from the
+    ValueError if any requested dumps to restore weren't restored, indicating that they were missing
-    configuration.
+    from the configuration.
     '''
-    if not dumps_actually_restored:
+    if not dumps_to_restore:
         raise ValueError('No data source dumps were found to restore')
 
     missing_dumps = sorted(
-        dumps_to_restore - dumps_actually_restored,
+        set(dumps_to_restore) - dumps_actually_restored,
         key=lambda dump: dump.data_source_name,
     )
 
@@ -536,96 +542,82 @@ def run_restore(
             config,
             working_directory,
         )
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
-            'remove_data_source_dumps',
-            config,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
-            borgmatic_runtime_directory,
-            patterns,
-            global_arguments.dry_run,
-        )
 
-        archive_name = borgmatic.borg.repo_list.resolve_archive_name(
+        with borgmatic.actions.dump.Dump_cleanup(
-            repository['path'],
+            config, borgmatic_runtime_directory, patterns, global_arguments.dry_run
-            restore_arguments.archive,
+        ):
-            config,
+            archive_name = borgmatic.borg.repo_list.resolve_archive_name(
-            local_borg_version,
+                repository['path'],
-            global_arguments,
+                restore_arguments.archive,
-            local_path,
-            remote_path,
-        )
-        dumps_from_archive = collect_dumps_from_archive(
-            repository['path'],
-            archive_name,
-            config,
-            local_borg_version,
-            global_arguments,
-            local_path,
-            remote_path,
-            borgmatic_runtime_directory,
-        )
-        dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
-
-        dumps_actually_restored = set()
-        connection_params = {
-            'container': restore_arguments.container,
-            'hostname': restore_arguments.hostname,
-            'port': restore_arguments.port,
-            'username': restore_arguments.username,
-            'password': restore_arguments.password,
-            'restore_path': restore_arguments.restore_path,
-        }
-
-        # Restore each dump.
-        for restore_dump in dumps_to_restore:
-            found_data_source = get_configured_data_source(
-                config,
-                restore_dump,
-            )
-            # For a dump that wasn't found via an exact match in the configuration, try to fallback
-            # to an "all" data source.
-            if not found_data_source:
-                found_data_source = get_configured_data_source(
-                    config,
-                    Dump(
-                        restore_dump.hook_name,
-                        'all',
-                        restore_dump.hostname,
-                        restore_dump.port,
-                        restore_dump.label,
-                        restore_dump.container,
-                    ),
-                )
-
-                if not found_data_source:
-                    continue
-
-                found_data_source = dict(found_data_source)
-                found_data_source['name'] = restore_dump.data_source_name
-
-            dumps_actually_restored.add(restore_dump)
-
-            restore_single_dump(
-                repository,
                 config,
                 local_borg_version,
                 global_arguments,
                 local_path,
                 remote_path,
+            )
+            dumps_from_archive = collect_dumps_from_archive(
+                repository['path'],
                 archive_name,
-                restore_dump.hook_name,
+                config,
-                dict(found_data_source, schemas=restore_arguments.schemas),
+                local_borg_version,
-                connection_params,
+                global_arguments,
+                local_path,
+                remote_path,
                 borgmatic_runtime_directory,
             )
+            dumps_to_restore = get_dumps_to_restore(restore_arguments, dumps_from_archive)
 
-        borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
+            dumps_actually_restored = set()
-            'remove_data_source_dumps',
+            connection_params = {
-            config,
+                'container': restore_arguments.container,
-            borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
+                'hostname': restore_arguments.hostname,
-            borgmatic_runtime_directory,
+                'port': restore_arguments.port,
-            patterns,
+                'username': restore_arguments.username,
-            global_arguments.dry_run,
+                'password': restore_arguments.password,
-        )
+                'restore_path': restore_arguments.restore_path,
+            }
+
+            # Restore each dump.
+            for restore_dump in dumps_to_restore:
+                found_data_source = get_configured_data_source(
+                    config,
+                    restore_dump,
+                )
+                # For a dump that wasn't found via an exact match in the configuration, try to fallback
+                # to an "all" data source.
+                if not found_data_source:
+                    found_data_source = get_configured_data_source(
+                        config,
+                        Dump(
+                            restore_dump.hook_name,
+                            'all',
+                            restore_dump.hostname,
+                            restore_dump.port,
+                            restore_dump.label,
+                            restore_dump.container,
+                        ),
+                    )
+
+                    if not found_data_source:
+                        continue
+
+                    found_data_source = dict(found_data_source)
+                    found_data_source['name'] = restore_dump.data_source_name
+
+                dumps_actually_restored.add(restore_dump)
+
+                restore_single_dump(
+                    repository,
+                    config,
+                    local_borg_version,
+                    global_arguments,
+                    local_path,
+                    remote_path,
+                    archive_name,
+                    restore_dump.hook_name,
+                    dict(found_data_source, schemas=restore_arguments.schemas),
+                    connection_params,
+                    borgmatic_runtime_directory,
+                )
 
     ensure_requested_dumps_restored(dumps_to_restore, dumps_actually_restored)

borgmatic/borg/break_lock.py

@@ -29,7 +29,7 @@ def break_lock(
         (local_path, 'break-lock')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())

borgmatic/borg/change_passphrase.py

@@ -32,7 +32,6 @@ def change_passphrase(
         (local_path, 'key', 'change-passphrase')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -66,5 +65,5 @@ def change_passphrase(
     )
 
     logger.answer(
-        f"{repository_path}: Don't forget to update your encryption_passphrase option (if needed)",
+        "Don't forget to update your encryption_passphrase option (if needed)",
     )

borgmatic/borg/check.py

@@ -176,7 +176,13 @@ def check_archives(
             + make_check_name_flags(checks_subset, archive_filter_flags)
             + (('--remote-path', remote_path) if remote_path else ())
             + (('--umask', str(umask)) if umask else ())
-            + (('--log-json',) if config.get('log_json') else ())
+            + (
+                ('--log-json',)
+                if (
+                    config.get('log_json') or not (check_arguments.repair or config.get('progress'))
+                )
+                else ()
+            )
             + (('--lock-wait', str(lock_wait)) if lock_wait else ())
             + verbosity_flags
             + (('--progress',) if config.get('progress') else ())

borgmatic/borg/compact.py

@@ -31,7 +31,7 @@ def compact_segments(
         (local_path, 'compact')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--progress',) if config.get('progress') else ())
         + (('--cleanup-commits',) if cleanup_commits else ())

borgmatic/borg/create.py

@@ -53,6 +53,7 @@ def validate_planned_backup_paths(
     local_path,
     working_directory,
     borgmatic_runtime_directory,
+    find_special_files=False,
 ):
     '''
     Given a dry-run flag, a Borg create command as a tuple, a configuration dict, a local Borg path,
@@ -61,6 +62,9 @@ def validate_planned_backup_paths(
     given runtime directory exists, validate that it will be included in a backup and hasn't been
     excluded.
 
+    If find special files is True, then return the subset of planned backup paths that are special
+    files. Otherwise, return an empty tuple.
+
     Raise ValueError if the runtime directory has been excluded via "exclude_patterns" or similar,
     because any features that rely on the runtime directory getting backed up will break. For
     instance, without the runtime directory, Borg can't consume any database dumps and borgmatic may
@@ -69,7 +73,7 @@ def validate_planned_backup_paths(
     # Omit "--exclude-nodump" from the Borg dry run command, because that flag causes Borg to open
     # files including any named pipe we've created. And omit "--filter" because that can break the
     # paths output parsing below such that path lines no longer start with the expected "- ".
-    paths_output = execute_command_and_capture_output(
+    path_lines = execute_command_and_capture_output(
         (
             *flags.omit_flag_and_value(
                 flags.omit_flag(
@@ -90,41 +94,68 @@ def validate_planned_backup_paths(
 
     # These are all the individual files that Borg is planning to backup as determined by the Borg
     # create dry run above.
-    paths = tuple(
+    paths = (
         path_line.split(' ', 1)[1]
-        for path_line in paths_output.split('\n')
+        for path_line in path_lines
         if path_line and path_line.startswith(('- ', '+ '))
     )
 
+    include_pattern_paths = {
+        pattern.path
+        for pattern in patterns
+        if pattern.type == borgmatic.borg.pattern.Pattern_type.INCLUDE
+    }
     runtime_directory_root_patterns = tuple(
         pattern
         for pattern in patterns
         if any_parent_directories(pattern.path, (borgmatic_runtime_directory,))
         if pattern.type == borgmatic.borg.pattern.Pattern_type.ROOT
+        # Skip root patterns that have corresponding include patterns, because those will "punch
+        # through" any subsequent excludes.
+        if pattern.path not in include_pattern_paths
     )
 
-    if not dry_run and os.path.exists(borgmatic_runtime_directory):
+    special_paths = []
-        # If there are any root patterns in the runtime directory that are missing from the paths
+    validate_runtime_directory = bool(
-        # Borg is planning to backup, then they must've gotten excluded, e.g. by user-configured
+        not dry_run
-        # excludes. Error accordingly.
+        and os.path.exists(borgmatic_runtime_directory)
-        for pattern in runtime_directory_root_patterns:
+        and runtime_directory_root_patterns
-            if not any(any_parent_directories(path, (pattern.path,)) for path in paths):
-                raise ValueError(
-                    f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.',
-                )
-
-    # Return the subset of output paths *not* contained within the borgmatic runtime directory. The
-    # intent is that any downstream checks using these paths should skip runtime paths that
-    # borgmatic uses for its own bookkeeping, instead focusing on user-configured paths.
-    return tuple(
-        path for path in paths if not any_parent_directories(path, (borgmatic_runtime_directory,))
     )
+    runtime_directory_in_path = False
+
+    # Do everything in this one loop because we only want to consume the paths generator once.
+    for path in paths:
+        # If all root patterns in the runtime directory are missing from the paths Borg is planning
+        # to backup, then they must've gotten excluded, e.g. by user-configured excludes. Warn
+        # accordingly (below).
+        if validate_runtime_directory and any(
+            any_parent_directories(path, (pattern.path,))
+            for pattern in runtime_directory_root_patterns
+        ):
+            runtime_directory_in_path = True
+
+        # Return the subset of output paths that are special files but *not* contained within the
+        # borgmatic runtime directory. The intent is to skip runtime paths that borgmatic uses for its
+        # own bookkeeping, instead focusing on user-configured paths.
+        if (
+            find_special_files
+            and not any_parent_directories(path, (borgmatic_runtime_directory,))
+            and special_file(path, working_directory)
+        ):
+            special_paths.append(path)
+
+    if validate_runtime_directory and not runtime_directory_in_path:
+        logger.warning(
+            f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes (or the snapshotted source directories are empty). Please ensure the runtime directory is not excluded.'
+        )
+
+    return tuple(special_paths)
 
 
 MAX_SPECIAL_FILE_PATHS_LENGTH = 1000
 
 
-def make_base_create_command(
+def make_base_create_command(  # noqa: PLR0912
     dry_run,
     repository_path,
     config,
@@ -149,6 +180,9 @@ def make_base_create_command(
     open pattern file handle).
     '''
     if config.get('source_directories_must_exist', False):
+        logger.warning(
+            'The "source_directories_must_exist" option is deprecated and will be removed from a future release; borgmatic now errors on missing files as Borg runs'
+        )
         borgmatic.borg.pattern.check_all_root_patterns_exist(patterns)
 
     patterns_file = borgmatic.borg.pattern.write_patterns_file(
@@ -217,8 +251,8 @@ def make_base_create_command(
         + (('--files-cache', files_cache) if files_cache else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (
             ('--list', '--filter', list_filter_flags)
             if config.get('list_details') and not json and not config.get('progress')
@@ -235,8 +269,16 @@ def make_base_create_command(
     )
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
+    if config.get('unsafe_skip_path_validation_before_create'):
+        logger.warning(
+            'Skipping pre-backup path validation due to "unsafe_skip_path_validation_before_create" option.'
+        )
+
+        return (create_flags, create_positional_arguments, patterns_file)
+
     logger.debug('Checking file paths Borg plans to include')
-    planned_backup_paths = validate_planned_backup_paths(
+
+    special_file_paths = validate_planned_backup_paths(
         dry_run,
         create_flags + create_positional_arguments,
         config,
@@ -244,44 +286,41 @@ def make_base_create_command(
         local_path,
         working_directory,
         borgmatic_runtime_directory=borgmatic_runtime_directory,
+        find_special_files=bool(stream_processes),
     )
 
-    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
+    if stream_processes and config.get('read_special') is False:
-    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
-    if stream_processes and not config.get('read_special'):
         logger.warning(
             'Ignoring configured "read_special" value of false, as true is needed for database hooks.',
         )
 
-        special_file_paths = tuple(
+    # If database hooks are enabled (as indicated by streaming processes), exclude files that might
-            path for path in planned_backup_paths if special_file(path, working_directory)
+    # cause Borg to hang. But skip this if the user has explicitly set the "read_special" to True.
+    if special_file_paths:
+        truncated_special_file_paths = textwrap.shorten(
+            ', '.join(special_file_paths),
+            width=MAX_SPECIAL_FILE_PATHS_LENGTH,
+            placeholder=' ...',
+        )
+        logger.warning(
+            f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
+        )
+        patterns_file = borgmatic.borg.pattern.write_patterns_file(
+            tuple(
+                borgmatic.borg.pattern.Pattern(
+                    special_file_path,
+                    borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
+                    borgmatic.borg.pattern.Pattern_style.FNMATCH,
+                    source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
+                )
+                for special_file_path in special_file_paths
+            ),
+            borgmatic_runtime_directory,
+            patterns_file=patterns_file,
         )
 
-        if special_file_paths:
+        if '--patterns-from' not in create_flags:
-            truncated_special_file_paths = textwrap.shorten(
+            create_flags += ('--patterns-from', patterns_file.name)
-                ', '.join(special_file_paths),
-                width=MAX_SPECIAL_FILE_PATHS_LENGTH,
-                placeholder=' ...',
-            )
-            logger.warning(
-                f'Excluding special files to prevent Borg from hanging: {truncated_special_file_paths}',
-            )
-            patterns_file = borgmatic.borg.pattern.write_patterns_file(
-                tuple(
-                    borgmatic.borg.pattern.Pattern(
-                        special_file_path,
-                        borgmatic.borg.pattern.Pattern_type.NO_RECURSE,
-                        borgmatic.borg.pattern.Pattern_style.FNMATCH,
-                        source=borgmatic.borg.pattern.Pattern_source.INTERNAL,
-                    )
-                    for special_file_path in special_file_paths
-                ),
-                borgmatic_runtime_directory,
-                patterns_file=patterns_file,
-            )
-
-            if '--patterns-from' not in create_flags:
-                create_flags += ('--patterns-from', patterns_file.name)
 
     return (create_flags, create_positional_arguments, patterns_file)
 
@@ -353,24 +392,28 @@ def create_archive(
     borg_exit_codes = config.get('borg_exit_codes')
 
     if stream_processes:
-        return execute_command_with_processes(
+        return '\n'.join(
-            create_flags + create_positional_arguments,
+            execute_command_with_processes(
-            stream_processes,
+                create_flags + create_positional_arguments,
-            output_log_level,
+                stream_processes,
-            output_file,
+                output_log_level,
-            working_directory=working_directory,
+                output_file,
-            environment=environment.make_environment(config),
+                working_directory=working_directory,
-            borg_local_path=local_path,
+                environment=environment.make_environment(config),
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     if output_log_level is None:
-        return execute_command_and_capture_output(
+        return '\n'.join(
-            create_flags + create_positional_arguments,
+            execute_command_and_capture_output(
-            working_directory=working_directory,
+                create_flags + create_positional_arguments,
-            environment=environment.make_environment(config),
+                working_directory=working_directory,
-            borg_local_path=local_path,
+                environment=environment.make_environment(config),
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     execute_command(

borgmatic/borg/delete.py

@@ -38,7 +38,7 @@ def make_delete_command(
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
         + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (

borgmatic/borg/export_key.py

@@ -48,7 +48,7 @@ def export_key(
         (local_path, 'key', 'export')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if output_file is None else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())

borgmatic/borg/export_tar.py

@@ -41,7 +41,7 @@ def export_tar_archive(
         (local_path, 'export-tar')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if destination_path != '-' else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--list',) if config.get('list_details') else ())

borgmatic/borg/extract.py

@@ -49,9 +49,10 @@ def extract_last_archive_dry_run(
     full_extract_command = (
         (local_path, 'extract', '--dry-run')
         + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not config.get('progress') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + verbosity_flags
+        + (('--progress',) if config.get('progress') else ())
         + list_flag
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_archive_flags(
@@ -61,13 +62,23 @@ def extract_last_archive_dry_run(
         )
     )
 
-    execute_command(
+    if config.get('progress'):
-        full_extract_command,
+        execute_command(
-        environment=environment.make_environment(config),
+            full_extract_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            output_file=DO_NOT_CAPTURE,
-        borg_local_path=local_path,
+            environment=environment.make_environment(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-    )
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
+    else:
+        execute_command(
+            full_extract_command,
+            environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
 
 
 def extract_archive(
@@ -97,9 +108,6 @@ def extract_archive(
     lock_wait = config.get('lock_wait', None)
     extra_borg_options = config.get('extra_borg_options', {}).get('extract', '')
 
-    if config.get('progress') and extract_to_stdout:
-        raise ValueError('progress and extract to stdout cannot both be set')
-
     if feature.available(feature.Feature.NUMERIC_IDS, local_borg_version):
         numeric_ids_flags = ('--numeric-ids',) if config.get('numeric_ids') else ()
     else:
@@ -127,13 +135,13 @@ def extract_archive(
         + (('--remote-path', remote_path) if remote_path else ())
         + numeric_ids_flags
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
         + (('--strip-components', str(strip_components)) if strip_components else ())
-        + (('--progress',) if config.get('progress') else ())
+        + (('--progress',) if config.get('progress') and not extract_to_stdout else ())
         + (('--stdout',) if extract_to_stdout else ())
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_archive_flags(
@@ -152,6 +160,17 @@ def extract_archive(
         os.path.join(working_directory or '', destination_path) if destination_path else None
     )
 
+    if extract_to_stdout:
+        return execute_command(
+            full_command,
+            output_file=subprocess.PIPE,
+            run_to_completion=False,
+            environment=environment.make_environment(config),
+            working_directory=full_destination_path,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
+
     # The progress output isn't compatible with captured and logged output, as progress messes with
     # the terminal directly.
     if config.get('progress'):
@@ -165,17 +184,6 @@ def extract_archive(
         )
         return None
 
-    if extract_to_stdout:
-        return execute_command(
-            full_command,
-            output_file=subprocess.PIPE,
-            run_to_completion=False,
-            environment=environment.make_environment(config),
-            working_directory=full_destination_path,
-            borg_local_path=local_path,
-            borg_exit_codes=borg_exit_codes,
-        )
-
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
     execute_command(

borgmatic/borg/import_key.py

@@ -4,7 +4,7 @@ import shlex
 
 import borgmatic.config.paths
 from borgmatic.borg import environment, flags
-from borgmatic.execute import DO_NOT_CAPTURE, execute_command
+from borgmatic.execute import execute_command
 
 logger = logging.getLogger(__name__)
 
@@ -32,19 +32,18 @@ def import_key(
     working_directory = borgmatic.config.paths.get_working_directory(config)
     extra_borg_options = config.get('extra_borg_options', {}).get('key_import', '')
 
-    if import_arguments.path and import_arguments.path != '-':
+    if (
-        if not os.path.exists(os.path.join(working_directory or '', import_arguments.path)):
+        import_arguments.path
-            raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
+        and import_arguments.path != '-'
-
+        and not os.path.exists(os.path.join(working_directory or '', import_arguments.path))
-        input_file = None
+    ):
-    else:
+        raise ValueError(f'Path {import_arguments.path} does not exist. Aborting.')
-        input_file = DO_NOT_CAPTURE
 
     full_command = (
         (local_path, 'key', 'import')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
@@ -54,7 +53,7 @@ def import_key(
             repository_path,
             local_borg_version,
         )
-        + ((import_arguments.path,) if input_file is None else ())
+        + (import_arguments.path or '-',)
     )
 
     if global_arguments.dry_run:
@@ -63,7 +62,6 @@ def import_key(
 
     execute_command(
         full_command,
-        input_file=input_file,
         output_log_level=logging.INFO,
         environment=environment.make_environment(config),
         working_directory=working_directory,

borgmatic/borg/info.py

@@ -40,7 +40,7 @@ def make_info_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
@@ -103,12 +103,14 @@ def display_archives_info(
     borg_exit_codes = config.get('borg_exit_codes')
     working_directory = borgmatic.config.paths.get_working_directory(config)
 
-    json_info = execute_command_and_capture_output(
+    json_info = '\n'.join(
-        json_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            json_command,
-        working_directory=working_directory,
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=working_directory,
-        borg_exit_codes=borg_exit_codes,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
     )
 
     if info_arguments.json:

borgmatic/borg/list.py

@@ -1,5 +1,6 @@
 import argparse
 import copy
+import json
 import logging
 import re
 import shlex
@@ -19,6 +20,7 @@ MAKE_FLAGS_EXCLUDES = (
     'paths',
     'find_paths',
     'format',
+    'json',
     *ARCHIVE_FILTER_FLAGS_MOVED_TO_REPO_LIST,
 )
 
@@ -53,7 +55,8 @@ def make_list_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
+        + flags.make_flags('json-lines', list_arguments.json)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + flags.make_flags('format', list_arguments.format or config.get('file_list_format'))
         + flags.make_flags_from_arguments(list_arguments, excludes=MAKE_FLAGS_EXCLUDES)
@@ -109,14 +112,15 @@ def capture_archive_listing(
     remote_path=None,
 ):
     '''
-    Given a local or remote repository path, an archive name, a configuration
+    Given a local or remote repository path, an archive name, a configuration dict, the local Borg
-    dict, the local Borg version, global arguments as an argparse.Namespace,
+    version, global arguments as an argparse.Namespace, the archive paths (or Borg patterns) in
-    the archive paths (or Borg patterns) in which to list files, the Borg path
+    which to list files, the Borg path format indicating keys to include in the output, and local
-    format to use for the output, and local and remote Borg paths, capture the
+    and remote Borg paths, capture the output of listing that archive and return it as a sequence of
-    output of listing that archive and return it as a list of file paths.
+    dicts, one per path.
     '''
     return tuple(
-        execute_command_and_capture_output(
+        json.loads(entry)
+        for entry in execute_command_and_capture_output(
             make_list_command(
                 repository_path,
                 config,
@@ -126,8 +130,8 @@ def capture_archive_listing(
                     archive=archive,
                     paths=list(list_paths) if list_paths else None,
                     find_paths=None,
-                    json=None,
+                    json=True,
-                    format=path_format or '{path}{NUL}',
+                    format=path_format or None,
                 ),
                 global_arguments,
                 local_path,
@@ -138,8 +142,6 @@ def capture_archive_listing(
             borg_local_path=local_path,
             borg_exit_codes=config.get('borg_exit_codes'),
         )
-        .strip('\0')
-        .split('\0'),
     )
 
 
@@ -196,11 +198,6 @@ def list_archive(
                     f"The --{name.replace('_', '-')} flag on the list action is ignored when using the --archive flag.",
                 )
 
-    if list_arguments.json:
-        raise ValueError(
-            'The --json flag on the list action is not supported when using the --archive/--find flags.',
-        )
-
     borg_exit_codes = config.get('borg_exit_codes')
 
     # If there are any paths to find (and there's not a single archive already selected), start by
@@ -219,24 +216,20 @@ def list_archive(
         )
 
         # Ask Borg to list archives. Capture its output for use below.
-        archive_lines = tuple(
+        archive_lines = execute_command_and_capture_output(
-            execute_command_and_capture_output(
+            repo_list.make_repo_list_command(
-                repo_list.make_repo_list_command(
+                repository_path,
-                    repository_path,
+                config,
-                    config,
+                local_borg_version,
-                    local_borg_version,
+                repo_list_arguments,
-                    repo_list_arguments,
+                global_arguments,
-                    global_arguments,
+                local_path,
-                    local_path,
+                remote_path,
-                    remote_path,
+            ),
-                ),
+            environment=environment.make_environment(config),
-                environment=environment.make_environment(config),
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-                working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=local_path,
-                borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
-                borg_exit_codes=borg_exit_codes,
-            )
-            .strip('\n')
-            .splitlines(),
         )
     else:
         archive_lines = (list_arguments.archive,)

borgmatic/borg/mount.py

@@ -32,7 +32,7 @@ def mount_archive(
         (local_path, 'mount')
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + (('--log-json',) if not mount_arguments.foreground else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())

borgmatic/borg/passcommand.py

@@ -17,9 +17,11 @@ def run_passcommand(passcommand, working_directory):
     Cache the results so that the passcommand only needs to run—and potentially prompt the user—once
     per borgmatic invocation.
     '''
-    return borgmatic.execute.execute_command_and_capture_output(
+    return '\n'.join(
-        shlex.split(passcommand),
+        borgmatic.execute.execute_command_and_capture_output(
-        working_directory=working_directory,
+            shlex.split(passcommand),
+            working_directory=working_directory,
+        )
     )
 
 

borgmatic/borg/prune.py

@@ -72,9 +72,9 @@ def prune_archives(
     full_command = (
         (local_path, 'prune')
         + make_prune_flags(config, prune_arguments, local_borg_version)
+        + ('--log-json',)
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (
             ('--stats',)

borgmatic/borg/recreate.py

@@ -46,7 +46,7 @@ def recreate_archive(
     recreate_command = (
         (local_path, 'recreate')
         + (('--remote-path', remote_path) if remote_path else ())
-        + (('--log-json',) if config.get('log_json') else ())
+        + ('--log-json',)
         + (('--lock-wait', str(lock_wait)) if lock_wait is not None else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())

borgmatic/borg/rename.py

@@ -1,6 +1,7 @@
 import logging
 import shlex
 
+import borgmatic.borg.environment
 import borgmatic.borg.flags
 
 logger = logging.getLogger(__name__)
@@ -25,7 +26,7 @@ def make_rename_command(
         + borgmatic.borg.flags.make_flags('dry-run', dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + borgmatic.borg.flags.make_repository_archive_flags(

borgmatic/borg/repo_create.py

@@ -89,7 +89,6 @@ def create_repository(
         + (('--make-parent-dirs',) if make_parent_directories else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
-        + (('--log-json',) if config.get('log_json') else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())

borgmatic/borg/repo_delete.py

@@ -21,11 +21,13 @@ def make_repo_delete_command(
     global_arguments,
     local_path,
     remote_path,
+    output_file,
 ):
     '''
     Given a local or remote repository dict, a configuration dict, the local Borg version, the
-    arguments to the repo_delete action as an argparse.Namespace, and global arguments, return a command
+    arguments to the repo_delete action as an argparse.Namespace, and global arguments, the Borg
-    as a tuple to repo_delete the entire repository.
+    local path, the Borg remote path, and an optional output file, return a command as a tuple to
+    repo_delete the entire repository.
     '''
     extra_borg_options = config.get('extra_borg_options', {}).get(
         'repo_delete'
@@ -51,7 +53,7 @@ def make_repo_delete_command(
         + borgmatic.borg.flags.make_flags('dry-run', global_arguments.dry_run)
         + borgmatic.borg.flags.make_flags('remote-path', remote_path)
         + borgmatic.borg.flags.make_flags('umask', config.get('umask'))
-        + borgmatic.borg.flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if output_file is None else ())
         + borgmatic.borg.flags.make_flags('lock-wait', config.get('lock_wait'))
         + borgmatic.borg.flags.make_flags('list', config.get('list_details'))
         + (
@@ -87,6 +89,14 @@ def delete_repository(
     '''
     borgmatic.logger.add_custom_log_levels()
 
+    # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+    # prompt won't work.
+    output_file = (
+        None
+        if repo_delete_arguments.force or repo_delete_arguments.cache_only
+        else borgmatic.execute.DO_NOT_CAPTURE
+    )
+
     command = make_repo_delete_command(
         repository,
         config,
@@ -95,18 +105,13 @@ def delete_repository(
         global_arguments,
         local_path,
         remote_path,
+        output_file,
     )
 
     borgmatic.execute.execute_command(
         command,
         output_log_level=logging.ANSWER,
-        # Don't capture output when Borg is expected to prompt for interactive confirmation, or the
+        output_file=output_file,
-        # prompt won't work.
-        output_file=(
-            None
-            if repo_delete_arguments.force or repo_delete_arguments.cache_only
-            else borgmatic.execute.DO_NOT_CAPTURE
-        ),
         environment=borgmatic.borg.environment.make_environment(config),
         working_directory=borgmatic.config.paths.get_working_directory(config),
         borg_local_path=local_path,

borgmatic/borg/repo_info.py

@@ -37,6 +37,7 @@ def display_repository_info(
             if feature.available(feature.Feature.REPO_INFO, local_borg_version)
             else ('info',)
         )
+        + (('--critical',) if repo_info_arguments.json else ())
         + (
             ('--info',)
             if logger.getEffectiveLevel() == logging.INFO and not repo_info_arguments.json
@@ -49,8 +50,8 @@ def display_repository_info(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
         + flags.make_flags('lock-wait', lock_wait)
+        + ('--log-json',)
         + (('--json',) if repo_info_arguments.json else ())
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())
         + flags.make_repository_flags(repository_path, local_borg_version)
@@ -60,12 +61,14 @@ def display_repository_info(
     borg_exit_codes = config.get('borg_exit_codes')
 
     if repo_info_arguments.json:
-        return execute_command_and_capture_output(
+        return '\n'.join(
-            full_command,
+            execute_command_and_capture_output(
-            environment=environment.make_environment(config),
+                full_command,
-            working_directory=working_directory,
+                environment=environment.make_environment(config),
-            borg_local_path=local_path,
+                working_directory=working_directory,
-            borg_exit_codes=borg_exit_codes,
+                borg_local_path=local_path,
+                borg_exit_codes=borg_exit_codes,
+            )
         )
 
     execute_command(

borgmatic/borg/repo_list.py

@@ -77,7 +77,7 @@ def get_latest_archive(
         ),
         *flags.make_flags('remote-path', remote_path),
         *flags.make_flags('umask', config.get('umask')),
-        *flags.make_flags('log-json', config.get('log_json')),
+        *('--log-json',),
         *flags.make_flags('lock-wait', config.get('lock_wait')),
         *(
             flags.make_flags('consider-checkpoints', consider_checkpoints)
@@ -90,12 +90,14 @@ def get_latest_archive(
         *flags.make_repository_flags(repository_path, local_borg_version),
     )
 
-    json_output = execute_command_and_capture_output(
+    json_output = '\n'.join(
-        full_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            full_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
     )
 
     archives = json.loads(json_output)['archives']
@@ -153,7 +155,7 @@ def make_repo_list_command(
         )
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + ('--log-json',)
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + (
             (
@@ -217,12 +219,14 @@ def list_repository(
     working_directory = borgmatic.config.paths.get_working_directory(config)
     borg_exit_codes = config.get('borg_exit_codes')
 
-    json_listing = execute_command_and_capture_output(
+    json_listing = '\n'.join(
-        json_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            json_command,
-        working_directory=working_directory,
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=working_directory,
-        borg_exit_codes=borg_exit_codes,
+            borg_local_path=local_path,
+            borg_exit_codes=borg_exit_codes,
+        )
     )
 
     if repo_list_arguments.json:

borgmatic/borg/transfer.py

@@ -33,7 +33,7 @@ def transfer_archives(
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + flags.make_flags('remote-path', remote_path)
         + flags.make_flags('umask', config.get('umask'))
-        + flags.make_flags('log-json', config.get('log_json'))
+        + (('--log-json',) if (config.get('log_json') or not config.get('progress')) else ())
         + flags.make_flags('lock-wait', config.get('lock_wait'))
         + flags.make_flags('progress', config.get('progress'))
         + (

borgmatic/borg/umount.py

@@ -15,6 +15,7 @@ def unmount_archive(config, mount_point, local_path='borg'):
     extra_borg_options = config.get('extra_borg_options', {}).get('umount', '')
     full_command = (
         (local_path, 'umount')
+        + ('--log-json',)
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (tuple(shlex.split(extra_borg_options)) if extra_borg_options else ())

borgmatic/borg/version.py

@@ -16,16 +16,19 @@ def local_borg_version(config, local_path='borg'):
     '''
     full_command = (
         (local_path, '--version')
+        + ('--log-json',)
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
     )
 
-    output = execute_command_and_capture_output(
+    output = '\n'.join(
-        full_command,
+        execute_command_and_capture_output(
-        environment=environment.make_environment(config),
+            full_command,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment.make_environment(config),
-        borg_local_path=local_path,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
-        borg_exit_codes=config.get('borg_exit_codes'),
+            borg_local_path=local_path,
+            borg_exit_codes=config.get('borg_exit_codes'),
+        )
     )
 
     try:

borgmatic/commands/borgmatic.py

@@ -660,6 +660,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                         levelno=logging.DEBUG,
                         levelname='DEBUG',
                         msg=f'{config_filename}: Loading configuration file',
+                        name=logger.name,
                     ),
                 ),
             ],
@@ -682,6 +683,7 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                             levelno=logging.CRITICAL,
                             levelname='CRITICAL',
                             msg=f'{config_filename}: Insufficient permissions to read configuration file',
+                            name=logger.name,
                         ),
                     ),
                 ],
@@ -694,10 +696,16 @@ def load_configurations(config_filenames, arguments, overrides=None, resolve_env
                             levelno=logging.CRITICAL,
                             levelname='CRITICAL',
                             msg=f'{config_filename}: Error parsing configuration file',
+                            name=logger.name,
                         ),
                     ),
                     logging.makeLogRecord(
-                        dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=str(error)),
+                        dict(
+                            levelno=logging.CRITICAL,
+                            levelname='CRITICAL',
+                            msg=str(error),
+                            name=logger.name,
+                        ),
                     ),
                 ],
             )
@@ -710,7 +718,7 @@ def log_record(suppress_log=False, **kwargs):
     Create a log record based on the given makeLogRecord() arguments, one of which must be
     named "levelno". Log the record (unless suppress log is set) and return it.
     '''
-    record = logging.makeLogRecord(kwargs)
+    record = logging.makeLogRecord(dict(kwargs, name=logger.name))
     if suppress_log:
         return record
 
@@ -823,6 +831,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='Bootstrap successful',
+                    name=logger.name,
                 ),
             )
         except (
@@ -845,6 +854,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='Generate successful',
+                    name=logger.name,
                 ),
             )
         except (
@@ -863,6 +873,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.CRITICAL,
                     levelname='CRITICAL',
                     msg='Configuration validation failed',
+                    name=logger.name,
                 ),
             )
 
@@ -876,6 +887,7 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
                     levelno=logging.ANSWER,
                     levelname='ANSWER',
                     msg='All configuration files are valid',
+                    name=logger.name,
                 ),
             )
         except (
@@ -968,6 +980,7 @@ def collect_configuration_run_summary_logs(configs, config_paths, arguments, log
                         levelno=logging.INFO,
                         levelname='INFO',
                         msg=f'{config_filename}: Successfully ran configuration file',
+                        name=logger.name,
                     ),
                 )
                 if results:
@@ -1067,6 +1080,23 @@ def get_singular_option_value(configs, option_name):
         return None
 
 
+def display_summary(summary_logs, log_json):  # pragma: no cover
+    summary_logs_max_level = max(log.levelno for log in summary_logs)
+
+    for message in ('summary:',) if log_json else ('', 'summary:'):
+        log_record(
+            levelno=summary_logs_max_level,
+            levelname=logging.getLevelName(summary_logs_max_level),
+            msg=message,
+        )
+
+    for log in summary_logs:
+        logger.handle(log)
+
+    if summary_logs_max_level >= logging.CRITICAL:
+        exit_with_help_link()
+
+
 def main(extra_summary_logs=()):  # pragma: no cover
     configure_signals()
     configure_delayed_logging()
@@ -1126,6 +1156,7 @@ def main(extra_summary_logs=()):  # pragma: no cover
         getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
     )
     log_file_path = get_singular_option_value(configs, 'log_file')
+    log_json = get_singular_option_value(configs, 'log_json')
 
     try:
         configure_logging(
@@ -1135,7 +1166,8 @@ def main(extra_summary_logs=()):  # pragma: no cover
             verbosity_to_log_level(get_verbosity(configs, 'monitoring_verbosity')),
             log_file_path,
             get_singular_option_value(configs, 'log_file_format'),
-            color_enabled=should_do_markup(configs, any_json_flags),
+            log_json,
+            color_enabled=should_do_markup(configs, any_json_flags or log_json),
         )
     except (FileNotFoundError, PermissionError) as error:
         configure_logging(logging.CRITICAL)
@@ -1163,17 +1195,5 @@ def main(extra_summary_logs=()):  # pragma: no cover
             )
         )
     )
-    summary_logs_max_level = max(log.levelno for log in summary_logs)
 
-    for message in ('', 'summary:'):
+    display_summary(summary_logs, log_json)
-        log_record(
-            levelno=summary_logs_max_level,
-            levelname=logging.getLevelName(summary_logs_max_level),
-            msg=message,
-        )
-
-    for log in summary_logs:
-        logger.handle(log)
-
-    if summary_logs_max_level >= logging.CRITICAL:
-        exit_with_help_link()

borgmatic/config/constants.py

@@ -1,4 +1,6 @@
 import contextlib
+import functools
+import re
 import shlex
 
 
@@ -21,7 +23,44 @@ def coerce_scalar(value):
         return value
 
 
-def apply_constants(value, constants, shell_escape=False):
+CONSTANT_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_constant(match, constants, command_hook):
+    '''
+    Given a re.Match instance of CONSTANT_PATTERN representing a matched constant name to be
+    interpolated, a constants dict, and whether this is for a command hook, lookup the matched
+    constant name within the given constants and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given constants, then return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name—unless this is for a command hook, in which case just return the literal
+    # string. That way, subsequent variable interpolation will still see the string as escaped
+    # instead of trying to interpolate it.
+    if match.group('left_escape') and match.group('right_escape'):
+        if command_hook:
+            return match.group(0)
+
+        return '{' + name + '}'
+
+    value = constants.get(name)
+
+    # The matched variable is in the constants, so return its value. And if this is for a command
+    # hook, then shell escape the value so as to prevent shell injection attacks.
+    if value is not None:
+        return shlex.quote(str(value)) if command_hook else str(value)
+
+    # The matched variable name isn't in the constants. Return the whole string unaltered.
+    return match.group(0)
+
+
+def apply_constants(value, constants, command_hook=False):
     '''
     Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
     replace any configuration string values of the form "{constant}" (or containing it) with the
@@ -39,26 +78,25 @@ def apply_constants(value, constants, shell_escape=False):
         return value
 
     if isinstance(value, str):
-        for constant_name, constant_value in constants.items():
-            value = value.replace(
-                '{' + constant_name + '}',
-                shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
-            )
-
         # Support constants within non-string scalars by coercing the value to its appropriate type.
-        value = coerce_scalar(value)
+        value = coerce_scalar(
+            CONSTANT_PATTERN.sub(
+                functools.partial(resolve_constant, constants=constants, command_hook=command_hook),
+                value,
+            )
+        )
     elif isinstance(value, list):
         for index, list_value in enumerate(value):
-            value[index] = apply_constants(list_value, constants, shell_escape)
+            value[index] = apply_constants(list_value, constants, command_hook)
     elif isinstance(value, dict):
         for option_name, option_value in value.items():
             value[option_name] = apply_constants(
                 option_value,
                 constants,
-                shell_escape=(
+                command_hook=(
-                    shell_escape
+                    command_hook
                     or option_name.startswith(('before_', 'after_'))
-                    or option_name == 'on_error'
+                    or option_name in {'on_error', 'run'}
                 ),
             )
 

borgmatic/config/schema.yaml

@@ -21,11 +21,7 @@ properties:
             type: string
         description: |
             List of source directories and files to back up. Globs and tildes
-            are expanded. Do not backslash spaces in path names. Be aware that
+            are expanded. Do not backslash spaces in path names.
-            by default, Borg treats missing source directories as warnings
-            rather than errors. If you'd like to change that behavior, see
-            https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
-            or the "source_directories_must_exist" option.
         example:
             - /home
             - /etc
@@ -34,8 +30,11 @@ properties:
     source_directories_must_exist:
         type: boolean
         description: |
-            If true, then source directories (and root pattern paths) must
+            Deprecated. Replaced by borgmatic treating Borg's "backup file not
-            exist. If they don't, an error is raised. Defaults to false.
+            found" warning as an error by default. But if
+            "source_directories_must_exist" is true, then source directories
+            (and root pattern paths) must exist before a backup begins. If they
+            don't, borgmatic errors. Defaults to false.
         example: true
     repositories:
         type: array
@@ -144,6 +143,23 @@ properties:
             false. But when a database hook is used, the setting here is ignored
             and read_special is considered true.
         example: true
+    unsafe_skip_path_validation_before_create:
+        type: boolean
+        description: |
+            Skip pre-backup safety validation: (1) runtime directory exclusion
+            detection, and (2) special file exclusion for database streaming.
+            Can significantly improve performance for large filesystems.
+
+            UNSAFE because, when this option is enabled, Borg may hang on
+            special files (named pipes, devices) and data from borgmatic data
+            source hooks may be silently excluded if your excluded patterns
+            match the runtime directory (see the "user_runtime_directory"
+            option).
+
+            Only enable if not using data source hooks and you manually verify
+            that your excludes don't affect the runtime directory. Defaults
+            to false.
+        example: true
     flags:
         type: boolean
         description: |
@@ -452,9 +468,9 @@ properties:
         description: |
             A list of Borg exit codes that should be elevated to errors or
             squashed to warnings as indicated. By default, Borg error exit codes
-            (2 to 99) are treated as errors while warning exit codes (1 and
+            (2 to 99) are treated as errors and most warning exit codes (1 and
-            100+) are treated as warnings. Exit codes other than 1 and 2 are
+            100+) are treated as errors as well. Exit codes other than 1 and 2
-            only present in Borg 1.4.0+.
+            are only present in Borg 1.4.0+.
         example:
             - code: 13
               treat_as: warning
@@ -2790,8 +2806,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to Apprise services as part of the
-                    "finish", "fail", and "log" states. Defaults to true.
+                    "finish", "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
             logs_size_limit:
                 type: integer
                 description: |
@@ -2905,8 +2921,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to Healthchecks as part of the "finish",
-                    "fail", and "log" states. Defaults to true.
+                    "fail", and "log" states. Defaults to false.
-                example: false
+                example: true
             ping_body_limit:
                 type: integer
                 description: |
@@ -3012,8 +3028,8 @@ properties:
                 type: boolean
                 description: |
                     Send borgmatic logs to PagerDuty when a backup errors.
-                    Defaults to true.
+                    Defaults to false.
-                example: false
+                example: true
         description: |
             Configuration for a monitoring integration with PagerDuty. Create an
             account at https://www.pagerduty.com if you'd like to use this
@@ -3041,7 +3057,7 @@ properties:
             url:
                 type: string
                 description: |
-                    Grafana loki log URL to notify when a backup begins,
+                    Grafana Loki log URL to notify when a backup begins,
                     ends, or fails.
                 example: "http://localhost:3100/loki/api/v1/push"
             labels:
@@ -3058,6 +3074,11 @@ properties:
                     app: "borgmatic"
                     config: "__config"
                     hostname: "__hostname"
+            send_logs:
+                type: boolean
+                description: |
+                    Send borgmatic logs to Loki. Defaults to false.
+                example: true
         description: |
             Configuration for a monitoring integration with Grafana Loki. You
             can send the logs to a self-hosted instance or create an account at

borgmatic/execute.py

@@ -1,9 +1,12 @@
 import collections
+import contextlib
 import enum
+import json
 import logging
 import select
 import subprocess
 import textwrap
+import time
 
 import borgmatic.logger
 
@@ -14,6 +17,9 @@ ERROR_OUTPUT_MAX_LINE_COUNT = 25
 BORG_ERROR_EXIT_CODE_START = 2
 BORG_ERROR_EXIT_CODE_END = 99
 
+# See https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids
+BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS = {101, 102, 104, 105, 106, 107}
+
 
 class Exit_status(enum.Enum):
     STILL_RUNNING = 1
@@ -22,7 +28,7 @@ class Exit_status(enum.Enum):
     ERROR = 4
 
 
-def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
+def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):  # noqa: PLR0911
     '''
     Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
     exit code. If a Borg local path is given and matches the process' command, then interpret the
@@ -53,8 +59,15 @@ def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_code
                     )
                     return Exit_status.WARNING
 
-        # If the exit code doesn't have explicit configuration, then fall back to the default Borg
+        # If the exit code doesn't have explicit configuration, then fall back to the default
-        # behavior.
+        # behavior of treating Borg errors as errors and some Borg warnings as errors.
+        if exit_code in BORG_WARNING_EXIT_CODES_TREATED_AS_ERRORS:
+            logger.error(
+                f'Treating exit code {exit_code} as an error, as per borgmatic defaults',
+            )
+
+            return Exit_status.ERROR
+
         return (
             Exit_status.ERROR
             if (
@@ -78,58 +91,146 @@ def command_for_process(process):
     return process.args if isinstance(process.args, str) else ' '.join(process.args)
 
 
-def output_buffer_for_process(process, exclude_stdouts):
+def output_buffers_for_process(process, exclude_stdouts):
     '''
     Given a process as an instance of subprocess.Popen and a sequence of stdouts to exclude, return
-    either the process's stdout or stderr. The idea is that if stdout is excluded for a process, we
+    the process stdout and stderr as a tuple—but exclude the stdout if it's in the given stdouts to
-    still have stderr to log.
+    exclude.
     '''
-    return process.stderr if process.stdout in exclude_stdouts else process.stdout
+    return tuple(
+        buffer for buffer in (process.stdout, process.stderr) if buffer not in exclude_stdouts
+    )
 
 
-def append_last_lines(last_lines, captured_output, line, output_log_level):
+def borg_json_log_line_to_record(line, log_level):
     '''
-    Given a rolling list of last lines, a list of captured output, a line to append, and an output
+    Given a single Borg "--log-json"-style log line and a log level, return the line converted to a
-    log level, append the line to the last lines and (if necessary) the captured output. Then log
+    logging.LogRecord instance. Return None if the line can't be parsed as JSON.
-    the line at the requested output log level.
     '''
-    last_lines.append(line)
+    with contextlib.suppress(json.JSONDecodeError, TypeError, KeyError, AttributeError):
+        log_data = json.loads(line)
+        log_type = log_data.get('type')
+
+        if log_type == 'log_message':
+            return logging.makeLogRecord(
+                dict(
+                    levelno=logging._nameToLevel.get(log_data.get('levelname')),
+                    created=log_data.get('time'),
+                    msg=log_data.get('message'),
+                    levelname=log_data.get('levelname'),
+                    name=log_data.get('name'),
+                )
+            )
+
+        if log_type == 'file_status':
+            return logging.makeLogRecord(
+                dict(
+                    levelno=log_level,
+                    created=time.time(),
+                    msg=f'{log_data.get("status")} {log_data.get("path")}',
+                    levelname=logging.getLevelName(log_level),
+                    name='borg.file_status',
+                )
+            )
+
+    return None
+
+
+def log_line_to_record(line, log_level):
+    '''
+    Given a log data dict for a single Borg log entry and a log level, return it converted to a
+    logging.LogRecord instance.
+    '''
+    return logging.makeLogRecord(
+        dict(
+            msg=line,
+            levelno=log_level,
+            levelname=logging.getLevelName(log_level),
+        )
+    )
+
+
+def parse_log_line(line, log_level, elevate_stderr, borg_local_path, command):
+    '''
+    Given a raw output line from an external program, whether this line came from stderr and should
+    be elevated to error/warning, the Borg local path, and the command as a sequence, return a
+    logging.LogRecord instance containing its parsed data.
+
+    If the command being run is Borg, and the log line is JSON-formatted log data, then grab the log
+    level from it and log the parsed JSON to be consumed later by a Python logging.Formatter.
+
+    But for non-Borg commands, elevate stderr-sourced logs to ERROR. The one exception is if the log
+    came from stderr and the string "warning:" appears at the start of the log line. In that case,
+    just elevate the log level to a WARN.
+    '''
+    if borg_local_path and command[0] == borg_local_path:
+        log_record = borg_json_log_line_to_record(line, log_level)
+
+        if log_record:
+            return log_record
+
+    if elevate_stderr:
+        return log_line_to_record(
+            line, logging.WARNING if line.lower().startswith('warning:') else logging.ERROR
+        )
+
+    return log_line_to_record(line, log_level)
+
+
+def handle_log_record(log_record, last_lines):
+    '''
+    Given a log record to be logged and a rolling list of last lines, append the record's message to
+    the last lines. Then (if the log level is not None), log the record.
+
+    Return the log record.
+    '''
+    log_message = log_record.getMessage()
+    last_lines.append(log_message)
 
     if len(last_lines) > ERROR_OUTPUT_MAX_LINE_COUNT:
         last_lines.pop(0)
 
-    if output_log_level is None:
+    if log_record.levelno is not None:
-        captured_output.append(line)
+        logger.handle(log_record)
-    else:
+
-        logger.log(output_log_level, line)
+    return log_record
 
 
-def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):  # noqa: PLR0912
+def log_outputs(  # noqa: PLR0912
+    processes,
+    exclude_stdouts,
+    output_log_level,
+    borg_local_path,
+    borg_exit_codes,
+    capture_stderr=False,
+):
     '''
-    Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
+    Given a sequence of subprocess.Popen() instances for multiple processes, log the outputs (stderr
-    process with the requested log level. Additionally, raise a CalledProcessError if a process
+    and stdout). Use the requested output log level for stdout, but always log stderr to the ERROR
-    exits with an error (or a warning for exit code 1, if that process does not match the Borg local
+    log level. Additionally, raise a CalledProcessError if a process exits with an error (or a
-    path).
+    warning for exit code 1, if that process does not match the Borg local path).
 
-    If output log level is None, then instead of logging, capture output for each process and return
+    If the output log level is None, then instead of logging, capture the output for the last
-    it as a dict from the process to its output. Use the given Borg local path and exit code
+    process given and yield it one line at a time. This includes stderr if capture stderr is set.
-    configuration to decide what's an error and what's a warning.
+    But if the output log level is not None, don't yield anything.
 
-    For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
+    This yielding means that this function is a generator, and must be consumed in order to execute.
-    stdouts are given to exclude, then for any matching processes, log from their stderr instead.
 
-    Note that stdout for a process can be None if output is intentionally not captured. In which
+    Use the given Borg local path and exit code configuration to decide what's an error and what's a
-    case it won't be logged.
+    warning. If any stdouts are given to exclude, then for any matching processes, ignore those
+    buffers. Also note that stdout for a process can be None if output is intentionally not
+    captured, in which case it won't be logged.
     '''
     # Map from output buffer to sequence of last lines.
-    buffer_last_lines = collections.defaultdict(list)
+    process_last_lines = collections.defaultdict(list)
     process_for_output_buffer = {
-        output_buffer_for_process(process, exclude_stdouts): process
+        buffer: process
         for process in processes
         if process.stdout or process.stderr
+        for buffer in output_buffers_for_process(process, exclude_stdouts)
     }
     output_buffers = list(process_for_output_buffer.keys())
-    captured_outputs = collections.defaultdict(list)
+    process_to_capture = processes[-1]
     still_running = True
 
     # Log output for each process until they all exit.
@@ -158,15 +259,30 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
                     if not line or not ready_process:
                         break
 
-                    # Keep the last few lines of output in case the process errors, and we need the
+                    command = (
-                    # output for the exception below.
+                        ready_process.args.split(' ')
-                    append_last_lines(
+                        if isinstance(ready_process.args, str)
-                        buffer_last_lines[ready_buffer],
+                        else ready_process.args
-                        captured_outputs[ready_process],
-                        line,
-                        output_log_level,
                     )
 
+                    # Keep the last few lines of output in case the process errors and we need the
+                    # output for the exception below.
+                    log_record = handle_log_record(
+                        parse_log_line(
+                            line=line,
+                            log_level=output_log_level,
+                            elevate_stderr=(
+                                ready_buffer == ready_process.stderr and not capture_stderr
+                            ),
+                            borg_local_path=borg_local_path,
+                            command=command,
+                        ),
+                        last_lines=process_last_lines[ready_process],
+                    )
+
+                    if log_record.levelno is None and ready_process == process_to_capture:
+                        yield log_record.getMessage()
+
         if not still_running:
             break
 
@@ -184,23 +300,32 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
             exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
 
             if exit_status in {Exit_status.ERROR, Exit_status.WARNING}:
+                last_lines = process_last_lines[process]
+
                 # If an error occurs, include its output in the raised exception so that we don't
                 # inadvertently hide error output.
-                output_buffer = output_buffer_for_process(process, exclude_stdouts)
+                for output_buffer in output_buffers_for_process(process, exclude_stdouts):
-                last_lines = buffer_last_lines[output_buffer] if output_buffer else []
+                    # Collect any straggling output lines that came in since we last gathered output.
+                    while output_buffer:  # pragma: no cover
+                        line = output_buffer.readline().rstrip().decode()
+                        if not line:
+                            break
 
-                # Collect any straggling output lines that came in since we last gathered output.
+                        log_record = handle_log_record(
-                while output_buffer:  # pragma: no cover
+                            parse_log_line(
-                    line = output_buffer.readline().rstrip().decode()
+                                line=line,
-                    if not line:
+                                log_level=output_log_level,
-                        break
+                                elevate_stderr=(
+                                    output_buffer == process.stderr and not capture_stderr
+                                ),
+                                borg_local_path=borg_local_path,
+                                command=command,
+                            ),
+                            last_lines=last_lines,
+                        )
 
-                    append_last_lines(
+                        if log_record.levelno is None and process == process_to_capture:
-                        last_lines,
+                            yield log_record.getMessage()
-                        captured_outputs[process],
-                        line,
-                        output_log_level,
-                    )
 
                 if len(last_lines) == ERROR_OUTPUT_MAX_LINE_COUNT:
                     last_lines.insert(0, '...')
@@ -222,13 +347,6 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, b
                 still_running = False
                 break
 
-    if captured_outputs:
-        return {
-            process: '\n'.join(output_lines) for process, output_lines in captured_outputs.items()
-        }
-
-    return None
-
 
 SECRET_COMMAND_FLAG_NAMES = {'--password'}
 
@@ -300,16 +418,16 @@ def execute_command(
     close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. If an open output file object is given, then write stdout to the file and only
+    the given log level. If an open output file object is given, then write stdout to the file and
-    log stderr. If an open input file object is given, then read stdin from the file. If shell is
+    only log stderr. If an open input file object is given, then read stdin from the file. If shell
-    True, execute the command within a shell. If an environment variables dict is given, then pass
+    is True, execute the command within a shell. If an environment variables dict is given, then
-    it into the command. If a working directory is given, use that as the present working directory
+    pass it into the command. If a working directory is given, use that as the present working
-    when running the command. If a Borg local path is given, and the command matches it (regardless
+    directory when running the command. If a Borg local path is given, and the command matches it
-    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    (regardless of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit
-    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    codes are given as a sequence of exit code configuration dicts, then use that configuration to
-    what's an error and what's a warning. If run to completion is False, then return the process for
+    decide what's an error and what's a warning. If run to completion is False, then return the
-    the command without executing it to completion.
+    process for the command without executing it to completion.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
@@ -321,7 +439,7 @@ def execute_command(
         command,
         stdin=input_file,
         stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-        stderr=None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT),
+        stderr=None if do_not_capture else subprocess.PIPE,
         shell=shell,
         env=environment,
         cwd=working_directory,
@@ -331,12 +449,14 @@ def execute_command(
         return process
 
     with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        log_outputs(
+        tuple(
-            (process,),
+            log_outputs(
-            (input_file, output_file),
+                (process,),
-            output_log_level,
+                (input_file, output_file),
-            borg_local_path,
+                output_log_level,
-            borg_exit_codes,
+                borg_local_path,
+                borg_exit_codes,
+            )
         )
 
     return None
@@ -355,14 +475,16 @@ def execute_command_and_capture_output(
 ):
     '''
     Execute the given command (a sequence of command/argument strings), capturing and returning its
-    output (stdout). If an input file descriptor is given, then pipe it to the command's stdin. If
+    output (stdout) as a generator that yields one line at a time. The generator must be consumed in
-    capture stderr is True, then capture and return stderr in addition to stdout. If shell is True,
+    order for the called command to execute.
-    execute the command within a shell. If an environment variables dict is given, then pass it into
+
-    the command. If a working directory is given, use that as the present working directory when
+    If an input file descriptor is given, then pipe it to the command's stdin. If capture stderr is
-    running the command. If a Borg local path is given, and the command matches it (regardless of
+    True, then capture stderr in addition to stdout. If shell is True, execute the command within a
-    arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are given
+    shell. If an environment variables dict is given, then pass it into the command. If a working
-    as a sequence of exit code configuration dicts, then use that configuration to decide what's an
+    directory is given, use that as the present working directory when running the command. If a
-    error and what's a warning.
+    Borg local path is given, and the command matches it (regardless of arguments), treat exit code
+    1 as a warning instead of an error. But if Borg exit codes are given as a sequence of exit code
+    configuration dicts, then use that configuration to decide what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command.
     '''
@@ -370,10 +492,11 @@ def execute_command_and_capture_output(
     command = ' '.join(full_command) if shell else full_command
 
     try:
-        output = subprocess.check_output(  # noqa: S603
+        process = subprocess.Popen(  # noqa: S603
             command,
             stdin=input_file,
-            stderr=subprocess.STDOUT if capture_stderr else None,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
             shell=shell,
             env=environment,
             cwd=working_directory,
@@ -386,9 +509,22 @@ def execute_command_and_capture_output(
         ):
             raise
 
-        output = error.output
+        if error.output is not None:
+            yield from iter(error.output.decode().splitlines())
 
-    return output.decode() if output is not None else None
+        return
+
+    with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
+        captured_lines = log_outputs(
+            (process,),
+            (input_file,),
+            None,
+            borg_local_path,
+            borg_exit_codes,
+            capture_stderr=capture_stderr,
+        )
+
+    yield from captured_lines
 
 
 def execute_command_with_processes(
@@ -405,20 +541,23 @@ def execute_command_with_processes(
     close_fds=False,  # Necessary for passing credentials via anonymous pipe.
 ):
     '''
-    Execute the given command (a sequence of command/argument strings) and log its output at the
+    Execute the given command (a sequence of command/argument strings) and log its stdout output at
-    given log level. Simultaneously, continue to poll one or more active processes so that they
+    the given log level. Simultaneously, continue to poll one or more active processes so that they
     run as well. This is useful, for instance, for processes that are streaming output to a named
     pipe that the given command is consuming from.
 
     If an open output file object is given, then write stdout to the file and only log stderr. But
     if output log level is None, instead suppress logging and return the captured output for (only)
-    the given command. If an open input file object is given, then read stdin from the file. If
+    the given command as a generator that yields one line at a time. The generator must be consumed
-    shell is True, execute the command within a shell. If an environment variables dict is given,
+    in order for the called command to execute—regardless of the output log level.
-    then pass it into the command. If a working directory is given, use that as the present working
+
-    directory when running the command. If a Borg local path is given, then for any matching command
+    If an open input file object is given, then read stdin from the file. If shell is True, execute
-    or process (regardless of arguments), treat exit code 1 as a warning instead of an error. But if
+    the command within a shell. If an environment variables dict is given, then pass it into the
-    Borg exit codes are given as a sequence of exit code configuration dicts, then use that
+    command. If a working directory is given, use that as the present working directory when running
-    configuration to decide what's an error and what's a warning.
+    the command. If a Borg local path is given, then for any matching command or process (regardless
+    of arguments), treat exit code 1 as a warning instead of an error. But if Borg exit codes are
+    given as a sequence of exit code configuration dicts, then use that configuration to decide
+    what's an error and what's a warning.
 
     Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
     upstream process.
@@ -432,9 +571,7 @@ def execute_command_with_processes(
             command,
             stdin=input_file,
             stdout=None if do_not_capture else (output_file or subprocess.PIPE),
-            stderr=(
+            stderr=None if do_not_capture else subprocess.PIPE,
-                None if do_not_capture else (subprocess.PIPE if output_file else subprocess.STDOUT)
-            ),
             shell=shell,
             env=environment,
             cwd=working_directory,
@@ -451,7 +588,7 @@ def execute_command_with_processes(
         raise
 
     with borgmatic.logger.Log_prefix(None):  # Log command output without any prefix.
-        captured_outputs = log_outputs(
+        captured_lines = log_outputs(
             (*processes, command_process),
             (input_file, output_file),
             output_log_level,
@@ -459,7 +596,4 @@ def execute_command_with_processes(
             borg_exit_codes,
         )
 
-    if output_log_level is None:
+    yield from captured_lines
-        return captured_outputs.get(command_process)
-
-    return None

borgmatic/hooks/command.py

@@ -13,39 +13,71 @@ logger = logging.getLogger(__name__)
 
 
 SOFT_FAIL_EXIT_CODE = 75
-BORG_PLACEHOLDERS = {
+BORG_PLACEHOLDER_NAMES = {
-    '{hostname}',
+    'hostname',
-    '{fqdn}',
+    'fqdn',
-    '{reverse-fqdn}',
+    'reverse-fqdn',
-    '{now}',
+    'now',
-    '{utcnow}',
+    'utcnow',
-    '{unixtime}',
+    'unixtime',
-    '{user}',
+    'user',
-    '{pid}',
+    'pid',
-    '{borgversion}',
+    'borgversion',
-    '{borgmajor}',
+    'borgmajor',
-    '{borgminor}',
+    'borgminor',
-    '{borgpatch}',
+    'borgpatch',
 }
 
 
+VARIABLE_PATTERN = re.compile(r'(?P<left_escape>\\)?\{(?P<name>[\w]+)(?P<right_escape>\\)?\}')
+
+
+def resolve_variable(match, context, hook_description):
+    '''
+    Given a re.Match instance of VARIABLE_PATTERN representing a matched variable name to be
+    interpolated, a context dict, and a description of the current command hook, lookup the matched
+    variable name within the given context and return its value.
+
+    If the match is escaped with backslashes, then instead of resolving the variable's value, strip
+    off the backslashing and return the literal value.
+
+    If the variable name isn't found in the given context (and isn't a Borg placeholder), then
+    warn and return the literal value.
+    '''
+    name = match.group('name')
+
+    # The would-be variable is escaped, so strip off the escaping and return the result without
+    # resolving the name.
+    if match.group('left_escape') and match.group('right_escape'):
+        return '{' + name + '}'
+
+    value = context.get(name)
+
+    # The matched variable is in the context, so return its value.
+    if value is not None:
+        return shlex.quote(str(value))
+
+    # The matched variable name isn't in the context. Warn about variables unknown to borgmatic, but
+    # don't warn if the variable name happens to be a Borg placeholder, as Borg should hopefully
+    # consume it.
+    if name not in BORG_PLACEHOLDER_NAMES:
+        logger.warning(
+            f'Variable "{name}" is not supported in the {hook_description} hook',
+        )
+
+    # Return the whole string unaltered.
+    return match.group(0)
+
+
 def interpolate_context(hook_description, command, context):
     '''
     Given a config filename, a hook description, a single hook command, and a dict of context
     names/values, interpolate the values by "{name}" into the command and return the result.
     '''
-    for name, value in context.items():
+    return VARIABLE_PATTERN.sub(
-        command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
+        functools.partial(resolve_variable, context=context, hook_description=hook_description),
-
+        command,
-    for unsupported_variable in re.findall(r'\{\w+\}', command):
+    )
-        # Warn about variables unknown to borgmatic, but don't warn if the variable name happens to
-        # be a Borg placeholder, as Borg should hopefully consume it.
-        if unsupported_variable not in BORG_PLACEHOLDERS:
-            logger.warning(
-                f'Variable "{unsupported_variable}" is not supported in the {hook_description} hook',
-            )
-
-    return command
 
 
 def make_environment(current_environment, sys_module=sys):
@@ -147,6 +179,7 @@ def execute_hooks(command_hooks, umask, working_directory, dry_run, **context):
                     shell=True,
                     environment=make_environment(os.environ),
                     working_directory=working_directory,
+                    close_fds=True,
                 )
         finally:
             if original_umask:

borgmatic/hooks/credential/keepassxc.py

@@ -43,4 +43,6 @@ def load_credential(hook_config, config, credential_parameters):
         + (expanded_database_path, attribute_name)  # Ensure database and entry are last.
     )
 
-    return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+    return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+        os.linesep
+    )

borgmatic/hooks/credential/systemd.py

@@ -48,7 +48,9 @@ def load_credential(hook_config, config, credential_parameters):
             ),
         )
 
-        return borgmatic.execute.execute_command_and_capture_output(command).rstrip(os.linesep)
+        return '\n'.join(borgmatic.execute.execute_command_and_capture_output(command)).rstrip(
+            os.linesep
+        )
 
     try:
         with open(

borgmatic/hooks/data_source/bootstrap.py

@@ -116,6 +116,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores are implemented via the separate, purpose-specific "bootstrap" action rather than the

borgmatic/hooks/data_source/btrfs.py

@@ -53,17 +53,19 @@ def get_subvolume_property(btrfs_command, subvolume_path, property_name):
     As a performance optimization, multiple calls to this function with the same arguments are
     cached.
     '''
-    output = borgmatic.execute.execute_command_and_capture_output(
+    output = '\n'.join(
-        (
+        borgmatic.execute.execute_command_and_capture_output(
-            *btrfs_command.split(' '),
+            (
-            'property',
+                *btrfs_command.split(' '),
-            'get',
+                'property',
-            '-t',  # Type.
+                'get',
-            'subvol',
+                '-t',  # Type.
-            subvolume_path,
+                'subvol',
-            property_name,
+                subvolume_path,
-        ),
+                property_name,
-        close_fds=True,
+            ),
+            close_fds=True,
+        )
     )
 
     try:
@@ -412,6 +414,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/data_source/config.py

@@ -72,13 +72,15 @@ def get_ip_from_container(container):
     last_error = None
     for engine in engines:
         try:
-            output = execute_command_and_capture_output(
+            output = '\n'.join(
-                (
+                execute_command_and_capture_output(
-                    engine,
+                    (
-                    'container',
+                        engine,
-                    'inspect',
+                        'container',
-                    '--format={{json .NetworkSettings}}',
+                        'inspect',
-                    container,
+                        '--format={{json .NetworkSettings}}',
+                        container,
+                    )
                 )
             )
         except subprocess.CalledProcessError as error:

borgmatic/hooks/data_source/dump.py

@@ -2,6 +2,7 @@ import fnmatch
 import json
 import logging
 import os
+import re
 import shutil
 
 import borgmatic.actions.restore
@@ -114,16 +115,33 @@ def remove_data_source_dumps(dump_path, data_source_type_name, dry_run):
         shutil.rmtree(dump_path)
 
 
+END_OF_STRING_PATTTERN = re.compile(r'\\z', flags=re.IGNORECASE)
+
+
 def convert_glob_patterns_to_borg_pattern(patterns):
     '''
     Convert a sequence of shell glob patterns like "/etc/*", "/tmp/*" to the corresponding Borg
     regular expression archive pattern as a single string like "re:etc/.*|tmp/.*".
     '''
-    # Remove the "\z" or "\Z" generated by fnmatch.translate() because we don't want the pattern to
+    # This deserves some explanation. Here's what this code is doing for each shell glob pattern:
-    # match only at the end of a path, as directory format dumps require extracting files with paths
+    #
-    # longer than the pattern. E.g., a pattern of "borgmatic/*/foo_databases/test" should also match
+    #  * Strip off the leading slash, since Borg doesn't store leading slashes in archives.
-    # paths like "borgmatic/*/foo_databases/test/toc.dat"
+    #  * Use fnmatch.translate() to convert the pattern to a Borg pattern.
+    #  * Remove the "\z" or \"Z" end-of-string special character generated by fnmatch.translate(),
+    #    because Borg doesn't like it. Replace it with a "$" end-of-string special character
+    #    instead. And yes, this is using a regular expression to modify a regular expression.
+    #  * Do the above for each of:
+    #    * The plain pattern. This supports the use case of a standard database where its name is a
+    #      filename in the path. Example: borgmatic/foo_databases/hostname/test
+    #    * The pattern with "/*" tacked onto the end of it. This supports the use case of a
+    #      directory-format database where its name is a directory name in the path. Example:
+    #      borgmatic/foo_databases/hostname/test/toc.dat
+    #
+    # Join the resulting transformed patterns together with "|" and return them as a string.
     return 're:' + '|'.join(
-        fnmatch.translate(pattern.lstrip('/')).replace('\\z', '').replace('\\Z', '')
+        re.sub(END_OF_STRING_PATTTERN, '$', fnmatch.translate(stripped))
+        + '|'
+        + re.sub(END_OF_STRING_PATTTERN, '$', fnmatch.translate(stripped + '/*'))
         for pattern in patterns
+        for stripped in (pattern.lstrip('/'),)
     )

borgmatic/hooks/data_source/lvm.py

@@ -44,17 +44,19 @@ def get_logical_volumes(lsblk_command, patterns=None):
     '''
     try:
         devices_info = json.loads(
-            borgmatic.execute.execute_command_and_capture_output(
+            '\n'.join(
-                # Use lsblk instead of lvs here because lvs can't show active mounts.
+                borgmatic.execute.execute_command_and_capture_output(
-                (
+                    # Use lsblk instead of lvs here because lvs can't show active mounts.
-                    *lsblk_command.split(' '),
+                    (
-                    '--output',
+                        *lsblk_command.split(' '),
-                    'name,path,mountpoint,type',
+                        '--output',
-                    '--json',
+                        'name,path,mountpoint,type',
-                    '--list',
+                        '--json',
+                        '--list',
+                    ),
+                    close_fds=True,
                 ),
-                close_fds=True,
+            )
-            ),
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {lsblk_command} JSON output: {error}')
@@ -321,19 +323,21 @@ def get_snapshots(lvs_command, snapshot_name=None):
     '''
     try:
         snapshot_info = json.loads(
-            borgmatic.execute.execute_command_and_capture_output(
+            '\n'.join(
-                # Use lvs instead of lsblk here because lsblk can't filter to just snapshots.
+                borgmatic.execute.execute_command_and_capture_output(
-                (
+                    # Use lvs instead of lsblk here because lsblk can't filter to just snapshots.
-                    *lvs_command.split(' '),
+                    (
-                    '--report-format',
+                        *lvs_command.split(' '),
-                    'json',
+                        '--report-format',
-                    '--options',
+                        'json',
-                    'lv_name,lv_path',
+                        '--options',
-                    '--select',
+                        'lv_name,lv_path',
-                    'lv_attr =~ ^s',  # Filter to just snapshots.
+                        '--select',
+                        'lv_attr =~ ^s',  # Filter to just snapshots.
+                    ),
+                    close_fds=True,
                 ),
-                close_fds=True,
+            )
-            ),
         )
     except json.JSONDecodeError as error:
         raise ValueError(f'Invalid {lvs_command} JSON output: {error}')
@@ -452,6 +456,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/data_source/mariadb.py

@@ -158,15 +158,15 @@ def database_names_to_dump(database, config, username, password, environment, dr
     if skip_names:
         logger.debug(f'Skipping database names: {", ".join(skip_names)}')
 
-    show_output = execute_command_and_capture_output(
+    show_lines = execute_command_and_capture_output(
         show_command,
         environment=environment,
         working_directory=borgmatic.config.paths.get_working_directory(config),
     )
 
     return tuple(
-        show_name
+        show_name.strip()
-        for show_name in show_output.strip().splitlines()
+        for show_name in show_lines
         if show_name not in SYSTEM_DATABASE_NAMES
         if not skip_names or show_name not in skip_names
     )
@@ -410,6 +410,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -419,16 +423,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -504,11 +516,14 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
-        environment=environment,
+            input_file=extract_process.stdout,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/mongodb.py

@@ -212,6 +212,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -221,16 +225,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -277,12 +289,15 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process] if extract_process else [],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process] if extract_process else [],
-        input_file=extract_process.stdout if extract_process else None,
+            output_log_level=logging.DEBUG,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            input_file=extract_process.stdout if extract_process else None,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )
 
 

borgmatic/hooks/data_source/mysql.py

@@ -87,15 +87,15 @@ def database_names_to_dump(database, config, username, password, environment, dr
     if skip_names:
         logger.debug(f'Skipping database names: {", ".join(skip_names)}')
 
-    show_output = execute_command_and_capture_output(
+    show_lines = execute_command_and_capture_output(
         show_command,
         environment=environment,
         working_directory=borgmatic.config.paths.get_working_directory(config),
     )
 
     return tuple(
-        show_name
+        show_name.strip()
-        for show_name in show_output.strip().splitlines()
+        for show_name in show_lines
         if show_name not in SYSTEM_DATABASE_NAMES
         if not skip_names or show_name not in skip_names
     )
@@ -341,6 +341,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -350,16 +354,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -441,11 +453,14 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
-        environment=environment,
+            input_file=extract_process.stdout,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/postgresql.py

@@ -103,7 +103,7 @@ def database_names_to_dump(database, config, environment, dry_run):
         + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
     )
     logger.debug('Querying for "all" PostgreSQL databases to dump')
-    list_output = execute_command_and_capture_output(
+    list_lines = execute_command_and_capture_output(
         list_command,
         environment=environment,
         working_directory=borgmatic.config.paths.get_working_directory(config),
@@ -111,7 +111,7 @@ def database_names_to_dump(database, config, environment, dry_run):
 
     return tuple(
         row[0]
-        for row in csv.reader(list_output.splitlines(), delimiter=',', quotechar='"')
+        for row in csv.reader(list_lines, delimiter=',', quotechar='"')
         if row[0] not in EXCLUDED_DATABASE_NAMES
     )
 
@@ -302,6 +302,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -311,16 +315,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -422,13 +434,16 @@ def restore_data_source_dump(
 
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process] if extract_process else [],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process] if extract_process else [],
-        input_file=extract_process.stdout if extract_process else None,
+            output_log_level=logging.DEBUG,
-        environment=environment,
+            input_file=extract_process.stdout if extract_process else None,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            environment=environment,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )
     execute_command(
         analyze_command,

borgmatic/hooks/data_source/sqlite.py

@@ -144,6 +144,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Given a sequence of configurations dicts, a configuration dict, the borgmatic runtime directory,
@@ -153,16 +157,24 @@ def make_data_source_dump_patterns(
     borgmatic_source_directory = borgmatic.config.paths.get_borgmatic_source_directory(config)
 
     return (
-        dump.make_data_source_dump_filename(make_dump_path('borgmatic'), name, label='*'),
+        dump.make_data_source_dump_filename(
+            make_dump_path('borgmatic'), name, hostname, port, container, label
+        ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_runtime_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
         dump.make_data_source_dump_filename(
             make_dump_path(borgmatic_source_directory),
             name,
-            label='*',
+            hostname,
+            port,
+            container,
+            label,
         ),
     )
 
@@ -203,12 +215,16 @@ def restore_data_source_dump(
         for part in shlex.split(data_source.get('sqlite_restore_command') or 'sqlite3')
     )
     restore_command = (*sqlite_restore_command, '-bail', shlex.quote(database_path))
+
     # Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
     # if the restore paths don't exist in the archive.
-    execute_command_with_processes(
+    tuple(
-        restore_command,
+        execute_command_with_processes(
-        [extract_process],
+            restore_command,
-        output_log_level=logging.DEBUG,
+            [extract_process],
-        input_file=extract_process.stdout,
+            output_log_level=logging.DEBUG,
-        working_directory=borgmatic.config.paths.get_working_directory(config),
+            input_file=extract_process.stdout,
+            working_directory=borgmatic.config.paths.get_working_directory(config),
+            borg_local_path=config.get('local_path', 'borg'),
+        )
     )

borgmatic/hooks/data_source/zfs.py

@@ -45,7 +45,7 @@ def get_datasets_to_backup(zfs_command, patterns):
 
     Return the result as a sequence of Dataset instances, sorted by mount point.
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *zfs_command.split(' '),
             'list',
@@ -65,7 +65,7 @@ def get_datasets_to_backup(zfs_command, patterns):
         datasets = sorted(
             (
                 Dataset(dataset_name, mount_point, (user_property_value == 'auto'), ())
-                for line in list_output.splitlines()
+                for line in list_lines
                 for (dataset_name, mount_point, can_mount, user_property_value) in (
                     line.rstrip().split('\t'),
                 )
@@ -125,7 +125,7 @@ def get_all_dataset_mount_points(zfs_command):
     '''
     Given a ZFS command to run, return all ZFS datasets as a sequence of sorted mount points.
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *zfs_command.split(' '),
             'list',
@@ -142,7 +142,7 @@ def get_all_dataset_mount_points(zfs_command):
         sorted(
             {
                 mount_point
-                for line in list_output.splitlines()
+                for line in list_lines
                 for mount_point in (line.rstrip(),)
                 if mount_point != 'none'
             },
@@ -344,7 +344,7 @@ def get_all_snapshots(zfs_command):
     Given a ZFS command to run, return all ZFS snapshots as a sequence of full snapshot names of the
     form "dataset@snapshot".
     '''
-    list_output = borgmatic.execute.execute_command_and_capture_output(
+    list_lines = borgmatic.execute.execute_command_and_capture_output(
         (
             *tuple(zfs_command.split(' ')),
             'list',
@@ -357,7 +357,7 @@ def get_all_snapshots(zfs_command):
         close_fds=True,
     )
 
-    return tuple(line.rstrip() for line in list_output.splitlines())
+    return tuple(line.rstrip() for line in list_lines)
 
 
 def remove_data_source_dumps(hook_config, config, borgmatic_runtime_directory, patterns, dry_run):  # noqa: PLR0912
@@ -453,6 +453,10 @@ def make_data_source_dump_patterns(
     config,
     borgmatic_runtime_directory,
     name=None,
+    hostname=None,
+    port=None,
+    container=None,
+    label=None,
 ):  # pragma: no cover
     '''
     Restores aren't implemented, because stored files can be extracted directly with "extract".

borgmatic/hooks/monitoring/apprise.py

@@ -17,7 +17,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to an Apprise notification service upon a finish or failure state. But skip
     this if the "send_logs" option is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     logs_size_limit = max(

borgmatic/hooks/monitoring/healthchecks.py

@@ -26,7 +26,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to Healthchecks upon a finish or failure state. But skip this if the
     "send_logs" option is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     ping_body_limit = max(

borgmatic/hooks/monitoring/loki.py

@@ -23,8 +23,8 @@ MAX_BUFFER_LINES = 100
 
 class Loki_log_buffer:
     '''
-    A log buffer that allows to output the logs as loki requests in json. Allows
+    A log buffer that allows to output the logs as Loki requests in json. Allows
-    adding labels to the log stream and takes care of communication with loki.
+    adding labels to the log stream and takes care of communication with Loki.
     '''
 
     def __init__(self, url, dry_run):
@@ -58,7 +58,7 @@ class Loki_log_buffer:
         if self.dry_run:
             # Just empty the buffer and skip
             self.root['streams'][0]['values'] = []
-            logger.info('Skipped uploading logs to loki due to dry run')
+            logger.info('Skipped uploading logs to Loki due to dry run')
             return
 
         if len(self) == 0:
@@ -80,23 +80,32 @@ class Loki_log_buffer:
             )
             result.raise_for_status()
         except requests.RequestException:
-            logger.warning('Failed to upload logs to loki')
+            logger.warning('Failed to upload logs to Loki')
 
 
 class Loki_log_handler(logging.Handler):
     '''
-    A log handler that sends logs to loki.
+    A log handler that sends logs to Loki.
     '''
 
-    def __init__(self, url, dry_run):
+    def __init__(self, url, send_logs, dry_run):
+        '''
+        Given a URL to send logs to, whether all borgmatic logs should be sent (or just explicitly
+        added messages from this hook), and whether this is a dry run, create an instance of
+        Loki_log_buffer.
+        '''
         super().__init__()
+
         self.buffer = Loki_log_buffer(url, dry_run)
+        self.send_logs = send_logs
 
     def emit(self, record):
         '''
-        Add a log record from the logging module to the stream.
+        Add a general log record from the logging module to the stream—but only if send logs is
+        enabled.
         '''
-        self.raw(record.getMessage())
+        if self.send_logs:
+            self.raw(record.getMessage())
 
     def add_label(self, key, value):
         '''
@@ -110,22 +119,26 @@ class Loki_log_handler(logging.Handler):
         '''
         self.buffer.add_value(msg)
 
-        if len(self.buffer) > MAX_BUFFER_LINES:
+        # If log sending is enabled, flush the buffer (and send data to Loki) once we accumulate
+        # enough log data in the buffer. But if log sending is disabled, flush immediately so that,
+        # for instance, start backup notifications are sent when the backup starts instead of after
+        # it finishes!
+        if len(self.buffer) > MAX_BUFFER_LINES or not self.send_logs:
             self.buffer.flush()
 
     def flush(self):
         '''
-        Send the logs to loki and empty the buffer.
+        Send the logs to Loki and empty the buffer.
         '''
         self.buffer.flush()
 
 
 def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
     '''
-    Add a handler to the root logger to regularly send the logs to loki.
+    Add a handler to the root logger to regularly send the logs to Loki.
     '''
     url = hook_config.get('url')
-    loki = Loki_log_handler(url, dry_run)
+    loki = Loki_log_handler(url, hook_config.get('send_logs', False), dry_run)
 
     for key, value in hook_config.get('labels').items():
         if value == '__hostname':
@@ -142,7 +155,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
 
 def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
     '''
-    Add an entry to the loki logger with the current state.
+    Add an entry to the Loki logger with the current state.
     '''
     for handler in tuple(logging.getLogger().handlers):
         if isinstance(handler, Loki_log_handler) and state in MONITOR_STATE_TO_LOKI:

borgmatic/hooks/monitoring/ntfy.py

@@ -46,14 +46,17 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
         topic = hook_config.get('topic')
 
         logger.info(f'Pinging ntfy topic {topic}{dry_run_label}')
-        logger.debug(f'Using Ntfy ping URL {base_url}/{topic}')
+        logger.debug(f'Using ntfy ping URL {base_url}')
 
         headers = {
             'User-Agent': 'borgmatic',
-            'X-Title': state_config.get('title'),
+        }
-            'X-Message': state_config.get('message'),
+        payload = {
-            'X-Priority': state_config.get('priority'),
+            'topic': topic,
-            'X-Tags': state_config.get('tags'),
+            'title': state_config.get('title'),
+            'message': state_config.get('message'),
+            'priority': state_config.get('priority'),
+            'tags': state_config.get('tags'),
         }
 
         try:
@@ -81,7 +84,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
                     'ntfy access_token is set but so is username/password, only using access_token',
                 )
 
-            auth = requests.auth.HTTPBasicAuth('', access_token)
+            headers['Authorization'] = f'Bearer {access_token}'
         elif (username and password) is not None:
             auth = requests.auth.HTTPBasicAuth(username, password)
             logger.info(f'Using basic auth with user {username} for ntfy')
@@ -94,10 +97,11 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
             logging.getLogger('urllib3').setLevel(logging.ERROR)
             try:
                 response = requests.post(
-                    f'{base_url}/{topic}',
+                    base_url,
                     auth=auth,
                     timeout=TIMEOUT_SECONDS,
                     headers=headers,
+                    json=payload,
                 )
                 if not response.ok:
                     response.raise_for_status()

borgmatic/hooks/monitoring/pagerduty.py

@@ -23,7 +23,7 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
     we can send them all to PagerDuty upon a failure state. But skip this if the "send_logs" option
     is false.
     '''
-    if hook_config.get('send_logs') is False:
+    if hook_config.get('send_logs', False) is False:
         return
 
     ping_body_limit = max(

borgmatic/logger.py

@@ -1,7 +1,9 @@
 import enum
+import json
 import logging
 import logging.handlers
 import os
+import socket
 import sys
 
 
@@ -85,6 +87,57 @@ class Multi_stream_handler(logging.Handler):
             handler.setLevel(level)
 
 
+DEFAULT_JOURNALD_PRIORITY = 6
+
+
+class JournaldHandler(logging.Handler):
+    def __init__(self, journald_socket_path):
+        super().__init__()
+
+        add_custom_log_levels()
+
+        self.journald_socket_path = journald_socket_path
+        self.log_level_to_journald_priority = {
+            logging.CRITICAL: 2,
+            logging.ERROR: 3,
+            logging.WARNING: 4,
+            logging.ANSWER: 5,
+            logging.INFO: 6,
+            logging.DEBUG: 7,
+        }
+
+    def emit(self, record):
+        sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+
+        try:
+            message_parts = []
+            entry = dict(
+                MESSAGE=record.getMessage(),
+                PRIORITY=self.log_level_to_journald_priority.get(
+                    record.levelno, DEFAULT_JOURNALD_PRIORITY
+                ),
+                SYSLOG_IDENTIFIER='borgmatic',
+                SYSLOG_PID=os.getpid(),
+            )
+
+            for key, value in entry.items():
+                encoded_key = key.upper().encode('utf-8')
+                encoded_value = str(value).encode('utf-8')
+
+                # Multi-line and single-line values use different formats on the wire.
+                if b'\n' in encoded_value:
+                    message_parts.extend((encoded_key, b'\n'))
+                    message_parts.extend(
+                        (len(encoded_value).to_bytes(8, 'little'), encoded_value, b'\n')
+                    )
+                else:
+                    message_parts.extend((encoded_key, b'=', encoded_value, b'\n'))
+
+            sock.sendto(b''.join(message_parts), self.journald_socket_path)
+        finally:
+            sock.close()
+
+
 class Log_prefix_formatter(logging.Formatter):
     def __init__(self, fmt='{prefix}{message}', *args, style='{', **kwargs):
         self.prefix = None
@@ -97,6 +150,29 @@ class Log_prefix_formatter(logging.Formatter):
         return super().format(record)
 
 
+def log_record_to_json(record):
+    '''
+    Given a logging.LogRecord, return it as a JSON-encoded string containing relevant attributes.
+    '''
+    return json.dumps(
+        dict(
+            type='log_message',
+            time=record.created,
+            message=record.getMessage(),
+            levelname=record.levelname,
+            name=record.name,
+        )
+    )
+
+
+class Json_formatter(logging.Formatter):
+    def __init__(self, fmt='{message}', *args, style='{', **kwargs):
+        super().__init__(*args, fmt=fmt, style=style, **kwargs)
+
+    def format(self, record):  # noqa: PLR6301
+        return log_record_to_json(record)
+
+
 class Color(enum.Enum):
     RESET = 0
     RED = 31
@@ -321,6 +397,10 @@ def flush_delayed_logging(target_handlers):
         root_logger.removeHandler(delayed_handler)
 
 
+JOURNALD_SOCKET_PATH = '/run/systemd/journal/socket'
+SYSLOG_PATHS = ('/dev/log', '/var/run/syslog', '/var/run/log')
+
+
 def configure_logging(
     console_log_level,
     syslog_log_level=None,
@@ -328,6 +408,7 @@ def configure_logging(
     monitoring_log_level=None,
     log_file=None,
     log_file_format=None,
+    log_json=False,
     color_enabled=True,
 ):
     '''
@@ -364,7 +445,9 @@ def configure_logging(
         },
     )
 
-    if color_enabled:
+    if log_json:
+        console_handler.setFormatter(Json_formatter())
+    elif color_enabled:
         console_handler.setFormatter(Console_color_formatter())
     else:
         console_handler.setFormatter(Log_prefix_formatter())
@@ -373,29 +456,32 @@ def configure_logging(
     handlers = [console_handler]
 
     if syslog_log_level != logging.DISABLED:
-        syslog_path = None
+        if os.path.exists(JOURNALD_SOCKET_PATH):
-
+            journald_handler = JournaldHandler(JOURNALD_SOCKET_PATH)
-        if os.path.exists('/dev/log'):
+            journald_handler.setLevel(syslog_log_level)
-            syslog_path = '/dev/log'
+            handlers.append(journald_handler)
-        elif os.path.exists('/var/run/syslog'):
+        else:
-            syslog_path = '/var/run/syslog'
+            syslog_path = next(
-        elif os.path.exists('/var/run/log'):
+                (path for path in SYSLOG_PATHS if os.path.exists(path)),
-            syslog_path = '/var/run/log'
+                None,
-
-        if syslog_path:
-            syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
-            syslog_handler.setFormatter(
-                Log_prefix_formatter(
-                    'borgmatic: {levelname} {prefix}{message}',
-                ),
             )
-            syslog_handler.setLevel(syslog_log_level)
+
-            handlers.append(syslog_handler)
+            if syslog_path:
+                syslog_handler = logging.handlers.SysLogHandler(address=syslog_path)
+                syslog_handler.setFormatter(
+                    Log_prefix_formatter(
+                        'borgmatic: {levelname} {prefix}{message}',
+                    ),
+                )
+                syslog_handler.setLevel(syslog_log_level)
+                handlers.append(syslog_handler)
 
     if log_file and log_file_log_level != logging.DISABLED:
         file_handler = logging.handlers.WatchedFileHandler(log_file)
         file_handler.setFormatter(
-            Log_prefix_formatter(
+            Json_formatter()
+            if log_json
+            else Log_prefix_formatter(
                 log_file_format or '[{asctime}] {levelname}: {prefix}{message}',
             ),
         )

docs/_includes/snippet/command-line/sample.md

@@ -0,0 +1,5 @@
+Here's the command-line help for this action in the [most recent version of
+borgmatic](https://projects.torsion.org/borgmatic-collective/borgmatic/releases).
+If you're using an older version, some of these flags may not work, and you
+should instead run the action with `--help` to see the flags specific to your
+borgmatic version.

docs/_includes/snippet/configuration/sample.md

@@ -0,0 +1,6 @@
+Here's an example configuration with all of the available options for this
+feature in the [most recent version of
+borgmatic](https://projects.torsion.org/borgmatic-collective/borgmatic/releases).
+If you're using an older version, some of these options may not work, and you
+should instead [generate a sample configuration file specific to your borgmatic
+version](https://torsion.org/borgmatic/how-to/set-up-backups/#configuration).

docs/how-to/customize-warnings-and-errors.md

@@ -5,40 +5,44 @@ eleventyNavigation:
   parent: How-to guides
   order: 13
 ---
-After Borg runs, it indicates whether it succeeded via its exit code, a
+
-numeric ID indicating success, warning, or error. borgmatic consumes this exit
+After Borg runs, it indicates whether it succeeded via its exit code, a numeric
-code to decide how to respond. Normally, a Borg error results in a borgmatic
+ID indicating success, warning, or error. borgmatic consumes this exit code to
-error, while a Borg warning or success doesn't.
+decide how to respond. By default, Borg errors (and some warnings) result
+in a borgmatic error, while Borg successes don't.
+
+<span class="minilink minilink-addedin">New in borgmatic version 2.1.0</span>
+borgmatic elevates most Borg warnings to errors by default. For instance, if a
+source directory is missing during backup, Borg indicates that with a warning
+exit code (`107`). And starting in borgmatic 2.1.0, that exit code is considered
+an error, so you'll actually find out about missing files.
 
 <span class="minilink minilink-addedin">With Borg version 1.4+</span> If the
 default behavior isn't sufficient for your needs, you can customize how
 borgmatic interprets [Borg's exit
-codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
+codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
 
-For instance, this borgmatic configuration elevates all Borg backup file
+For instance, this borgmatic configuration elevates a Borg warning about source files
-permission warnings (exit code `105`)—and only those warnings—to errors:
+changes during backup (exit code `100`)—and only those warnings—to
+errors:
 
 ```yaml
 borg_exit_codes:
-    - code: 105
+    - code: 100
       treat_as: error
 ```
 
-The following configuration does that *and* elevates backup file not found
+The following configuration does that *and* treats Borg's backup file not found
-warnings (exit code `107`) to errors as well:
+(exit code `107`) as a warning:
 
 ```yaml
 borg_exit_codes:
-    - code: 105
+    - code: 100
       treat_as: error
     - code: 107
-      treat_as: error
+      treat_as: warning
 ```
 
-See the full list of [Borg 1.4 error and warning exit
-codes](https://borgbackup.readthedocs.io/en/stable/internals/frontends.html#message-ids).
-The `rc:` numeric value there tells you the exit code for each.
-
 If you don't know the exit code for a particular Borg error or warning you're
 experiencing, you can usually find it in your borgmatic output when `--verbosity
 2` is enabled. For instance, here's a snippet of that output when a backup file
@@ -50,8 +54,8 @@ is not found:
 terminating with warning status, rc 107
 ```
 
-So if you want to configure borgmatic to treat this as an error instead of a
+So if you want to configure borgmatic to treat this as an warning instead of an
-warning, the exit status to use is `107`.
+error, the exit status to use is `107`.
 
 <span class="minilink minilink-addedin">With Borg version 1.2 and earlier</span>
 Older versions of Borg didn't support granular exit codes, but still
@@ -67,15 +71,4 @@ borg_exit_codes:
 
 Be aware though that Borg exits with a warning code for a variety of benign
 situations such as files changing while they're being read, so this example
-may not meet your needs.
+may not meet your needs. Upgrading to Borg 1.4+ is recommended.
-
-Here's another Borg 1.2 example that squashes Borg errors to warnings:
-
-```yaml
-borg_exit_codes:
-    - code: 2
-      treat_as: warning
-```
-
-Be careful with this example though, because it prevents borgmatic from
-erroring when Borg errors, which may not be desirable.

docs/how-to/deal-with-very-large-backups.md

@@ -39,6 +39,31 @@ documentation](https://torsion.org/borgmatic/reference/configuration/consistency
 for details.
 
 
+### Pre-backup safety validation
+
+Before running a backup during the `create` action, borgmatic automatically runs
+through some pre-backup safety validation to ensure that:
+
+1. You haven't accidentally excluded borgmatic's [runtime
+directory](https://torsion.org/borgmatic/reference/configuration/runtime-directory/)
+from the backup, which would break things like database dumps, filesystem
+snapshots, etc.
+2. Special files are auto-excluded to prevent Borg from hanging.
+
+This validation does have a cost: performance. On a large filesystem, it can
+take a while to run. So if you are absolutely sure that you aren't excluding
+borgmatic's runtime directory, and you also aren't including any special files
+that might cause Borg to hang, you can disable the pre-backup validation as
+follows:
+
+```yaml
+unsafe_skip_path_validation_before_create: true
+```
+
+However, this is indeed unsafe, and could lead to hangs or data being left out
+of backups. Use this option at your own risk.
+
+
 ## Troubleshooting
 
 ### Broken pipe with remote repository

docs/how-to/snapshot-your-filesystems.md

@@ -20,10 +20,9 @@ snapshots.
 
 ### ZFS
 
-<span class="minilink minilink-addedin">New in version 1.9.3</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.3</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+supports taking snapshots with the [ZFS filesystem](https://openzfs.org/) and
-taking snapshots with the [ZFS filesystem](https://openzfs.org/) and sending
+sending those snapshots to Borg for backup.
-those snapshots to Borg for backup.
 
 To use this feature, add one or more ZFS dataset paths to your
 `source_directories`. Then, enable borgmatic's ZFS snapshotting of those
@@ -62,10 +61,10 @@ extract them.
 
 ### Btrfs
 
-<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.4</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports taking
+supports taking snapshots with the [Btrfs
-snapshots with the [Btrfs filesystem](https://btrfs.readthedocs.io/) and sending
+filesystem](https://btrfs.readthedocs.io/) and sending those snapshots to Borg
-those snapshots to Borg for backup.
+for backup.
 
 To use this feature, add one or more subvolume paths to your
 `source_directories`. Then, enable Btrfs within borgmatic by adding the
@@ -104,12 +103,11 @@ them.
 
 ### LVM
 
-<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.4</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+supports taking snapshots with [LVM](https://sourceware.org/lvm2/) (Linux
-taking snapshots with [LVM](https://sourceware.org/lvm2/) (Linux Logical
+Logical Volume Manager) and sending those snapshots to Borg for backup. LVM
-Volume Manager) and sending those snapshots to Borg for backup. LVM isn't
+isn't itself a filesystem, but it can take snapshots at the layer right below
-itself a filesystem, but it can take snapshots at the layer right below your
+your filesystem.
-filesystem.
 
 Note that, due to Borg being a file-level backup, this feature is really only
 suitable for filesystems, not whole disk or raw images containing multiple

docs/reference/command-line/actions/borg.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/borg.txt %}
 ```

docs/reference/command-line/actions/break-lock.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/break-lock.txt %}
 ```

docs/reference/command-line/actions/check.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/check.txt %}
 ```

docs/reference/command-line/actions/compact.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/compact.txt %}
 ```

docs/reference/command-line/actions/config-bootstrap.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/config-bootstrap.txt %}
 ```

docs/reference/command-line/actions/config-generate.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/config-generate.txt %}
 ```

docs/reference/command-line/actions/config-validate.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/config-validate.txt %}
 ```

docs/reference/command-line/actions/config.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/config.txt %}
 ```

docs/reference/command-line/actions/create.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/create.txt %}
 ```

docs/reference/command-line/actions/delete.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/delete.txt %}
 ```

docs/reference/command-line/actions/export-tar.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/export-tar.txt %}
 ```

docs/reference/command-line/actions/extract.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/extract.txt %}
 ```

docs/reference/command-line/actions/info.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/info.txt %}
 ```

docs/reference/command-line/actions/key-change-passphrase.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/key-change-passphrase.txt %}
 ```

docs/reference/command-line/actions/key-export.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/key-export.txt %}
 ```

docs/reference/command-line/actions/key-import.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/key-import.txt %}
 ```

docs/reference/command-line/actions/list.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/list.txt %}
 ```

docs/reference/command-line/actions/mount.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/mount.txt %}
 ```

docs/reference/command-line/actions/prune.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/prune.txt %}
 ```

docs/reference/command-line/actions/recreate.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/recreate.txt %}
 ```

docs/reference/command-line/actions/repo-create.md

@@ -4,6 +4,9 @@ eleventyNavigation:
   key: repo-create
   parent: 🎬 Actions
 ---
+
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/repo-create.txt %}
 ```

docs/reference/command-line/actions/repo-delete.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/repo-delete.txt %}
 ```

docs/reference/command-line/actions/repo-info.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/repo-info.txt %}
 ```

docs/reference/command-line/actions/repo-list.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/repo-list.txt %}
 ```

docs/reference/command-line/actions/restore.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/restore.txt %}
 ```

docs/reference/command-line/actions/transfer.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/transfer.txt %}
 ```

docs/reference/command-line/actions/umount.md

@@ -5,6 +5,8 @@ eleventyNavigation:
   parent: 🎬 Actions
 ---
 
+{% include snippet/command-line/sample.md %}
+
 ```
 {% include borgmatic/command-line/umount.txt %}
 ```

docs/reference/command-line/logging.md

@@ -34,6 +34,13 @@ Additionally, for the `create` action in particular, you can include the
 that are new or changed since the last backup.
 
 
+### JSON logging
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span>With the
+`--log-json` flag, borgmatic logs both its own logs and Borg's output as JSON.
+The `--log-json` flag applies to console output and any log file (see below).
+
+
 ## Logging to syslog
 
 By default, borgmatic only logs its output to the console. You can enable
@@ -66,6 +73,40 @@ systemd, try running `journalctl -xe`. Otherwise, try viewing
 logged to syslog by default whenever run at a non-interactive console.
 
 
+### Systemd journal
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span>If the syslog
+verbosity is set and systemd's journal is present, then borgmatic sends
+structured logs to the journal instead of plain text. This allows you to query
+logs by particular fields. For instance:
+
+```bash
+journalctl --catalog --pager-end SYSLOG_IDENTIFIER=borgmatic
+```
+
+Or to view just borgmatic warning and error logs:
+
+```bash
+journalctl --catalog --pager-end SYSLOG_IDENTIFIER=borgmatic --priority warning..alert
+```
+
+The fields borgmatic sends are:
+
+ * `MESSAGE`: the log message
+ * `PRIORITY`: the [numeric priority level](https://wiki.archlinux.org/title/Systemd/Journal#Priority_level) for the log entry
+ * `SYSLOG_IDENTIFIER`: `borgmatic`
+ * `SYSLOG_PID`: borgmatic's process ID
+
+You can also output these structured logs as JSON:
+
+```bash
+journalctl --output=json SYSLOG_IDENTIFIER=borgmatic
+```
+
+Note that systemd's journal adds its own fields to the JSON on top of the fields
+that borgmatic logs.
+
+
 ### Rate limiting
 
 If you are using rsyslog or systemd's journal, be aware that by default they

docs/reference/configuration/command-hooks.md

@@ -135,6 +135,12 @@ don't run in the context of a single repository. But the deprecated command
 hooks (`before_backup`, `on_error`, etc.) do generally support variable
 interpolation.
 
+Note that you can also interpolate [arbitrary environment
+variables](https://torsion.org/borgmatic/reference/configuration/environment-variables/).
+
+
+### Shell escaping
+
 borgmatic automatically escapes these interpolated values to prevent shell
 injection attacks. One implication is that you shouldn't wrap the interpolated
 values in your own quotes, as that will interfere with the quoting performed by
@@ -158,8 +164,26 @@ commands:
           - send-text-message.sh {error}
 ```
 
-Note that you can also interpolate [arbitrary environment
+
-variables](https://torsion.org/borgmatic/reference/configuration/environment-variables/).
+### Disabling variable interpolation
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span> To prevent
+borgmatic from attempting variable interpolation on a specific would-be variable
+name—or warning about unknown variables—you can backslash its curly brackets.
+For instance:
+
+```yaml
+commands:
+    - after: action
+      when: [prune]
+      run:
+          - record-prune.sh \{name\}
+```
+
+This tells borgmatic to skip variable interpolation for `{name}` and instead
+pass the `{name}` literal to the command. This is handy if you've got a command
+containing literal curly brackets around a name that appears to reference a
+variable—but actually doesn't.
 
 
 ## Soft failure
@@ -213,6 +237,8 @@ There are some caveats you should be aware of with this feature.
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/commands.yaml %}
 ```

docs/reference/configuration/constants.md

@@ -75,3 +75,22 @@ that uses that constant.
 
 An alternate to constants is passing in your values via [environment
 variables](https://torsion.org/borgmatic/reference/configuration/environment-variables/).
+
+
+## Disabling constants
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span> To prevent
+borgmatic from attempting constant interpolation on a specific would-be
+constant name, you can backslash its curly brackets. For instance:
+
+```yaml
+constants:
+    name: foo
+
+source_directories:
+    - /home/user/\{name\}
+```
+
+This tells borgmatic to skip constant interpolation for `{name}` and instead use
+the `{name}` literal. This is handy if you've got a filename that has literal
+curly brackets around a name that happens to match a constant.

docs/reference/configuration/data-sources/btrfs.md

@@ -4,10 +4,10 @@ eleventyNavigation:
   key: Btrfs
   parent: 🗄️ Data sources
 ---
-<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.4</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports taking
+supports taking snapshots with the [Btrfs
-snapshots with the [Btrfs filesystem](https://btrfs.readthedocs.io/) and sending
+filesystem](https://btrfs.readthedocs.io/) and sending those snapshots to Borg
-those snapshots to Borg for backup.
+for backup.
 
 The minimum configuration to enable Btrfs support is:
 
@@ -114,6 +114,8 @@ subvolume, which is necessary for the borgmatic Btrfs hook to work.
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/btrfs.yaml %}
 ```

docs/reference/configuration/data-sources/lvm.md

@@ -4,12 +4,11 @@ eleventyNavigation:
   key: LVM
   parent: 🗄️ Data sources
 ---
-<span class="minilink minilink-addedin">New in version 1.9.4</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.4</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+supports taking snapshots with [LVM](https://sourceware.org/lvm2/) (Linux
-taking snapshots with [LVM](https://sourceware.org/lvm2/) (Linux Logical
+Logical Volume Manager) and sending those snapshots to Borg for backup. LVM
-Volume Manager) and sending those snapshots to Borg for backup. LVM isn't
+isn't itself a filesystem, but it can take snapshots at the layer right below
-itself a filesystem, but it can take snapshots at the layer right below your
+your filesystem.
-filesystem.
 
 The minimum configuration to enable LVM support is:
 
@@ -125,6 +124,8 @@ work. See the comments in the sample systemd service file for details.
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/lvm.yaml %}
 ```

docs/reference/configuration/data-sources/mariadb.md

@@ -18,6 +18,8 @@ mariadb_databases:
 
 ### Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/mariadb_databases.yaml %}
 ```

docs/reference/configuration/data-sources/mongodb.md

@@ -16,6 +16,8 @@ mongodb_databases:
 
 ### Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/mongodb_databases.yaml %}
 ```

docs/reference/configuration/data-sources/mysql.md

@@ -16,6 +16,8 @@ mysql_databases:
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/mysql_databases.yaml %}
 ```

docs/reference/configuration/data-sources/postgresql.md

@@ -16,6 +16,8 @@ postgresql_databases:
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/postgresql_databases.yaml %}
 ```

docs/reference/configuration/data-sources/sqlite.md

@@ -17,6 +17,8 @@ sqlite_databases:
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/sqlite_databases.yaml %}
 ```

docs/reference/configuration/data-sources/zfs.md

@@ -4,10 +4,9 @@ eleventyNavigation:
   key: ZFS
   parent: 🗄️ Data sources
 ---
-<span class="minilink minilink-addedin">New in version 1.9.3</span> <span
+<span class="minilink minilink-addedin">New in version 1.9.3</span> borgmatic
-class="minilink minilink-addedin">Beta feature</span> borgmatic supports
+supports taking snapshots with the [ZFS filesystem](https://openzfs.org/) and
-taking snapshots with the [ZFS filesystem](https://openzfs.org/) and sending
+sending those snapshots to Borg for backup.
-those snapshots to Borg for backup.
 
 The minimum configuration to enable ZFS support is:
 
@@ -114,6 +113,8 @@ the sample systemd service file for details.
 
 ## Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/zfs.yaml %}
 ```

docs/reference/configuration/monitoring/apprise.md

@@ -66,9 +66,28 @@ apprise:
         - fail
 ```
 
+
+### Sending logs
+
 <span class="minilink minilink-addedin">New in version 1.8.9</span> borgmatic
-logs are automatically included in the body data sent to your Apprise services
+can include logs in the body data sent to your Apprise services when a backup
-when a backup finishes or fails.
+finishes or fails. There is a `send_logs` option to enable or disable log
+sending:
+
+```yaml
+apprise:
+    services:
+        - url: gotify://hostname/token
+          label: gotify
+    send_logs: true
+```
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span> To avoid
+revealing private log information to third-party services, logs are no longer
+sent by default when `send_logs` is omitted.
+
+<span class="minilink minilink-addedin">Prior to version 2.1.0</span> Logs were
+sent by default when `send_logs` was omitted.
 
 You can customize the verbosity of the logs that are sent with borgmatic's
 `--monitoring-verbosity` flag. The `--list` and `--stats` flags may also be of
@@ -80,36 +99,25 @@ for more information.
 defaults for these flags in your borgmatic configuration via the
 `monitoring_verbosity`, `list`, and `statistics` options.
 
-If you don't want any logs sent, you can disable log sending by setting
+To limit the size of logs sent to Apprise services, set `logs_size_limit` to a
-`send_logs` to `false`:
+number of bytes:
-
-```yaml
-apprise:
-    services:
-        - url: gotify://hostname/token
-          label: gotify
-    send_logs: false
-```
-
-Or to limit the size of logs sent to Apprise services:
 
 ```yaml
 apprise:
     services:
         - url: gotify://hostname/token
           label: gotify
+    send_logs: true
     logs_size_limit: 500
 ```
 
 This may be necessary for some services that reject large requests.
 
-See the [configuration
-reference](https://torsion.org/borgmatic/reference/configuration/) for
-details.
-
 
 ### Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/apprise.yaml %}
 ```

docs/reference/configuration/monitoring/healthchecks.md

@@ -23,11 +23,34 @@ With this configuration, borgmatic pings your Healthchecks project when a
 backup begins, ends, or errors, but only when any of the `create`, `prune`,
 `compact`, or `check` actions are run.
 
-Then, if the actions complete successfully, borgmatic notifies Healthchecks of
+You can configure Healthchecks to notify you by a [variety of
-the success and includes borgmatic logs in the payload data sent to
+mechanisms](https://healthchecks.io/#welcome-integrations) when backups fail
-Healthchecks. This means that borgmatic logs show up in the Healthchecks UI,
+or it doesn't hear from borgmatic for a certain period of time.
-although be aware that Healthchecks currently has a 100-kilobyte limit for the
+
-logs in each ping.
+
+### Sending logs
+
+<span class="minilink minilink-addedin">New in version 1.4.11</span> If the
+actions complete successfully, borgmatic can notify Healthchecks of the success
+and includes borgmatic logs in the payload data sent to Healthchecks.  This
+means that borgmatic logs can show up in the Healthchecks UI, although be aware
+that Healthchecks currently has a 100-kilobyte limit for the logs in each ping.
+
+<span class="minilink minilink-addedin">New in version 1.6.1</span> borgmatic
+has a `send_logs` option to enable or disable log sending:
+
+```yaml
+healthchecks:
+    ping_url: https://hc-ping.com/addffa72-da17-40ae-be9c-ff591afb942a
+    send_logs: true
+```
+
+<span class="minilink minilink-addedin">New in version 2.1.0</span> To avoid
+revealing private log information to a third-party service, logs are no longer
+sent by default when `send_logs` is omitted.
+
+<span class="minilink minilink-addedin">Prior to version 2.1.0</span> Logs were
+sent by default when `send_logs` was omitted.
 
 If an error occurs during any action or hook, borgmatic notifies Healthchecks,
 also tacking on logs including the error itself. But the logs are only
@@ -38,21 +61,17 @@ You can customize the verbosity of the logs that are sent to Healthchecks with
 borgmatic's `--monitoring-verbosity` flag. The `--list` and `--stats` flags may
 also be of use. See [create action
 documentation](https://torsion.org/borgmatic/reference/command-line/actions/create/)
-for more information.  Additionally, see the [borgmatic configuration
+for more information.
-file](https://torsion.org/borgmatic/reference/configuration/) for additional
-Healthchecks options.
 
 <span class="minilink minilink-addedin">New in version 2.0.0</span>Set the
 defaults for these flags in your borgmatic configuration via the
 `monitoring_verbosity`, `list`, and `statistics` options.
 
-You can configure Healthchecks to notify you by a [variety of
-mechanisms](https://healthchecks.io/#welcome-integrations) when backups fail
-or it doesn't hear from borgmatic for a certain period of time.
-
 
 ### Full configuration
 
+{% include snippet/configuration/sample.md %}
+
 ```yaml
 {% include borgmatic/healthchecks.yaml %}
 ```