Compare commits

..
23 Commits
Author SHA1 Message Date
Dan Helfman 5e7d14e419 Bump version for release. 2025-11-26 15:31:04 -08:00
Dan Helfman 8f2ae5e9a1 Fix a regression in the Btrfs hook in which subvolume snapshots didn't get cleaned up until the start of the next borgmatic run (#1186). 2025-11-26 13:59:23 -08:00
Dan Helfman 928bb3fb8c Fix an unreleased regression in the KeePassXC hook in which it raised when it was used without explicit "keepassxc:" configuration (#1181). 2025-11-25 22:48:05 -08:00
Dan Helfman 31623f9c91 Add borgmatic release artifacts to each release on the releases page. 2025-11-25 20:41:25 -08:00
Dan Helfman 47ce12e4b6 Explicitly note Borg v1 caching doesn't work with Btrfs (#1187).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1187
2025-11-21 06:55:39 +00:00
Jackson e72ace5294 remove lingering ) 2025-11-21 05:17:44 +01:00
Jackson 72c3e524be docs: btrfs: note that btrfs hook doesn't cache even with runtime dir 2025-11-21 05:17:44 +01:00
Dan Helfman 399576e7c9 Add data hooks working directory fix to NEWS (#1127). 2025-11-19 10:42:11 -08:00
Dan Helfman a7356f2360 Fix for the database dumps hooks not respecting the "working_directory" option (#1127).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1129
2025-11-19 18:40:50 +00:00
Dan Helfman 1f8c39fb17 Add fish shell completion fix to NEWS (#1184). 2025-11-18 22:29:19 -08:00
Dan Helfman 66e567f78c Hard-code default configuration paths in command-line flag help (#1184).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1184
2025-11-19 06:25:56 +00:00
Tim Ruffing ea45a37db8 Omit "-c" when describing default value for -c 2025-11-18 09:55:54 +01:00
Dan Helfman a229fc7f92 Give the borgmatic logo a white circle background so it actually shows up properly in dark mode. 2025-11-17 22:20:27 -08:00
Tim Ruffing 573405bb88 Hard-code default config paths in help strings
This fixes a bug with fish completions. Before this commit,
`borgmatic --fish-completion` would generate a fish completion script
with an expanded XDG_CONFIG_HOME (if this variable is set).

This is problematic for package maintainers if the variable is set
during the creation of a package.

Moreover, it is problematic if the user has set this variable because
the fish completion scripts checks its own consistency with `borg
--fish-completion` during startup to detect version mismatches. If the
user has a packaged fish completion file with "~/.config/$HOME" in it,
this won't match the output of `borg --fish-completion` because the
latter contains the expansion of $XDG_CONFIG_HOME. As a result, the
script will incorrectly conclude that it is outdated.
2025-11-17 15:42:30 +01:00
Dan Helfman 94dc4c497a Note in the KeePassXC documentation which features are in which versions of borgmatic (#1181). 2025-11-14 21:27:07 -08:00
Dan Helfman 0e9193c46b Add an "ask_for_password" option to the KeePassXC credential hook for disabling KeePassXC's password prompt (#1181). 2025-11-14 21:20:21 -08:00
Dan Helfman 151becbaea Move shell autocompletion documentation out to reference docs. 2025-11-11 10:21:01 -08:00
bockulator ce7f0226be Mock get_working_directory() calls in data source hook tests 2025-11-11 15:36:51 +01:00
bockulator 51ef9c7708 Fix database dumps hooks not respecting working_directory 2025-11-11 14:22:34 +01:00
Dan Helfman 14a8055e71 In the SQLite database hook, run SQLite such that it exits upon encountering an error instead of, you know, not doing that. 2025-11-07 23:22:44 -08:00
Dan Helfman fcfc7ee726 Emojis everywhere! 2025-11-04 23:04:16 -08:00
Dan Helfman 859c338d06 Improve documentation search results for individual configuration options. 2025-11-04 22:34:15 -08:00
Dan Helfman d1f2d8a061 Add documentation on repositories, including SSH, Rclone, S3, and B2. 2025-11-03 13:56:15 -08:00
68 changed files with 841 additions and 133 deletions
-2
View File
@@ -29,5 +29,3 @@ jobs:
PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
- run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
- run: podman push "$IMAGE_NAME"
- run: scripts/export-docs-from-image
- run: curl --user "${{ secrets.REGISTRY_USERNAME }}:${{ secrets.REGISTRY_PASSWORD }}" --upload-file borgmatic-docs.tar.gz https://projects.torsion.org/api/packages/borgmatic-collective/generic/borgmatic-docs/$(head --lines=1 NEWS)/borgmatic-docs.tar.gz
+16
View File
@@ -1,3 +1,19 @@
2.0.12
* #1127: Fix for the database hooks not respecting the "working_directory" option.
* #1181: Add an "ask_for_password" option to the KeePassXC credential hook for disabling
KeePassXC's password prompt, e.g. if you're only using a key file to decrypt your database.
* #1184: Fix the fish shell completion's detection of version mismatches.
* #1186: Fix a regression in the Btrfs hook in which subvolume snapshots didn't get cleaned up
until the start of the next borgmatic run.
* In the SQLite database hook, run SQLite such that it exits upon encountering an error instead of,
you know, not doing that.
* Add documentation on repositories, including SSH, Rclone, S3, and B2:
https://torsion.org/borgmatic/reference/configuration/repositories/
* Improve documentation search results for individual configuration options.
* Add borgmatic release artifacts (wheel and tarball) to each release on the releases page:
https://projects.torsion.org/borgmatic-collective/borgmatic/releases
* Move the tarball of borgmatic's HTML documentation from the packages page to the releases page.
2.0.11
* #957: Document borgmatic's limitations around parallelism—both its own and Borg's. See the
documentation for more information:
+5 -2
View File
@@ -49,12 +49,13 @@ def run_create(
working_directory,
borgmatic_runtime_directory,
)
original_patterns = list(patterns)
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_data_source_dumps',
config,
borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
borgmatic_runtime_directory,
patterns,
original_patterns,
global_arguments.dry_run,
)
active_dumps = borgmatic.hooks.dispatch.call_hooks(
@@ -129,12 +130,14 @@ def run_create(
yield output
# Use the original patterns so as to disregard any modifications made by any data source
# hooks, e.g. via dump_data_sources() above.
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_data_source_dumps',
config,
borgmatic.hooks.dispatch.Hook_type.DATA_SOURCE,
borgmatic_runtime_directory,
patterns,
original_patterns,
global_arguments.dry_run,
)
+4 -5
View File
@@ -568,8 +568,7 @@ def make_parsers(schema, unparsed_arguments): # noqa: PLR0915
ignoring actions, and the combined parser is handy for displaying help that includes everything:
global flags, a list of actions, etc.
'''
config_paths = collect.get_default_config_paths(expand_home=True)
unexpanded_config_paths = collect.get_default_config_paths(expand_home=False)
config_paths = collect.get_default_config_paths()
# Using allow_abbrev=False here prevents the global parser from erroring about "ambiguous"
# options like --encryption. Such options are intended for an action parser rather than the
@@ -582,7 +581,7 @@ def make_parsers(schema, unparsed_arguments): # noqa: PLR0915
'--config',
dest='config_paths',
action='append',
help=f"Configuration filename or directory, can specify flag multiple times, defaults to: -c {' -c '.join(unexpanded_config_paths)}",
help='Configuration filename or directory, can specify flag multiple times, defaults to /etc/borgmatic/config.yaml, /etc/borgmatic.d, $XDG_CONFIG_HOME/borgmatic/config.yaml, and $XDG_CONFIG_HOME/borgmatic.d, where $XDG_CONFIG_HOME defaults to $HOME/.config',
)
global_group.add_argument(
'-n',
@@ -1215,7 +1214,7 @@ def make_parsers(schema, unparsed_arguments): # noqa: PLR0915
'--destination',
dest='destination_path',
default=config_paths[0],
help=f'Destination configuration file (or directory if using --split), default: {unexpanded_config_paths[0]}',
help='Destination configuration file (or directory if using --split), default: /etc/borgmatic/config.yaml',
)
config_generate_group.add_argument(
'--overwrite',
@@ -2015,7 +2014,7 @@ def parse_arguments(schema, *unparsed_arguments):
)
if not arguments['global'].config_paths:
arguments['global'].config_paths = collect.get_default_config_paths(expand_home=True)
arguments['global'].config_paths = collect.get_default_config_paths()
for action_name in ('bootstrap', 'generate', 'validate'):
if action_name in arguments and len(arguments) > HIGHLANDER_ACTION_ARGUMENTS_COUNT:
+6 -9
View File
@@ -1,17 +1,14 @@
import os
def get_default_config_paths(expand_home=True):
def get_default_config_paths():
'''
Based on the value of the XDG_CONFIG_HOME and HOME environment variables, return a list of
default configuration paths. This includes both system-wide configuration and configuration in
the current user's home directory.
Don't expand the home directory ($HOME) if the expand home flag is False.
Return a list of default configuration paths. This includes both system-wide
configuration and configuration in the current user's home directory.
'''
user_config_directory = os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
if expand_home:
user_config_directory = os.path.expandvars(user_config_directory)
user_config_directory = os.path.expandvars(
os.getenv('XDG_CONFIG_HOME') or os.path.join('$HOME', '.config')
)
return [
'/etc/borgmatic/config.yaml',
+28 -3
View File
@@ -1,6 +1,7 @@
import collections
import contextlib
import io
import json
import os
import re
@@ -316,6 +317,24 @@ def merge_source_configuration_into_destination(destination_config, source_confi
return destination_config
def get_configuration_subset(config, option_name): # pragma: no cover
'''
Given configuration as a ruamel.yaml.CommentedMap and an option name found within it at the top
level, return a new CommentedMap containing a subset of the configuration with only the given
option and no other top-level options.
This is useful when generating the sample configuration for a single option instead of a whole
configuration file.
'''
option_config = ruamel.yaml.CommentedMap({option_name: config[option_name]})
# Due to a quirk of ruamel.yaml, the comment right before a top-level key is not on that key and
# needs to get copied separately.
option_config.ca.items[option_name] = config.ca.items[option_name]
return option_config
def generate_sample_configuration(
dry_run,
source_filename,
@@ -359,16 +378,22 @@ def generate_sample_configuration(
os.makedirs(destination_path, exist_ok=True)
for option_name, option_config in destination_config.items():
for option_name in destination_config:
write_configuration(
os.path.join(destination_path, f'{option_name}.yaml'),
transform_optional_configuration(
render_configuration({option_name: option_config}),
render_configuration(get_configuration_subset(destination_config, option_name)),
comment_out=False,
),
).strip(),
overwrite=overwrite,
)
# Also dump a manifest listing all the options we've written.
json.dump(
{'option_names': list(destination_config.keys())},
open(os.path.join(destination_path, 'options.json'), 'w', encoding='utf-8'),
)
return
if os.path.exists(destination_path) and not os.path.isfile(destination_path):
+8
View File
@@ -3172,6 +3172,14 @@ properties:
description: |
Command to use instead of "keepassxc-cli".
example: /usr/local/bin/keepassxc-cli
ask_for_password:
type: boolean
description: |
Whether keepassxc-cli should prompt the user for a password.
Disabling this is only really useful if you're unlocking
your KeePassXC database with a key file instead of a
password. Defaults to true.
example: false
key_file:
type: string
description: |
+1
View File
@@ -29,6 +29,7 @@ def load_credential(hook_config, config, credential_parameters):
command = (
tuple(shlex.split((hook_config or {}).get('keepassxc_cli_command', 'keepassxc-cli')))
+ ('show', '--show-protected', '--attributes', 'Password')
+ (('--no-password',) if not (hook_config or {}).get('ask_for_password', True) else ())
+ (
('--key-file', hook_config['key_file'])
if hook_config and hook_config.get('key_file')
+2 -2
View File
@@ -146,8 +146,8 @@ Subvolume = collections.namedtuple('Subvolume', ('path', 'contained_patterns'),
def get_subvolumes(btrfs_command, patterns):
'''
Given a Btrfs command to run and a sequence of configured patterns, find the intersection
between the current Btrfs filesystem and subvolume paths and the paths of any patterns. The
idea is that these pattern paths represent the requested subvolumes to snapshot.
between the current Btrfs filesystem/subvolume paths and the paths of any patterns. The idea is
that these pattern paths represent the requested subvolumes to snapshot.
Only include subvolumes that contain at least one root pattern sourced from borgmatic
configuration (as opposed to generated elsewhere in borgmatic).
+7 -1
View File
@@ -155,7 +155,11 @@ def database_names_to_dump(database, config, username, password, environment, dr
if skip_names:
logger.debug(f'Skipping database names: {", ".join(skip_names)}')
show_output = execute_command_and_capture_output(show_command, environment=environment)
show_output = execute_command_and_capture_output(
show_command,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
return tuple(
show_name
@@ -239,6 +243,7 @@ def execute_dump_command(
dump_command,
environment=environment,
run_to_completion=False,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
@@ -495,4 +500,5 @@ def restore_data_source_dump(
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
+12 -2
View File
@@ -90,11 +90,20 @@ def dump_data_sources(
if dump_format == 'directory':
dump.create_parent_directory_for_dump(dump_filename)
execute_command(command, shell=True) # noqa: S604
execute_command( # noqa: S604
command,
shell=True,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
else:
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(command, shell=True, run_to_completion=False), # noqa: S604
execute_command( # noqa: S604
command,
shell=True,
run_to_completion=False,
working_directory=borgmatic.config.paths.get_working_directory(config),
),
)
if not dry_run:
@@ -273,6 +282,7 @@ def restore_data_source_dump(
[extract_process] if extract_process else [],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout if extract_process else None,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
+7 -1
View File
@@ -84,7 +84,11 @@ def database_names_to_dump(database, config, username, password, environment, dr
if skip_names:
logger.debug(f'Skipping database names: {", ".join(skip_names)}')
show_output = execute_command_and_capture_output(show_command, environment=environment)
show_output = execute_command_and_capture_output(
show_command,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
return tuple(
show_name
@@ -170,6 +174,7 @@ def execute_dump_command(
dump_command,
environment=environment,
run_to_completion=False,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
@@ -432,4 +437,5 @@ def restore_data_source_dump(
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
+13 -2
View File
@@ -103,7 +103,11 @@ def database_names_to_dump(database, config, environment, dry_run):
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
)
logger.debug('Querying for "all" PostgreSQL databases to dump')
list_output = execute_command_and_capture_output(list_command, environment=environment)
list_output = execute_command_and_capture_output(
list_command,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
return tuple(
row[0]
@@ -245,6 +249,7 @@ def dump_data_sources(
command,
shell=True,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
else:
dump.create_named_pipe_for_dump(dump_filename)
@@ -254,6 +259,7 @@ def dump_data_sources(
shell=True,
environment=environment,
run_to_completion=False,
working_directory=borgmatic.config.paths.get_working_directory(config),
),
)
@@ -422,5 +428,10 @@ def restore_data_source_dump(
output_log_level=logging.DEBUG,
input_file=extract_process.stdout if extract_process else None,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
execute_command(
analyze_command,
environment=environment,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
execute_command(analyze_command, environment=environment)
+9 -2
View File
@@ -86,6 +86,7 @@ def dump_data_sources(
)
command = (
*sqlite_command,
'-bail',
shlex.quote(database_path),
'.dump',
'>',
@@ -100,7 +101,12 @@ def dump_data_sources(
dump.create_named_pipe_for_dump(dump_filename)
processes.append(
execute_command(command, shell=True, run_to_completion=False), # noqa: S604
execute_command( # noqa: S604
command,
shell=True,
run_to_completion=False,
working_directory=borgmatic.config.paths.get_working_directory(config),
),
)
if not dry_run:
@@ -196,7 +202,7 @@ def restore_data_source_dump(
shlex.quote(part)
for part in shlex.split(data_source.get('sqlite_restore_command') or 'sqlite3')
)
restore_command = (*sqlite_restore_command, shlex.quote(database_path))
restore_command = (*sqlite_restore_command, '-bail', shlex.quote(database_path))
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command_with_processes(
@@ -204,4 +210,5 @@ def restore_data_source_dump(
[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=borgmatic.config.paths.get_working_directory(config),
)
+1
View File
@@ -24,6 +24,7 @@ RUN npm install @11ty/eleventy \
markdown-it-anchor \
markdown-it-replace-link
COPY --from=borgmatic /etc/borgmatic/* /source/docs/_includes/borgmatic/
COPY --from=borgmatic /etc/borgmatic/options.json /source/docs/reference/configuration/index.json
COPY --from=borgmatic /command-line/* /source/docs/_includes/borgmatic/command-line/
COPY --from=borgmatic /contributors.html /source/docs/_includes/borgmatic/contributors.html
COPY . /source
@@ -1,5 +1,5 @@
---
title: How to add preparation and cleanup steps to backups
title: 🧹 How to add preparation and cleanup steps to backups
eleventyNavigation:
key: 🧹 Add preparation and cleanup steps
parent: How-to guides
@@ -1,5 +1,5 @@
---
title: How to backup to a removable drive or an intermittent server
title: 💾 How to backup to a removable drive or an intermittent server
eleventyNavigation:
key: 💾 Backup to a removable drive/server
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to backup your databases
title: 🗄️ How to backup your databases
eleventyNavigation:
key: 🗄️ Backup your databases
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to customize warnings and errors
title: 💥 How to customize warnings and errors
eleventyNavigation:
key: 💥 Customize warnings/errors
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to deal with very large backups
title: 📏 How to deal with very large backups
eleventyNavigation:
key: 📏 Deal with very large backups
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to develop on borgmatic
title: 🏗️ How to develop on borgmatic
eleventyNavigation:
key: 🏗️ Develop on borgmatic
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to extract a backup
title: 📤 How to extract a backup
eleventyNavigation:
key: 📤 Extract a backup
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to inspect your backups
title: 🔎 How to inspect your backups
eleventyNavigation:
key: 🔎 Inspect your backups
parent: How-to guides
+4 -3
View File
@@ -1,5 +1,5 @@
---
title: How to make backups redundant
title: ☁️ How to make backups redundant
eleventyNavigation:
key: ☁️ Make backups redundant
parent: How-to guides
@@ -40,10 +40,11 @@ Here's a way of visualizing what borgmatic does with the above configuration:
This gives you redundancy of your data across repositories and even
potentially across providers.
See [Borg repository URLs
documentation](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls)
See [repositories
documentation](https://torsion.org/borgmatic/reference/configuration/repositories/)
for more information on how to specify local and remote repository paths.
### Different options per repository
What if you want borgmatic to backup to multiple repositories—while also
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to make per-application backups
title: 🗂️ How to make per-application backups
eleventyNavigation:
key: 🗂️ Make per-application backups
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to monitor your backups
title: 🚨 How to monitor your backups
eleventyNavigation:
key: 🚨 Monitor your backups
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to provide your passwords
title: 🔒 How to provide your passwords
eleventyNavigation:
key: 🔒 Provide your passwords
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to run arbitrary Borg commands
title: 🔧 How to run arbitrary Borg commands
eleventyNavigation:
key: 🔧 Run arbitrary Borg commands
parent: How-to guides
+19 -44
View File
@@ -1,5 +1,5 @@
---
title: How to set up backups
title: 📥 How to set up backups
eleventyNavigation:
key: 📥 Set up backups
parent: How-to guides
@@ -214,15 +214,20 @@ edits are valid.
Before you can create backups with borgmatic, you first need to create a Borg
repository so you have a destination for your backup archives. (But skip this
step if you already have a Borg repository.) To create a repository, run a
command like the following with Borg 1.x:
step if you already have a Borg repository.) To create a repository, assuming
it's already [specified in borgmatic's
configuration](https://torsion.org/borgmatic/reference/configuration/repositories/),
run a command like the following with Borg 1.x:
```bash
sudo borgmatic init --encryption repokey
sudo borgmatic repo-create --encryption repokey
```
<span class="minilink minilink-addedin">New in borgmatic version 1.9.0</span>
Or, with Borg 2.x:
<span class="minilink minilink-addedin">Prior to borgmatic version 1.9.0</span>
The `repo-create` action was called `init`.
<span class="minilink minilink-addedin">With Borg version 2.x</span> Borg 2.x
uses more specific encryption modes like `repokey-aes-ocb`. For example:
```bash
sudo borgmatic repo-create --encryption repokey-aes-ocb
@@ -350,48 +355,18 @@ Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293).
## Niceties
<a id="shell-completion"></a>
<a id="bash"></a>
<a id="fish"></a>
### Shell completion
borgmatic includes a shell completion script (currently only for Bash and Fish) to
support tab-completing borgmatic command-line actions and flags. Depending on
how you installed borgmatic, this may be enabled by default.
### Command-line autocompletion
#### Bash
borgmatic includes autocompletion scripts for various command-line shells to
support tab-completing command-line actions and flags. See the [autocompletion
documentation](https://torsion.org/borgmatic/reference/command-line/autocompletion/)
for details.
If completions aren't enabled, start by installing the `bash-completion` Linux package or the
[`bash-completion@2`](https://formulae.brew.sh/formula/bash-completion@2)
macOS Homebrew formula. Then, install the shell completion script globally:
```bash
sudo su -c "borgmatic --bash-completion > $(pkg-config --variable=completionsdir bash-completion)/borgmatic"
```
If you don't have `pkg-config` installed, you can try the following path
instead:
```bash
sudo su -c "borgmatic --bash-completion > /usr/share/bash-completion/completions/borgmatic"
```
Or, if you'd like to install the script for only the current user:
```bash
mkdir --parents ~/.local/share/bash-completion/completions
borgmatic --bash-completion > ~/.local/share/bash-completion/completions/borgmatic
```
Finally, restart your shell (`exit` and open a new shell) so the completions
take effect.
#### fish
To add completions for fish, install the completions file globally:
```fish
borgmatic --fish-completion | sudo tee /usr/share/fish/vendor_completions.d/borgmatic.fish
source /usr/share/fish/vendor_completions.d/borgmatic.fish
```
### Colored output
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to snapshot your filesystems
title: 📸 How to snapshot your filesystems
eleventyNavigation:
key: 📸 Snapshot your filesystems
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: How to upgrade borgmatic and Borg
title: 📦 How to upgrade borgmatic and Borg
eleventyNavigation:
key: 📦 Upgrade borgmatic/Borg
parent: How-to guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Actions
title: 🎬 Actions
eleventyNavigation:
key: 🎬 Actions
parent: 💻 Command-line
@@ -0,0 +1,36 @@
---
title: bash
eleventyNavigation:
key: bash
parent: ➡️ Autocompletion
---
borgmatic includes an autocompletion script for
[Bash](https://www.gnu.org/software/bash/) to support tab-completing
command-line actions and flags. Depending on how you installed borgmatic, this
may be enabled by default.
If completions aren't enabled, start by installing the `bash-completion` Linux
package or the
[`bash-completion@2`](https://formulae.brew.sh/formula/bash-completion@2) macOS
Homebrew formula. Then, install the shell completion script globally:
```bash
sudo su -c "borgmatic --bash-completion > $(pkg-config --variable=completionsdir bash-completion)/borgmatic"
```
If you don't have `pkg-config` installed, you can try the following path
instead:
```bash
sudo su -c "borgmatic --bash-completion > /usr/share/bash-completion/completions/borgmatic"
```
Or, if you'd like to install the script for only the current user:
```bash
mkdir --parents ~/.local/share/bash-completion/completions
borgmatic --bash-completion > ~/.local/share/bash-completion/completions/borgmatic
```
Finally, restart your shell (`exit` and open a new shell) so the completions
take effect.
@@ -0,0 +1,17 @@
---
title: fish
eleventyNavigation:
key: fish
parent: ➡️ Autocompletion
---
borgmatic includes an autocompletion script for [fish
shell](https://fishshell.com/) to support tab-completing command-line actions
and flags. Depending on how you installed borgmatic, this may be enabled by
default.
If completions aren't enabled, install the completions file globally:
```shell
borgmatic --fish-completion | sudo tee /usr/share/fish/vendor_completions.d/borgmatic.fish
source /usr/share/fish/vendor_completions.d/borgmatic.fish
```
@@ -0,0 +1,13 @@
---
title: ➡️ Autocompletion
eleventyNavigation:
key: ➡️ Autocompletion
parent: 💻 Command-line
---
borgmatic includes autocompletion scripts for various command-line shells to
support tab-completing command-line actions and flags. Depending on how you
installed borgmatic, this may be enabled by default. Here are the supported
shells and how to configure borgmatic autocompletion for each:
{{ collections.all | eleventyNavigation: "➡️ Autocompletion" | eleventyNavigationToHtml | replace: 'href="/reference/', 'href="/borgmatic/reference/' }}
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Command-line
title: 💻 Command-line
eleventyNavigation:
key: 💻 Command-line
parent: Reference guides
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Logging
title: 🪵 Logging
eleventyNavigation:
key: 🪵 Logging
parent: 💻 Command-line
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Overrides
title: 🔄 Overrides
eleventyNavigation:
key: 🔄 Overrides
parent: 💻 Command-line
@@ -1,5 +1,5 @@
---
title: Archive name format
title: 📛 Archive name format
eleventyNavigation:
key: 📛 Archive name format
parent: ⚙️ Configuration
@@ -1,5 +1,5 @@
---
title: Command hooks
title: 🪝 Command hooks
eleventyNavigation:
key: 🪝 Command hooks
parent: ⚙️ Configuration
@@ -1,5 +1,5 @@
---
title: Consistency checks
title: Consistency checks
eleventyNavigation:
key: ✅ Consistency checks
parent: ⚙️ Configuration
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Constants
title: 🟰 Constants
eleventyNavigation:
key: 🟰 Constants
parent: ⚙️ Configuration
@@ -1,5 +1,5 @@
---
title: Credentials
title: 🔒 Credentials
eleventyNavigation:
key: 🔒 Credentials
parent: ⚙️ Configuration
@@ -42,6 +42,9 @@ postgresql_databases:
password: "{credential keepassxc /etc/keys.kdbx database}"
```
### Custom command
You can also optionally override the `keepassxc-cli` command that borgmatic calls to load
passwords:
@@ -49,3 +52,49 @@ passwords:
keepassxc:
keepassxc_cli_command: /usr/local/bin/keepassxc-cli
```
Another example:
```yaml
keepassxc:
keepassxc_cli_command: docker exec keepassxc keepassxc-cli
```
### Key file
<span class="minilink minilink-addedin">New in version 2.0.0</span>KeePassXC
supports unlocking a database with a separate [key
file](https://keepassxc.org/docs/#faq-keyfile-howto) instead of or in addition
to a password. To configure borgmatic for that, use the `key_file` option:
```yaml
keepassxc:
key_file: /path/to/keyfile
```
<span class="minilink minilink-addedin">New in version 2.0.12</span>By default,
keepassxc-cli prompts the user for the password to unlock a database. But if you
only want to provide a key file to unlock your database and not a password, for
instance to support unattended backups, use the `ask_for_password` option:
```yaml
keepassxc:
ask_for_password: false
key_file: /path/to/keyfile
```
### YubiKey
<span class="minilink minilink-addedin">New in version 2.0.0</span>KeePassXC
also supports unlocking a database with the help of [a
YubiKey](https://keepassxc.org/docs/#faq-yubikey-2fa). To configure borgmatic
for that, use the `yubikey` option:
```yaml
keepassxc:
yubikey: 1:7370001
```
The value here is the YubiKey slot number (e.g., `1` or `2`) and optional serial
number (e.g., `7370001`) used to access the KeePassXC database. Join the two
values with a colon, but omit the colon if you're leaving out the serial number.
@@ -76,6 +76,14 @@ next, the [Borg file
cache](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#cache)
will never get cache hits on snapshotted files. This makes backing up Btrfs
snapshots a little slower than non-snapshotted files that have consistent paths.
**It is also not possible to mitigate cache misses**, as the Btrfs hook uses
snapshot paths which change between borgmatic invocations, and the snapshots
are located outside the [runtime
directory](https://torsion.org/borgmatic/reference/configuration/runtime-directory/),
contrary to
[ZFS](https://torsion.org/borgmatic/reference/configuration/data-sources/zfs/#performance)
and
[LVM](https://torsion.org/borgmatic/reference/configuration/data-sources/lvm/#performance).
<span class="minilink minilink-addedin">With Borg version 2.x</span> Even
snapshotted files should get cache hits, because Borg 2.x is smarter about how
@@ -1,5 +1,5 @@
---
title: Data sources
title: 🗄️ Data sources
eleventyNavigation:
key: 🗄️ Data sources
parent: ⚙️ Configuration
@@ -91,7 +91,7 @@ cache](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.htm
may not get cache hits on snapshotted files. This makes backing up ZFS snapshots
a little slower than non-snapshotted files that have consistent paths. You can
mitigate this by setting a fixed [runtime
directory](https://torsion.org/borgmatic/reference/configuration/runtime-directory/))
directory](https://torsion.org/borgmatic/reference/configuration/runtime-directory/)
(that's not located in `/tmp`). This allows borgmatic to use a consistent
snapshot path from one run to the next, thereby resulting in Borg files cache
hits.
@@ -1,5 +1,5 @@
---
title: Environment variables
title: 💲 Environment variables
eleventyNavigation:
key: 💲 Environment variables
parent: ⚙️ Configuration
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Includes
title: ❗ Includes
eleventyNavigation:
key: ❗ Includes
parent: ⚙️ Configuration
+10 -5
View File
@@ -1,14 +1,14 @@
---
title: Configuration
title: ⚙️ Configuration
eleventyNavigation:
key: ⚙️ Configuration
parent: Reference guides
order: 0
---
Below is a sample borgmatic configuration file including all available options
for the [most recent version of
Below is a sample borgmatic configuration snippet for every available option in
the [most recent version of
borgmatic](https://projects.torsion.org/borgmatic-collective/borgmatic/releases).
This file is also [available for
A full example configuration file is also [available for
download](https://torsion.org/borgmatic/reference/config.yaml).
If you're using an older version of borgmatic, some of these options may not
@@ -16,6 +16,11 @@ work, and you should instead [generate a sample configuration file specific to
your borgmatic
version](https://torsion.org/borgmatic/how-to/set-up-backups/#configuration).
<span data-pagefind-weight="6.0">
{% for option_name in option_names %}
### {{ option_name }} option
```yaml
{% include borgmatic/config.yaml %}
{% include borgmatic/{{ option_name }}.yaml %}
```
{% endfor %}
</span>
@@ -1,5 +1,5 @@
---
title: Monitoring
title: 🚨 Monitoring
eleventyNavigation:
key: 🚨 Monitoring
parent: ⚙️ Configuration
@@ -1,5 +1,5 @@
---
title: Patterns and excludes
title: Patterns and excludes
eleventyNavigation:
key: ⛔ Patterns and excludes
parent: ⚙️ Configuration
@@ -0,0 +1,138 @@
---
title: 🗃️ Repositories
eleventyNavigation:
key: 🗃️ Repositories
parent: ⚙️ Configuration
---
Borg repositories are where your backups get stored. You can define them in
borgmatic's configuration via the `repositories` option, something like:
```yaml
repositories:
- path: /path/to/first.borg
label: first
- path: /path/to/second.borg
label: second
```
Each repository has a `path` and an optional `label`. The [Borg repository URLs
documentation](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls)
has examples of valid repositories paths, but see below for some
borgmatic-specific examples.
The `label` shows up in [logged
messages](https://torsion.org/borgmatic/reference/command-line/logging/) about
the repository and also serves as a way to refer to the repository via
the `--repository` flag on the command-line for supported
[actions](https://torsion.org/borgmatic/reference/command-line/actions/).
When you run borgmatic's [`create`
action](https://torsion.org/borgmatic/reference/command-line/actions/create/),
it invokes Borg once for each configured repository in sequence. (So, not in
parallel.) That means—in each repository—borgmatic creates a single new backup
archive containing all of your [source
directories](https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/).
## SSH
Backing up to a remote server via
[SSH](https://en.wikipedia.org/wiki/Secure_Shell) looks like:
```yaml
repositories:
- path: ssh://user@host:port/./absolute/path/to/repo
```
Or relative to the remote user's home directory:
```yaml
repositories:
- path: ssh://user@host:port/~/relative/path/to/repo
```
This assumes that you've already configured SSH access (e.g. public keys, known
hosts, authorized hosts, etc.) outside of borgmatic and that Borg is installed
on the server.
<span class="minilink minilink-addedin">With Borg version 2.x</span>The SSH
syntax is a little different:
```yaml
repositories:
- path: ssh://user@host:port//absolute/path/to/repo
```
Or relative to the remote user's home directory:
```yaml
repositories:
- path: ssh://user@host:port/relative/path/to/repo
```
Also see the [`ssh_command` configuration
option](https://torsion.org/borgmatic/reference/configuration/) for overriding
the path to the SSH binary or passing it custom flags. For example:
```yaml
ssh_command: ssh -i /path/to/private/key
```
### SFTP
[SFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) repositories
work just like SSH repositories, but with `sftp://` substituted for `ssh://`.
## Rclone
<span class="minilink minilink-addedin">New in Borg version 2.x</span> If you're
using Borg 2, you can backup to repositories via [Rclone](https://rclone.org/),
which supports a large number of [cloud
providers](https://rclone.org/#providers). This means that Borg, via Rclone,
backs up directly to a cloud provider without having to create an intermediate
repository.
The borgmatic configuration for Rclone looks like:
```yaml
repositories:
- path: rclone:remote:path
```
Note the lack of "`//`" after `rclone:`.
This configuration assumes that you've already [configured a corresponding
Rclone remote](https://rclone.org/docs/).
## S3 / B2
<span class="minilink minilink-addedin">New in Borg version 2.x</span> Borg 2
supports storing repositories directly on [Amazon
S3](https://aws.amazon.com/s3/), [Backblaze
B2](https://www.backblaze.com/cloud-storage), or an S3-alike service, even
without the use of Rclone or an intermediate repository. The configuration for
that might look like one of the following:
```yaml
repositories:
- path: s3:access_key_id:access_key_secret@/bucket/path
- path: b2:access_key_id:access_key_secret@schema://hostname:port/bucket/path
```
Note the lack of "`//`" after `s3:` or `b2:`.
When selecting your cloud hosting provider, be aware that Amazon in particular
has [financially
supported](https://en.wikipedia.org/wiki/White_House_State_Ballroom) the Trump
regime.
## Related documentation
* [How to make backups redundant](https://torsion.org/borgmatic/how-to/make-backups-redundant/)
* [How to provide your passwords](https://torsion.org/borgmatic/how-to/provide-your-passwords/)
@@ -1,5 +1,5 @@
---
title: Runtime directory
title: 📁 Runtime directory
eleventyNavigation:
key: 📁 Runtime directory
parent: ⚙️ Configuration
+1 -1
View File
@@ -1,5 +1,5 @@
---
title: Source code
title: 🐍 Source code
eleventyNavigation:
key: 🐍 Source code
parent: Reference guides
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 10 KiB

+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "borgmatic"
version = "2.0.11"
version = "2.0.12"
authors = [
{ name="Dan Helfman", email="witten@torsion.org" },
]
+13 -4
View File
@@ -2,10 +2,19 @@
set -e
docs_container_id=$(podman create "$IMAGE_NAME")
podman cp $docs_container_id:/usr/share/nginx/html - > borgmatic-docs-dump.tar
USER_PODMAN_SOCKET_PATH=/run/user/$UID/podman/podman.sock
if [ -e "$USER_PODMAN_SOCKET_PATH" ]; then
export DOCKER_HOST="unix://$USER_PODMAN_SOCKET_PATH"
export CONTAINER_SOCKET_PATH="$USER_PODMAN_SOCKET_PATH"
fi
podman build --tag borgmatic-release-docs --file docs/Dockerfile .
docs_container_id=$(podman create borgmatic-release-docs)
podman cp "$docs_container_id":/usr/share/nginx/html - > borgmatic-docs-dump.tar
tar xf borgmatic-docs-dump.tar
rm borgmatic-docs-dump.tar
mv html borgmatic-docs
tar cfz borgmatic-docs.tar.gz borgmatic-docs
podman rm --volumes $docs_container_id
rm -f dist/borgmatic-docs.tar.gz
tar cfz dist/borgmatic-docs.tar.gz borgmatic-docs
podman rm --volumes "$docs_container_id"
+25 -3
View File
@@ -34,18 +34,40 @@ git push github $version
# Build borgmatic and publish to pypi.
rm -fr dist
uv build
twine upload -r pypi --username __token__ dist/borgmatic-*.tar.gz
twine upload -r pypi --username __token__ dist/borgmatic-*-py3-none-any.whl
tarball_path=$(ls dist/borgmatic-*.tar.gz)
wheel_path=$(ls dist/borgmatic-*-py3-none-any.whl)
twine upload -r pypi --username __token__ "$tarball_path"
twine upload -r pypi --username __token__ "$wheel_path"
# Build docs and extract HTML.
scripts/export-docs-from-image
docs_path=dist/borgmatic-docs.tar.gz
# Set release changelogs on projects.torsion.org and GitHub.
release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
curl --silent --request POST \
release_id=$(curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases" \
--header "Authorization: token $projects_token" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data "{\"body\": \"$escaped_release_changelog\", \"draft\": false, \"name\": \"borgmatic $version\", \"prerelease\": false, \"tag_name\": \"$version\"}"
| jq ".id")
curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases/$release_id/assets?name=$(basename $wheel_path)" \
--header "Authorization: token $projects_token" \
--header "Accept: application/json" \
--form attachment=@"$wheel_path"
curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases/$release_id/assets?name=$(basename $tarball_path)" \
--header "Authorization: token $projects_token" \
--header "Accept: application/json" \
--form attachment=@"$tarball_path"
curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases/$release_id/assets?name=$(basename $docs_path)" \
--header "Authorization: token $projects_token" \
--header "Accept: application/json" \
--form attachment=@"$docs_path"
github-release create --token="$github_token" --owner=witten --repo=borgmatic --tag="$version" --target_commit="main" \
--name="borgmatic $version" --body="$release_changelog"
+11 -3
View File
@@ -512,6 +512,12 @@ def test_generate_sample_configuration_with_dry_run_does_not_write_file():
def test_generate_sample_configuration_with_split_writes_each_option_to_file():
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('schema.yaml', encoding='utf-8').and_return('')
builtins.should_receive('open').with_args(
'dest/options.json', 'w', encoding='utf-8'
).and_return(flexmock())
flexmock(module.json).should_receive('dump').with_args(
{'option_names': ['foo', 'bar']}, object
).once()
flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
flexmock(load=lambda filename: {})
)
@@ -520,17 +526,18 @@ def test_generate_sample_configuration_with_split_writes_each_option_to_file():
{'foo': 1, 'bar': 2}
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module).should_receive('get_configuration_subset')
flexmock(module).should_receive('render_configuration')
flexmock(module).should_receive('transform_optional_configuration')
flexmock(module).should_receive('transform_optional_configuration').and_return(' ')
flexmock(module.os).should_receive('makedirs')
flexmock(module).should_receive('write_configuration').with_args(
'dest/foo.yaml',
None,
'',
overwrite=False,
).once()
flexmock(module).should_receive('write_configuration').with_args(
'dest/bar.yaml',
None,
'',
overwrite=False,
).once()
@@ -549,6 +556,7 @@ def test_generate_sample_configuration_with_split_and_file_destination_errors():
)
flexmock(module.os.path).should_receive('exists').and_return(True)
flexmock(module.os.path).should_receive('isdir').and_return(False)
flexmock(module).should_receive('get_configuration_subset').never()
flexmock(module).should_receive('render_configuration').never()
flexmock(module).should_receive('transform_optional_configuration').never()
flexmock(module.os).should_receive('makedirs').never()
+85
View File
@@ -4,6 +4,7 @@ import os
import pytest
from flexmock import flexmock
import borgmatic.borg.pattern
from borgmatic.actions import create as module
@@ -296,6 +297,90 @@ def test_run_create_with_active_dumps_json_updates_archive_info():
) == [expected_create_result]
def mock_call_hooks(
function_name, config, hook_type, config_paths, borgmatic_runtime_directory, patterns, dry_run
):
'''
Simulate a dump_data_sources() call that mutates the given patterns.
'''
mock_dump_process = flexmock()
mock_dump_process.should_receive('poll').and_return(None).and_return(0)
patterns[0] = borgmatic.borg.pattern.Pattern('/mutated/pattern/path')
return {'dump': mock_dump_process}
def mock_call_hooks_even_if_unconfigured(
function_name, config, hook_type, borgmatic_runtime_directory, patterns, dry_run
):
'''
Assert that we're dealing with the original patterns here, not the mutated patterns.
'''
assert patterns[0].path == 'foo'
return {}
def test_run_create_with_active_dumps_removes_data_source_dumps_with_original_patterns():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.paths).should_receive('Runtime_directory').and_return(
flexmock(),
)
flexmock(module.borgmatic.borg.create).should_receive('create_archive').once()
flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').replace_with(
mock_call_hooks
)
flexmock(module.borgmatic.hooks.dispatch).should_receive(
'call_hooks_even_if_unconfigured',
).replace_with(mock_call_hooks_even_if_unconfigured)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module.borgmatic.actions.pattern).should_receive('collect_patterns').and_return(
(borgmatic.borg.pattern.Pattern('foo'), borgmatic.borg.pattern.Pattern('bar'))
)
flexmock(module.borgmatic.actions.pattern).should_receive('process_patterns').replace_with(
lambda patterns, *args, **kwargs: list(patterns)
)
flexmock(os.path).should_receive('join').and_return('/run/borgmatic/bootstrap')
flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
{'id': 'id1', 'name': 'archive.checkpoint'},
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
flexmock(module).should_receive('rename_checkpoint_archive').with_args(
repository_path='repo',
global_arguments=global_arguments,
config={},
local_borg_version=None,
local_path=None,
remote_path=None,
).once()
create_arguments = flexmock(
repository=None,
progress=flexmock(),
statistics=flexmock(),
json=False,
comment=None,
list_details=flexmock(),
)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
local_borg_version=None,
create_arguments=create_arguments,
global_arguments=global_arguments,
dry_run_label='',
local_path=None,
remote_path=None,
),
)
def test_rename_checkpoint_archive_renames_archive_using_name():
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
flexmock(module.borgmatic.borg.repo_list).should_receive('get_latest_archive').and_return(
-8
View File
@@ -21,14 +21,6 @@ def test_get_default_config_paths_prefers_xdg_config_home_for_user_config_path()
assert '/home/user/.etc/borgmatic/config.yaml' in config_paths
def test_get_default_config_paths_does_not_expand_home_when_false():
flexmock(module.os, environ={'HOME': '/home/user'})
config_paths = module.get_default_config_paths(expand_home=False)
assert '$HOME/.config/borgmatic/config.yaml' in config_paths
def test_collect_config_filenames_collects_yml_file_endings():
config_paths = ('config.yaml', '/etc/borgmatic.d')
mock_path = flexmock(module.os.path)
@@ -151,6 +151,38 @@ def test_load_credential_with_key_file():
)
def test_load_credential_with_key_file_and_ask_for_password_false():
flexmock(module.os.path).should_receive('expanduser').with_args('database.kdbx').and_return(
'database.kdbx',
)
flexmock(module.os.path).should_receive('exists').and_return(True)
flexmock(module.borgmatic.execute).should_receive(
'execute_command_and_capture_output',
).with_args(
(
'keepassxc-cli',
'show',
'--show-protected',
'--attributes',
'Password',
'--no-password',
'--key-file',
'/path/to/keyfile',
'database.kdbx',
'mypassword',
),
).and_return('password').once()
assert (
module.load_credential(
hook_config={'key_file': '/path/to/keyfile', 'ask_for_password': False},
config={},
credential_parameters=('database.kdbx', 'mypassword'),
)
== 'password'
)
def test_load_credential_with_yubikey():
flexmock(module.os.path).should_receive('expanduser').with_args('database.kdbx').and_return(
'database.kdbx',
@@ -186,6 +186,9 @@ def test_database_names_to_dump_queries_mariadb_for_database_names():
'trustsome1',
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
'/path/to/working/dir'
)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -196,6 +199,7 @@ def test_database_names_to_dump_queries_mariadb_for_database_names():
'show schemas',
),
environment=environment,
working_directory='/path/to/working/dir',
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -221,6 +225,7 @@ def test_database_names_to_dump_with_database_name_all_and_skip_names_filters_ou
'trustsome1',
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -231,6 +236,7 @@ def test_database_names_to_dump_with_database_name_all_and_skip_names_filters_ou
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nbaz\nmysql\n').once()
names = module.database_names_to_dump(
@@ -252,6 +258,7 @@ def test_database_names_to_dump_with_environment_password_transport_skips_defaul
).replace_with(lambda value, config: value)
flexmock(module).should_receive('parse_extra_options').and_return((), None)
flexmock(module).should_receive('make_defaults_file_options').never()
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -263,6 +270,7 @@ def test_database_names_to_dump_with_environment_password_transport_skips_defaul
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -288,6 +296,7 @@ def test_database_names_to_dump_runs_mariadb_with_tls():
'trustsome1',
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -299,6 +308,7 @@ def test_database_names_to_dump_runs_mariadb_with_tls():
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -324,6 +334,7 @@ def test_database_names_to_dump_runs_mariadb_without_tls():
'trustsome1',
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -335,6 +346,7 @@ def test_database_names_to_dump_runs_mariadb_without_tls():
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -699,6 +711,7 @@ def test_database_names_to_dump_runs_mariadb_with_list_options():
'trustsome1',
'mariadb.cnf',
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mariadb',
@@ -710,6 +723,7 @@ def test_database_names_to_dump_runs_mariadb_with_list_options():
'show schemas',
),
environment=None,
working_directory=None,
).and_return('foo\nbar').once()
assert module.database_names_to_dump(database, {}, 'root', 'trustsome1', None, '') == (
@@ -733,6 +747,7 @@ def test_database_names_to_dump_runs_non_default_mariadb_with_list_options():
'trustsome1',
'mariadb.cnf',
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
environment=None,
full_command=(
@@ -744,6 +759,7 @@ def test_database_names_to_dump_runs_non_default_mariadb_with_list_options():
'--execute',
'show schemas',
),
working_directory=None,
).and_return('foo\nbar').once()
assert module.database_names_to_dump(database, {}, 'root', 'trustsome1', None, '') == (
@@ -766,6 +782,7 @@ def test_execute_dump_command_runs_mariadb_dump():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -779,6 +796,7 @@ def test_execute_dump_command_runs_mariadb_dump():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -807,6 +825,7 @@ def test_execute_dump_command_with_environment_password_transport_skips_defaults
flexmock(module).should_receive('parse_extra_options').and_return((), None)
flexmock(module).should_receive('make_defaults_file_options').never()
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -821,6 +840,7 @@ def test_execute_dump_command_with_environment_password_transport_skips_defaults
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -853,6 +873,7 @@ def test_execute_dump_command_runs_mariadb_dump_without_add_drop_database():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -865,6 +886,7 @@ def test_execute_dump_command_runs_mariadb_dump_without_add_drop_database():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -897,6 +919,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_hostname_and_port():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -916,6 +939,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_hostname_and_port():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -948,6 +972,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_tls():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -962,6 +987,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_tls():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -994,6 +1020,7 @@ def test_execute_dump_command_runs_mariadb_dump_without_tls():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1008,6 +1035,7 @@ def test_execute_dump_command_runs_mariadb_dump_without_tls():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1040,6 +1068,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_username_and_password():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1053,6 +1082,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_username_and_password():
),
environment={},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1085,6 +1115,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_options():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1099,6 +1130,7 @@ def test_execute_dump_command_runs_mariadb_dump_with_options():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1131,6 +1163,7 @@ def test_execute_dump_command_runs_non_default_mariadb_dump_with_options():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1145,6 +1178,7 @@ def test_execute_dump_command_runs_non_default_mariadb_dump_with_options():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1241,12 +1275,14 @@ def test_restore_data_source_dump_runs_mariadb_to_restore():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mariadb', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1279,12 +1315,14 @@ def test_restore_data_source_dump_runs_mariadb_with_options():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mariadb', '--harder', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1319,12 +1357,14 @@ def test_restore_data_source_dump_runs_non_default_mariadb_with_options():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('custom_mariadb', '--harder', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1357,6 +1397,7 @@ def test_restore_data_source_dump_runs_mariadb_with_hostname_and_port():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mariadb',
@@ -1372,6 +1413,7 @@ def test_restore_data_source_dump_runs_mariadb_with_hostname_and_port():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1404,6 +1446,7 @@ def test_restore_data_source_dump_runs_mariadb_with_tls():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mariadb',
@@ -1414,6 +1457,7 @@ def test_restore_data_source_dump_runs_mariadb_with_tls():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1446,6 +1490,7 @@ def test_restore_data_source_dump_runs_mariadb_without_tls():
None,
).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mariadb',
@@ -1456,6 +1501,7 @@ def test_restore_data_source_dump_runs_mariadb_without_tls():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1488,12 +1534,14 @@ def test_restore_data_source_dump_runs_mariadb_with_username_and_password():
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mariadb', '--defaults-extra-file=/dev/fd/99', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1531,12 +1579,14 @@ def test_restore_data_source_with_environment_password_transport_skips_defaults_
flexmock(module.os).should_receive('environ').and_return(
{'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mariadb', '--batch', '--user', 'root'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1579,6 +1629,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mariadb',
@@ -1595,6 +1646,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1641,6 +1693,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mariadb',
@@ -1658,6 +1711,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -31,12 +31,16 @@ def test_dump_data_sources_runs_mongodump_for_each_database():
'databases/localhost/foo',
).and_return('databases/localhost/bar')
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
'/path/to/working/dir'
)
for name, process in zip(('foo', 'bar'), processes):
flexmock(module).should_receive('execute_command').with_args(
('mongodump', '--db', name, '--archive', '>', f'databases/localhost/{name}'),
shell=True,
run_to_completion=False,
working_directory='/path/to/working/dir',
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
@@ -100,6 +104,7 @@ def test_dump_data_sources_runs_mongodump_with_hostname_and_port():
'databases/database.example.org/foo',
)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -116,6 +121,7 @@ def test_dump_data_sources_runs_mongodump_with_hostname_and_port():
),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -165,6 +171,7 @@ def test_dump_data_sources_runs_mongodump_with_username_and_password():
'/dev/fd/99',
)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -183,6 +190,7 @@ def test_dump_data_sources_runs_mongodump_with_username_and_password():
),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -217,10 +225,12 @@ def test_dump_data_sources_runs_mongodump_with_directory_format():
)
flexmock(module.dump).should_receive('create_parent_directory_for_dump')
flexmock(module.dump).should_receive('create_named_pipe_for_dump').never()
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
('mongodump', '--out', 'databases/localhost/foo', '--db', 'foo'),
shell=True,
working_directory=None,
).and_return(flexmock()).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -258,6 +268,7 @@ def test_dump_data_sources_runs_mongodump_with_options():
'databases/localhost/foo',
)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -271,6 +282,7 @@ def test_dump_data_sources_runs_mongodump_with_options():
),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -305,11 +317,13 @@ def test_dump_data_sources_runs_mongodumpall_for_all_databases():
'databases/localhost/all',
)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
('mongodump', '--archive', '>', 'databases/localhost/all'),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -373,11 +387,13 @@ def test_restore_data_source_dump_runs_mongorestore():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
['mongorestore', '--archive', '--drop'],
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -407,6 +423,7 @@ def test_restore_data_source_dump_runs_mongorestore_with_hostname_and_port():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'mongorestore',
@@ -420,6 +437,7 @@ def test_restore_data_source_dump_runs_mongorestore_with_hostname_and_port():
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -458,6 +476,7 @@ def test_restore_data_source_dump_runs_mongorestore_with_username_and_password()
flexmock(module).should_receive('make_password_config_file').with_args('trustsome1').and_return(
'/dev/fd/99',
)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'mongorestore',
@@ -473,6 +492,7 @@ def test_restore_data_source_dump_runs_mongorestore_with_username_and_password()
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -515,6 +535,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
flexmock(module).should_receive('make_password_config_file').with_args(
'clipassword',
).and_return('/dev/fd/99')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'mongorestore',
@@ -534,6 +555,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -576,6 +598,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
flexmock(module).should_receive('make_password_config_file').with_args(
'restorepass',
).and_return('/dev/fd/99')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'mongorestore',
@@ -595,6 +618,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -622,11 +646,13 @@ def test_restore_data_source_dump_runs_mongorestore_with_options():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
['mongorestore', '--archive', '--drop', '--harder'],
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -654,6 +680,7 @@ def test_restore_databases_dump_runs_mongorestore_with_schemas():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'mongorestore',
@@ -667,6 +694,7 @@ def test_restore_databases_dump_runs_mongorestore_with_schemas():
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -694,11 +722,13 @@ def test_restore_data_source_dump_runs_psql_for_all_database_dump():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
['mongorestore', '--archive'],
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -751,11 +781,13 @@ def test_restore_data_source_dump_without_extract_process_restores_from_disk():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
['mongorestore', '--dir', '/dump/path', '--drop'],
processes=[],
output_log_level=logging.DEBUG,
input_file=None,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -785,6 +817,7 @@ def test_dump_data_sources_uses_custom_mongodump_command():
'databases/localhost/foo',
)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -797,6 +830,7 @@ def test_dump_data_sources_uses_custom_mongodump_command():
),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -855,6 +889,7 @@ def test_restore_data_source_dump_uses_custom_mongorestore_command():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
[
'custom_mongorestore', # Should use custom command instead of default
@@ -865,6 +900,7 @@ def test_restore_data_source_dump_uses_custom_mongorestore_command():
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -68,6 +68,7 @@ def test_database_names_to_dump_queries_mysql_for_database_names():
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -78,6 +79,7 @@ def test_database_names_to_dump_queries_mysql_for_database_names():
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -107,6 +109,7 @@ def test_database_names_to_dump_with_database_name_all_and_skip_names_filters_ou
'trustsome1',
None,
).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -117,6 +120,7 @@ def test_database_names_to_dump_with_database_name_all_and_skip_names_filters_ou
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nbaz\nmysql\n').once()
names = module.database_names_to_dump(
@@ -142,6 +146,7 @@ def test_database_names_to_dump_with_environment_password_transport_skips_defaul
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).never()
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -153,6 +158,7 @@ def test_database_names_to_dump_with_environment_password_transport_skips_defaul
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -178,6 +184,7 @@ def test_database_names_to_dump_runs_mysql_with_tls():
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -189,6 +196,7 @@ def test_database_names_to_dump_runs_mysql_with_tls():
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -214,6 +222,7 @@ def test_database_names_to_dump_runs_mysql_without_tls():
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -225,6 +234,7 @@ def test_database_names_to_dump_runs_mysql_without_tls():
'show schemas',
),
environment=environment,
working_directory=None,
).and_return('foo\nbar\nmysql\n').once()
names = module.database_names_to_dump(
@@ -584,6 +594,7 @@ def test_database_names_to_dump_runs_mysql_with_list_options():
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).with_args('root', 'trustsome1', 'my.cnf').and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'mysql',
@@ -595,6 +606,7 @@ def test_database_names_to_dump_runs_mysql_with_list_options():
'show schemas',
),
environment=None,
working_directory=None,
).and_return('foo\nbar').once()
assert module.database_names_to_dump(database, {}, 'root', 'trustsome1', None, '') == (
@@ -615,6 +627,7 @@ def test_database_names_to_dump_runs_non_default_mysql_with_list_options():
flexmock(module.borgmatic.hooks.data_source.mariadb).should_receive(
'make_defaults_file_options',
).with_args('root', 'trustsome1', 'my.cnf').and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
environment=None,
full_command=(
@@ -626,6 +639,7 @@ def test_database_names_to_dump_runs_non_default_mysql_with_list_options():
'--execute',
'show schemas',
),
working_directory=None,
).and_return('foo\nbar').once()
assert module.database_names_to_dump(database, {}, 'root', 'trustsome1', None, '') == (
@@ -648,6 +662,7 @@ def test_execute_dump_command_runs_mysqldump():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -661,6 +676,7 @@ def test_execute_dump_command_runs_mysqldump():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -693,6 +709,7 @@ def test_execute_dump_command_with_environment_password_transport_skips_defaults
'make_defaults_file_options',
).never()
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -707,6 +724,7 @@ def test_execute_dump_command_with_environment_password_transport_skips_defaults
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -739,6 +757,7 @@ def test_execute_dump_command_runs_mysqldump_without_add_drop_database():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -751,6 +770,7 @@ def test_execute_dump_command_runs_mysqldump_without_add_drop_database():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -783,6 +803,7 @@ def test_execute_dump_command_runs_mysqldump_with_hostname_and_port():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -802,6 +823,7 @@ def test_execute_dump_command_runs_mysqldump_with_hostname_and_port():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -834,6 +856,7 @@ def test_execute_dump_command_runs_mysqldump_with_tls():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -848,6 +871,7 @@ def test_execute_dump_command_runs_mysqldump_with_tls():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -880,6 +904,7 @@ def test_execute_dump_command_runs_mysqldump_without_tls():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -894,6 +919,7 @@ def test_execute_dump_command_runs_mysqldump_without_tls():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -926,6 +952,7 @@ def test_execute_dump_command_runs_mysqldump_with_username_and_password():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -939,6 +966,7 @@ def test_execute_dump_command_runs_mysqldump_with_username_and_password():
),
environment={},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -971,6 +999,7 @@ def test_execute_dump_command_runs_mysqldump_with_options():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -985,6 +1014,7 @@ def test_execute_dump_command_runs_mysqldump_with_options():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1017,6 +1047,7 @@ def test_execute_dump_command_runs_non_default_mysqldump():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1030,6 +1061,7 @@ def test_execute_dump_command_runs_non_default_mysqldump():
),
environment=None,
run_to_completion=False,
working_directory=None,
).and_return(process).once()
assert (
@@ -1125,12 +1157,14 @@ def test_restore_data_source_dump_runs_mysql_to_restore():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mysql', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1163,12 +1197,14 @@ def test_restore_data_source_dump_runs_mysql_with_options():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mysql', '--harder', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1201,12 +1237,14 @@ def test_restore_data_source_dump_runs_non_default_mysql_with_options():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('custom_mysql', '--harder', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1239,6 +1277,7 @@ def test_restore_data_source_dump_runs_mysql_with_hostname_and_port():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mysql',
@@ -1254,6 +1293,7 @@ def test_restore_data_source_dump_runs_mysql_with_hostname_and_port():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1286,6 +1326,7 @@ def test_restore_data_source_dump_runs_mysql_with_tls():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mysql',
@@ -1296,6 +1337,7 @@ def test_restore_data_source_dump_runs_mysql_with_tls():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1328,6 +1370,7 @@ def test_restore_data_source_dump_runs_mysql_without_tls():
'make_defaults_file_options',
).with_args(None, None, None).and_return(())
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mysql',
@@ -1338,6 +1381,7 @@ def test_restore_data_source_dump_runs_mysql_without_tls():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1370,12 +1414,14 @@ def test_restore_data_source_dump_runs_mysql_with_username_and_password():
'make_defaults_file_options',
).with_args('root', 'trustsome1', None).and_return(('--defaults-extra-file=/dev/fd/99',))
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mysql', '--defaults-extra-file=/dev/fd/99', '--batch'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1417,12 +1463,14 @@ def test_restore_data_source_with_environment_password_transport_skips_defaults_
flexmock(module.os).should_receive('environ').and_return(
{'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('mysql', '--batch', '--user', 'root'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root', 'MYSQL_PWD': 'trustsome1'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1467,6 +1515,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
('--defaults-extra-file=/dev/fd/99',),
)
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mysql',
@@ -1483,6 +1532,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1531,6 +1581,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
('--defaults-extra-file=/dev/fd/99',),
)
flexmock(module.os).should_receive('environ').and_return({'USER': 'root'})
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'mysql',
@@ -1548,6 +1599,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'USER': 'root'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -117,6 +117,9 @@ def test_database_names_to_dump_with_all_and_format_lists_databases_with_hostnam
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
'/path/to/working/dir'
)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'psql',
@@ -131,6 +134,7 @@ def test_database_names_to_dump_with_all_and_format_lists_databases_with_hostnam
'1234',
),
environment=object,
working_directory='/path/to/working/dir',
).and_return('foo,test,\nbar,test,"stuff and such"')
assert module.database_names_to_dump(database, {}, flexmock(), dry_run=False) == (
@@ -144,6 +148,7 @@ def test_database_names_to_dump_with_all_and_format_lists_databases_with_usernam
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'psql',
@@ -156,6 +161,7 @@ def test_database_names_to_dump_with_all_and_format_lists_databases_with_usernam
'postgres',
),
environment=object,
working_directory=None,
).and_return('foo,test,\nbar,test,"stuff and such"')
assert module.database_names_to_dump(database, {}, flexmock(), dry_run=False) == (
@@ -169,9 +175,11 @@ def test_database_names_to_dump_with_all_and_format_lists_databases_with_options
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('psql', '--list', '--no-password', '--no-psqlrc', '--csv', '--tuples-only', '--harder'),
environment=object,
working_directory=None,
).and_return('foo,test,\nbar,test,"stuff and such"')
assert module.database_names_to_dump(database, {}, flexmock(), dry_run=False) == (
@@ -201,6 +209,7 @@ def test_database_names_to_dump_with_all_and_psql_command_uses_custom_command():
flexmock(module.borgmatic.hooks.credential.parse).should_receive(
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'docker',
@@ -216,6 +225,7 @@ def test_database_names_to_dump_with_all_and_psql_command_uses_custom_command():
'--tuples-only',
),
environment=object,
working_directory=None,
).and_return('foo,text').once()
assert module.database_names_to_dump(database, {}, flexmock(), dry_run=False) == ('foo',)
@@ -255,6 +265,7 @@ def test_dump_data_sources_runs_pg_dump_for_each_database():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
for name, process in zip(('foo', 'bar'), processes):
flexmock(module).should_receive('execute_command').with_args(
@@ -272,6 +283,7 @@ def test_dump_data_sources_runs_pg_dump_for_each_database():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -428,6 +440,7 @@ def test_dump_data_sources_runs_pg_dump_with_hostname_and_port():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -448,6 +461,7 @@ def test_dump_data_sources_runs_pg_dump_with_hostname_and_port():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -492,6 +506,7 @@ def test_dump_data_sources_runs_pg_dump_with_username_and_password():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -510,6 +525,7 @@ def test_dump_data_sources_runs_pg_dump_with_username_and_password():
shell=True,
environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -552,6 +568,7 @@ def test_dump_data_sources_with_username_injection_attack_gets_escaped():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -570,6 +587,7 @@ def test_dump_data_sources_with_username_injection_attack_gets_escaped():
shell=True,
environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -610,6 +628,7 @@ def test_dump_data_sources_runs_pg_dump_with_directory_format():
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_parent_directory_for_dump')
flexmock(module.dump).should_receive('create_named_pipe_for_dump').never()
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -625,6 +644,7 @@ def test_dump_data_sources_runs_pg_dump_with_directory_format():
),
shell=True,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).and_return(flexmock()).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -668,6 +688,7 @@ def test_dump_data_sources_runs_pg_dump_with_string_compression():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -686,6 +707,7 @@ def test_dump_data_sources_runs_pg_dump_with_string_compression():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(processes[0]).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -729,6 +751,7 @@ def test_dump_data_sources_runs_pg_dump_with_integer_compression():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -747,6 +770,7 @@ def test_dump_data_sources_runs_pg_dump_with_integer_compression():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(processes[0]).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -790,6 +814,7 @@ def test_dump_data_sources_runs_pg_dump_with_options():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -807,6 +832,7 @@ def test_dump_data_sources_runs_pg_dump_with_options():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -847,12 +873,14 @@ def test_dump_data_sources_runs_pg_dumpall_for_all_databases():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
('pg_dumpall', '--no-password', '--clean', '--if-exists', '>', 'databases/localhost/all'),
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -893,6 +921,7 @@ def test_dump_data_sources_runs_non_default_pg_dump():
'resolve_credential',
).replace_with(lambda value, config: value)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
@@ -911,6 +940,7 @@ def test_dump_data_sources_runs_non_default_pg_dump():
shell=True,
environment={'PGSSLMODE': 'disable'},
run_to_completion=False,
working_directory=None,
).and_return(process).once()
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -947,6 +977,7 @@ def test_restore_data_source_dump_runs_pg_restore():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -961,6 +992,7 @@ def test_restore_data_source_dump_runs_pg_restore():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -974,6 +1006,7 @@ def test_restore_data_source_dump_runs_pg_restore():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1004,6 +1037,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_hostname_and_port():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1022,6 +1056,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_hostname_and_port():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1039,6 +1074,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_hostname_and_port():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1071,6 +1107,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_username_and_password():
)
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1087,6 +1124,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_username_and_password():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1102,6 +1140,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_username_and_password():
'ANALYZE',
),
environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1145,6 +1184,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
)
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1165,6 +1205,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGPASSWORD': 'clipassword', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1184,6 +1225,7 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
'ANALYZE',
),
environment={'PGPASSWORD': 'clipassword', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1227,6 +1269,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
)
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1247,6 +1290,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGPASSWORD': 'restorepassword', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1266,6 +1310,7 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
'ANALYZE',
),
environment={'PGPASSWORD': 'restorepassword', 'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1301,6 +1346,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_options():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1316,6 +1362,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_options():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1330,6 +1377,7 @@ def test_restore_data_source_dump_runs_pg_restore_with_options():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1358,6 +1406,7 @@ def test_restore_data_source_dump_runs_psql_for_all_database_dump():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'psql',
@@ -1368,10 +1417,12 @@ def test_restore_data_source_dump_runs_psql_for_all_database_dump():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
('psql', '--no-password', '--no-psqlrc', '--quiet', '--command', 'ANALYZE'),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1400,12 +1451,14 @@ def test_restore_data_source_dump_runs_psql_for_plain_database_dump():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
('psql', '--no-password', '--no-psqlrc', '--dbname', 'foo'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1419,6 +1472,7 @@ def test_restore_data_source_dump_runs_psql_for_plain_database_dump():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1454,6 +1508,7 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'docker',
@@ -1473,6 +1528,7 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1491,6 +1547,7 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1545,6 +1602,7 @@ def test_restore_data_source_dump_without_extract_process_restores_from_disk():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('/dump/path')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1560,6 +1618,7 @@ def test_restore_data_source_dump_without_extract_process_restores_from_disk():
output_log_level=logging.DEBUG,
input_file=None,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1573,6 +1632,7 @@ def test_restore_data_source_dump_without_extract_process_restores_from_disk():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -1600,6 +1660,7 @@ def test_restore_data_source_dump_with_schemas_restores_schemas():
flexmock(module).should_receive('make_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path')
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('/dump/path')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'pg_restore',
@@ -1619,6 +1680,7 @@ def test_restore_data_source_dump_with_schemas_restores_schemas():
output_log_level=logging.DEBUG,
input_file=None,
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
(
@@ -1632,6 +1694,7 @@ def test_restore_data_source_dump_with_schemas_restores_schemas():
'ANALYZE',
),
environment={'PGSSLMODE': 'disable'},
working_directory=None,
).once()
module.restore_data_source_dump(
@@ -111,9 +111,13 @@ def test_dump_data_sources_with_path_injection_attack_gets_escaped():
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(
'/path/to/working/dir'
)
flexmock(module).should_receive('execute_command').with_args(
(
'sqlite3',
'-bail',
"'/path/to/database1; naughty-command'",
'.dump',
'>',
@@ -121,6 +125,7 @@ def test_dump_data_sources_with_path_injection_attack_gets_escaped():
),
shell=True,
run_to_completion=False,
working_directory='/path/to/working/dir',
).and_return(processes[0])
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -166,10 +171,12 @@ def test_dump_data_sources_runs_non_default_sqlite_with_path_injection_attack_ge
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command').with_args(
(
'custom_sqlite', # custom sqlite command
"'*'", # Should get shell escaped to prevent injection attacks.
'-bail',
"'/path/to/database1; naughty-command'",
'.dump',
'>',
@@ -177,6 +184,7 @@ def test_dump_data_sources_runs_non_default_sqlite_with_path_injection_attack_ge
),
shell=True,
run_to_completion=False,
working_directory=None,
).and_return(processes[0])
flexmock(module.dump).should_receive('write_data_source_dumps_metadata').with_args(
'/run/borgmatic',
@@ -322,14 +330,17 @@ def test_restore_data_source_dump_restores_database():
hook_config = [{'path': '/path/to/database', 'name': 'database'}, {'name': 'other'}]
extract_process = flexmock(stdout=flexmock())
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'sqlite3',
'-bail',
'/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()
@@ -356,15 +367,18 @@ def test_restore_data_source_dump_runs_non_default_sqlite_restores_database():
]
extract_process = flexmock(stdout=flexmock())
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'custom_sqlite',
"'*'", # Should get shell escaped to prevent injection attacks.
'-bail',
'/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()
@@ -390,14 +404,17 @@ def test_restore_data_source_dump_with_connection_params_uses_connection_params_
]
extract_process = flexmock(stdout=flexmock())
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'sqlite3',
'-bail',
'cli/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()
@@ -423,14 +440,17 @@ def test_restore_data_source_dump_runs_non_default_sqlite_with_connection_params
]
extract_process = flexmock(stdout=flexmock())
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'custom_sqlite',
'-bail',
'cli/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()
@@ -462,11 +482,13 @@ def test_restore_data_source_dump_without_connection_params_uses_restore_params_
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'sqlite3',
'-bail',
'config/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()
@@ -493,14 +515,17 @@ def test_restore_data_source_dump_runs_non_default_sqlite_without_connection_par
]
extract_process = flexmock(stdout=flexmock())
flexmock(module.borgmatic.config.paths).should_receive('get_working_directory').and_return(None)
flexmock(module).should_receive('execute_command_with_processes').with_args(
(
'custom_sqlite',
'-bail',
'config/path/to/database',
),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
working_directory=None,
).once()
flexmock(module.os).should_receive('remove').once()