Compare commits

...
58 Commits
Author SHA1 Message Date
Dan Helfman 8c301ba688 Bump version for release. 2024-03-11 13:27:08 -07:00
Dan Helfman 035e96156a Add an "access_token" option to the ntfy monitoring hook for authenticating without username/password (#811). 2024-03-11 12:48:58 -07:00
Dan Helfman a08c7fc77a When running the "rcreate" action and the repository already exists but with a different encryption mode than requested, error (#840). 2024-03-11 11:24:36 -07:00
Dan Helfman cf9e387811 Document a potentially breaking shell quoting edge case within error hooks (#839). 2024-03-11 10:42:51 -07:00
Dan Helfman e37224606a Clarify dev-CI parity. 2024-03-10 19:14:18 -07:00
Dan Helfman 9647301b99 Add log sending for the Apprise logging hook, enabled by default. 2024-03-10 16:18:49 -07:00
Dan Helfman a0e5dbff96 Remove list of command in Bash script. 2024-03-06 21:24:44 -08:00
Dan Helfman 86117edccf Remove build.torsion.org references from documentation. 2024-03-06 20:01:32 -08:00
Dan Helfman 440f3eeb63 Remove Drone configuration/tests. 2024-03-06 19:04:29 -08:00
Dan Helfman 181051eae1 Add new build server to NEWS. 2024-03-06 18:52:27 -08:00
Dan Helfman ec0ee971ed Attempt to use secrets. 2024-03-06 18:38:45 -08:00
Dan Helfman b83ffa0cf6 Attempt to fix trigger. 2024-03-06 16:53:41 -08:00
Dan Helfman cf88665d37 Fix typo. 2024-03-06 16:52:33 -08:00
Dan Helfman b233adba63 Fix build? 2024-03-06 16:51:49 -08:00
Dan Helfman 018f5e3315 Merge workflows, since Gitea doesn't yet support workflow_run. 2024-03-06 16:49:50 -08:00
Dan Helfman 284f26b49d Only run tests on pushes to main branch. 2024-03-06 16:40:39 -08:00
Dan Helfman 11b437794e Attempt to build documentation. 2024-03-06 16:38:34 -08:00
Dan Helfman 0665b50d57 Fixed debugging. 2024-03-06 16:17:12 -08:00
Dan Helfman 0586b80e5b More debugging. 2024-03-06 15:53:30 -08:00
Dan Helfman 272a7b4866 Actually kill other containers after tests finish. 2024-03-06 15:41:03 -08:00
Dan Helfman 98d4a59459 Another iteration. 2024-03-06 15:29:56 -08:00
Dan Helfman 744139cf97 Disable progress. 2024-03-06 15:21:45 -08:00
Dan Helfman 1339509e9b Flag order apparently matters to Docker Compose. 2024-03-06 14:55:55 -08:00
Dan Helfman e14f61415b Fix spew in test script. 2024-03-06 14:54:53 -08:00
Dan Helfman 98cf8f7e20 Another try at exiting tests properly. 2024-03-06 14:42:06 -08:00
Dan Helfman 5f16b64639 Attempt to exit test containers on tests exit while also showing test output. 2024-03-06 14:39:23 -08:00
Dan Helfman fe62a81151 Add missing service name to test scrits. 2024-03-06 14:32:26 -08:00
Dan Helfman 585b1573ae Attempt to make containers stop after tests run. 2024-03-06 14:30:49 -08:00
Dan Helfman 141ba2771d Attempt to fix and debug read-only filesystem issue at build. 2024-03-06 11:10:20 -08:00
Dan Helfman a527f76d08 Add back checkout now that NodeJS is installed on the host. 2024-03-06 08:49:53 -08:00
Dan Helfman a97c68b4c8 Debugging ls. 2024-03-06 08:35:50 -08:00
Dan Helfman ef07005a75 Remove duplicative(?) checkout step. 2024-03-06 08:35:05 -08:00
Dan Helfman 43c7c3b6be First attempt at using Gitea Actions to run tests. 2024-03-06 08:32:55 -08:00
Dan Helfman 2f6ad9d173 Add NO_COLOR support to NEWS (#835). 2024-03-04 13:49:54 -08:00
Dan HelfmanandGitHub 16bc0de3fb Support for NO_COLOR environment variable (#835).
Merge pull request #82 from shivansh02/feature/support-no-color-env-var
2024-03-04 13:46:09 -08:00
shivansh02 458d157e62 NO_COLOR set to any value returns false 2024-03-05 00:15:52 +05:30
shivansh02 40c3a28620 support for NO_COLOR env var 2024-03-04 18:21:28 +05:30
Dan Helfman 60107f1ee8 Add custom dump/restore command options for MySQL and MariaDB (#311). 2024-03-03 14:32:49 -08:00
Dan HelfmanandGitHub a1153a21fa Custom dump command options for MySQL and MariaDB.
Merge pull request #81 from shivansh02/feature/custom-dump-restore-commands-mysql
2024-03-03 14:27:14 -08:00
shivansh02 b6cb7da98e custom dump commands for mariadb 2024-03-04 00:24:22 +05:30
shivansh02 9e3d19a406 custom commands escaped 2024-03-03 23:31:02 +05:30
shivansh02 2b755d8ade custom show command for mysql and schema description 2024-03-03 23:15:07 +05:30
shivansh02 925f99cfef custom dump command for mysql 2024-03-03 03:47:02 +05:30
Dan Helfman c9f20eb260 Fix "--override" values containing deprecated section headers not actually overriding configuration options under deprecated section headers (#829). 2024-02-15 21:12:42 -08:00
Dan Helfman f4744826fe When the "--json" flag is given, suppress console escape codes so as not to interfere with JSON output (#827). 2024-02-11 17:44:43 -08:00
Dan Helfman 5586aab967 Clarify documentation about restoring a database: borgmatic does not create the database upon restore. 2024-02-09 15:35:29 -08:00
Dan Helfman 6fa5dff79b Fix broken escaping logic for "pg_dump_command" (#822) + bonus shell injection fixes. 2024-01-31 10:53:32 -08:00
Dan Helfman 75d11aa9cd Pass the PostgreSQL "PGSSLMODE" environment variable through to Borg (#370). 2024-01-25 14:18:01 -08:00
Dan Helfman ad1d104d65 Fix broken repository detection in the "rcreate" action with Borg 1.4 (#820). 2024-01-24 15:45:51 -08:00
Dan Helfman 009062128d Remove Python 3.8+ restriction, as only Python 3.8+ is supported. 2024-01-22 09:41:43 -08:00
Dan Helfman e9813d2539 Allow the "--repository" flag to match across multiple configuration files (#818). 2024-01-21 18:25:44 -08:00
Dan Helfman f9998b50e8 Rephrase documentation and link to docs on exit codes feature (#798). 2024-01-21 14:47:21 -08:00
Dan Helfman 5f921a7f80 Add documentation heading (#798). 2024-01-21 11:48:23 -08:00
Dan Helfman abf2b3a8c7 Elevate specific Borg warnings to errors or squash errors to warnings (#798). 2024-01-21 11:34:40 -08:00
Dan Helfman 34f3c2bb16 Clarify "--override" command-line help (#814) 2024-01-19 11:55:00 -08:00
Dan Helfman 4d79f582df Fix a traceback when providing an invalid "--override" value for a list option (#814). 2024-01-18 10:39:40 -08:00
Dan Helfman 63198088c4 Store included configuration files within each backup archive in support of the "config bootstrap" action (#736). 2024-01-09 13:47:20 -08:00
Dan Helfman 3c22a8ec16 Prevent various shell injection attacks (#810). 2024-01-07 10:21:49 -08:00
101 changed files with 2959 additions and 842 deletions
-95
View File
@@ -1,95 +0,0 @@
---
kind: pipeline
name: python-3-8-alpine-3-13
services:
- name: postgresql
image: docker.io/postgres:13.1-alpine
environment:
POSTGRES_PASSWORD: test
POSTGRES_DB: test
- name: postgresql2
image: docker.io/postgres:13.1-alpine
environment:
POSTGRES_PASSWORD: test2
POSTGRES_DB: test
commands:
- docker-entrypoint.sh -p 5433
- name: mariadb
image: docker.io/mariadb:10.11.4
environment:
MARIADB_ROOT_PASSWORD: test
MARIADB_DATABASE: test
- name: mariadb2
image: docker.io/mariadb:10.11.4
environment:
MARIADB_ROOT_PASSWORD: test2
MARIADB_DATABASE: test
commands:
- docker-entrypoint.sh --port=3307
- name: not-actually-mysql
image: docker.io/mariadb:10.11.4
environment:
MARIADB_ROOT_PASSWORD: test
MARIADB_DATABASE: test
- name: not-actually-mysql2
image: docker.io/mariadb:10.11.4
environment:
MARIADB_ROOT_PASSWORD: test2
MARIADB_DATABASE: test
commands:
- docker-entrypoint.sh --port=3307
- name: mongodb
image: docker.io/mongo:5.0.5
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: test
- name: mongodb2
image: docker.io/mongo:5.0.5
environment:
MONGO_INITDB_ROOT_USERNAME: root2
MONGO_INITDB_ROOT_PASSWORD: test2
commands:
- docker-entrypoint.sh --port=27018
clone:
skip_verify: true
steps:
- name: build
image: docker.io/alpine:3.13
environment:
TEST_CONTAINER: true
pull: always
commands:
- scripts/run-full-tests
---
kind: pipeline
name: documentation
type: exec
platform:
os: linux
arch: amd64
clone:
skip_verify: true
steps:
- name: build
environment:
USERNAME:
from_secret: docker_username
PASSWORD:
from_secret: docker_password
IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
commands:
- podman login --username "$USERNAME" --password "$PASSWORD" projects.torsion.org
- podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
- podman push "$IMAGE_NAME"
trigger:
repo:
- borgmatic-collective/borgmatic
branch:
- main
+28
View File
@@ -0,0 +1,28 @@
name: build
run-name: ${{ gitea.actor }} is building
on:
push:
branches: [main]
jobs:
test:
runs-on: host
steps:
- uses: actions/checkout@v4
- run: scripts/run-end-to-end-tests
docs:
needs: [test]
runs-on: host
env:
IMAGE_NAME: projects.torsion.org/borgmatic-collective/borgmatic:docs
steps:
- uses: actions/checkout@v4
- run: podman login --username "$USERNAME" --password "$PASSWORD" projects.torsion.org
env:
USERNAME: "${{ secrets.REGISTRY_USERNAME }}"
PASSWORD: "${{ secrets.REGISTRY_PASSWORD }}"
- run: podman build --tag "$IMAGE_NAME" --file docs/Dockerfile --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" .
- run: podman push "$IMAGE_NAME"
+43
View File
@@ -1,3 +1,46 @@
1.8.9
* #311: Add custom dump/restore command options for MySQL and MariaDB.
* #811: Add an "access_token" option to the ntfy monitoring hook for authenticating
without username/password.
* #827: When the "--json" flag is given, suppress console escape codes so as not to
interfere with JSON output.
* #829: Fix "--override" values containing deprecated section headers not actually overriding
configuration options under deprecated section headers.
* #835: Add support for the NO_COLOR environment variable. See the documentation for more
information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#colored-output
* #839: Add log sending for the Apprise logging hook, enabled by default. See the documentation for
more information:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook
* #839: Document a potentially breaking shell quoting edge case within error hooks:
https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#error-hooks
* #840: When running the "rcreate" action and the repository already exists but with a different
encryption mode than requested, error.
* Switch from Drone to Gitea Actions for continuous integration.
* Rename scripts/run-end-to-end-dev-tests to scripts/run-end-to-end-tests and use it in both dev
and CI for better dev-CI parity.
* Clarify documentation about restoring a database: borgmatic does not create the database upon
restore.
1.8.8
* #370: For the PostgreSQL hook, pass the "PGSSLMODE" environment variable through to Borg when the
database's configuration omits the "ssl_mode" option.
* #818: Allow the "--repository" flag to match across multiple configuration files.
* #820: Fix broken repository detection in the "rcreate" action with Borg 1.4. The issue did not
occur with other versions of Borg.
* #822: Fix broken escaping logic in the PostgreSQL hook's "pg_dump_command" option.
* SECURITY: Prevent additional shell injection attacks within the PostgreSQL hook.
1.8.7
* #736: Store included configuration files within each backup archive in support of the "config
bootstrap" action. Previously, only top-level configuration files were stored.
* #798: Elevate specific Borg warnings to errors or squash errors to
* warnings. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
* #810: SECURITY: Prevent shell injection attacks within the PostgreSQL hook, the MongoDB hook, the
SQLite hook, the "borgmatic borg" action, and command hook variable/constant interpolation.
* #814: Fix a traceback when providing an invalid "--override" value for a list option.
1.8.6
* #767: Add an "--ssh-command" flag to the "config bootstrap" action for setting a custom SSH
command, as no configuration is available (including the "ssh_command" option) until
-3
View File
@@ -154,6 +154,3 @@ general, contributions are very welcome. We don't bite!
Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/main)</a>
+7 -9
View File
@@ -13,14 +13,11 @@ logger = logging.getLogger(__name__)
def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
'''
Given:
The bootstrap arguments, which include the repository and archive name, borgmatic source directory,
destination directory, and whether to strip components.
The global arguments, which include the dry run flag
and the local borg version,
Return:
The config paths from the manifest.json file in the borgmatic source directory after extracting it from the
repository.
Given the bootstrap arguments as an argparse.Namespace (containing the repository and archive
name, borgmatic source directory, destination directory, and whether to strip components), the
global arguments as an argparse.Namespace (containing the dry run flag and the local borg
version), return the config paths from the manifest.json file in the borgmatic source directory
after extracting it from the repository.
Raise ValueError if the manifest JSON is missing, can't be decoded, or doesn't contain the
expected configuration path data.
@@ -32,6 +29,7 @@ def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
)
config = {'ssh_command': bootstrap_arguments.ssh_command}
extract_process = borgmatic.borg.extract.extract_archive(
global_arguments.dry_run,
bootstrap_arguments.repository,
@@ -48,8 +46,8 @@ def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
global_arguments,
extract_to_stdout=True,
)
manifest_json = extract_process.stdout.read()
if not manifest_json:
raise ValueError(
'Cannot read configuration paths from archive due to missing bootstrap manifest'
+5 -1
View File
@@ -47,6 +47,7 @@ def run_create(
config_filename,
repository,
config,
config_paths,
hook_context,
local_borg_version,
create_arguments,
@@ -90,7 +91,9 @@ def run_create(
)
if config.get('store_config_files', True):
create_borgmatic_manifest(
config, global_arguments.used_config_paths, global_arguments.dry_run
config,
config_paths,
global_arguments.dry_run,
)
stream_processes = [process for processes in active_dumps.values() for process in processes]
@@ -98,6 +101,7 @@ def run_create(
global_arguments.dry_run,
repository['path'],
config,
config_paths,
local_borg_version,
global_arguments,
local_path=local_path,
+4 -2
View File
@@ -1,4 +1,5 @@
import logging
import shlex
import borgmatic.commands.arguments
import borgmatic.logger
@@ -56,9 +57,8 @@ def run_arbitrary_borg(
)
return execute_command(
full_command,
tuple(shlex.quote(part) for part in full_command),
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
shell=True,
extra_environment=dict(
(environment.make_environment(config) or {}),
@@ -67,4 +67,6 @@ def run_arbitrary_borg(
'ARCHIVE': archive if archive else '',
},
),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+6 -1
View File
@@ -34,4 +34,9 @@ def break_lock(
)
borg_environment = environment.make_environment(config)
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+12 -2
View File
@@ -434,15 +434,25 @@ def check_archives(
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The Borg repair option triggers an interactive prompt, which won't work when output is
# captured. And progress messes with the terminal directly.
if check_arguments.repair or check_arguments.progress:
execute_command(
full_command, output_file=DO_NOT_CAPTURE, extra_environment=borg_environment
full_command,
output_file=DO_NOT_CAPTURE,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(full_command, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
for check in checks:
write_check_time(
+2 -1
View File
@@ -48,6 +48,7 @@ def compact_segments(
execute_command(
full_command,
output_log_level=logging.INFO,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+14 -10
View File
@@ -272,7 +272,7 @@ def any_parent_directories(path, candidate_parents):
def collect_special_file_paths(
create_command, local_path, working_directory, borg_environment, skip_directories
create_command, config, local_path, working_directory, borg_environment, skip_directories
):
'''
Given a Borg create command as a tuple, a local Borg path, a working directory, a dict of
@@ -290,6 +290,7 @@ def collect_special_file_paths(
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
paths = tuple(
@@ -323,6 +324,7 @@ def create_archive(
dry_run,
repository_path,
config,
config_paths,
local_borg_version,
global_arguments,
local_path='borg',
@@ -334,8 +336,9 @@ def create_archive(
stream_processes=None,
):
'''
Given vebosity/dry-run flags, a local or remote repository path, and a configuration dict,
create a Borg archive and return Borg's JSON output (if any).
Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
sequence of loaded configuration paths, the local Borg version, and global arguments as an
argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
create command while also triggering the given processes to produce output.
@@ -351,11 +354,7 @@ def create_archive(
expand_directories(
tuple(config.get('source_directories', ()))
+ borgmatic_source_directories
+ tuple(
global_arguments.used_config_paths
if config.get('store_config_files', True)
else ()
)
+ tuple(config_paths if config.get('store_config_files', True) else ())
)
),
additional_directory_devices=map_directories_to_devices(
@@ -471,6 +470,7 @@ def create_archive(
logger.debug(f'{repository_path}: Collecting special file paths')
special_file_paths = collect_special_file_paths(
create_flags + create_positional_arguments,
config,
local_path,
working_directory,
borg_environment,
@@ -496,6 +496,7 @@ def create_archive(
+ (('--progress',) if progress else ())
+ (('--json',) if json else ())
)
borg_exit_codes = config.get('borg_exit_codes')
if stream_processes:
return execute_command_with_processes(
@@ -503,9 +504,10 @@ def create_archive(
stream_processes,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
elif output_log_level is None:
return execute_command_and_capture_output(
@@ -513,13 +515,15 @@ def create_archive(
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
create_flags + create_positional_arguments,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+4
View File
@@ -50,4 +50,8 @@ def make_environment(config):
if value is not None:
environment[environment_variable_name] = 'YES' if value else 'NO'
# On Borg 1.4.0a1+, take advantage of more specific exit codes. No effect on
# older versions of Borg.
environment['BORG_EXIT_CODES'] = 'modern'
return environment
+2 -1
View File
@@ -65,6 +65,7 @@ def export_key(
full_command,
output_file=output_file,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+2 -1
View File
@@ -69,6 +69,7 @@ def export_tar_archive(
full_command,
output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
output_log_level=output_log_level,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+15 -2
View File
@@ -57,7 +57,11 @@ def extract_last_archive_dry_run(
)
execute_command(
full_extract_command, working_directory=None, extra_environment=borg_environment
full_extract_command,
working_directory=None,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
@@ -127,6 +131,7 @@ def extract_archive(
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The progress output isn't compatible with captured and logged output, as progress messes with
# the terminal directly.
@@ -136,6 +141,8 @@ def extract_archive(
output_file=DO_NOT_CAPTURE,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
return None
@@ -146,10 +153,16 @@ def extract_archive(
working_directory=destination_path,
run_to_completion=False,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command(
full_command, working_directory=destination_path, extra_environment=borg_environment
full_command,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+4 -1
View File
@@ -95,11 +95,13 @@ def display_archives_info(
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_info = execute_command_and_capture_output(
json_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if info_arguments.json:
@@ -110,6 +112,7 @@ def display_archives_info(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+5 -1
View File
@@ -124,6 +124,7 @@ def capture_archive_listing(
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
.strip('\n')
.split('\n')
@@ -189,6 +190,7 @@ def list_archive(
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# If there are any paths to find (and there's not a single archive already selected), start by
# getting a list of archives to search.
@@ -219,6 +221,7 @@ def list_archive(
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
.strip('\n')
.split('\n')
@@ -251,6 +254,7 @@ def list_archive(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+8 -2
View File
@@ -65,9 +65,15 @@ def mount_archive(
execute_command(
full_command,
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
return
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+2 -1
View File
@@ -94,6 +94,7 @@ def prune_archives(
execute_command(
full_command,
output_log_level=output_log_level,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+26 -11
View File
@@ -1,4 +1,5 @@
import argparse
import json
import logging
import subprocess
@@ -8,7 +9,7 @@ from borgmatic.execute import DO_NOT_CAPTURE, execute_command
logger = logging.getLogger(__name__)
RINFO_REPOSITORY_NOT_FOUND_EXIT_CODE = 2
RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES = {2, 13}
def create_repository(
@@ -31,21 +32,34 @@ def create_repository(
version, a Borg encryption mode, the path to another repo whose key material should be reused,
whether the repository should be append-only, and the storage quota to use, create the
repository. If the repository already exists, then log and skip creation.
Raise ValueError if the requested encryption mode does not match that of the repository.
Raise json.decoder.JSONDecodeError if the "borg info" JSON outputcannot be decoded.
Raise subprocess.CalledProcessError if "borg info" returns an error exit code.
'''
try:
rinfo.display_repository_info(
repository_path,
config,
local_borg_version,
argparse.Namespace(json=True),
global_arguments,
local_path,
remote_path,
info_data = json.loads(
rinfo.display_repository_info(
repository_path,
config,
local_borg_version,
argparse.Namespace(json=True),
global_arguments,
local_path,
remote_path,
)
)
repository_encryption_mode = info_data.get('encryption', {}).get('mode')
if repository_encryption_mode != encryption_mode:
raise ValueError(
f'Requested encryption mode "{encryption_mode}" does not match existing repository encryption mode "{repository_encryption_mode}"'
)
logger.info(f'{repository_path}: Repository already exists. Skipping creation.')
return
except subprocess.CalledProcessError as error:
if error.returncode != RINFO_REPOSITORY_NOT_FOUND_EXIT_CODE:
if error.returncode not in RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES:
raise
lock_wait = config.get('lock_wait')
@@ -81,6 +95,7 @@ def create_repository(
execute_command(
rcreate_command,
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+4 -1
View File
@@ -49,17 +49,20 @@ def display_repository_info(
)
extra_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
if rinfo_arguments.json:
return execute_command_and_capture_output(
full_command,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
full_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+8 -2
View File
@@ -45,6 +45,7 @@ def resolve_archive_name(
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:
latest_archive = output.strip().splitlines()[-1]
@@ -147,9 +148,13 @@ def list_repository(
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_listing = execute_command_and_capture_output(
json_command, extra_environment=borg_environment, borg_local_path=local_path
json_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if rlist_arguments.json:
@@ -160,6 +165,7 @@ def list_repository(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+1
View File
@@ -56,5 +56,6 @@ def transfer_archives(
output_log_level=logging.ANSWER,
output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
extra_environment=environment.make_environment(config),
)
+4 -2
View File
@@ -5,7 +5,7 @@ from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def unmount_archive(mount_point, local_path='borg'):
def unmount_archive(config, mount_point, local_path='borg'):
'''
Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
from the mount point.
@@ -17,4 +17,6 @@ def unmount_archive(mount_point, local_path='borg'):
+ (mount_point,)
)
execute_command(full_command)
execute_command(
full_command, borg_local_path=local_path, borg_exit_codes=config.get('borg_exit_codes')
)
+1
View File
@@ -22,6 +22,7 @@ def local_borg_version(config, local_path='borg'):
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:
+1 -1
View File
@@ -302,7 +302,7 @@ def make_parsers():
metavar='OPTION.SUBOPTION=VALUE',
dest='overrides',
action='append',
help='Configuration file option to override with specified value, can specify flag multiple times',
help='Configuration file option to override with specified value, see documentation for overriding list or key/value options, can specify flag multiple times',
)
global_group.add_argument(
'--no-environment-interpolation',
+29 -24
View File
@@ -58,11 +58,11 @@ def get_skip_actions(config, arguments):
return skip_actions
def run_configuration(config_filename, config, arguments):
def run_configuration(config_filename, config, config_paths, arguments):
'''
Given a config filename, the corresponding parsed config dict, and command-line arguments as a
dict from subparser name to a namespace of parsed arguments, execute the defined create, prune,
compact, check, and/or other actions.
Given a config filename, the corresponding parsed config dict, a sequence of loaded
configuration paths, and command-line arguments as a dict from subparser name to a namespace of
parsed arguments, execute the defined create, prune, compact, check, and/or other actions.
Yield a combination of:
@@ -144,6 +144,7 @@ def run_configuration(config_filename, config, arguments):
arguments=arguments,
config_filename=config_filename,
config=config,
config_paths=config_paths,
local_path=local_path,
remote_path=remote_path,
local_borg_version=local_borg_version,
@@ -179,7 +180,7 @@ def run_configuration(config_filename, config, arguments):
try:
if monitoring_hooks_are_activated:
# send logs irrespective of error
# Send logs irrespective of error.
dispatch.call_hooks(
'ping_monitor',
config,
@@ -264,6 +265,7 @@ def run_actions(
arguments,
config_filename,
config,
config_paths,
local_path,
remote_path,
local_borg_version,
@@ -271,9 +273,9 @@ def run_actions(
):
'''
Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
filename, several different configuration dicts, local and remote paths to Borg, a local Borg
version string, and a repository name, run all actions from the command-line arguments on the
given repository.
filename, a configuration dict, a sequence of loaded configuration paths, local and remote paths
to Borg, a local Borg version string, and a repository name, run all actions from the
command-line arguments on the given repository.
Yield JSON output strings from executing any actions that produce JSON.
@@ -328,6 +330,7 @@ def run_actions(
config_filename,
repository,
config,
config_paths,
hook_context,
local_borg_version,
action_arguments,
@@ -502,13 +505,15 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
'''
Given a sequence of configuration filenames, load and validate each configuration file. Return
the results as a tuple of: dict of configuration filename to corresponding parsed configuration,
and sequence of logging.LogRecord instances containing any parse errors.
a sequence of paths for all loaded configuration files (including includes), and a sequence of
logging.LogRecord instances containing any parse errors.
Log records are returned here instead of being logged directly because logging isn't yet
initialized at this point!
'''
# Dict mapping from config filename to corresponding parsed config dict.
configs = collections.OrderedDict()
config_paths = set()
logs = []
# Parse and load each configuration file.
@@ -525,9 +530,10 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
]
)
try:
configs[config_filename], parse_logs = validate.parse_configuration(
configs[config_filename], paths, parse_logs = validate.parse_configuration(
config_filename, validate.schema_filename(), overrides, resolve_env
)
config_paths.update(paths)
logs.extend(parse_logs)
except PermissionError:
logs.extend(
@@ -557,7 +563,7 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
]
)
return (configs, logs)
return (configs, sorted(config_paths), logs)
def log_record(suppress_log=False, **kwargs):
@@ -724,12 +730,12 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
return
def collect_configuration_run_summary_logs(configs, arguments):
def collect_configuration_run_summary_logs(configs, config_paths, arguments):
'''
Given a dict of configuration filename to corresponding parsed configuration and parsed
command-line arguments as a dict from subparser name to a parsed namespace of arguments, run
each configuration file and yield a series of logging.LogRecord instances containing summary
information about each run.
Given a dict of configuration filename to corresponding parsed configuration, a sequence of
loaded configuration paths, and parsed command-line arguments as a dict from subparser name to a
parsed namespace of arguments, run each configuration file and yield a series of
logging.LogRecord instances containing summary information about each run.
As a side effect of running through these configuration files, output their JSON results, if
any, to stdout.
@@ -774,7 +780,7 @@ def collect_configuration_run_summary_logs(configs, arguments):
# Execute the actions corresponding to each configuration file.
json_results = []
for config_filename, config in configs.items():
results = list(run_configuration(config_filename, config, arguments))
results = list(run_configuration(config_filename, config, config_paths, arguments))
error_logs = tuple(result for result in results if isinstance(result, logging.LogRecord))
if error_logs:
@@ -795,6 +801,7 @@ def collect_configuration_run_summary_logs(configs, arguments):
logger.info(f"Unmounting mount point {arguments['umount'].mount_point}")
try:
borg_umount.unmount_archive(
config,
mount_point=arguments['umount'].mount_point,
local_path=get_local_path(configs),
)
@@ -855,8 +862,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
sys.exit(0)
config_filenames = tuple(collect.collect_config_filenames(global_arguments.config_paths))
global_arguments.used_config_paths = list(config_filenames)
configs, parse_logs = load_configurations(
configs, config_paths, parse_logs = load_configurations(
config_filenames, global_arguments.overrides, global_arguments.resolve_env
)
configuration_parse_errors = (
@@ -866,10 +872,8 @@ def main(extra_summary_logs=[]): # pragma: no cover
any_json_flags = any(
getattr(sub_arguments, 'json', False) for sub_arguments in arguments.values()
)
colorama.init(
autoreset=True,
strip=not should_do_markup(global_arguments.no_color or any_json_flags, configs),
)
color_enabled = should_do_markup(global_arguments.no_color or any_json_flags, configs)
colorama.init(autoreset=color_enabled, strip=not color_enabled)
try:
configure_logging(
verbosity_to_log_level(global_arguments.verbosity),
@@ -878,6 +882,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
verbosity_to_log_level(global_arguments.monitoring_verbosity),
global_arguments.log_file,
global_arguments.log_file_format,
color_enabled=color_enabled,
)
except (FileNotFoundError, PermissionError) as error:
configure_logging(logging.CRITICAL)
@@ -893,7 +898,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
configs, arguments, configuration_parse_errors
)
)
or list(collect_configuration_run_summary_logs(configs, arguments))
or list(collect_configuration_run_summary_logs(configs, config_paths, arguments))
)
)
summary_logs_max_level = max(log.levelno for log in summary_logs)
+18 -4
View File
@@ -1,3 +1,6 @@
import shlex
def coerce_scalar(value):
'''
Given a configuration value, coerce it to an integer or a boolean as appropriate and return the
@@ -16,7 +19,7 @@ def coerce_scalar(value):
return value
def apply_constants(value, constants):
def apply_constants(value, constants, shell_escape=False):
'''
Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
replace any configuration string values of the form "{constant}" (or containing it) with the
@@ -26,6 +29,8 @@ def apply_constants(value, constants):
For instance, if a configuration value contains "{foo}", replace it with the value of the "foo"
key found within the configuration's "constants".
If shell escape is True, then escape the constant's value before applying it.
Return the configuration value and modify the original.
'''
if not value or not constants:
@@ -33,15 +38,24 @@ def apply_constants(value, constants):
if isinstance(value, str):
for constant_name, constant_value in constants.items():
value = value.replace('{' + constant_name + '}', str(constant_value))
value = value.replace(
'{' + constant_name + '}',
shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
)
# Support constants within non-string scalars by coercing the value to its appropriate type.
value = coerce_scalar(value)
elif isinstance(value, list):
for index, list_value in enumerate(value):
value[index] = apply_constants(list_value, constants)
value[index] = apply_constants(list_value, constants, shell_escape)
elif isinstance(value, dict):
for option_name, option_value in value.items():
value[option_name] = apply_constants(option_value, constants)
shell_escape = (
shell_escape
or option_name.startswith('before_')
or option_name.startswith('after_')
or option_name == 'on_error'
)
value[option_name] = apply_constants(option_value, constants, shell_escape)
return value
+1 -2
View File
@@ -225,8 +225,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
favoring values from the source when there are collisions.
The purpose of this is to upgrade configuration files from old versions of borgmatic by adding
new
configuration keys and comments.
new configuration keys and comments.
'''
if not source_config:
return destination_config
+39 -25
View File
@@ -9,18 +9,18 @@ import ruamel.yaml
logger = logging.getLogger(__name__)
def probe_and_include_file(filename, include_directories):
def probe_and_include_file(filename, include_directories, config_paths):
'''
Given a filename to include and a list of include directories to search for matching files,
probe for the file, load it, and return the loaded configuration as a data structure of nested
dicts, lists, etc.
Given a filename to include, a list of include directories to search for matching files, and a
set of configuration paths, probe for the file, load it, and return the loaded configuration as
a data structure of nested dicts, lists, etc. Add the filename to the given configuration paths.
Raise FileNotFoundError if the included file was not found.
'''
expanded_filename = os.path.expanduser(filename)
if os.path.isabs(expanded_filename):
return load_configuration(expanded_filename)
return load_configuration(expanded_filename, config_paths)
candidate_filenames = {
os.path.join(directory, expanded_filename) for directory in include_directories
@@ -28,32 +28,33 @@ def probe_and_include_file(filename, include_directories):
for candidate_filename in candidate_filenames:
if os.path.exists(candidate_filename):
return load_configuration(candidate_filename)
return load_configuration(candidate_filename, config_paths)
raise FileNotFoundError(
f'Could not find include {filename} at {" or ".join(candidate_filenames)}'
)
def include_configuration(loader, filename_node, include_directory):
def include_configuration(loader, filename_node, include_directory, config_paths):
'''
Given a ruamel.yaml.loader.Loader, a ruamel.yaml.nodes.ScalarNode containing the included
filename (or a list containing multiple such filenames), and an include directory path to search
for matching files, load the given YAML filenames (ignoring the given loader so we can use our
own) and return their contents as data structure of nested dicts, lists, etc. If the given
filename (or a list containing multiple such filenames), an include directory path to search for
matching files, and a set of configuration paths, load the given YAML filenames (ignoring the
given loader so we can use our own) and return their contents as data structure of nested dicts,
lists, etc. Add the names of included files to the given configuration paths. If the given
filename node's value is a scalar string, then the return value will be a single value. But if
the given node value is a list, then the return value will be a list of values, one per loaded
configuration file.
If a filename is relative, probe for it within 1. the current working directory and 2. the given
include directory.
If a filename is relative, probe for it within: 1. the current working directory and 2. the
given include directory.
Raise FileNotFoundError if an included file was not found.
'''
include_directories = [os.getcwd(), os.path.abspath(include_directory)]
if isinstance(filename_node.value, str):
return probe_and_include_file(filename_node.value, include_directories)
return probe_and_include_file(filename_node.value, include_directories, config_paths)
if (
isinstance(filename_node.value, list)
@@ -63,7 +64,7 @@ def include_configuration(loader, filename_node, include_directory):
# Reversing the values ensures the correct ordering if these includes are subsequently
# merged together.
return [
probe_and_include_file(node.value, include_directories)
probe_and_include_file(node.value, include_directories, config_paths)
for node in reversed(filename_node.value)
]
@@ -109,11 +110,17 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
separate YAML configuration files. Example syntax: `option: !include common.yaml`
'''
def __init__(self, preserve_quotes=None, loader=None, include_directory=None):
def __init__(
self, preserve_quotes=None, loader=None, include_directory=None, config_paths=None
):
super(Include_constructor, self).__init__(preserve_quotes, loader)
self.add_constructor(
'!include',
functools.partial(include_configuration, include_directory=include_directory),
functools.partial(
include_configuration,
include_directory=include_directory,
config_paths=config_paths,
),
)
# These are catch-all error handlers for tags that don't get applied and removed by
@@ -155,26 +162,33 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
node.value = deep_merge_nodes(node.value)
def load_configuration(filename):
def load_configuration(filename, config_paths=None):
'''
Load the given configuration file and return its contents as a data structure of nested dicts
and lists.
and lists. Add the filename to the given configuration paths set, and also add any included
configuration filenames.
Raise ruamel.yaml.error.YAMLError if something goes wrong parsing the YAML, or RecursionError
if there are too many recursive includes.
'''
if config_paths is None:
config_paths = set()
# Use an embedded derived class for the include constructor so as to capture the filename
# value. (functools.partial doesn't work for this use case because yaml.Constructor has to be
# an actual class.)
class Include_constructor_with_include_directory(Include_constructor):
# Use an embedded derived class for the include constructor so as to capture the include
# directory and configuration paths values. (functools.partial doesn't work for this use case
# because yaml.Constructor has to be an actual class.)
class Include_constructor_with_extras(Include_constructor):
def __init__(self, preserve_quotes=None, loader=None):
super(Include_constructor_with_include_directory, self).__init__(
preserve_quotes, loader, include_directory=os.path.dirname(filename)
super(Include_constructor_with_extras, self).__init__(
preserve_quotes,
loader,
include_directory=os.path.dirname(filename),
config_paths=config_paths,
)
yaml = ruamel.yaml.YAML(typ='safe')
yaml.Constructor = Include_constructor_with_include_directory
yaml.Constructor = Include_constructor_with_extras
config_paths.add(filename)
with open(filename) as file:
return yaml.load(file.read())
+11 -1
View File
@@ -13,6 +13,11 @@ def set_values(config, keys, value):
first_key = keys[0]
if len(keys) == 1:
if isinstance(config, list):
raise ValueError(
'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)'
)
config[first_key] = value
return
@@ -98,7 +103,7 @@ def parse_overrides(raw_overrides, schema):
for raw_override in raw_overrides:
try:
raw_keys, value = raw_override.split('=', 1)
keys = strip_section_names(tuple(raw_keys.split('.')))
keys = tuple(raw_keys.split('.'))
option_type = type_for_option(schema, keys)
parsed_overrides.append(
@@ -122,8 +127,13 @@ def apply_overrides(config, schema, raw_overrides):
Given a configuration dict, a corresponding configuration schema dict, and a sequence of
configuration file override strings in the form of "option.suboption=value", parse each override
and set it into the configuration dict.
Set the overrides into the configuration both with and without deprecated section names (if
used), so that the overrides work regardless of whether the configuration is also using
deprecated section names.
'''
overrides = parse_overrides(raw_overrides, schema)
for keys, value in overrides:
set_values(config, keys, value)
set_values(config, strip_section_names(keys), value)
+100 -5
View File
@@ -341,6 +341,37 @@ properties:
Path for Borg encryption key files. Defaults to
$borg_base_directory/.config/borg/keys
example: /path/to/base/config/keys
borg_exit_codes:
type: array
items:
type: object
required: ['code', 'treat_as']
additionalProperties: false
properties:
code:
type: integer
not: {enum: [0]}
description: |
The exit code for an existing Borg warning or error.
example: 100
treat_as:
type: string
enum: ['error', 'warning']
description: |
Whether to consider the exit code as an error or as a
warning in borgmatic.
example: error
description: |
A list of Borg exit codes that should be elevated to errors or
squashed to warnings as indicated. By default, Borg error exit codes
(2 to 99) are treated as errors while warning exit codes (1 and
100+) are treated as warnings. Exit codes other than 1 and 2 are
only present in Borg 1.4.0+.
example:
- code: 13
treat_as: warning
- code: 100
treat_as: error
umask:
type: integer
description: |
@@ -940,6 +971,20 @@ properties:
a password will only work if MariaDB is configured to
trust the configured username without a password.
example: trustsome1
mariadb_dump_command:
type: string
description: |
Command to use instead of "mariadb-dump". This can be
used to run a specific mariadb_dump version (e.g., one
inside a running container). Defaults to "mariadb-dump".
example: docker exec mariadb_container mariadb-dump
mariadb_command:
type: string
description: |
Command to run instead of "mariadb". This can be used to
run a specific mariadb version (e.g., one inside a
running container). Defaults to "mariadb".
example: docker exec mariadb_container mariadb
restore_password:
type: string
description: |
@@ -1056,6 +1101,20 @@ properties:
Password with which to connect to the restore database.
Defaults to the "password" option.
example: trustsome1
mysql_dump_command:
type: string
description: |
Command to use instead of "mysqldump". This can be used
to run a specific mysql_dump version (e.g., one inside a
running container). Defaults to "mysqldump".
example: docker exec mysql_container mysqldump
mysql_command:
type: string
description: |
Command to run instead of "mysql". This can be used to
run a specific mysql version (e.g., one inside a running
container). Defaults to "mysql".
example: docker exec mysql_container mysql
format:
type: string
enum: ['sql']
@@ -1259,6 +1318,12 @@ properties:
description: |
The password used for authentication.
example: fakepassword
access_token:
type: string
description: |
An ntfy access token to authenticate with instead of
username/password.
example: tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2
start:
type: object
properties:
@@ -1373,6 +1438,19 @@ properties:
label: kodi
- url: "line://Token@User"
label: line
send_logs:
type: boolean
description: |
Send borgmatic logs to Apprise services as part the
"finish", "fail", and "log" states. Defaults to true.
example: false
logs_size_limit:
type: integer
description: |
Number of bytes of borgmatic logs to send to Apprise
services. Set to 0 to send all logs and disable this
truncation. Defaults to 1500.
example: 100000
start:
type: object
required: ['body']
@@ -1418,6 +1496,21 @@ properties:
description: |
Specify the message body.
example: Your backups have failed.
log:
type: object
required: ['body']
properties:
title:
type: string
description: |
Specify the message title. If left unspecified, no
title is sent.
example: Ping!
body:
type: string
description: |
Specify the message body.
example: Here is some info about your backups.
states:
type: array
items:
@@ -1426,13 +1519,15 @@ properties:
- start
- finish
- fail
- log
uniqueItems: true
description: |
List of one or more monitoring states to ping for: "start",
"finish", and/or "fail". Defaults to pinging for failure
only. For each selected state, corresponding configuration
for the message title and body should be given. If any is
left unspecified, a generic message is emitted instead.
List of one or more monitoring states to ping for:
"start", "finish", "fail", and/or "log". Defaults to
pinging for failure only. For each selected state,
corresponding configuration for the message title and body
should be given. If any is left unspecified, a generic
message is emitted instead.
example:
- start
- finish
+9 -10
View File
@@ -97,14 +97,16 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
'checks': ['repository', 'archives'],
}
Also return a sequence of logging.LogRecord instances containing any warnings about the
configuration.
Also return a set of loaded configuration paths and a sequence of logging.LogRecord instances
containing any warnings about the configuration.
Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
have permissions to read the file, or Validation_error if the config does not match the schema.
'''
config_paths = set()
try:
config = load.load_configuration(config_filename)
config = load.load_configuration(config_filename, config_paths)
schema = load.load_configuration(schema_filename)
except (ruamel.yaml.error.YAMLError, RecursionError) as error:
raise Validation_error(config_filename, (str(error),))
@@ -130,7 +132,7 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
apply_logical_validation(config_filename, config)
return config, logs
return config, config_paths, logs
def normalize_repository_path(repository):
@@ -165,11 +167,10 @@ def repositories_match(first, second):
def guard_configuration_contains_repository(repository, configurations):
'''
Given a repository path and a dict mapping from config filename to corresponding parsed config
dict, ensure that the repository is declared exactly once in all of the configurations. If no
dict, ensure that the repository is declared at least once in all of the configurations. If no
repository is given, skip this check.
Raise ValueError if the repository is not found in a configuration, or is declared multiple
times.
Raise ValueError if the repository is not found in any configurations.
'''
if not repository:
return
@@ -184,9 +185,7 @@ def guard_configuration_contains_repository(repository, configurations):
)
if count == 0:
raise ValueError(f'Repository {repository} not found in configuration files')
if count > 1:
raise ValueError(f'Repository {repository} found in multiple configuration files')
raise ValueError(f'Repository "{repository}" not found in configuration files')
def guard_single_repository_selected(repository, configurations):
+85 -22
View File
@@ -1,4 +1,5 @@
import collections
import enum
import logging
import os
import select
@@ -8,22 +9,61 @@ logger = logging.getLogger(__name__)
ERROR_OUTPUT_MAX_LINE_COUNT = 25
BORG_ERROR_EXIT_CODE = 2
BORG_ERROR_EXIT_CODE_START = 2
BORG_ERROR_EXIT_CODE_END = 99
def exit_code_indicates_error(command, exit_code, borg_local_path=None):
class Exit_status(enum.Enum):
STILL_RUNNING = 1
SUCCESS = 2
WARNING = 3
ERROR = 4
def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
'''
Return True if the given exit code from running a command corresponds to an error. If a Borg
local path is given and matches the process' command, then treat exit code 1 as a warning
instead of an error.
Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
exit code. If a Borg local path is given and matches the process' command, then interpret the
exit code based on Borg's documented exit code semantics. And if Borg exit codes are given as a
sequence of exit code configuration dicts, then take those configured preferences into account.
'''
if exit_code is None:
return False
return Exit_status.STILL_RUNNING
if exit_code == 0:
return Exit_status.SUCCESS
if borg_local_path and command[0] == borg_local_path:
return bool(exit_code < 0 or exit_code >= BORG_ERROR_EXIT_CODE)
# First try looking for the exit code in the borg_exit_codes configuration.
for entry in borg_exit_codes or ():
if entry.get('code') == exit_code:
treat_as = entry.get('treat_as')
return bool(exit_code != 0)
if treat_as == 'error':
logger.error(
f'Treating exit code {exit_code} as an error, as per configuration'
)
return Exit_status.ERROR
elif treat_as == 'warning':
logger.warning(
f'Treating exit code {exit_code} as a warning, as per configuration'
)
return Exit_status.WARNING
# If the exit code doesn't have explicit configuration, then fall back to the default Borg
# behavior.
return (
Exit_status.ERROR
if (
exit_code < 0
or (
exit_code >= BORG_ERROR_EXIT_CODE_START
and exit_code <= BORG_ERROR_EXIT_CODE_END
)
)
else Exit_status.WARNING
)
return Exit_status.ERROR
def command_for_process(process):
@@ -60,7 +100,7 @@ def append_last_lines(last_lines, captured_output, line, output_log_level):
logger.log(output_log_level, line)
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):
'''
Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
process with the requested log level. Additionally, raise a CalledProcessError if a process
@@ -68,7 +108,8 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
path).
If output log level is None, then instead of logging, capture output for each process and return
it as a dict from the process to its output.
it as a dict from the process to its output. Use the given Borg local path and exit code
configuration to decide what's an error and what's a warning.
For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
stdouts are given to exclude, then for any matching processes, log from their stderr instead.
@@ -132,11 +173,13 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
if exit_code is None:
still_running = True
command = process.args.split(' ') if isinstance(process.args, str) else process.args
continue
command = process.args.split(' ') if isinstance(process.args, str) else process.args
exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
# If any process errors, then raise accordingly.
if exit_code_indicates_error(command, exit_code, borg_local_path):
if exit_status in (Exit_status.ERROR, Exit_status.WARNING):
# If an error occurs, include its output in the raised exception so that we don't
# inadvertently hide error output.
output_buffer = output_buffer_for_process(process, exclude_stdouts)
@@ -162,9 +205,13 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
other_process.stdout.read(0)
other_process.kill()
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
if exit_status == Exit_status.ERROR:
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
still_running = False
break
if captured_outputs:
return {
@@ -199,6 +246,7 @@ def execute_command(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
run_to_completion=True,
):
'''
@@ -209,8 +257,9 @@ def execute_command(
augment the current environment, and pass the result into the command. If a working directory is
given, use that as the present working directory when running the command. If a Borg local path
is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
instead of an error. If run to completion is False, then return the process for the command
without executing it to completion.
instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
dicts, then use that configuration to decide what's an error and what's a warning. If run to
completion is False, then return the process for the command without executing it to completion.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
@@ -232,7 +281,11 @@ def execute_command(
return process
log_outputs(
(process,), (input_file, output_file), output_log_level, borg_local_path=borg_local_path
(process,),
(input_file, output_file),
output_log_level,
borg_local_path,
borg_exit_codes,
)
@@ -243,6 +296,7 @@ def execute_command_and_capture_output(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings), capturing and returning its
@@ -251,7 +305,9 @@ def execute_command_and_capture_output(
given, then use it to augment the current environment, and pass the result into the command. If
a working directory is given, use that as the present working directory when running the
command. If a Borg local path is given, and the command matches it (regardless of arguments),
treat exit code 1 as a warning instead of an error.
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
@@ -268,7 +324,10 @@ def execute_command_and_capture_output(
cwd=working_directory,
)
except subprocess.CalledProcessError as error:
if exit_code_indicates_error(command, error.returncode, borg_local_path):
if (
interpret_exit_code(command, error.returncode, borg_local_path, borg_exit_codes)
== Exit_status.ERROR
):
raise
output = error.output
@@ -285,6 +344,7 @@ def execute_command_with_processes(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -299,7 +359,9 @@ def execute_command_with_processes(
use it to augment the current environment, and pass the result into the command. If a working
directory is given, use that as the present working directory when running the command. If a
Borg local path is given, then for any matching command or process (regardless of arguments),
treat exit code 1 as a warning instead of an error.
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
upstream process.
@@ -334,7 +396,8 @@ def execute_command_with_processes(
tuple(processes) + (command_process,),
(input_file, output_file),
output_log_level,
borg_local_path=borg_local_path,
borg_local_path,
borg_exit_codes,
)
if output_log_level is None:
+41 -11
View File
@@ -1,16 +1,36 @@
import logging
import operator
import borgmatic.hooks.logs
import borgmatic.hooks.monitor
logger = logging.getLogger(__name__)
def initialize_monitor(
ping_url, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
DEFAULT_LOGS_SIZE_LIMIT_BYTES = 100000
HANDLER_IDENTIFIER = 'apprise'
def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
No initialization is necessary for this monitor.
Add a handler to the root logger that stores in memory the most recent logs emitted. That way,
we can send them all to an Apprise notification service upon a finish or failure state. But skip
this if the "send_logs" option is false.
'''
pass
if hook_config.get('send_logs') is False:
return
logs_size_limit = max(
hook_config.get('logs_size_limit', DEFAULT_LOGS_SIZE_LIMIT_BYTES)
- len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
0,
)
borgmatic.hooks.logs.add_handler(
borgmatic.hooks.logs.Forgetful_buffering_handler(
HANDLER_IDENTIFIER, logs_size_limit, monitoring_log_level
)
)
def ping_monitor(hook_config, config, config_filename, state, monitoring_log_level, dry_run):
@@ -59,9 +79,20 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
if dry_run:
return
body = state_config.get('body')
if state in (
borgmatic.hooks.monitor.State.FINISH,
borgmatic.hooks.monitor.State.FAIL,
borgmatic.hooks.monitor.State.LOG,
):
formatted_logs = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
if formatted_logs:
body += f'\n\n{formatted_logs}'
result = apprise_object.notify(
title=state_config.get('title', ''),
body=state_config.get('body'),
body=body,
body_format=NotifyFormat.TEXT,
notify_type=state_to_notify_type[state.name.lower()],
)
@@ -70,10 +101,9 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
logger.warning(f'{config_filename}: Error sending some Apprise notifications')
def destroy_monitor(
ping_url_or_uuid, config, config_filename, monitoring_log_level, dry_run
): # pragma: no cover
def destroy_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
'''
No destruction is necessary for this monitor.
Remove the monitor handler that was added to the root logger. This prevents the handler from
getting reused by other instances of this monitor.
'''
pass
borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)
+2 -1
View File
@@ -1,6 +1,7 @@
import logging
import os
import re
import shlex
from borgmatic import execute
@@ -16,7 +17,7 @@ def interpolate_context(config_filename, hook_description, command, context):
names/values, interpolate the values by "{name}" into the command and return the result.
'''
for name, value in context.items():
command = command.replace(f'{{{name}}}', str(value))
command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
for unsupported_variable in re.findall(r'{\w+}', command):
logger.warning(
+10 -64
View File
@@ -2,6 +2,7 @@ import logging
import requests
import borgmatic.hooks.logs
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
@@ -13,61 +14,8 @@ MONITOR_STATE_TO_HEALTHCHECKS = {
monitor.State.LOG: 'log',
}
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
DEFAULT_PING_BODY_LIMIT_BYTES = 100000
class Forgetful_buffering_handler(logging.Handler):
'''
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached. But if the given byte capacity is zero,
don't throw away any messages.
'''
def __init__(self, byte_capacity, log_level):
super().__init__()
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
self.setLevel(log_level)
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
if not self.byte_capacity:
return
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def format_buffered_logs_for_payload():
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler)
)
except StopIteration:
# No handler means no payload.
return ''
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
DEFAULT_PING_BODY_LIMIT_BYTES = 1500
HANDLER_IDENTIFIER = 'healthchecks'
def initialize_monitor(hook_config, config, config_filename, monitoring_log_level, dry_run):
@@ -81,12 +29,14 @@ def initialize_monitor(hook_config, config, config_filename, monitoring_log_leve
ping_body_limit = max(
hook_config.get('ping_body_limit', DEFAULT_PING_BODY_LIMIT_BYTES)
- len(PAYLOAD_TRUNCATION_INDICATOR),
- len(borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
0,
)
logging.getLogger().addHandler(
Forgetful_buffering_handler(ping_body_limit, monitoring_log_level)
borgmatic.hooks.logs.add_handler(
borgmatic.hooks.logs.Forgetful_buffering_handler(
HANDLER_IDENTIFIER, ping_body_limit, monitoring_log_level
)
)
@@ -117,7 +67,7 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
logger.debug(f'{config_filename}: Using Healthchecks ping URL {ping_url}')
if state in (monitor.State.FINISH, monitor.State.FAIL, monitor.State.LOG):
payload = format_buffered_logs_for_payload()
payload = borgmatic.hooks.logs.format_buffered_logs_for_payload(HANDLER_IDENTIFIER)
else:
payload = ''
@@ -138,8 +88,4 @@ def destroy_monitor(hook_config, config, config_filename, monitoring_log_level,
Remove the monitor handler that was added to the root logger. This prevents the handler from
getting reused by other instances of this monitor.
'''
logger = logging.getLogger()
for handler in tuple(logger.handlers):
if isinstance(handler, Forgetful_buffering_handler):
logger.removeHandler(handler)
borgmatic.hooks.logs.remove_handler(HANDLER_IDENTIFIER)
+91
View File
@@ -0,0 +1,91 @@
import logging
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
class Forgetful_buffering_handler(logging.Handler):
'''
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached. But if the given byte capacity is zero,
don't throw away any messages.
The given identifier is used to distinguish the instance of this handler used for one monitoring
hook from those instances used for other monitoring hooks.
'''
def __init__(self, identifier, byte_capacity, log_level):
super().__init__()
self.identifier = identifier
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
self.setLevel(log_level)
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
if not self.byte_capacity:
return
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def add_handler(handler): # pragma: no cover
'''
Add the given handler to the global logger.
'''
logging.getLogger().addHandler(handler)
def get_handler(identifier):
'''
Given the identifier for an existing Forgetful_buffering_handler instance, return the handler.
Raise ValueError if the handler isn't found.
'''
try:
return next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler) and handler.identifier == identifier
)
except StopIteration:
raise ValueError(f'A buffering handler for {identifier} was not found')
def format_buffered_logs_for_payload(identifier):
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = get_handler(identifier)
except ValueError:
# No handler means no payload.
return ''
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
def remove_handler(identifier):
'''
Given the identifier for an existing Forgetful_buffering_handler instance, remove it.
'''
logger = logging.getLogger()
try:
logger.removeHandler(get_handler(identifier))
except ValueError:
pass
+15 -3
View File
@@ -1,6 +1,7 @@
import copy
import logging
import os
import shlex
from borgmatic.execute import (
execute_command,
@@ -35,8 +36,11 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
if dry_run:
return ()
mariadb_show_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mariadb_command') or 'mariadb')
)
show_command = (
('mariadb',)
mariadb_show_command
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
@@ -79,8 +83,12 @@ def execute_dump_command(
)
return None
mariadb_dump_command = tuple(
shlex.quote(part)
for part in shlex.split(database.get('mariadb_dump_command') or 'mariadb-dump')
)
dump_command = (
('mariadb-dump',)
mariadb_dump_command
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
@@ -208,8 +216,12 @@ def restore_data_source_dump(
'restore_password', data_source.get('password')
)
mariadb_restore_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('mariadb_command') or 'mariadb')
)
restore_command = (
('mariadb', '--batch')
mariadb_restore_command
+ ('--batch',)
+ (
tuple(data_source['restore_options'].split(' '))
if 'restore_options' in data_source
+14 -9
View File
@@ -1,4 +1,5 @@
import logging
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
@@ -62,19 +63,23 @@ def build_dump_command(database, dump_filename, dump_format):
return (
('mongodump',)
+ (('--out', dump_filename) if dump_format == 'directory' else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--password', database['password']) if 'password' in database else ())
+ (('--out', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (('--username', shlex.quote(database['username'])) if 'username' in database else ())
+ (('--password', shlex.quote(database['password'])) if 'password' in database else ())
+ (
('--authenticationDatabase', database['authentication_database'])
('--authenticationDatabase', shlex.quote(database['authentication_database']))
if 'authentication_database' in database
else ()
)
+ (('--db', database['name']) if not all_databases else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (('--archive', '>', dump_filename) if dump_format != 'directory' else ())
+ (('--db', shlex.quote(database['name'])) if not all_databases else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (('--archive', '>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
+14 -3
View File
@@ -1,6 +1,7 @@
import copy
import logging
import os
import shlex
from borgmatic.execute import (
execute_command,
@@ -35,8 +36,11 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
if dry_run:
return ()
mysql_show_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mysql_command') or 'mysql')
)
show_command = (
('mysql',)
mysql_show_command
+ (tuple(database['list_options'].split(' ')) if 'list_options' in database else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
@@ -79,8 +83,11 @@ def execute_dump_command(
)
return None
mysql_dump_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('mysql_dump_command') or 'mysqldump')
)
dump_command = (
('mysqldump',)
mysql_dump_command
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (('--add-drop-database',) if database.get('add_drop_database', True) else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
@@ -207,8 +214,12 @@ def restore_data_source_dump(
'restore_password', data_source.get('password')
)
mysql_restore_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('mysql_command') or 'mysql')
)
restore_command = (
('mysql', '--batch')
mysql_restore_command
+ ('--batch',)
+ (
tuple(data_source['restore_options'].split(' '))
if 'restore_options' in data_source
+9 -2
View File
@@ -50,9 +50,16 @@ def ping_monitor(hook_config, config, config_filename, state, monitoring_log_lev
username = hook_config.get('username')
password = hook_config.get('password')
access_token = hook_config.get('access_token')
auth = None
if (username and password) is not None:
if access_token is not None:
if username or password:
logger.warning(
f'{config_filename}: ntfy access_token is set but so is username/password, only using access_token'
)
auth = requests.auth.HTTPBasicAuth('', access_token)
elif (username and password) is not None:
auth = requests.auth.HTTPBasicAuth(username, password)
logger.info(f'{config_filename}: Using basic auth with user {username} for ntfy')
elif username is not None:
+40 -20
View File
@@ -25,8 +25,8 @@ def make_dump_path(config): # pragma: no cover
def make_extra_environment(database, restore_connection_params=None):
'''
Make the extra_environment dict from the given database configuration.
If restore connection params are given, this is for a restore operation.
Make the extra_environment dict from the given database configuration. If restore connection
params are given, this is for a restore operation.
'''
extra = dict()
@@ -40,7 +40,8 @@ def make_extra_environment(database, restore_connection_params=None):
except (AttributeError, KeyError):
pass
extra['PGSSLMODE'] = database.get('ssl_mode', 'disable')
if 'ssl_mode' in database:
extra['PGSSLMODE'] = database['ssl_mode']
if 'ssl_cert' in database:
extra['PGSSLCERT'] = database['ssl_cert']
if 'ssl_key' in database:
@@ -49,6 +50,7 @@ def make_extra_environment(database, restore_connection_params=None):
extra['PGSSLROOTCERT'] = database['ssl_root_cert']
if 'ssl_crl' in database:
extra['PGSSLCRL'] = database['ssl_crl']
return extra
@@ -71,9 +73,11 @@ def database_names_to_dump(database, extra_environment, log_prefix, dry_run):
if dry_run:
return ()
psql_command = shlex.split(database.get('psql_command') or 'psql')
psql_command = tuple(
shlex.quote(part) for part in shlex.split(database.get('psql_command') or 'psql')
)
list_command = (
tuple(psql_command)
psql_command
+ ('--list', '--no-password', '--no-psqlrc', '--csv', '--tuples-only')
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
@@ -125,7 +129,10 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
for database_name in dump_database_names:
dump_format = database.get('format', None if database_name == 'all' else 'custom')
default_dump_command = 'pg_dumpall' if database_name == 'all' else 'pg_dump'
dump_command = database.get('pg_dump_command') or default_dump_command
dump_command = tuple(
shlex.quote(part)
for part in shlex.split(database.get('pg_dump_command') or default_dump_command)
)
dump_filename = dump.make_data_source_dump_filename(
dump_path, database_name, database.get('hostname')
)
@@ -136,24 +143,32 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
continue
command = (
(
dump_command,
dump_command
+ (
'--no-password',
'--clean',
'--if-exists',
)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (
('--username', shlex.quote(database['username']))
if 'username' in database
else ()
)
+ (('--no-owner',) if database.get('no_owner', False) else ())
+ (('--format', dump_format) if dump_format else ())
+ (('--file', dump_filename) if dump_format == 'directory' else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (() if database_name == 'all' else (database_name,))
+ (('--format', shlex.quote(dump_format)) if dump_format else ())
+ (('--file', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (() if database_name == 'all' else (shlex.quote(database_name),))
# Use shell redirection rather than the --file flag to sidestep synchronization issues
# when pg_dump/pg_dumpall tries to write to a named pipe. But for the directory dump
# format in a particular, a named destination is required, and redirection doesn't work.
+ (('>', dump_filename) if dump_format != 'directory' else ())
+ (('>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
logger.debug(
@@ -232,9 +247,11 @@ def restore_data_source_dump(
dump_filename = dump.make_data_source_dump_filename(
make_dump_path(config), data_source['name'], data_source.get('hostname')
)
psql_command = shlex.split(data_source.get('psql_command') or 'psql')
psql_command = tuple(
shlex.quote(part) for part in shlex.split(data_source.get('psql_command') or 'psql')
)
analyze_command = (
tuple(psql_command)
psql_command
+ ('--no-password', '--no-psqlrc', '--quiet')
+ (('--host', hostname) if hostname else ())
+ (('--port', port) if port else ())
@@ -248,9 +265,12 @@ def restore_data_source_dump(
+ ('--command', 'ANALYZE')
)
use_psql_command = all_databases or data_source.get('format') == 'plain'
pg_restore_command = shlex.split(data_source.get('pg_restore_command') or 'pg_restore')
pg_restore_command = tuple(
shlex.quote(part)
for part in shlex.split(data_source.get('pg_restore_command') or 'pg_restore')
)
restore_command = (
tuple(psql_command if use_psql_command else pg_restore_command)
(psql_command if use_psql_command else pg_restore_command)
+ ('--no-password',)
+ (('--no-psqlrc',) if use_psql_command else ('--if-exists', '--exit-on-error', '--clean'))
+ (('--dbname', data_source['name']) if not all_databases else ())
+3 -2
View File
@@ -1,5 +1,6 @@
import logging
import os
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
@@ -51,10 +52,10 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
command = (
'sqlite3',
database_path,
shlex.quote(database_path),
'.dump',
'>',
dump_filename,
shlex.quote(dump_filename),
)
logger.debug(
f'{log_prefix}: Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
+10 -2
View File
@@ -41,6 +41,10 @@ def should_do_markup(no_color, configs):
if any(config.get('output', {}).get('color') is False for config in configs.values()):
return False
no_color_env = os.environ.get('NO_COLOR', None)
if no_color_env is not None:
return False
py_colors = os.environ.get('PY_COLORS', None)
if py_colors is not None:
@@ -159,10 +163,11 @@ def configure_logging(
monitoring_log_level=None,
log_file=None,
log_file_format=None,
color_enabled=True,
):
'''
Configure logging to go to both the console and (syslog or log file). Use the given log levels,
respectively.
respectively. If color is enabled, set up log formatting accordingly.
Raise FileNotFoundError or PermissionError if the log file could not be opened for writing.
'''
@@ -191,7 +196,10 @@ def configure_logging(
logging.DEBUG: console_standard_handler,
}
)
console_handler.setFormatter(Console_color_formatter())
if color_enabled:
console_handler.setFormatter(Console_color_formatter())
console_handler.setLevel(console_log_level)
handlers = [console_handler]
+4
View File
@@ -262,6 +262,10 @@ systems that you'd like supported.
## Database restoration
When you want to replace an existing database with its backed-up contents, you
can restore it with borgmatic. Note that the database must already exist;
borgmatic does not currently create a database upon restore.
To restore a database dump from an archive, use the `borgmatic restore`
action. But the first step is to figure out which archive to restore from. A
good way to do that is to use the `rlist` action:
@@ -0,0 +1,86 @@
---
title: How to customize warnings and errors
eleventyNavigation:
key: 💥 Customize warnings/errors
parent: How-to guides
order: 12
---
## When things go wrong
After Borg runs, it indicates whether it succeeded via its exit code, a
numeric ID indicating success, warning, or error. borgmatic consumes this exit
code to decide how to respond. Normally, a Borg error results in a borgmatic
error, while a Borg warning or success doesn't.
But if that default behavior isn't sufficient for your needs, you can
customize how borgmatic interprets [Borg's exit
codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
For instance, to elevate Borg warnings to errors, thereby causing borgmatic to
error on them, use the following borgmatic configuration:
```yaml
borg_exit_codes:
- exit_code: 1
treat_as: error
```
Be aware though that Borg exits with a warning code for a variety of benign
situations such as files changing while they're being read, so this example
may not meet your needs. Keep reading though for more granular exit code
configuration.
Here's an example that squashes Borg errors to warnings:
```yaml
borg_exit_codes:
- exit_code: 2
treat_as: warning
```
Be careful with this example though, because it prevents borgmatic from
erroring when Borg errors, which may not be desirable.
### More granular configuration
<span class="minilink minilink-addedin">New in Borg version 1.4</span> Borg
support for [more granular exit
codes](https://borgbackup.readthedocs.io/en/1.4-maint/usage/general.html#return-codes)
means that you can configure borgmatic to respond to specific Borg conditions.
See the full list of [Borg 1.4 error and warning exit
codes](https://borgbackup.readthedocs.io/en/1.4.0b1/internals/frontends.html#message-ids).
The `rc:` numeric value there tells you the exit code for each.
For instance, this borgmatic configuration elevates all Borg backup file
permission warnings (exit code `105`)—and only those warnings—to errors:
```yaml
borg_exit_codes:
- exit_code: 105
treat_as: error
```
The following configuration does that *and* elevates backup file not found
warnings (exit code `107`) to errors as well:
```yaml
borg_exit_codes:
- exit_code: 105
treat_as: error
- exit_code: 107
treat_as: error
```
If you don't know the exit code for a particular Borg error or warning you're
experiencing, you can usually find it in your borgmatic output when
`--verbosity 2` is enabled. For instance, here's a snippet of that output when
a backup file is not found:
```
/noexist: stat: [Errno 2] No such file or directory: '/noexist'
...
terminating with warning status, rc 107
```
So if you want to configure borgmatic to treat this as an error instead of a
warning, the exit status to use is `107`.
+23 -7
View File
@@ -3,7 +3,7 @@ title: How to develop on borgmatic
eleventyNavigation:
key: 🏗️ Develop on borgmatic
parent: How-to guides
order: 13
order: 14
---
## Source code
@@ -35,6 +35,14 @@ pipx ensurepath
pipx install --editable .
```
Or to work on the [Apprise
hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook),
change that last line to:
```bash
pipx install --editable .[Apprise]
```
To get oriented with the borgmatic source code, have a look at the [source
code reference](https://torsion.org/borgmatic/docs/reference/source-code/).
@@ -92,14 +100,14 @@ with Borg and supported databases for a few representative scenarios. These
tests don't run by default when running `tox`, because they're relatively slow
and depend on containers for runtime dependencies. These tests do run on the
continuous integration (CI) server, and running them on your developer machine
is the closest thing to CI-test parity.
is the closest thing to dev-CI parity.
If you would like to run the full test suite, first install Docker (or Podman;
see below) and [Docker Compose](https://docs.docker.com/compose/install/).
Then run:
```bash
scripts/run-end-to-end-dev-tests
scripts/run-end-to-end-tests
```
This script assumes you have permission to run `docker`. If you don't, then
@@ -141,6 +149,9 @@ the following deviations from it:
separate from their contents.
* Within multiline constructs, use standard four-space indentation. Don't align
indentation with an opening delimiter.
* In general, spell out words in variable names instead of shortening them.
So, think `index` instead of `idx`. There are some notable exceptions to
this though (like `config`).
borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
formatter, the [Flake8](http://flake8.pycqa.org/en/latest/) code checker, and
@@ -148,12 +159,17 @@ the [isort](https://github.com/timothycrosley/isort) import orderer, so
certain code style requirements will be enforced when running automated tests.
See the Black, Flake8, and isort documentation for more information.
## Continuous integration
Each pull request triggers a continuous integration build which runs the test
suite. You can view these builds on
[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic),
and they're also linked from the commits list on each pull request.
Each commit to
[main](https://projects.torsion.org/borgmatic-collective/borgmatic/branches)
triggers [a continuous integration
build](https://projects.torsion.org/borgmatic-collective/borgmatic/actions)
which runs the test suite and updates
[documentation](https://torsion.org/borgmatic/). These builds are also linked
from the [commits for the main
branch](https://projects.torsion.org/borgmatic-collective/borgmatic/commits/branch/main).
## Documentation development
+5
View File
@@ -191,3 +191,8 @@ for bootstrapping.
borgmatic configuration files, for instance if they contain sensitive
information you don't want to store even inside your encrypted backups. If you
do this though, the `config bootstrap` action will no longer work.
<span class="minilink minilink-addedin">New in version 1.8.7</span> Included
configuration files are stored in each backup archive. This means that the
`config bootstrap` action not only extracts the top-level configuration files
but also the includes they depend upon.
+17 -13
View File
@@ -139,8 +139,8 @@ Some borgmatic command-line actions also have a `--match-archives` flag that
overrides both the auto-matching behavior and the `match_archives`
configuration option.
<span class="minilink minilink-addedin">Prior to 1.7.11</span> The way to
limit the archives used for the `prune` action was a `prefix` option in the
<span class="minilink minilink-addedin">Prior to version 1.7.11</span> The way
to limit the archives used for the `prune` action was a `prefix` option in the
`retention` section for matching against the start of archive names. And the
option for limiting the archives used for the `check` action was a separate
`prefix` in the `consistency` section. Both of these options are deprecated in
@@ -495,21 +495,29 @@ borgmatic create --override parent_option.option1=value1 --override parent_optio
forget to specify the section that an option is in. That looks like a prefix
on the option name, e.g. `location.repositories`.
Note that each value is parsed as an actual YAML string, so you can even set
list values by using brackets. For instance:
Note that each value is parsed as an actual YAML string, so you can set list
values by using brackets. For instance:
```bash
borgmatic create --override repositories=[test1.borg,test2.borg]
```
Or even a single list element:
Or a single list element:
```bash
borgmatic create --override repositories=[/root/test.borg]
```
If your override value contains special YAML characters like colons, then
you'll need quotes for it to parse correctly:
Or a single list element that is a key/value pair:
```bash
borgmatic create --override repositories="[{path: test.borg, label: test}]"
```
If your override value contains characters like colons or spaces, then you'll
need to use quotes for it to parse correctly.
Another example:
```bash
borgmatic create --override repositories="['user@server:test.borg']"
@@ -518,16 +526,12 @@ borgmatic create --override repositories="['user@server:test.borg']"
There is not currently a way to override a single element of a list without
replacing the whole list.
Note that if you override an option of the list type (like
`location.repositories`), you do need to use the `[ ]` list syntax. See the
[configuration
Using the `[ ]` list syntax is required when overriding an option of the list
type (like `location.repositories`). See the [configuration
reference](https://torsion.org/borgmatic/docs/reference/configuration/) for
which options are list types. (YAML list values look like `- this` with an
indentation and a leading dash.)
Be sure to quote your overrides if they contain spaces or other characters
that your shell may interpret.
An alternate to command-line overrides is passing in your values via
[environment
variables](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
+74 -12
View File
@@ -101,7 +101,7 @@ script to handle the alerting:
```yaml
on_error:
- send-text-message.sh "{configuration_filename}" "{repository}"
- send-text-message.sh {configuration_filename} {repository}
```
In this example, when the error occurs, borgmatic interpolates runtime values
@@ -124,6 +124,27 @@ actions. borgmatic does not run `on_error` hooks if an error occurs within a
documentation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/),
especially the security information.
<span class="minilink minilink-addedin">New in version 1.8.7</span> borgmatic
automatically escapes these interpolated values to prevent shell injection
attacks. One implication of this change is that you shouldn't wrap the
interpolated values in your own quotes, as that will interfere with the
quoting performed by borgmatic and result in your command receiving incorrect
arguments. For instance, this won't work:
```yaml
on_error:
# Don't do this! It won't work, as the {error} value is already quoted.
- send-text-message.sh "Uh oh: {error}"
```
Do this instead:
```yaml
on_error:
- send-text-message.sh {error}
```
## Healthchecks hook
@@ -149,7 +170,7 @@ backup begins, ends, or errors, but only when any of the `create`, `prune`,
Then, if the actions complete successfully, borgmatic notifies Healthchecks of
the success and includes borgmatic logs in the payload data sent to
Healthchecks. This means that borgmatic logs show up in the Healthchecks UI,
although be aware that Healthchecks currently has a 10-kilobyte limit for the
although be aware that Healthchecks currently has a 100-kilobyte limit for the
logs in each ping.
If an error occurs during any action or hook, borgmatic notifies Healthchecks,
@@ -277,22 +298,22 @@ platforms including [web](https://ntfy.sh/stats),
[iOS](https://apps.apple.com/us/app/ntfy/id1625396347).
Since push notifications for regular events might soon become quite annoying,
this hook only fires on any errors by default in order to instantly alert you to issues.
The `states` list can override this.
this hook only fires on any errors by default in order to instantly alert you
to issues. The `states` list can override this. Each state can have its own
custom messages, priorities and tags or, if none are provided, will use the
default.
As ntfy is unauthenticated, it isn't a suitable channel for any private information
so the default messages are intentionally generic. These can be overridden, depending
on your risk assessment. Each `state` can have its own custom messages, priorities and tags
or, if none are provided, will use the default.
An example configuration is shown here, with all the available options, including
[priorities](https://ntfy.sh/docs/publish/#message-priority) and
An example configuration is shown here with all the available options,
including [priorities](https://ntfy.sh/docs/publish/#message-priority) and
[tags](https://ntfy.sh/docs/publish/#tags-emojis):
```yaml
ntfy:
topic: my-unique-topic
server: https://ntfy.my-domain.com
username: myuser
password: secret
start:
title: A borgmatic backup started
message: Watch this space...
@@ -317,6 +338,16 @@ ntfy:
<span class="minilink minilink-addedin">Prior to version 1.8.0</span> Put
the `ntfy:` option in the `hooks:` section of your configuration.
<span class="minilink minilink-addedin">New in version 1.8.9</span> Instead of
`username`/`password`, you can specify an [ntfy access
token](https://docs.ntfy.sh/config/#access-tokens):
```yaml
ntfy:
topic: my-unique-topic
server: https://ntfy.my-domain.com
access_token: tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2
````
## Loki hook
@@ -385,7 +416,7 @@ pipx](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#installation),
run the following to install Apprise so borgmatic can use it:
```bash
sudo pipx install --editable --force borgmatic[Apprise]
sudo pipx install --force borgmatic[Apprise]
```
Omit `sudo` if borgmatic is installed as a non-root user.
@@ -429,6 +460,37 @@ apprise:
- fail
```
<span class="minilink minilink-addedin">New in version 1.8.9</span> borgmatic
logs are automatically included in the body data sent to your Apprise services
when a backup finishes or fails.
You can customize the verbosity of the logs that are sent with borgmatic's
`--monitoring-verbosity` flag. The `--list` and `--stats` flags may also be of
use. See `borgmatic create --help` for more information.
If you don't want any logs sent, you can disable this feature by setting
`send_logs` to `false`:
```yaml
apprise:
services:
- url: gotify://hostname/token
label: gotify
send_logs: false
```
Or to limit the size of logs sent to Apprise services:
```yaml
apprise:
services:
- url: gotify://hostname/token
label: gotify
logs_size_limit: 500
```
This may be necessary for some services that reject large requests.
See the [configuration
reference](https://torsion.org/borgmatic/docs/reference/configuration/) for
details.
+3 -2
View File
@@ -406,8 +406,9 @@ source /usr/share/fish/vendor_completions.d/borgmatic.fish
borgmatic produces colored terminal output by default. It is disabled when a
non-interactive terminal is detected (like a cron job), or when you use the
`--json` flag. Otherwise, you can disable it by passing the `--no-color` flag,
setting the environment variable `PY_COLORS=False`, or setting the `color`
option to `false` in the `output` section of configuration.
setting the environment variables `PY_COLORS=False` or `NO_COLOR=True`, or
setting the `color` option to `false` in the `output` section of
configuration.
## Troubleshooting
+1 -1
View File
@@ -3,7 +3,7 @@ title: How to upgrade borgmatic and Borg
eleventyNavigation:
key: 📦 Upgrade borgmatic/Borg
parent: How-to guides
order: 12
order: 13
---
## Upgrading borgmatic
@@ -16,5 +16,7 @@ if [ -e "$USER_PODMAN_SOCKET_PATH" ]; then
export DOCKER_HOST="unix://$USER_PODMAN_SOCKET_PATH"
fi
docker-compose --file tests/end-to-end/docker-compose.yaml up --force-recreate \
--renew-anon-volumes --abort-on-container-exit
docker-compose --file tests/end-to-end/docker-compose.yaml --progress quiet up --force-recreate \
--renew-anon-volumes --detach
docker-compose --file tests/end-to-end/docker-compose.yaml --progress quiet attach tests
docker-compose --file tests/end-to-end/docker-compose.yaml --progress quiet down
+2 -5
View File
@@ -3,7 +3,7 @@
# This script installs test dependencies and runs all tests, including end-to-end tests. It
# is designed to run inside a test container, and presumes that other test infrastructure like
# databases are already running. Therefore, on a developer machine, you should not run this script
# directly. Instead, run scripts/run-end-to-end-dev-tests
# directly. Instead, run scripts/run-end-to-end-tests
#
# For more information, see:
# https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/
@@ -25,8 +25,5 @@ python3 -m pip install --no-cache --upgrade pip==22.2.2 setuptools==64.0.1
pip3 install --ignore-installed tox==4.11.3
export COVERAGE_FILE=/tmp/.coverage
if [ "$1" != "--end-to-end-only" ]; then
tox --workdir /tmp/.tox --sitepackages
fi
tox --workdir /tmp/.tox --sitepackages
tox --workdir /tmp/.tox --sitepackages -e end-to-end
+1 -1
View File
@@ -1,6 +1,6 @@
from setuptools import find_packages, setup
VERSION = '1.8.6'
VERSION = '1.8.9'
setup(
+4 -4
View File
@@ -1,10 +1,10 @@
appdirs==1.4.4; python_version >= '3.8'
appdirs==1.4.4
apprise==1.3.0
attrs==22.2.0; python_version >= '3.8'
black==23.3.0; python_version >= '3.8'
attrs==22.2.0
black==23.3.0
certifi==2023.7.22
chardet==5.1.0
click==8.1.3; python_version >= '3.8'
click==8.1.3
codespell==2.2.4
colorama==0.4.6
coverage==7.2.3
+1 -2
View File
@@ -49,14 +49,13 @@ services:
environment:
TEST_CONTAINER: true
volumes:
- "../..:/app:ro"
- "../..:/app"
tmpfs:
- "/app/borgmatic.egg-info"
- "/app/build"
tty: true
working_dir: /app
entrypoint: /app/scripts/run-full-tests
command: --end-to-end-only
depends_on:
- postgresql
- postgresql2
@@ -1,38 +0,0 @@
import ruamel.yaml
def test_dev_docker_compose_has_same_services_as_build_server_configuration():
'''
The end-to-end test configuration for local development and the build server's test
configuration use two different mechanisms for configuring and spinning up "services"—the
database containers upon which the end-to-end tests are reliant. The dev configuration uses
Docker Compose, while the Drone build server configuration uses its own similar-but-different
configuration file format.
Therefore, to ensure dev-build parity, these tests assert that the services are the same across
the dev and build configurations. This includes service name, container image, environment
variables, and commands.
This test only compares services and does not assert anything else about the respective testing
environments.
'''
yaml = ruamel.yaml.YAML(typ='safe')
dev_services = {
name: service
for name, service in yaml.load(open('tests/end-to-end/docker-compose.yaml').read())[
'services'
].items()
if name != 'tests'
}
build_server_services = tuple(yaml.load_all(open('.drone.yml').read()))[0]['services']
assert len(dev_services) == len(build_server_services)
for build_service in build_server_services:
dev_service = dev_services[build_service['name']]
assert dev_service['image'] == build_service['image']
assert dev_service['environment'] == build_service['environment']
if 'command' in dev_service or 'commands' in build_service:
assert len(build_service['commands']) <= 1
assert dev_service['command'] == build_service['commands'][0]
+63 -20
View File
@@ -12,7 +12,10 @@ def test_load_configuration_parses_contents():
config_file = io.StringIO('key: value')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == {'key': 'value'}
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', 'other.yaml'}
def test_load_configuration_with_only_integer_value_does_not_raise():
@@ -20,7 +23,10 @@ def test_load_configuration_with_only_integer_value_does_not_raise():
config_file = io.StringIO('33')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == 33
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml', config_paths) == 33
assert config_paths == {'config.yaml', 'other.yaml'}
def test_load_configuration_inlines_include_relative_to_current_directory():
@@ -34,8 +40,10 @@ def test_load_configuration_inlines_include_relative_to_current_directory():
config_file = io.StringIO('key: !include include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': 'value'}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_inlines_include_relative_to_config_parent_directory():
@@ -56,8 +64,10 @@ def test_load_configuration_inlines_include_relative_to_config_parent_directory(
config_file = io.StringIO('key: !include include.yaml')
config_file.name = '/etc/config.yaml'
builtins.should_receive('open').with_args('/etc/config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('/etc/config.yaml') == {'key': 'value'}
assert module.load_configuration('/etc/config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'/etc/config.yaml', '/etc/include.yaml', 'other.yaml'}
def test_load_configuration_raises_if_relative_include_does_not_exist():
@@ -70,9 +80,10 @@ def test_load_configuration_raises_if_relative_include_does_not_exist():
config_file = io.StringIO('key: !include include.yaml')
config_file.name = '/etc/config.yaml'
builtins.should_receive('open').with_args('/etc/config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(FileNotFoundError):
module.load_configuration('/etc/config.yaml')
module.load_configuration('/etc/config.yaml', config_paths)
def test_load_configuration_inlines_absolute_include():
@@ -86,8 +97,10 @@ def test_load_configuration_inlines_absolute_include():
config_file = io.StringIO('key: !include /root/include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': 'value'}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', '/root/include.yaml', 'other.yaml'}
def test_load_configuration_raises_if_absolute_include_does_not_exist():
@@ -98,9 +111,10 @@ def test_load_configuration_raises_if_absolute_include_does_not_exist():
config_file = io.StringIO('key: !include /root/include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(FileNotFoundError):
assert module.load_configuration('config.yaml')
assert module.load_configuration('config.yaml', config_paths)
def test_load_configuration_inlines_multiple_file_include_as_list():
@@ -117,8 +131,15 @@ def test_load_configuration_inlines_multiple_file_include_as_list():
config_file = io.StringIO('key: !include [/root/include1.yaml, /root/include2.yaml]')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': ['value2', 'value1']}
assert module.load_configuration('config.yaml', config_paths) == {'key': ['value2', 'value1']}
assert config_paths == {
'config.yaml',
'/root/include1.yaml',
'/root/include2.yaml',
'other.yaml',
}
def test_load_configuration_include_with_unsupported_filename_type_raises():
@@ -129,9 +150,10 @@ def test_load_configuration_include_with_unsupported_filename_type_raises():
config_file = io.StringIO('key: !include {path: /root/include.yaml}')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_merges_include():
@@ -155,8 +177,13 @@ def test_load_configuration_merges_include():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'foo': 'override', 'baz': 'quux'}
assert module.load_configuration('config.yaml', config_paths) == {
'foo': 'override',
'baz': 'quux',
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_merges_multiple_file_include():
@@ -188,12 +215,14 @@ def test_load_configuration_merges_multiple_file_include():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {
assert module.load_configuration('config.yaml', config_paths) == {
'foo': 'override',
'baz': 'second',
'original': 'yes',
}
assert config_paths == {'config.yaml', '/tmp/include1.yaml', '/tmp/include2.yaml', 'other.yaml'}
def test_load_configuration_with_retain_tag_merges_include_but_keeps_local_values():
@@ -226,11 +255,13 @@ def test_load_configuration_with_retain_tag_merges_include_but_keeps_local_value
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {
assert module.load_configuration('config.yaml', config_paths) == {
'stuff': {'foo': 'override'},
'other': {'a': 'override', 'c': 'd'},
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_retain_tag_but_without_merge_include_raises():
@@ -256,9 +287,10 @@ def test_load_configuration_with_retain_tag_but_without_merge_include_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_retain_tag_on_scalar_raises():
@@ -284,9 +316,10 @@ def test_load_configuration_with_retain_tag_on_scalar_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_merges_include_and_omits_requested_values():
@@ -315,8 +348,10 @@ def test_load_configuration_with_omit_tag_merges_include_and_omits_requested_val
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'stuff': ['a', 'c', 'x', 'y']}
assert module.load_configuration('config.yaml', config_paths) == {'stuff': ['a', 'c', 'x', 'y']}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_omit_tag_on_unknown_value_merges_include_and_does_not_raise():
@@ -345,8 +380,12 @@ def test_load_configuration_with_omit_tag_on_unknown_value_merges_include_and_do
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'stuff': ['a', 'b', 'c', 'x', 'y']}
assert module.load_configuration('config.yaml', config_paths) == {
'stuff': ['a', 'b', 'c', 'x', 'y']
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_omit_tag_on_non_list_item_raises():
@@ -374,9 +413,10 @@ def test_load_configuration_with_omit_tag_on_non_list_item_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_on_non_scalar_list_item_raises():
@@ -403,9 +443,10 @@ def test_load_configuration_with_omit_tag_on_non_scalar_list_item_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_but_without_merge_raises():
@@ -433,9 +474,10 @@ def test_load_configuration_with_omit_tag_but_without_merge_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_does_not_merge_include_list():
@@ -460,9 +502,10 @@ def test_load_configuration_does_not_merge_include_list():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(module.ruamel.yaml.error.YAMLError):
assert module.load_configuration('config.yaml')
assert module.load_configuration('config.yaml', config_paths)
@pytest.mark.parametrize(
@@ -29,6 +29,7 @@ def test_apply_overrides_updates_config():
'section.key=value1',
'other_section.thing=value2',
'section.nested.key=value3',
'location.no_longer_in_location=value4',
'new.foo=bar',
'new.mylist=[baz]',
'new.nonlist=[quux]',
@@ -36,6 +37,7 @@ def test_apply_overrides_updates_config():
config = {
'section': {'key': 'value', 'other': 'other_value'},
'other_section': {'thing': 'thing_value'},
'no_longer_in_location': 'because_location_is_deprecated',
}
schema = {
'properties': {
@@ -49,4 +51,6 @@ def test_apply_overrides_updates_config():
'section': {'key': 'value1', 'other': 'other_value', 'nested': {'key': 'value3'}},
'other_section': {'thing': 'value2'},
'new': {'foo': 'bar', 'mylist': ['baz'], 'nonlist': '[quux]'},
'location': {'no_longer_in_location': 'value4'},
'no_longer_in_location': 'value4',
}
+16 -7
View File
@@ -58,7 +58,7 @@ def test_parse_configuration_transforms_file_into_mapping():
'''
)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home', '/etc'],
@@ -68,6 +68,7 @@ def test_parse_configuration_transforms_file_into_mapping():
'keep_minutely': 60,
'checks': [{'name': 'repository'}, {'name': 'archives'}],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
@@ -84,12 +85,13 @@ def test_parse_configuration_passes_through_quoted_punctuation():
'''
)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': [f'/home/{string.punctuation}'],
'repositories': [{'path': 'test.borg'}],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
@@ -141,7 +143,7 @@ def test_parse_configuration_inlines_include_inside_deprecated_section():
include_file.name = 'include.yaml'
builtins.should_receive('open').with_args('/tmp/include.yaml').and_return(include_file)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
@@ -149,6 +151,7 @@ def test_parse_configuration_inlines_include_inside_deprecated_section():
'keep_daily': 7,
'keep_hourly': 24,
}
assert config_paths == {'/tmp/include.yaml', '/tmp/config.yaml'}
assert len(logs) == 1
@@ -175,7 +178,7 @@ def test_parse_configuration_merges_include():
include_file.name = 'include.yaml'
builtins.should_receive('open').with_args('/tmp/include.yaml').and_return(include_file)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
@@ -183,6 +186,7 @@ def test_parse_configuration_merges_include():
'keep_daily': 1,
'keep_hourly': 24,
}
assert config_paths == {'/tmp/include.yaml', '/tmp/config.yaml'}
assert logs == []
@@ -194,6 +198,9 @@ def test_parse_configuration_raises_for_missing_config_file():
def test_parse_configuration_raises_for_missing_schema_file():
mock_config_and_schema('')
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('/tmp/config.yaml').and_return(
io.StringIO('foo: bar')
)
builtins.should_receive('open').with_args('/tmp/schema.yaml').and_raise(FileNotFoundError)
with pytest.raises(FileNotFoundError):
@@ -233,8 +240,8 @@ def test_parse_configuration_applies_overrides():
'''
)
config, logs = module.parse_configuration(
'/tmp/config.yaml', '/tmp/schema.yaml', overrides=['location.local_path=borg2']
config, config_paths, logs = module.parse_configuration(
'/tmp/config.yaml', '/tmp/schema.yaml', overrides=['local_path=borg2']
)
assert config == {
@@ -242,6 +249,7 @@ def test_parse_configuration_applies_overrides():
'repositories': [{'path': 'hostname.borg'}],
'local_path': 'borg2',
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
@@ -260,11 +268,12 @@ def test_parse_configuration_applies_normalization_after_environment_variable_in
)
flexmock(os).should_receive('getenv').replace_with(lambda variable_name, default: default)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
'repositories': [{'path': 'ssh://user@hostname/./repo'}],
'exclude_if_present': ['.nobackup'],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs
+28
View File
@@ -0,0 +1,28 @@
import logging
from flexmock import flexmock
from borgmatic.hooks import apprise as module
def test_destroy_monitor_removes_apprise_handler():
logger = logging.getLogger()
original_handlers = list(logger.handlers)
module.borgmatic.hooks.logs.add_handler(
module.borgmatic.hooks.logs.Forgetful_buffering_handler(
identifier=module.HANDLER_IDENTIFIER, byte_capacity=100, log_level=1
)
)
module.destroy_monitor(flexmock(), flexmock(), flexmock(), flexmock(), flexmock())
assert logger.handlers == original_handlers
def test_destroy_monitor_without_apprise_handler_does_not_raise():
logger = logging.getLogger()
original_handlers = list(logger.handlers)
module.destroy_monitor(flexmock(), flexmock(), flexmock(), flexmock(), flexmock())
assert logger.handlers == original_handlers
+5 -1
View File
@@ -8,7 +8,11 @@ from borgmatic.hooks import healthchecks as module
def test_destroy_monitor_removes_healthchecks_handler():
logger = logging.getLogger()
original_handlers = list(logger.handlers)
logger.addHandler(module.Forgetful_buffering_handler(byte_capacity=100, log_level=1))
module.borgmatic.hooks.logs.add_handler(
module.borgmatic.hooks.logs.Forgetful_buffering_handler(
identifier=module.HANDLER_IDENTIFIER, byte_capacity=100, log_level=1
)
)
module.destroy_monitor(flexmock(), flexmock(), flexmock(), flexmock(), flexmock())
+118 -30
View File
@@ -11,7 +11,7 @@ from borgmatic import execute as module
def test_log_outputs_logs_each_line_separately():
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'hi').once()
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'there').once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=subprocess.PIPE)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -28,13 +28,14 @@ def test_log_outputs_logs_each_line_separately():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_skips_logs_for_process_with_none_stdout():
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'hi').never()
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'there').once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=None)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -51,12 +52,13 @@ def test_log_outputs_skips_logs_for_process_with_none_stdout():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=subprocess.PIPE)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -73,6 +75,7 @@ def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
exclude_stdouts=(),
output_log_level=None,
borg_local_path='borg',
borg_exit_codes=None,
)
assert captured_outputs == {hi_process: 'hi', there_process: 'there'}
@@ -80,7 +83,7 @@ def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
def test_log_outputs_includes_error_output_in_exception():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
@@ -88,7 +91,11 @@ def test_log_outputs_includes_error_output_in_exception():
with pytest.raises(subprocess.CalledProcessError) as error:
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.output
@@ -100,7 +107,7 @@ def test_log_outputs_logs_multiline_error_output():
of a process' traceback.
'''
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
@@ -111,13 +118,17 @@ def test_log_outputs_logs_multiline_error_output():
with pytest.raises(subprocess.CalledProcessError):
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_skips_error_output_in_exception_for_process_with_none_stdout():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=None)
@@ -125,30 +136,43 @@ def test_log_outputs_skips_error_output_in_exception_for_process_with_none_stdou
with pytest.raises(subprocess.CalledProcessError) as error:
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
assert not error.value.output
def test_log_outputs_kills_other_processes_when_one_errors():
def test_log_outputs_kills_other_processes_and_raises_when_one_errors():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], 2, 'borg'
).and_return(True)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.ERROR)
other_process = subprocess.Popen(
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['sleep', '2'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('interpret_exit_code').with_args(
['sleep', '2'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('output_buffer_for_process').with_args(process, ()).and_return(
process.stdout
)
@@ -163,12 +187,56 @@ def test_log_outputs_kills_other_processes_when_one_errors():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
assert error.value.output
def test_log_outputs_kills_other_processes_and_returns_when_one_exits_with_warning():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.WARNING)
other_process = subprocess.Popen(
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('interpret_exit_code').with_args(
['sleep', '2'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('output_buffer_for_process').with_args(process, ()).and_return(
process.stdout
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, ()
).and_return(other_process.stdout)
flexmock(other_process).should_receive('kill').once()
module.log_outputs(
(process, other_process),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_vents_other_processes_when_one_exits():
'''
Execute a command to generate a longish random string and pipe it into another command that
@@ -204,6 +272,7 @@ def test_log_outputs_vents_other_processes_when_one_exits():
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
@@ -235,6 +304,7 @@ def test_log_outputs_does_not_error_when_one_process_exits():
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
@@ -243,17 +313,27 @@ def test_log_outputs_truncates_long_error_output():
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], 2, 'borg'
).and_return(True)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
with pytest.raises(subprocess.CalledProcessError) as error:
flexmock(module, ERROR_OUTPUT_MAX_LINE_COUNT=0).log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
@@ -262,24 +342,32 @@ def test_log_outputs_truncates_long_error_output():
def test_log_outputs_with_no_output_logs_nothing():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
process = subprocess.Popen(['true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_with_unfinished_process_re_polls():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
process = subprocess.Popen(['true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(process).should_receive('poll').and_return(None).and_return(0).times(3)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
+10 -5
View File
@@ -22,13 +22,14 @@ def test_run_create_executes_and_calls_hooks_for_configured_repository():
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -57,13 +58,14 @@ def test_run_create_with_store_config_files_false_does_not_create_borgmatic_mani
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={'store_config_files': False},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -94,13 +96,14 @@ def test_run_create_runs_with_selected_repository():
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -126,13 +129,14 @@ def test_run_create_bails_if_repository_does_not_match():
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository='repo',
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -167,13 +171,14 @@ def test_run_create_produces_json():
json=True,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
assert list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
+61 -1
View File
@@ -16,6 +16,7 @@ def test_run_arbitrary_borg_calls_borg_with_flags():
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -37,6 +38,7 @@ def test_run_arbitrary_borg_with_log_info_calls_borg_with_info_flag():
('borg', 'break-lock', '--info', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -59,6 +61,7 @@ def test_run_arbitrary_borg_with_log_debug_calls_borg_with_debug_flag():
('borg', 'break-lock', '--debug', '--show-rc', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -84,6 +87,7 @@ def test_run_arbitrary_borg_with_lock_wait_calls_borg_with_lock_wait_flags():
('borg', 'break-lock', '--lock-wait', '5', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -102,9 +106,10 @@ def test_run_arbitrary_borg_with_archive_calls_borg_with_archive_flag():
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '::$ARCHIVE'),
('borg', 'break-lock', "'::$ARCHIVE'"),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': 'archive'},
)
@@ -127,6 +132,7 @@ def test_run_arbitrary_borg_with_local_path_calls_borg_via_local_path():
('borg1', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg1',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -140,6 +146,29 @@ def test_run_arbitrary_borg_with_local_path_calls_borg_via_local_path():
)
def test_run_arbitrary_borg_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
module.run_arbitrary_borg(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
options=['break-lock', '::'],
)
def test_run_arbitrary_borg_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -151,6 +180,32 @@ def test_run_arbitrary_borg_with_remote_path_calls_borg_with_remote_path_flags()
('borg', 'break-lock', '--remote-path', 'borg1', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
module.run_arbitrary_borg(
repository_path='repo',
config={},
local_borg_version='1.2.3',
options=['break-lock', '::'],
remote_path='borg1',
)
def test_run_arbitrary_borg_with_remote_path_injection_attack_gets_escaped():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(
('--remote-path', 'borg1; naughty-command')
).and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '--remote-path', "'borg1; naughty-command'", '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -173,6 +228,7 @@ def test_run_arbitrary_borg_passes_borg_specific_flags_to_borg():
('borg', 'list', '--progress', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -194,6 +250,7 @@ def test_run_arbitrary_borg_omits_dash_dash_in_flags_passed_to_borg():
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -215,6 +272,7 @@ def test_run_arbitrary_borg_without_borg_specific_flags_does_not_raise():
('borg',),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -236,6 +294,7 @@ def test_run_arbitrary_borg_passes_key_sub_command_to_borg_before_injected_flags
('borg', 'key', 'export', '--info', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -258,6 +317,7 @@ def test_run_arbitrary_borg_passes_debug_sub_command_to_borg_before_injected_fla
('borg', 'debug', 'dump-manifest', '--info', '::', 'path'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
+29 -2
View File
@@ -7,11 +7,12 @@ from borgmatic.borg import break_lock as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -28,6 +29,32 @@ def test_break_lock_calls_borg_with_required_flags():
)
def test_break_lock_calls_borg_with_local_path():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg1', 'break-lock', 'repo'))
module.break_lock(
repository_path='repo',
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_break_lock_calls_borg_using_exit_codes():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg1', 'break-lock', 'repo'))
module.break_lock(
repository_path='repo',
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_break_lock_calls_borg_with_remote_path_flags():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg', 'break-lock', '--remote-path', 'borg1', 'repo'))
+41 -2
View File
@@ -8,10 +8,13 @@ from borgmatic.borg import check as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command, extra_environment=None
command,
extra_environment=None,
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
).once()
@@ -689,6 +692,8 @@ def test_check_archives_with_progress_passes_through_to_borg():
('borg', 'check', '--progress', 'repo'),
output_file=module.DO_NOT_CAPTURE,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
@@ -723,6 +728,8 @@ def test_check_archives_with_repair_passes_through_to_borg():
('borg', 'check', '--repair', 'repo'),
output_file=module.DO_NOT_CAPTURE,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
@@ -963,6 +970,36 @@ def test_check_archives_with_local_path_calls_borg_via_local_path():
)
def test_check_archives_with_exit_codes_calls_borg_using_them():
checks = ('repository',)
check_last = flexmock()
borg_exit_codes = flexmock()
config = {'check_last': check_last, 'borg_exit_codes': borg_exit_codes}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
'{"repository": {"id": "repo"}}'
)
flexmock(module).should_receive('upgrade_check_times')
flexmock(module).should_receive('parse_checks')
flexmock(module).should_receive('make_archive_filter_flags').and_return(())
flexmock(module).should_receive('make_archives_check_id').and_return(None)
flexmock(module).should_receive('filter_checks_on_frequency').and_return(checks)
flexmock(module).should_receive('make_check_flags').with_args(checks, ()).and_return(())
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg', 'check', 'repo'), borg_exit_codes=borg_exit_codes)
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
module.check_archives(
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_remote_path_passes_through_to_borg():
checks = ('repository',)
check_last = flexmock()
@@ -1128,6 +1165,8 @@ def test_check_archives_with_match_archives_passes_through_to_borg():
flexmock(module).should_receive('execute_command').with_args(
('borg', 'check', '--match-archives', 'foo-*', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
+18 -1
View File
@@ -7,12 +7,13 @@ from borgmatic.borg import compact as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(compact_command, output_log_level):
def insert_execute_command_mock(compact_command, output_log_level, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
compact_command,
output_log_level=output_log_level,
borg_local_path=compact_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -87,6 +88,22 @@ def test_compact_segments_with_local_path_calls_borg_via_local_path():
)
def test_compact_segments_with_exit_codes_calls_borg_using_them():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
borg_exit_codes = flexmock()
insert_execute_command_mock(
COMPACT_COMMAND + ('repo',), logging.INFO, borg_exit_codes=borg_exit_codes
)
module.compact_segments(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_compact_segments_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(COMPACT_COMMAND + ('--remote-path', 'borg1', 'repo'), logging.INFO)
+209 -55
View File
@@ -402,6 +402,7 @@ def test_collect_special_file_paths_parses_special_files_from_borg_dry_run_file_
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -420,6 +421,7 @@ def test_collect_special_file_paths_excludes_requested_directories():
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -438,6 +440,7 @@ def test_collect_special_file_paths_excludes_non_special_files():
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -452,12 +455,14 @@ def test_collect_special_file_paths_omits_exclude_no_dump_flag_from_command():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('Processing files ...\n- /foo\n+ /bar\n- /baz').once()
flexmock(module).should_receive('special_file').and_return(True)
flexmock(module).should_receive('any_parent_directories').and_return(False)
module.collect_special_file_paths(
('borg', 'create', '--exclude-nodump'),
config={},
local_path='borg',
working_directory=None,
borg_environment=None,
@@ -494,6 +499,7 @@ def test_create_archive_calls_borg_with_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -506,8 +512,9 @@ def test_create_archive_calls_borg_with_parameters():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -537,6 +544,7 @@ def test_create_archive_calls_borg_with_environment():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -549,8 +557,9 @@ def test_create_archive_calls_borg_with_environment():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -582,6 +591,7 @@ def test_create_archive_with_patterns_calls_borg_with_patterns_including_convert
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -594,23 +604,24 @@ def test_create_archive_with_patterns_calls_borg_with_patterns_including_convert
'repositories': ['repo'],
'patterns': ['pattern'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sources_and_config_paths():
def test_create_archive_with_sources_and_config_paths_calls_borg_with_sources_and_config_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
flexmock(module).should_receive('deduplicate_directories').and_return(
('foo', 'bar', '/etc/borgmatic/config.yaml')
('foo', 'bar', '/tmp/test.yaml')
)
flexmock(module).should_receive('map_directories_to_devices').and_return({})
flexmock(module).should_receive('expand_directories').with_args([]).and_return(())
flexmock(module).should_receive('expand_directories').with_args(
('foo', 'bar', '/etc/borgmatic/config.yaml')
).and_return(('foo', 'bar', '/etc/borgmatic/config.yaml'))
('foo', 'bar', '/tmp/test.yaml')
).and_return(('foo', 'bar', '/tmp/test.yaml'))
flexmock(module).should_receive('expand_directories').with_args([]).and_return(())
flexmock(module).should_receive('pattern_root_directories').and_return([])
flexmock(module.os.path).should_receive('expanduser').and_raise(TypeError)
@@ -627,10 +638,11 @@ def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sourc
environment = {'BORG_THINGY': 'YUP'}
flexmock(module.environment).should_receive('make_environment').and_return(environment)
flexmock(module).should_receive('execute_command').with_args(
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS + ('/etc/borgmatic/config.yaml',),
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS + ('/tmp/test.yaml',),
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -642,12 +654,13 @@ def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sourc
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=['/etc/borgmatic/config.yaml']),
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_sources_and_used_config_paths_with_store_config_files_false_calls_borg_with_sources_and_no_config_paths():
def test_create_archive_with_sources_and_config_paths_with_store_config_files_false_calls_borg_with_sources_and_no_config_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
@@ -677,6 +690,7 @@ def test_create_archive_with_sources_and_used_config_paths_with_store_config_fil
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -689,8 +703,9 @@ def test_create_archive_with_sources_and_used_config_paths_with_store_config_fil
'repositories': ['repo'],
'store_config_files': False,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=['/etc/borgmatic/config.yaml']),
global_arguments=flexmock(log_json=False),
)
@@ -722,6 +737,7 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -734,8 +750,9 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
'repositories': ['repo'],
'exclude_patterns': ['exclude'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -764,6 +781,7 @@ def test_create_archive_with_log_info_calls_borg_with_info_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -777,8 +795,9 @@ def test_create_archive_with_log_info_calls_borg_with_info_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -807,6 +826,7 @@ def test_create_archive_with_log_info_and_json_suppresses_most_borg_output():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
)
insert_logging_mock(logging.INFO)
@@ -818,8 +838,9 @@ def test_create_archive_with_log_info_and_json_suppresses_most_borg_output():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -849,6 +870,7 @@ def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -862,8 +884,9 @@ def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -892,6 +915,7 @@ def test_create_archive_with_log_debug_and_json_suppresses_most_borg_output():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
)
insert_logging_mock(logging.DEBUG)
@@ -903,8 +927,9 @@ def test_create_archive_with_log_debug_and_json_suppresses_most_borg_output():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -934,6 +959,7 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -946,8 +972,9 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -978,6 +1005,7 @@ def test_create_archive_with_stats_and_dry_run_calls_borg_without_stats_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -991,8 +1019,9 @@ def test_create_archive_with_stats_and_dry_run_calls_borg_without_stats_paramete
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stats=True,
)
@@ -1022,6 +1051,7 @@ def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_inte
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1035,8 +1065,9 @@ def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_inte
'exclude_patterns': None,
'checkpoint_interval': 600,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1065,6 +1096,7 @@ def test_create_archive_with_checkpoint_volume_calls_borg_with_checkpoint_volume
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1078,8 +1110,9 @@ def test_create_archive_with_checkpoint_volume_calls_borg_with_checkpoint_volume
'exclude_patterns': None,
'checkpoint_volume': 1024,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1108,6 +1141,7 @@ def test_create_archive_with_chunker_params_calls_borg_with_chunker_params_param
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1121,8 +1155,9 @@ def test_create_archive_with_chunker_params_calls_borg_with_chunker_params_param
'exclude_patterns': None,
'chunker_params': '1,2,3,4',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1151,6 +1186,7 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1164,8 +1200,9 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
'exclude_patterns': None,
'compression': 'rle',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1200,6 +1237,7 @@ def test_create_archive_with_upload_rate_limit_calls_borg_with_upload_ratelimit_
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1213,8 +1251,9 @@ def test_create_archive_with_upload_rate_limit_calls_borg_with_upload_ratelimit_
'exclude_patterns': None,
'upload_rate_limit': 100,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1245,6 +1284,7 @@ def test_create_archive_with_working_directory_calls_borg_with_working_directory
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory='/working/dir',
extra_environment=None,
)
@@ -1258,8 +1298,9 @@ def test_create_archive_with_working_directory_calls_borg_with_working_directory
'working_directory': '/working/dir',
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1288,6 +1329,7 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1301,8 +1343,9 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
'one_file_system': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1337,6 +1380,7 @@ def test_create_archive_with_numeric_ids_calls_borg_with_numeric_ids_parameter(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1350,8 +1394,9 @@ def test_create_archive_with_numeric_ids_calls_borg_with_numeric_ids_parameter(
'numeric_ids': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1382,6 +1427,7 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1390,6 +1436,7 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1403,8 +1450,9 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
'read_special': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1445,6 +1493,7 @@ def test_create_archive_with_basic_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1458,8 +1507,9 @@ def test_create_archive_with_basic_option_calls_borg_with_corresponding_paramete
option_name: option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1499,6 +1549,7 @@ def test_create_archive_with_atime_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1512,8 +1563,9 @@ def test_create_archive_with_atime_option_calls_borg_with_corresponding_paramete
'atime': option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1553,6 +1605,7 @@ def test_create_archive_with_flags_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1566,8 +1619,9 @@ def test_create_archive_with_flags_option_calls_borg_with_corresponding_paramete
'flags': option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1596,6 +1650,7 @@ def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1609,8 +1664,9 @@ def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters(
'files_cache': 'ctime,size',
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1639,6 +1695,7 @@ def test_create_archive_with_local_path_calls_borg_via_local_path():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg1',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1651,12 +1708,59 @@ def test_create_archive_with_local_path_calls_borg_via_local_path():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_create_archive_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
flexmock(module).should_receive('deduplicate_directories').and_return(('foo', 'bar'))
flexmock(module).should_receive('map_directories_to_devices').and_return({})
flexmock(module).should_receive('expand_directories').and_return(())
flexmock(module).should_receive('pattern_root_directories').and_return([])
flexmock(module.os.path).should_receive('expanduser').and_raise(TypeError)
flexmock(module).should_receive('expand_home_directories').and_return(())
flexmock(module).should_receive('write_pattern_file').and_return(None)
flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module).should_receive('ensure_files_readable')
flexmock(module).should_receive('make_pattern_flags').and_return(())
flexmock(module).should_receive('make_exclude_flags').and_return(())
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
(f'repo::{DEFAULT_ARCHIVE_NAME}',)
)
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS,
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
working_directory=None,
extra_environment=None,
)
module.create_archive(
dry_run=False,
repository_path='repo',
config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
'borg_exit_codes': borg_exit_codes,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -1682,6 +1786,7 @@ def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1694,8 +1799,9 @@ def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters(
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
remote_path='borg1',
)
@@ -1725,6 +1831,7 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1738,8 +1845,9 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
'exclude_patterns': None,
'umask': 740,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1768,6 +1876,7 @@ def test_create_archive_with_log_json_calls_borg_with_log_json_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1780,8 +1889,9 @@ def test_create_archive_with_log_json_calls_borg_with_log_json_parameters():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=True, used_config_paths=[]),
global_arguments=flexmock(log_json=True),
)
@@ -1810,6 +1920,7 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1823,8 +1934,9 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
'exclude_patterns': None,
'lock_wait': 5,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1853,6 +1965,7 @@ def test_create_archive_with_stats_calls_borg_with_stats_parameter_and_answer_ou
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1865,8 +1978,9 @@ def test_create_archive_with_stats_calls_borg_with_stats_parameter_and_answer_ou
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stats=True,
)
@@ -1896,6 +2010,7 @@ def test_create_archive_with_files_calls_borg_with_list_parameter_and_answer_out
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1908,8 +2023,9 @@ def test_create_archive_with_files_calls_borg_with_list_parameter_and_answer_out
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
list_files=True,
)
@@ -1939,6 +2055,7 @@ def test_create_archive_with_progress_and_log_info_calls_borg_with_progress_para
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1952,8 +2069,9 @@ def test_create_archive_with_progress_and_log_info_calls_borg_with_progress_para
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
)
@@ -1983,6 +2101,7 @@ def test_create_archive_with_progress_calls_borg_with_progress_parameter():
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1995,8 +2114,9 @@ def test_create_archive_with_progress_calls_borg_with_progress_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
)
@@ -2036,6 +2156,7 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2045,6 +2166,7 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2057,8 +2179,9 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
stream_processes=processes,
)
@@ -2099,6 +2222,7 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2108,6 +2232,7 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2121,8 +2246,9 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
'exclude_patterns': None,
'read_special': False,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2167,6 +2293,7 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2176,6 +2303,7 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2188,8 +2316,9 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2230,6 +2359,7 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2239,6 +2369,7 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2252,8 +2383,9 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
'exclude_patterns': None,
'read_special': True,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2283,6 +2415,7 @@ def test_create_archive_with_json_calls_borg_with_json_parameter():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
json_output = module.create_archive(
@@ -2293,8 +2426,9 @@ def test_create_archive_with_json_calls_borg_with_json_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -2326,6 +2460,7 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_parameter()
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
json_output = module.create_archive(
@@ -2336,8 +2471,9 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_parameter()
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
stats=True,
)
@@ -2370,6 +2506,7 @@ def test_create_archive_with_source_directories_glob_expands():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2383,8 +2520,9 @@ def test_create_archive_with_source_directories_glob_expands():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2413,6 +2551,7 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2426,8 +2565,9 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2456,6 +2596,7 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2468,8 +2609,9 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2498,6 +2640,7 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2511,8 +2654,9 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
'exclude_patterns': None,
'archive_name_format': 'ARCHIVE_NAME',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2542,6 +2686,7 @@ def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2555,8 +2700,9 @@ def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
'exclude_patterns': None,
'archive_name_format': 'Documents_{hostname}-{now}', # noqa: FS003
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2586,6 +2732,7 @@ def test_create_archive_with_repository_accepts_borg_placeholders():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2599,8 +2746,9 @@ def test_create_archive_with_repository_accepts_borg_placeholders():
'exclude_patterns': None,
'archive_name_format': 'Documents_{hostname}-{now}', # noqa: FS003
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2629,6 +2777,7 @@ def test_create_archive_with_extra_borg_options_calls_borg_with_extra_options():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2642,8 +2791,9 @@ def test_create_archive_with_extra_borg_options_calls_borg_with_extra_options():
'exclude_patterns': None,
'extra_borg_options': {'create': '--extra --options'},
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2681,6 +2831,7 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2690,6 +2841,7 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2702,8 +2854,9 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2727,8 +2880,9 @@ def test_create_archive_with_non_existent_directory_and_source_directories_must_
'exclude_patterns': None,
'source_directories_must_exist': True,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
+2 -1
View File
@@ -22,7 +22,8 @@ def test_make_environment_with_ssh_command_should_set_environment():
def test_make_environment_without_configuration_should_not_set_environment():
environment = module.make_environment({})
assert environment == {}
# borgmatic always sets this Borg environment variable.
assert environment == {'BORG_EXIT_CODES': 'modern'}
def test_make_environment_with_relocated_repo_access_true_should_set_environment_yes():
+33 -2
View File
@@ -9,7 +9,7 @@ from borgmatic.borg import export_key as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE):
def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE, borg_exit_codes=None):
borgmatic.logger.add_custom_log_levels()
flexmock(module.environment).should_receive('make_environment')
@@ -17,7 +17,8 @@ def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE):
command,
output_file=output_file,
output_log_level=module.logging.ANSWER,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -36,6 +37,36 @@ def test_export_key_calls_borg_with_required_flags():
)
def test_export_key_calls_borg_with_local_path():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
insert_execute_command_mock(('borg1', 'key', 'export', 'repo'))
module.export_key(
repository_path='repo',
config={},
local_borg_version='1.2.3',
export_arguments=flexmock(paper=False, qr_html=False, path=None),
global_arguments=flexmock(dry_run=False, log_json=False),
local_path='borg1',
)
def test_export_key_calls_borg_using_exit_codes():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
borg_exit_codes = flexmock()
insert_execute_command_mock(('borg', 'key', 'export', 'repo'), borg_exit_codes=borg_exit_codes)
module.export_key(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
export_arguments=flexmock(paper=False, qr_html=False, path=None),
global_arguments=flexmock(dry_run=False, log_json=False),
)
def test_export_key_calls_borg_with_remote_path_flags():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
+37 -8
View File
@@ -8,7 +8,11 @@ from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(
command, output_log_level=logging.INFO, borg_local_path='borg', capture=True
command,
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
capture=True,
):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
@@ -16,11 +20,12 @@ def insert_execute_command_mock(
output_file=None if capture else module.DO_NOT_CAPTURE,
output_log_level=output_log_level,
borg_local_path=borg_local_path,
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
def test_export_tar_archive_calls_borg_with_path_parameters():
def test_export_tar_archive_calls_borg_with_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -42,7 +47,7 @@ def test_export_tar_archive_calls_borg_with_path_parameters():
)
def test_export_tar_archive_calls_borg_with_local_path_parameters():
def test_export_tar_archive_calls_borg_with_local_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -65,7 +70,31 @@ def test_export_tar_archive_calls_borg_with_local_path_parameters():
)
def test_export_tar_archive_calls_borg_with_remote_path_parameters():
def test_export_tar_archive_calls_borg_using_exit_codes():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'export-tar', 'repo::archive', 'test.tar'),
borg_exit_codes=borg_exit_codes,
)
module.export_tar_archive(
dry_run=False,
repository_path='repo',
archive='archive',
paths=None,
destination_path='test.tar',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_export_tar_archive_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -88,7 +117,7 @@ def test_export_tar_archive_calls_borg_with_remote_path_parameters():
)
def test_export_tar_archive_calls_borg_with_umask_parameters():
def test_export_tar_archive_calls_borg_with_umask_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -130,7 +159,7 @@ def test_export_tar_archive_calls_borg_with_log_json_parameter():
)
def test_export_tar_archive_calls_borg_with_lock_wait_parameters():
def test_export_tar_archive_calls_borg_with_lock_wait_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -173,7 +202,7 @@ def test_export_tar_archive_with_log_info_calls_borg_with_info_parameter():
)
def test_export_tar_archive_with_log_debug_calls_borg_with_debug_parameters():
def test_export_tar_archive_with_log_debug_calls_borg_with_debug_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -216,7 +245,7 @@ def test_export_tar_archive_calls_borg_with_dry_run_parameter():
)
def test_export_tar_archive_calls_borg_with_tar_filter_parameters():
def test_export_tar_archive_calls_borg_with_tar_filter_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+76 -1
View File
@@ -8,12 +8,14 @@ from borgmatic.borg import extract as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command, working_directory=None):
def insert_execute_command_mock(command, working_directory=None, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
working_directory=working_directory,
extra_environment=None,
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
).once()
@@ -99,6 +101,25 @@ def test_extract_last_archive_dry_run_calls_borg_via_local_path():
)
def test_extract_last_archive_dry_run_calls_borg_using_exit_codes():
flexmock(module.rlist).should_receive('resolve_archive_name').and_return('archive')
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'extract', '--dry-run', 'repo::archive'), borg_exit_codes=borg_exit_codes
)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
module.extract_last_archive_dry_run(
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
repository_path='repo',
lock_wait=None,
)
def test_extract_last_archive_dry_run_calls_borg_with_remote_path_flags():
flexmock(module.rlist).should_receive('resolve_archive_name').and_return('archive')
insert_execute_command_mock(
@@ -174,6 +195,54 @@ def test_extract_archive_calls_borg_with_path_flags():
)
def test_extract_archive_calls_borg_with_local_path():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
insert_execute_command_mock(('borg1', 'extract', 'repo::archive'))
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
flexmock(module.borgmatic.config.validate).should_receive(
'normalize_repository_path'
).and_return('repo')
module.extract_archive(
dry_run=False,
repository='repo',
archive='archive',
paths=None,
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_extract_archive_calls_borg_with_exit_codes():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'extract', 'repo::archive'), borg_exit_codes=borg_exit_codes
)
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
flexmock(module.borgmatic.config.validate).should_receive(
'normalize_repository_path'
).and_return('repo')
module.extract_archive(
dry_run=False,
repository='repo',
archive='archive',
paths=None,
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_extract_archive_calls_borg_with_remote_path_flags():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
insert_execute_command_mock(('borg', 'extract', '--remote-path', 'borg1', 'repo::archive'))
@@ -470,6 +539,8 @@ def test_extract_archive_calls_borg_with_progress_parameter():
output_file=module.DO_NOT_CAPTURE,
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -518,6 +589,8 @@ def test_extract_archive_calls_borg_with_stdout_parameter_and_returns_process():
working_directory=None,
run_to_completion=False,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(process).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -549,6 +622,8 @@ def test_extract_archive_skips_abspath_for_remote_repository():
('borg', 'extract', 'server:repo::archive'),
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+58 -1
View File
@@ -293,7 +293,7 @@ def test_capture_archive_listing_does_not_raise():
module.capture_archive_listing(
repository_path='repo',
archive='archive',
config=flexmock(),
config={},
local_borg_version=flexmock(),
global_arguments=flexmock(log_json=False),
)
@@ -332,6 +332,7 @@ def test_list_archive_calls_borg_with_flags():
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -395,6 +396,7 @@ def test_list_archive_calls_borg_with_local_path():
('borg2', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg2',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -408,6 +410,53 @@ def test_list_archive_calls_borg_with_local_path():
)
def test_list_archive_calls_borg_using_exit_codes():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.logger).answer = lambda message: None
list_arguments = argparse.Namespace(
archive='archive',
paths=None,
json=False,
find_paths=None,
prefix=None,
match_archives=None,
sort_by=None,
first=None,
last=None,
)
global_arguments = flexmock(log_json=False)
flexmock(module.feature).should_receive('available').and_return(False)
borg_exit_codes = flexmock()
flexmock(module).should_receive('make_list_command').with_args(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
list_arguments=list_arguments,
global_arguments=global_arguments,
local_path='borg',
remote_path=None,
).and_return(('borg', 'list', 'repo::archive'))
flexmock(module).should_receive('make_find_paths').and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
module.list_archive(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
list_arguments=list_arguments,
global_arguments=global_arguments,
)
def test_list_archive_calls_borg_multiple_times_with_find_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -430,6 +479,7 @@ def test_list_archive_calls_borg_multiple_times_with_find_paths():
('borg', 'list', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('archive1\narchive2').once()
flexmock(module).should_receive('make_list_command').and_return(
('borg', 'list', 'repo::archive1')
@@ -440,12 +490,14 @@ def test_list_archive_calls_borg_multiple_times_with_find_paths():
('borg', 'list', 'repo::archive1') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', 'repo::archive2') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -491,6 +543,7 @@ def test_list_archive_calls_borg_with_archive():
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -611,6 +664,7 @@ def test_list_archive_with_archive_ignores_archive_filter_flag(
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -669,6 +723,7 @@ def test_list_archive_with_find_paths_allows_archive_filter_flag_but_only_passes
('borg', 'rlist', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('archive1\narchive2').once()
flexmock(module).should_receive('make_list_command').with_args(
@@ -715,12 +770,14 @@ def test_list_archive_with_find_paths_allows_archive_filter_flag_but_only_passes
('borg', 'list', '--repo', 'repo', 'archive1') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', '--repo', 'repo', 'archive2') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
+46 -2
View File
@@ -7,11 +7,12 @@ from borgmatic.borg import mount as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -93,6 +94,47 @@ def test_mount_archive_calls_borg_with_path_flags():
)
def test_mount_archive_calls_borg_with_local_path():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
insert_execute_command_mock(('borg1', 'mount', 'repo::archive', '/mnt'))
mount_arguments = flexmock(mount_point='/mnt', options=None, paths=None, foreground=False)
module.mount_archive(
repository_path='repo',
archive='archive',
mount_arguments=mount_arguments,
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_mount_archive_calls_borg_using_exit_codes():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'mount', 'repo::archive', '/mnt'),
borg_exit_codes=borg_exit_codes,
)
mount_arguments = flexmock(mount_point='/mnt', options=None, paths=None, foreground=False)
module.mount_archive(
repository_path='repo',
archive='archive',
mount_arguments=mount_arguments,
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_mount_archive_calls_borg_with_remote_path_flags():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -216,6 +258,7 @@ def test_mount_archive_calls_borg_with_foreground_parameter():
('borg', 'mount', '--foreground', 'repo::archive', '/mnt'),
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -288,6 +331,7 @@ def test_mount_archive_with_date_based_matching_calls_borg_with_date_based_flags
'/mnt',
),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+24 -1
View File
@@ -7,12 +7,13 @@ from borgmatic.borg import prune as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(prune_command, output_log_level):
def insert_execute_command_mock(prune_command, output_log_level, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
prune_command,
output_log_level=output_log_level,
borg_local_path=prune_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -227,6 +228,27 @@ def test_prune_archives_with_local_path_calls_borg_via_local_path():
)
def test_prune_archives_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('make_prune_flags').and_return(BASE_PRUNE_FLAGS)
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg',) + PRUNE_COMMAND[1:] + ('repo',), logging.INFO, borg_exit_codes
)
prune_arguments = flexmock(stats=False, list_archives=False)
module.prune_archives(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
prune_arguments=prune_arguments,
)
def test_prune_archives_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -403,6 +425,7 @@ def test_prune_archives_with_date_based_matching_calls_borg_with_date_based_flag
),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+51 -3
View File
@@ -13,21 +13,24 @@ RCREATE_COMMAND = ('borg', 'rcreate', '--encryption', 'repokey')
def insert_rinfo_command_found_mock():
flexmock(module.rinfo).should_receive('display_repository_info')
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
'{"encryption": {"mode": "repokey"}}'
)
def insert_rinfo_command_not_found_mock():
flexmock(module.rinfo).should_receive('display_repository_info').and_raise(
subprocess.CalledProcessError(module.RINFO_REPOSITORY_NOT_FOUND_EXIT_CODE, [])
subprocess.CalledProcessError(sorted(module.RINFO_REPOSITORY_NOT_FOUND_EXIT_CODES)[0], [])
)
def insert_rcreate_command_mock(rcreate_command, **kwargs):
def insert_rcreate_command_mock(rcreate_command, borg_exit_codes=None, **kwargs):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
rcreate_command,
output_file=module.DO_NOT_CAPTURE,
borg_local_path=rcreate_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -119,6 +122,27 @@ def test_create_repository_skips_creation_when_repository_already_exists():
)
def test_create_repository_errors_when_repository_with_differing_encryption_mode_already_exists():
insert_rinfo_command_found_mock()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_flags').and_return(
(
'--repo',
'repo',
)
)
with pytest.raises(ValueError):
module.create_repository(
dry_run=False,
repository_path='repo',
config={},
local_borg_version='2.3.4',
global_arguments=flexmock(log_json=False),
encryption_mode='repokey-blake2',
)
def test_create_repository_raises_for_unknown_rinfo_command_error():
flexmock(module.rinfo).should_receive('display_repository_info').and_raise(
subprocess.CalledProcessError(RINFO_SOME_UNKNOWN_EXIT_CODE, [])
@@ -353,6 +377,30 @@ def test_create_repository_with_local_path_calls_borg_via_local_path():
)
def test_create_repository_with_exit_codes_calls_borg_using_them():
borg_exit_codes = flexmock()
insert_rinfo_command_not_found_mock()
insert_rcreate_command_mock(
('borg',) + RCREATE_COMMAND[1:] + ('--repo', 'repo'), borg_exit_codes=borg_exit_codes
)
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_flags').and_return(
(
'--repo',
'repo',
)
)
module.create_repository(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
global_arguments=flexmock(log_json=False),
encryption_mode='repokey',
)
def test_create_repository_with_remote_path_calls_borg_with_remote_path_flag():
insert_rinfo_command_not_found_mock()
insert_rcreate_command_mock(RCREATE_COMMAND + ('--remote-path', 'borg1', '--repo', 'repo'))
+55
View File
@@ -21,6 +21,7 @@ def test_display_repository_info_calls_borg_with_flags():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -28,6 +29,7 @@ def test_display_repository_info_calls_borg_with_flags():
('borg', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -49,6 +51,7 @@ def test_display_repository_info_without_borg_features_calls_borg_with_info_sub_
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -56,6 +59,7 @@ def test_display_repository_info_without_borg_features_calls_borg_with_info_sub_
('borg', 'info', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -82,6 +86,7 @@ def test_display_repository_info_with_log_info_calls_borg_with_info_flag():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--info', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -89,6 +94,7 @@ def test_display_repository_info_with_log_info_calls_borg_with_info_flag():
('borg', 'rinfo', '--info', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.INFO)
@@ -116,6 +122,7 @@ def test_display_repository_info_with_log_info_and_json_suppresses_most_borg_out
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -145,6 +152,7 @@ def test_display_repository_info_with_log_debug_calls_borg_with_debug_flag():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--debug', '--show-rc', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -152,6 +160,7 @@ def test_display_repository_info_with_log_debug_calls_borg_with_debug_flag():
('borg', 'rinfo', '--debug', '--show-rc', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.DEBUG)
@@ -180,6 +189,7 @@ def test_display_repository_info_with_log_debug_and_json_suppresses_most_borg_ou
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -210,6 +220,7 @@ def test_display_repository_info_with_json_calls_borg_with_json_flag():
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -239,12 +250,14 @@ def test_display_repository_info_with_local_path_calls_borg_via_local_path():
('borg1', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg1', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg1',
borg_exit_codes=None,
extra_environment=None,
)
@@ -258,6 +271,42 @@ def test_display_repository_info_with_local_path_calls_borg_via_local_path():
)
def test_display_repository_info_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_flags').and_return(
(
'--repo',
'repo',
)
)
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
)
module.display_repository_info(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
rinfo_arguments=flexmock(json=False),
global_arguments=flexmock(log_json=False),
)
def test_display_repository_info_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -273,12 +322,14 @@ def test_display_repository_info_with_remote_path_calls_borg_with_remote_path_fl
('borg', 'rinfo', '--remote-path', 'borg1', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--remote-path', 'borg1', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -307,12 +358,14 @@ def test_display_repository_info_with_log_json_calls_borg_with_log_json_flags():
('borg', 'rinfo', '--log-json', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--log-json', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -341,12 +394,14 @@ def test_display_repository_info_with_lock_wait_calls_borg_with_lock_wait_flags(
('borg', 'rinfo', '--lock-wait', '5', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--lock-wait', '5', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+31
View File
@@ -38,6 +38,7 @@ def test_resolve_archive_name_calls_borg_with_flags():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -59,6 +60,7 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
insert_logging_mock(logging.INFO)
@@ -81,6 +83,7 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
insert_logging_mock(logging.DEBUG)
@@ -103,6 +106,7 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
('borg1', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg1',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -118,6 +122,29 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
)
def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
expected_archive = 'archive-name'
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
).and_return(expected_archive + '\n')
assert (
module.resolve_archive_name(
'repo',
'latest',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
== expected_archive
)
def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags():
expected_archive = 'archive-name'
flexmock(module.environment).should_receive('make_environment')
@@ -125,6 +152,7 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
('borg', 'list', '--remote-path', 'borg1') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -146,6 +174,7 @@ def test_resolve_archive_name_without_archives_raises():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('')
with pytest.raises(ValueError):
@@ -166,6 +195,7 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
('borg', 'list', '--log-json') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -188,6 +218,7 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
('borg', 'list', '--lock-wait', 'okay') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
+45
View File
@@ -21,6 +21,7 @@ def test_transfer_archives_calls_borg_with_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -52,6 +53,7 @@ def test_transfer_archives_with_dry_run_calls_borg_with_dry_run_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -80,6 +82,7 @@ def test_transfer_archives_with_log_info_calls_borg_with_info_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.INFO)
@@ -108,6 +111,7 @@ def test_transfer_archives_with_log_debug_calls_borg_with_debug_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.DEBUG)
@@ -139,6 +143,7 @@ def test_transfer_archives_with_archive_calls_borg_with_match_archives_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -169,6 +174,7 @@ def test_transfer_archives_with_match_archives_calls_borg_with_match_archives_fl
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -199,6 +205,7 @@ def test_transfer_archives_with_archive_name_format_calls_borg_with_match_archiv
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -227,6 +234,7 @@ def test_transfer_archives_with_local_path_calls_borg_via_local_path():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg2',
borg_exit_codes=None,
extra_environment=None,
)
@@ -243,6 +251,36 @@ def test_transfer_archives_with_local_path_calls_borg_via_local_path():
)
def test_transfer_archives_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flexmock(module.flags).should_receive('make_flags_from_arguments').and_return(())
flexmock(module.flags).should_receive('make_repository_flags').and_return(('--repo', 'repo'))
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'transfer', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
)
module.transfer_archives(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
transfer_arguments=flexmock(
archive=None, progress=None, match_archives=None, source_repository=None
),
global_arguments=flexmock(log_json=False),
)
def test_transfer_archives_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -259,6 +297,7 @@ def test_transfer_archives_with_remote_path_calls_borg_with_remote_path_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -291,6 +330,7 @@ def test_transfer_archives_with_log_json_calls_borg_with_log_json_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -323,6 +363,7 @@ def test_transfer_archives_with_lock_wait_calls_borg_with_lock_wait_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -351,6 +392,7 @@ def test_transfer_archives_with_progress_calls_borg_with_progress_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -383,6 +425,7 @@ def test_transfer_archives_passes_through_arguments_to_borg(argument_name):
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -417,6 +460,7 @@ def test_transfer_archives_with_source_repository_calls_borg_with_other_repo_fla
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -459,6 +503,7 @@ def test_transfer_archives_with_date_based_matching_calls_borg_with_date_based_f
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+20 -5
View File
@@ -7,25 +7,40 @@ from borgmatic.borg import umount as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
flexmock(module).should_receive('execute_command').with_args(command).once()
def insert_execute_command_mock(command, borg_local_path='borg', borg_exit_codes=None):
flexmock(module).should_receive('execute_command').with_args(
command, borg_local_path=borg_local_path, borg_exit_codes=borg_exit_codes
).once()
def test_unmount_archive_calls_borg_with_required_parameters():
insert_execute_command_mock(('borg', 'umount', '/mnt'))
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_with_log_info_calls_borg_with_info_parameter():
insert_execute_command_mock(('borg', 'umount', '--info', '/mnt'))
insert_logging_mock(logging.INFO)
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_with_log_debug_calls_borg_with_debug_parameters():
insert_execute_command_mock(('borg', 'umount', '--debug', '--show-rc', '/mnt'))
insert_logging_mock(logging.DEBUG)
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_calls_borg_with_local_path():
insert_execute_command_mock(('borg1', 'umount', '/mnt'), borg_local_path='borg1')
module.unmount_archive(config={}, mount_point='/mnt', local_path='borg1')
def test_unmount_archive_calls_borg_with_exit_codes():
borg_exit_codes = flexmock()
insert_execute_command_mock(('borg', 'umount', '/mnt'), borg_exit_codes=borg_exit_codes)
module.unmount_archive(config={'borg_exit_codes': borg_exit_codes}, mount_point='/mnt')
+12 -1
View File
@@ -11,13 +11,14 @@ VERSION = '1.2.3'
def insert_execute_command_and_capture_output_mock(
command, borg_local_path='borg', version_output=f'borg {VERSION}'
command, borg_local_path='borg', borg_exit_codes=None, version_output=f'borg {VERSION}'
):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
command,
extra_environment=None,
borg_local_path=borg_local_path,
borg_exit_codes=borg_exit_codes,
).once().and_return(version_output)
@@ -51,6 +52,16 @@ def test_local_borg_version_with_local_borg_path_calls_borg_with_it():
assert module.local_borg_version({}, 'borg1') == VERSION
def test_local_borg_version_with_borg_exit_codes_calls_using_with_them():
borg_exit_codes = flexmock()
insert_execute_command_and_capture_output_mock(
('borg', '--version'), borg_exit_codes=borg_exit_codes
)
flexmock(module.environment).should_receive('make_environment')
assert module.local_borg_version({'borg_exit_codes': borg_exit_codes}) == VERSION
def test_local_borg_version_with_invalid_version_raises():
insert_execute_command_and_capture_output_mock(('borg', '--version'), version_output='wtf')
flexmock(module.environment).should_receive('make_environment')
+96 -40
View File
@@ -38,7 +38,7 @@ def test_run_configuration_runs_actions_for_each_repository():
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
arguments = {'global': flexmock(monitoring_verbosity=1)}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -51,7 +51,7 @@ def test_run_configuration_with_skip_actions_does_not_raise():
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}], 'skip_actions': ['compact']}
arguments = {'global': flexmock(monitoring_verbosity=1)}
list(module.run_configuration('test.yaml', config, arguments))
list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
def test_run_configuration_with_invalid_borg_version_errors():
@@ -64,7 +64,7 @@ def test_run_configuration_with_invalid_borg_version_errors():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'prune': flexmock()}
list(module.run_configuration('test.yaml', config, arguments))
list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
def test_run_configuration_logs_monitor_start_error():
@@ -80,7 +80,7 @@ def test_run_configuration_logs_monitor_start_error():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -96,7 +96,7 @@ def test_run_configuration_bails_for_monitor_start_soft_failure():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -113,7 +113,7 @@ def test_run_configuration_logs_actions_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False)}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -130,7 +130,7 @@ def test_run_configuration_bails_for_actions_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -148,7 +148,7 @@ def test_run_configuration_logs_monitor_log_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -167,7 +167,7 @@ def test_run_configuration_bails_for_monitor_log_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -185,7 +185,7 @@ def test_run_configuration_logs_monitor_finish_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -204,7 +204,7 @@ def test_run_configuration_bails_for_monitor_finish_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -219,7 +219,7 @@ def test_run_configuration_does_not_call_monitoring_hooks_if_monitoring_hooks_ar
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=-2, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -236,7 +236,7 @@ def test_run_configuration_logs_on_error_hook_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -253,7 +253,7 @@ def test_run_configuration_bails_for_on_error_hook_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -268,7 +268,7 @@ def test_run_configuration_retries_soft_error():
flexmock(module).should_receive('log_error_records').and_return([flexmock()]).once()
config = {'repositories': [{'path': 'foo'}], 'retries': 1}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
@@ -292,7 +292,7 @@ def test_run_configuration_retries_hard_error():
).and_return(error_logs)
config = {'repositories': [{'path': 'foo'}], 'retries': 1}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
@@ -311,7 +311,7 @@ def test_run_configuration_repos_ordered():
).and_return(expected_results[1:]).ordered()
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -346,7 +346,7 @@ def test_run_configuration_retries_round_robin():
'retries': 1,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == foo_error_logs + bar_error_logs
@@ -379,7 +379,7 @@ def test_run_configuration_retries_one_passes():
'retries': 1,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
@@ -423,7 +423,7 @@ def test_run_configuration_retry_wait():
'retry_wait': 10,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
@@ -463,7 +463,7 @@ def test_run_configuration_retries_timeout_multiple_repos():
'retry_wait': 10,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
@@ -478,6 +478,7 @@ def test_run_actions_runs_rcreate():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rcreate': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -495,6 +496,7 @@ def test_run_actions_adds_log_file_to_hook_context():
config_filename=object,
repository={'path': 'repo'},
config={'repositories': []},
config_paths=[],
hook_context={'repository': 'repo', 'repositories': '', 'log_file': 'foo'},
local_borg_version=object,
create_arguments=object,
@@ -509,6 +511,7 @@ def test_run_actions_adds_log_file_to_hook_context():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -529,6 +532,7 @@ def test_run_actions_runs_transfer():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'transfer': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -549,6 +553,7 @@ def test_run_actions_runs_create():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -569,6 +574,7 @@ def test_run_actions_with_skip_actions_skips_create():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['create']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -588,6 +594,7 @@ def test_run_actions_runs_prune():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'prune': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -607,6 +614,7 @@ def test_run_actions_with_skip_actions_skips_prune():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'prune': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['prune']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -626,6 +634,7 @@ def test_run_actions_runs_compact():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'compact': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -645,6 +654,7 @@ def test_run_actions_with_skip_actions_skips_compact():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'compact': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['compact']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -665,6 +675,7 @@ def test_run_actions_runs_check_when_repository_enabled_for_checks():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -685,6 +696,7 @@ def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -705,6 +717,7 @@ def test_run_actions_with_skip_actions_skips_check():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['check']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -724,6 +737,7 @@ def test_run_actions_runs_extract():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'extract': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -743,6 +757,7 @@ def test_run_actions_runs_export_tar():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'export-tar': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -762,6 +777,7 @@ def test_run_actions_runs_mount():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'mount': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -781,6 +797,7 @@ def test_run_actions_runs_restore():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'restore': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -801,6 +818,7 @@ def test_run_actions_runs_rlist():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rlist': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -822,6 +840,7 @@ def test_run_actions_runs_list():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'list': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -843,6 +862,7 @@ def test_run_actions_runs_rinfo():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rinfo': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -864,6 +884,7 @@ def test_run_actions_runs_info():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'info': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -884,6 +905,7 @@ def test_run_actions_runs_break_lock():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'break-lock': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -903,6 +925,7 @@ def test_run_actions_runs_export_key():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'export': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -922,6 +945,7 @@ def test_run_actions_runs_borg():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'borg': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -946,6 +970,7 @@ def test_run_actions_runs_multiple_actions_in_argument_order():
},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -960,30 +985,33 @@ def test_load_configurations_collects_parsed_configurations_and_logs():
test_expected_logs = [flexmock(), flexmock()]
other_expected_logs = [flexmock(), flexmock()]
flexmock(module.validate).should_receive('parse_configuration').and_return(
configuration, test_expected_logs
).and_return(other_configuration, other_expected_logs)
configuration, ['/tmp/test.yaml'], test_expected_logs
).and_return(other_configuration, ['/tmp/other.yaml'], other_expected_logs)
configs, logs = tuple(module.load_configurations(('test.yaml', 'other.yaml')))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml', 'other.yaml')))
assert configs == {'test.yaml': configuration, 'other.yaml': other_configuration}
assert config_paths == ['/tmp/other.yaml', '/tmp/test.yaml']
assert set(logs) >= set(test_expected_logs + other_expected_logs)
def test_load_configurations_logs_warning_for_permission_error():
flexmock(module.validate).should_receive('parse_configuration').and_raise(PermissionError)
configs, logs = tuple(module.load_configurations(('test.yaml',)))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml',)))
assert configs == {}
assert config_paths == []
assert max(log.levelno for log in logs) == logging.WARNING
def test_load_configurations_logs_critical_for_parse_error():
flexmock(module.validate).should_receive('parse_configuration').and_raise(ValueError)
configs, logs = tuple(module.load_configurations(('test.yaml',)))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml',)))
assert configs == {}
assert config_paths == []
assert max(log.levelno for log in logs) == logging.CRITICAL
@@ -1214,7 +1242,9 @@ def test_collect_configuration_run_summary_logs_info_for_success():
arguments = {}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1226,7 +1256,9 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1239,7 +1271,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_extract():
arguments = {'extract': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1254,7 +1288,9 @@ def test_collect_configuration_run_summary_logs_extract_with_repository_error():
arguments = {'extract': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1267,7 +1303,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_mount():
arguments = {'mount': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1282,7 +1320,9 @@ def test_collect_configuration_run_summary_logs_mount_with_repository_error():
arguments = {'mount': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1293,7 +1333,9 @@ def test_collect_configuration_run_summary_logs_missing_configs_error():
expected_logs = (flexmock(),)
flexmock(module).should_receive('log_error_records').and_return(expected_logs)
logs = tuple(module.collect_configuration_run_summary_logs({}, arguments=arguments))
logs = tuple(
module.collect_configuration_run_summary_logs({}, config_paths=[], arguments=arguments)
)
assert logs == expected_logs
@@ -1305,7 +1347,9 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1320,7 +1364,9 @@ def test_collect_configuration_run_summary_logs_post_hook_error():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert expected_logs[0] in logs
@@ -1335,7 +1381,9 @@ def test_collect_configuration_run_summary_logs_for_list_with_archive_and_reposi
arguments = {'list': flexmock(repository='repo', archive='test')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1347,7 +1395,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_list():
arguments = {'list': flexmock(repository='repo', archive=None)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1362,7 +1412,9 @@ def test_collect_configuration_run_summary_logs_run_configuration_error():
arguments = {}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.CRITICAL}
@@ -1378,7 +1430,9 @@ def test_collect_configuration_run_summary_logs_run_umount_error():
arguments = {'umount': flexmock(mount_point='/mnt')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO, logging.CRITICAL}
@@ -1396,6 +1450,8 @@ def test_collect_configuration_run_summary_logs_outputs_merged_json_results():
tuple(
module.collect_configuration_run_summary_logs(
{'test.yaml': {}, 'test2.yaml': {}}, arguments=arguments
{'test.yaml': {}, 'test2.yaml': {}},
config_paths=['/tmp/test.yaml', '/tmp/test2.yaml'],
arguments=arguments,
)
)
+11 -1
View File
@@ -46,6 +46,10 @@ def test_apply_constants_with_empty_constants_passes_through_value():
(['{foo}', '{baz}'], ['bar', 'quux']),
({'key': 'value'}, {'key': 'value'}),
({'key': '{foo}'}, {'key': 'bar'}),
({'key': '{inject}'}, {'key': 'echo hi; naughty-command'}),
({'before_backup': '{inject}'}, {'before_backup': "'echo hi; naughty-command'"}),
({'after_backup': '{inject}'}, {'after_backup': "'echo hi; naughty-command'"}),
({'on_error': '{inject}'}, {'on_error': "'echo hi; naughty-command'"}),
(3, 3),
(True, True),
(False, False),
@@ -53,6 +57,12 @@ def test_apply_constants_with_empty_constants_passes_through_value():
)
def test_apply_constants_makes_string_substitutions(value, expected_value):
flexmock(module).should_receive('coerce_scalar').replace_with(lambda value: value)
constants = {'foo': 'bar', 'baz': 'quux', 'int': 3, 'bool': True}
constants = {
'foo': 'bar',
'baz': 'quux',
'int': 3,
'bool': True,
'inject': 'echo hi; naughty-command',
}
assert module.apply_constants(value, constants) == expected_value
+19 -11
View File
@@ -6,27 +6,35 @@ from borgmatic.config import load as module
def test_probe_and_include_file_with_absolute_path_skips_probing():
config = flexmock()
flexmock(module).should_receive('load_configuration').with_args('/etc/include.yaml').and_return(
config
).once()
config_paths = set()
flexmock(module).should_receive('load_configuration').with_args(
'/etc/include.yaml', config_paths
).and_return(config).once()
assert module.probe_and_include_file('/etc/include.yaml', ['/etc', '/var']) == config
assert (
module.probe_and_include_file('/etc/include.yaml', ['/etc', '/var'], config_paths) == config
)
def test_probe_and_include_file_with_relative_path_probes_include_directories():
config = flexmock()
config = {'foo': 'bar'}
config_paths = set()
flexmock(module.os.path).should_receive('exists').with_args('/etc/include.yaml').and_return(
False
)
flexmock(module.os.path).should_receive('exists').with_args('/var/include.yaml').and_return(
True
)
flexmock(module).should_receive('load_configuration').with_args('/etc/include.yaml').never()
flexmock(module).should_receive('load_configuration').with_args('/var/include.yaml').and_return(
config
).once()
flexmock(module).should_receive('load_configuration').with_args(
'/etc/include.yaml', config_paths
).never()
flexmock(module).should_receive('load_configuration').with_args(
'/var/include.yaml', config_paths
).and_return(config).once()
assert module.probe_and_include_file('include.yaml', ['/etc', '/var']) == config
assert module.probe_and_include_file('include.yaml', ['/etc', '/var'], config_paths) == {
'foo': 'bar',
}
def test_probe_and_include_file_with_relative_path_and_missing_files_raises():
@@ -34,4 +42,4 @@ def test_probe_and_include_file_with_relative_path_and_missing_files_raises():
flexmock(module).should_receive('load_configuration').never()
with pytest.raises(FileNotFoundError):
module.probe_and_include_file('include.yaml', ['/etc', '/var'])
module.probe_and_include_file('include.yaml', ['/etc', '/var'], config_paths=set())
+7
View File
@@ -44,6 +44,13 @@ def test_set_values_with_multiple_keys_updates_hierarchy():
assert config == {'option': {'key': 'value', 'other': 'other_value'}}
def test_set_values_with_key_when_list_index_expected_errors():
config = {'option': ['foo', 'bar', 'baz']}
with pytest.raises(ValueError):
module.set_values(config, keys=('option', 'key'), value='value')
@pytest.mark.parametrize(
'schema,option_keys,expected_type',
(
-15
View File
@@ -184,21 +184,6 @@ def test_guard_configuration_contains_repository_errors_when_repository_missing_
)
def test_guard_configuration_contains_repository_errors_when_repository_matches_config_twice():
flexmock(module).should_receive('repositories_match').replace_with(
lambda first, second: first == second
)
with pytest.raises(ValueError):
module.guard_configuration_contains_repository(
repository='repo',
configurations={
'config.yaml': {'repositories': ['repo', 'repo2']},
'other.yaml': {'repositories': ['repo']},
},
)
def test_guard_single_repository_selected_raises_when_multiple_repositories_configured_and_none_selected():
with pytest.raises(ValueError):
module.guard_single_repository_selected(
+159 -7
View File
@@ -13,10 +13,63 @@ def mock_apprise():
add=lambda servers: None, notify=lambda title, body, body_format, notify_type: None
)
flexmock(apprise.Apprise).new_instances(apprise_mock)
return apprise_mock
def test_ping_monitor_adheres_dry_run():
def test_initialize_monitor_with_send_logs_false_does_not_add_handler():
flexmock(module.borgmatic.hooks.logs).should_receive('add_handler').never()
module.initialize_monitor(
hook_config={'send_logs': False},
config={},
config_filename='test.yaml',
monitoring_log_level=1,
dry_run=False,
)
def test_initialize_monitor_with_send_logs_true_adds_handler_with_default_log_size_limit():
truncation_indicator_length = 4
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').with_args(
module.HANDLER_IDENTIFIER,
module.DEFAULT_LOGS_SIZE_LIMIT_BYTES - truncation_indicator_length,
1,
).once()
flexmock(module.borgmatic.hooks.logs).should_receive('add_handler').once()
module.initialize_monitor(
hook_config={'send_logs': True},
config={},
config_filename='test.yaml',
monitoring_log_level=1,
dry_run=False,
)
def test_initialize_monitor_without_send_logs_adds_handler_with_default_log_size_limit():
truncation_indicator_length = 4
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').with_args(
module.HANDLER_IDENTIFIER,
module.DEFAULT_LOGS_SIZE_LIMIT_BYTES - truncation_indicator_length,
1,
).once()
flexmock(module.borgmatic.hooks.logs).should_receive('add_handler').once()
module.initialize_monitor(
hook_config={},
config={},
config_filename='test.yaml',
monitoring_log_level=1,
dry_run=False,
)
def test_ping_monitor_respects_dry_run():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('loggy log')
mock_apprise().should_receive('notify').never()
module.ping_monitor(
@@ -29,7 +82,9 @@ def test_ping_monitor_adheres_dry_run():
)
def test_ping_monitor_does_not_hit_with_no_states():
def test_ping_monitor_with_no_states_does_not_notify():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler').never()
flexmock(module.borgmatic.hooks.logs).should_receive('format_buffered_logs_for_payload').never()
mock_apprise().should_receive('notify').never()
module.ping_monitor(
@@ -42,7 +97,11 @@ def test_ping_monitor_does_not_hit_with_no_states():
)
def test_ping_monitor_hits_fail_by_default():
def test_ping_monitor_notifies_fail_by_default():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='A borgmatic FAIL event happened',
body='A borgmatic FAIL event happened',
@@ -61,7 +120,34 @@ def test_ping_monitor_hits_fail_by_default():
)
def test_ping_monitor_hits_with_finish_default_config():
def test_ping_monitor_with_logs_appends_logs_to_body():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('loggy log')
mock_apprise().should_receive('notify').with_args(
title='A borgmatic FAIL event happened',
body='A borgmatic FAIL event happened\n\nloggy log',
body_format=NotifyFormat.TEXT,
notify_type=NotifyType.FAILURE,
).once()
for state in borgmatic.hooks.monitor.State:
module.ping_monitor(
{'services': [{'url': f'ntfys://{TOPIC}', 'label': 'ntfys'}]},
{},
'config.yaml',
state,
monitoring_log_level=1,
dry_run=False,
)
def test_ping_monitor_with_finish_default_config_notifies():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='A borgmatic FINISH event happened',
body='A borgmatic FINISH event happened',
@@ -79,7 +165,9 @@ def test_ping_monitor_hits_with_finish_default_config():
)
def test_ping_monitor_hits_with_start_default_config():
def test_ping_monitor_with_start_default_config_notifies():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler').never()
flexmock(module.borgmatic.hooks.logs).should_receive('format_buffered_logs_for_payload').never()
mock_apprise().should_receive('notify').with_args(
title='A borgmatic START event happened',
body='A borgmatic START event happened',
@@ -97,7 +185,11 @@ def test_ping_monitor_hits_with_start_default_config():
)
def test_ping_monitor_hits_with_fail_default_config():
def test_ping_monitor_with_fail_default_config_notifies():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='A borgmatic FAIL event happened',
body='A borgmatic FAIL event happened',
@@ -115,7 +207,11 @@ def test_ping_monitor_hits_with_fail_default_config():
)
def test_ping_monitor_hits_with_log_default_config():
def test_ping_monitor_with_log_default_config_notifies():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='A borgmatic LOG event happened',
body='A borgmatic LOG event happened',
@@ -134,6 +230,10 @@ def test_ping_monitor_hits_with_log_default_config():
def test_ping_monitor_passes_through_custom_message_title():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='foo',
body='bar',
@@ -156,6 +256,10 @@ def test_ping_monitor_passes_through_custom_message_title():
def test_ping_monitor_passes_through_custom_message_body():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').with_args(
title='',
body='baz',
@@ -177,7 +281,37 @@ def test_ping_monitor_passes_through_custom_message_body():
)
def test_ping_monitor_passes_through_custom_message_body_and_appends_logs():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('loggy log')
mock_apprise().should_receive('notify').with_args(
title='',
body='baz\n\nloggy log',
body_format=NotifyFormat.TEXT,
notify_type=NotifyType.FAILURE,
).once()
module.ping_monitor(
{
'services': [{'url': f'ntfys://{TOPIC}', 'label': 'ntfys'}],
'states': ['fail'],
'fail': {'body': 'baz'},
},
{},
'config.yaml',
borgmatic.hooks.monitor.State.FAIL,
monitoring_log_level=1,
dry_run=False,
)
def test_ping_monitor_pings_multiple_services():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('add').with_args([f'ntfys://{TOPIC}', f'ntfy://{TOPIC}']).once()
module.ping_monitor(
@@ -196,6 +330,8 @@ def test_ping_monitor_pings_multiple_services():
def test_ping_monitor_logs_info_for_no_services():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler').never()
flexmock(module.borgmatic.hooks.logs).should_receive('format_buffered_logs_for_payload').never()
flexmock(module.logger).should_receive('info').once()
module.ping_monitor(
@@ -209,6 +345,10 @@ def test_ping_monitor_logs_info_for_no_services():
def test_ping_monitor_logs_warning_when_notify_fails():
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return('')
mock_apprise().should_receive('notify').and_return(False)
flexmock(module.logger).should_receive('warning').once()
@@ -221,3 +361,15 @@ def test_ping_monitor_logs_warning_when_notify_fails():
monitoring_log_level=1,
dry_run=False,
)
def test_destroy_monitor_does_not_raise():
flexmock(module.borgmatic.hooks.logs).should_receive('remove_handler')
module.destroy_monitor(
hook_config={},
config={},
config_filename='test.yaml',
monitoring_log_level=1,
dry_run=False,
)
+10
View File
@@ -25,6 +25,16 @@ def test_interpolate_context_interpolates_variables():
)
def test_interpolate_context_escapes_interpolated_variables():
command = 'ls {foo} {inject}' # noqa: FS003
context = {'foo': 'bar', 'inject': 'hi; naughty-command'}
assert (
module.interpolate_context('test.yaml', 'pre-backup', command, context)
== "ls bar 'hi; naughty-command'"
)
def test_execute_hook_invokes_each_command():
flexmock(module).should_receive('interpolate_context').replace_with(
lambda config_file, hook_description, command, context: command
+42 -86
View File
@@ -3,72 +3,6 @@ from flexmock import flexmock
from borgmatic.hooks import healthchecks as module
def test_forgetful_buffering_handler_emit_collects_log_records():
handler = module.Forgetful_buffering_handler(byte_capacity=100, log_level=1)
handler.emit(flexmock(getMessage=lambda: 'foo'))
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
assert not handler.forgot
def test_forgetful_buffering_handler_emit_collects_log_records_with_zero_byte_capacity():
handler = module.Forgetful_buffering_handler(byte_capacity=0, log_level=1)
handler.emit(flexmock(getMessage=lambda: 'foo'))
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
assert not handler.forgot
def test_forgetful_buffering_handler_emit_forgets_log_records_when_capacity_reached():
handler = module.Forgetful_buffering_handler(byte_capacity=len('foo\nbar\n'), log_level=1)
handler.emit(flexmock(getMessage=lambda: 'foo'))
assert handler.buffer == ['foo\n']
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
handler.emit(flexmock(getMessage=lambda: 'baz'))
assert handler.buffer == ['bar\n', 'baz\n']
handler.emit(flexmock(getMessage=lambda: 'quux'))
assert handler.buffer == ['quux\n']
assert handler.forgot
def test_format_buffered_logs_for_payload_flattens_log_buffer():
handler = module.Forgetful_buffering_handler(byte_capacity=100, log_level=1)
handler.buffer = ['foo\n', 'bar\n']
logger = flexmock(handlers=[handler])
logger.should_receive('removeHandler')
flexmock(module.logging).should_receive('getLogger').and_return(logger)
payload = module.format_buffered_logs_for_payload()
assert payload == 'foo\nbar\n'
def test_format_buffered_logs_for_payload_inserts_truncation_indicator_when_logs_forgotten():
handler = module.Forgetful_buffering_handler(byte_capacity=100, log_level=1)
handler.buffer = ['foo\n', 'bar\n']
handler.forgot = True
logger = flexmock(handlers=[handler])
logger.should_receive('removeHandler')
flexmock(module.logging).should_receive('getLogger').and_return(logger)
payload = module.format_buffered_logs_for_payload()
assert payload == '...\nfoo\nbar\n'
def test_format_buffered_logs_for_payload_without_handler_produces_empty_payload():
logger = flexmock(handlers=[module.logging.Handler()])
logger.should_receive('removeHandler')
flexmock(module.logging).should_receive('getLogger').and_return(logger)
payload = module.format_buffered_logs_for_payload()
assert payload == ''
def mock_logger():
logger = flexmock()
logger.should_receive('addHandler')
@@ -81,8 +15,10 @@ def test_initialize_monitor_creates_log_handler_with_ping_body_limit():
monitoring_log_level = 1
mock_logger()
flexmock(module).should_receive('Forgetful_buffering_handler').with_args(
ping_body_limit - len(module.PAYLOAD_TRUNCATION_INDICATOR), monitoring_log_level
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').with_args(
module.HANDLER_IDENTIFIER,
ping_body_limit - len(module.borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
monitoring_log_level,
).once()
module.initialize_monitor(
@@ -94,8 +30,10 @@ def test_initialize_monitor_creates_log_handler_with_default_ping_body_limit():
monitoring_log_level = 1
mock_logger()
flexmock(module).should_receive('Forgetful_buffering_handler').with_args(
module.DEFAULT_PING_BODY_LIMIT_BYTES - len(module.PAYLOAD_TRUNCATION_INDICATOR),
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').with_args(
module.HANDLER_IDENTIFIER,
module.DEFAULT_PING_BODY_LIMIT_BYTES
- len(module.borgmatic.hooks.logs.PAYLOAD_TRUNCATION_INDICATOR),
monitoring_log_level,
).once()
@@ -107,8 +45,8 @@ def test_initialize_monitor_creates_log_handler_with_zero_ping_body_limit():
monitoring_log_level = 1
mock_logger()
flexmock(module).should_receive('Forgetful_buffering_handler').with_args(
ping_body_limit, monitoring_log_level
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').with_args(
module.HANDLER_IDENTIFIER, ping_body_limit, monitoring_log_level
).once()
module.initialize_monitor(
@@ -118,7 +56,7 @@ def test_initialize_monitor_creates_log_handler_with_zero_ping_body_limit():
def test_initialize_monitor_creates_log_handler_when_send_logs_true():
mock_logger()
flexmock(module).should_receive('Forgetful_buffering_handler').once()
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').once()
module.initialize_monitor(
{'send_logs': True}, {}, 'test.yaml', monitoring_log_level=1, dry_run=False
@@ -127,7 +65,7 @@ def test_initialize_monitor_creates_log_handler_when_send_logs_true():
def test_initialize_monitor_bails_when_send_logs_false():
mock_logger()
flexmock(module).should_receive('Forgetful_buffering_handler').never()
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
module.initialize_monitor(
{'send_logs': False}, {}, 'test.yaml', monitoring_log_level=1, dry_run=False
@@ -135,7 +73,7 @@ def test_initialize_monitor_bails_when_send_logs_false():
def test_ping_monitor_hits_ping_url_for_start_state():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com'}
flexmock(module.requests).should_receive('post').with_args(
'https://example.com/start', data=''.encode('utf-8'), verify=True
@@ -154,7 +92,10 @@ def test_ping_monitor_hits_ping_url_for_start_state():
def test_ping_monitor_hits_ping_url_for_finish_state():
hook_config = {'ping_url': 'https://example.com'}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://example.com', data=payload.encode('utf-8'), verify=True
).and_return(flexmock(ok=True))
@@ -172,7 +113,10 @@ def test_ping_monitor_hits_ping_url_for_finish_state():
def test_ping_monitor_hits_ping_url_for_fail_state():
hook_config = {'ping_url': 'https://example.com'}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://example.com/fail', data=payload.encode('utf'), verify=True
).and_return(flexmock(ok=True))
@@ -190,7 +134,10 @@ def test_ping_monitor_hits_ping_url_for_fail_state():
def test_ping_monitor_hits_ping_url_for_log_state():
hook_config = {'ping_url': 'https://example.com'}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://example.com/log', data=payload.encode('utf'), verify=True
).and_return(flexmock(ok=True))
@@ -208,7 +155,10 @@ def test_ping_monitor_hits_ping_url_for_log_state():
def test_ping_monitor_with_ping_uuid_hits_corresponding_url():
hook_config = {'ping_url': 'abcd-efgh-ijkl-mnop'}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
f"https://hc-ping.com/{hook_config['ping_url']}",
data=payload.encode('utf-8'),
@@ -228,7 +178,10 @@ def test_ping_monitor_with_ping_uuid_hits_corresponding_url():
def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
hook_config = {'ping_url': 'https://example.com', 'verify_tls': False}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://example.com', data=payload.encode('utf-8'), verify=False
).and_return(flexmock(ok=True))
@@ -246,7 +199,10 @@ def test_ping_monitor_skips_ssl_verification_when_verify_tls_false():
def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
hook_config = {'ping_url': 'https://example.com', 'verify_tls': True}
payload = 'data'
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.borgmatic.hooks.logs).should_receive('get_handler')
flexmock(module.borgmatic.hooks.logs).should_receive(
'format_buffered_logs_for_payload'
).and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://example.com', data=payload.encode('utf-8'), verify=True
).and_return(flexmock(ok=True))
@@ -262,7 +218,7 @@ def test_ping_monitor_executes_ssl_verification_when_verify_tls_true():
def test_ping_monitor_dry_run_does_not_hit_ping_url():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com'}
flexmock(module.requests).should_receive('post').never()
@@ -277,7 +233,7 @@ def test_ping_monitor_dry_run_does_not_hit_ping_url():
def test_ping_monitor_does_not_hit_ping_url_when_states_not_matching():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com', 'states': ['finish']}
flexmock(module.requests).should_receive('post').never()
@@ -292,7 +248,7 @@ def test_ping_monitor_does_not_hit_ping_url_when_states_not_matching():
def test_ping_monitor_hits_ping_url_when_states_matching():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com', 'states': ['start', 'finish']}
flexmock(module.requests).should_receive('post').with_args(
'https://example.com/start', data=''.encode('utf-8'), verify=True
@@ -309,7 +265,7 @@ def test_ping_monitor_hits_ping_url_when_states_matching():
def test_ping_monitor_with_connection_error_logs_warning():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com'}
flexmock(module.requests).should_receive('post').with_args(
'https://example.com/start', data=''.encode('utf-8'), verify=True
@@ -327,7 +283,7 @@ def test_ping_monitor_with_connection_error_logs_warning():
def test_ping_monitor_with_other_error_logs_warning():
flexmock(module).should_receive('Forgetful_buffering_handler')
flexmock(module.borgmatic.hooks.logs).should_receive('Forgetful_buffering_handler').never()
hook_config = {'ping_url': 'https://example.com'}
response = flexmock(ok=False)
response.should_receive('raise_for_status').and_raise(
+103
View File
@@ -0,0 +1,103 @@
import pytest
from flexmock import flexmock
from borgmatic.hooks import logs as module
def test_forgetful_buffering_handler_emit_collects_log_records():
handler = module.Forgetful_buffering_handler(identifier='test', byte_capacity=100, log_level=1)
handler.emit(flexmock(getMessage=lambda: 'foo'))
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
assert not handler.forgot
def test_forgetful_buffering_handler_emit_collects_log_records_with_zero_byte_capacity():
handler = module.Forgetful_buffering_handler(identifier='test', byte_capacity=0, log_level=1)
handler.emit(flexmock(getMessage=lambda: 'foo'))
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
assert not handler.forgot
def test_forgetful_buffering_handler_emit_forgets_log_records_when_capacity_reached():
handler = module.Forgetful_buffering_handler(
identifier='test', byte_capacity=len('foo\nbar\n'), log_level=1
)
handler.emit(flexmock(getMessage=lambda: 'foo'))
assert handler.buffer == ['foo\n']
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
handler.emit(flexmock(getMessage=lambda: 'baz'))
assert handler.buffer == ['bar\n', 'baz\n']
handler.emit(flexmock(getMessage=lambda: 'quux'))
assert handler.buffer == ['quux\n']
assert handler.forgot
def test_get_handler_matches_by_identifier():
handlers = [
flexmock(),
flexmock(),
module.Forgetful_buffering_handler(identifier='other', byte_capacity=100, log_level=1),
module.Forgetful_buffering_handler(identifier='test', byte_capacity=100, log_level=1),
flexmock(),
]
flexmock(module.logging.getLogger(), handlers=handlers)
assert module.get_handler('test') == handlers[3]
def test_get_handler_without_match_raises():
handlers = [
flexmock(),
module.Forgetful_buffering_handler(identifier='other', byte_capacity=100, log_level=1),
]
flexmock(module.logging.getLogger(), handlers=handlers)
with pytest.raises(ValueError):
assert module.get_handler('test')
def test_format_buffered_logs_for_payload_flattens_log_buffer():
handler = module.Forgetful_buffering_handler(identifier='test', byte_capacity=100, log_level=1)
handler.buffer = ['foo\n', 'bar\n']
flexmock(module).should_receive('get_handler').and_return(handler)
payload = module.format_buffered_logs_for_payload(identifier='test')
assert payload == 'foo\nbar\n'
def test_format_buffered_logs_for_payload_inserts_truncation_indicator_when_logs_forgotten():
handler = module.Forgetful_buffering_handler(identifier='test', byte_capacity=100, log_level=1)
handler.buffer = ['foo\n', 'bar\n']
handler.forgot = True
flexmock(module).should_receive('get_handler').and_return(handler)
payload = module.format_buffered_logs_for_payload(identifier='test')
assert payload == '...\nfoo\nbar\n'
def test_format_buffered_logs_for_payload_without_handler_produces_empty_payload():
flexmock(module).should_receive('get_handler').and_raise(ValueError)
payload = module.format_buffered_logs_for_payload(identifier='test')
assert payload == ''
def test_remove_handler_with_matching_handler_does_not_raise():
flexmock(module).should_receive('get_handler').and_return(flexmock())
flexmock(module.logging.getLogger()).should_receive('removeHandler')
module.remove_handler('test')
def test_remove_handler_without_matching_handler_does_not_raise():
flexmock(module).should_receive('get_handler').and_raise(ValueError)
module.remove_handler('test')
+89
View File
@@ -142,6 +142,27 @@ def test_database_names_to_dump_runs_mariadb_with_list_options():
assert module.database_names_to_dump(database, None, 'test.yaml', '') == ('foo', 'bar')
def test_database_names_to_dump_runs_non_default_mariadb_with_list_options():
database = {
'name': 'all',
'list_options': '--defaults-extra-file=mariadb.cnf',
'mariadb_command': 'custom_mariadb',
}
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
extra_environment=None,
full_command=(
'custom_mariadb', # Custom MariaDB command
'--defaults-extra-file=mariadb.cnf',
'--skip-column-names',
'--batch',
'--execute',
'show schemas',
),
).and_return(('foo\nbar')).once()
assert module.database_names_to_dump(database, None, 'test.yaml', '') == ('foo', 'bar')
def test_execute_dump_command_runs_mariadb_dump():
process = flexmock()
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -315,6 +336,44 @@ def test_execute_dump_command_runs_mariadb_dump_with_options():
)
def test_execute_dump_command_runs_non_default_mariadb_dump_with_options():
process = flexmock()
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module).should_receive('execute_command').with_args(
(
'custom_mariadb_dump', # Custom MariaDB dump command
'--stuff=such',
'--add-drop-database',
'--databases',
'foo',
'--result-file',
'dump',
),
extra_environment=None,
run_to_completion=False,
).and_return(process).once()
assert (
module.execute_dump_command(
database={
'name': 'foo',
'mariadb_dump_command': 'custom_mariadb_dump',
'options': '--stuff=such',
}, # Custom MariaDB dump command specified
log_prefix='log',
dump_path=flexmock(),
database_names=('foo',),
extra_environment=None,
dry_run=False,
dry_run_label='',
)
== process
)
def test_execute_dump_command_with_duplicate_dump_skips_mariadb_dump():
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
flexmock(module.os.path).should_receive('exists').and_return(True)
@@ -435,6 +494,36 @@ def test_restore_data_source_dump_runs_mariadb_with_options():
)
def test_restore_data_source_dump_runs_non_default_mariadb_with_options():
hook_config = [
{'name': 'foo', 'restore_options': '--harder', 'mariadb_command': 'custom_mariadb'}
]
extract_process = flexmock(stdout=flexmock())
flexmock(module).should_receive('execute_command_with_processes').with_args(
('custom_mariadb', '--batch', '--harder'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
extra_environment=None,
).once()
module.restore_data_source_dump(
hook_config,
{},
'test.yaml',
data_source=hook_config[0],
dry_run=False,
extract_process=extract_process,
connection_params={
'hostname': None,
'port': None,
'username': None,
'password': None,
},
)
def test_restore_data_source_dump_runs_mariadb_with_hostname_and_port():
hook_config = [{'name': 'foo', 'hostname': 'database.example.org', 'port': 5433}]
extract_process = flexmock(stdout=flexmock())
+8
View File
@@ -164,6 +164,14 @@ def test_dump_data_sources_runs_mongodumpall_for_all_databases():
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_build_dump_command_with_username_injection_attack_gets_escaped():
database = {'name': 'test', 'username': 'bob; naughty-command'}
command = module.build_dump_command(database, dump_filename='test', dump_format='archive')
assert "'bob; naughty-command'" in command
def test_restore_data_source_dump_runs_mongorestore():
hook_config = [{'name': 'foo', 'schemas': None}, {'name': 'bar'}]
extract_process = flexmock(stdout=flexmock())
+85
View File
@@ -142,6 +142,27 @@ def test_database_names_to_dump_runs_mysql_with_list_options():
assert module.database_names_to_dump(database, None, 'test.yaml', '') == ('foo', 'bar')
def test_database_names_to_dump_runs_non_default_mysql_with_list_options():
database = {
'name': 'all',
'list_options': '--defaults-extra-file=my.cnf',
'mysql_command': 'custom_mysql',
}
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
extra_environment=None,
full_command=(
'custom_mysql', # Custom MySQL command
'--defaults-extra-file=my.cnf',
'--skip-column-names',
'--batch',
'--execute',
'show schemas',
),
).and_return(('foo\nbar')).once()
assert module.database_names_to_dump(database, None, 'test.yaml', '') == ('foo', 'bar')
def test_execute_dump_command_runs_mysqldump():
process = flexmock()
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
@@ -315,6 +336,42 @@ def test_execute_dump_command_runs_mysqldump_with_options():
)
def test_execute_dump_command_runs_non_default_mysqldump():
process = flexmock()
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module).should_receive('execute_command').with_args(
(
'custom_mysqldump', # Custom MySQL dump command
'--add-drop-database',
'--databases',
'foo',
'--result-file',
'dump',
),
extra_environment=None,
run_to_completion=False,
).and_return(process).once()
assert (
module.execute_dump_command(
database={
'name': 'foo',
'mysql_dump_command': 'custom_mysqldump',
}, # Custom MySQL dump command specified
log_prefix='log',
dump_path=flexmock(),
database_names=('foo',),
extra_environment=None,
dry_run=False,
dry_run_label='',
)
== process
)
def test_execute_dump_command_with_duplicate_dump_skips_mysqldump():
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return('dump')
flexmock(module.os.path).should_receive('exists').and_return(True)
@@ -435,6 +492,34 @@ def test_restore_data_source_dump_runs_mysql_with_options():
)
def test_restore_data_source_dump_runs_non_default_mysql_with_options():
hook_config = [{'name': 'foo', 'mysql_command': 'custom_mysql', 'restore_options': '--harder'}]
extract_process = flexmock(stdout=flexmock())
flexmock(module).should_receive('execute_command_with_processes').with_args(
('custom_mysql', '--batch', '--harder'),
processes=[extract_process],
output_log_level=logging.DEBUG,
input_file=extract_process.stdout,
extra_environment=None,
).once()
module.restore_data_source_dump(
hook_config,
{},
'test.yaml',
data_source=hook_config[0],
dry_run=False,
extract_process=extract_process,
connection_params={
'hostname': None,
'port': None,
'username': None,
'password': None,
},
)
def test_restore_data_source_dump_runs_mysql_with_hostname_and_port():
hook_config = [{'name': 'foo', 'hostname': 'database.example.org', 'port': 5433}]
extract_process = flexmock(stdout=flexmock())
+48 -3
View File
@@ -52,7 +52,52 @@ def test_ping_monitor_minimal_config_hits_hosted_ntfy_on_fail():
)
def test_ping_monitor_with_auth_hits_hosted_ntfy_on_fail():
def test_ping_monitor_with_access_token_hits_hosted_ntfy_on_fail():
hook_config = {
'topic': topic,
'access_token': 'abc123',
}
flexmock(module.requests).should_receive('post').with_args(
f'{default_base_url}/{topic}',
headers=return_default_message_headers(borgmatic.hooks.monitor.State.FAIL),
auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
).and_return(flexmock(ok=True)).once()
module.ping_monitor(
hook_config,
{},
'config.yaml',
borgmatic.hooks.monitor.State.FAIL,
monitoring_log_level=1,
dry_run=False,
)
def test_ping_monitor_with_username_password_and_access_token_ignores_username_password():
hook_config = {
'topic': topic,
'username': 'testuser',
'password': 'fakepassword',
'access_token': 'abc123',
}
flexmock(module.requests).should_receive('post').with_args(
f'{default_base_url}/{topic}',
headers=return_default_message_headers(borgmatic.hooks.monitor.State.FAIL),
auth=module.requests.auth.HTTPBasicAuth('', 'abc123'),
).and_return(flexmock(ok=True)).once()
flexmock(module.logger).should_receive('warning').once()
module.ping_monitor(
hook_config,
{},
'config.yaml',
borgmatic.hooks.monitor.State.FAIL,
monitoring_log_level=1,
dry_run=False,
)
def test_ping_monitor_with_username_password_hits_hosted_ntfy_on_fail():
hook_config = {
'topic': topic,
'username': 'testuser',
@@ -74,7 +119,7 @@ def test_ping_monitor_with_auth_hits_hosted_ntfy_on_fail():
)
def test_ping_monitor_auth_with_no_username_warning():
def test_ping_monitor_with_password_but_no_username_warns():
hook_config = {'topic': topic, 'password': 'fakepassword'}
flexmock(module.requests).should_receive('post').with_args(
f'{default_base_url}/{topic}',
@@ -93,7 +138,7 @@ def test_ping_monitor_auth_with_no_username_warning():
)
def test_ping_monitor_auth_with_no_password_warning():
def test_ping_monitor_with_username_but_no_password_warns():
hook_config = {'topic': topic, 'username': 'testuser'}
flexmock(module.requests).should_receive('post').with_args(
f'{default_base_url}/{topic}',
+60 -4
View File
@@ -50,6 +50,14 @@ def test_make_extra_environment_without_cli_password_or_configured_password_does
assert 'PGPASSWORD' not in extra
def test_make_extra_environment_without_ssl_mode_does_not_set_ssl_mode():
database = {'name': 'foo'}
extra = module.make_extra_environment(database)
assert 'PGSSLMODE' not in extra
def test_database_names_to_dump_passes_through_individual_database_name():
database = {'name': 'foo'}
@@ -164,11 +172,17 @@ def test_database_names_to_dump_with_all_and_format_excludes_particular_database
def test_database_names_to_dump_with_all_and_psql_command_uses_custom_command():
database = {'name': 'all', 'format': 'custom', 'psql_command': 'docker exec mycontainer psql'}
database = {
'name': 'all',
'format': 'custom',
'psql_command': 'docker exec --workdir * mycontainer psql',
}
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
(
'docker',
'exec',
'--workdir',
"'*'", # Should get shell escaped to prevent injection attacks.
'mycontainer',
'psql',
'--list',
@@ -345,6 +359,42 @@ def test_dump_data_sources_runs_pg_dump_with_username_and_password():
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_dump_data_sources_with_username_injection_attack_gets_escaped():
databases = [{'name': 'foo', 'username': 'postgres; naughty-command', 'password': 'trustsome1'}]
process = flexmock()
flexmock(module).should_receive('make_extra_environment').and_return(
{'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'}
)
flexmock(module).should_receive('make_dump_path').and_return('')
flexmock(module).should_receive('database_names_to_dump').and_return(('foo',))
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
'databases/localhost/foo'
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module).should_receive('execute_command').with_args(
(
'pg_dump',
'--no-password',
'--clean',
'--if-exists',
'--username',
"'postgres; naughty-command'",
'--format',
'custom',
'foo',
'>',
'databases/localhost/foo',
),
shell=True,
extra_environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
run_to_completion=False,
).and_return(process).once()
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_dump_data_sources_runs_pg_dump_with_directory_format():
databases = [{'name': 'foo', 'format': 'directory'}]
flexmock(module).should_receive('make_extra_environment').and_return({'PGSSLMODE': 'disable'})
@@ -432,7 +482,7 @@ def test_dump_data_sources_runs_pg_dumpall_for_all_databases():
def test_dump_data_sources_runs_non_default_pg_dump():
databases = [{'name': 'foo', 'pg_dump_command': 'special_pg_dump'}]
databases = [{'name': 'foo', 'pg_dump_command': 'special_pg_dump --compress *'}]
process = flexmock()
flexmock(module).should_receive('make_extra_environment').and_return({'PGSSLMODE': 'disable'})
flexmock(module).should_receive('make_dump_path').and_return('')
@@ -446,6 +496,8 @@ def test_dump_data_sources_runs_non_default_pg_dump():
flexmock(module).should_receive('execute_command').with_args(
(
'special_pg_dump',
'--compress',
"'*'", # Should get shell escaped to prevent injection attacks.
'--no-password',
'--clean',
'--if-exists',
@@ -943,8 +995,8 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
hook_config = [
{
'name': 'foo',
'pg_restore_command': 'docker exec mycontainer pg_restore',
'psql_command': 'docker exec mycontainer psql',
'pg_restore_command': 'docker exec --workdir * mycontainer pg_restore',
'psql_command': 'docker exec --workdir * mycontainer psql',
'schemas': None,
}
]
@@ -957,6 +1009,8 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
(
'docker',
'exec',
'--workdir',
"'*'", # Should get shell escaped to prevent injection attacks.
'mycontainer',
'pg_restore',
'--no-password',
@@ -975,6 +1029,8 @@ def test_restore_data_source_dump_runs_non_default_pg_restore_and_psql():
(
'docker',
'exec',
'--workdir',
"'*'", # Should get shell escaped to prevent injection attacks.
'mycontainer',
'psql',
'--no-password',
+27
View File
@@ -39,6 +39,33 @@ def test_dump_data_sources_dumps_each_database():
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == processes
def test_dump_data_sources_with_path_injection_attack_gets_escaped():
databases = [
{'path': '/path/to/database1; naughty-command', 'name': 'database1'},
]
processes = [flexmock()]
flexmock(module).should_receive('make_dump_path').and_return('/path/to/dump')
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
'/path/to/dump/database'
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module).should_receive('execute_command').with_args(
(
'sqlite3',
"'/path/to/database1; naughty-command'",
'.dump',
'>',
'/path/to/dump/database',
),
shell=True,
run_to_completion=False,
).and_return(processes[0])
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == processes
def test_dump_data_sources_with_non_existent_path_warns_and_dumps_database():
databases = [
{'path': '/path/to/database1', 'name': 'database1'},
+46 -25
View File
@@ -7,32 +7,49 @@ from borgmatic import execute as module
@pytest.mark.parametrize(
'command,exit_code,borg_local_path,expected_result',
'command,exit_code,borg_local_path,borg_exit_codes,expected_result',
(
(['grep'], 2, None, True),
(['grep'], 2, 'borg', True),
(['borg'], 2, 'borg', True),
(['borg1'], 2, 'borg1', True),
(['grep'], 1, None, True),
(['grep'], 1, 'borg', True),
(['borg'], 1, 'borg', False),
(['borg1'], 1, 'borg1', False),
(['grep'], 0, None, False),
(['grep'], 0, 'borg', False),
(['borg'], 0, 'borg', False),
(['borg1'], 0, 'borg1', False),
(['grep'], 2, None, None, module.Exit_status.ERROR),
(['grep'], 2, 'borg', None, module.Exit_status.ERROR),
(['borg'], 2, 'borg', None, module.Exit_status.ERROR),
(['borg1'], 2, 'borg1', None, module.Exit_status.ERROR),
(['grep'], 1, None, None, module.Exit_status.ERROR),
(['grep'], 1, 'borg', None, module.Exit_status.ERROR),
(['borg'], 1, 'borg', None, module.Exit_status.WARNING),
(['borg1'], 1, 'borg1', None, module.Exit_status.WARNING),
(['grep'], 100, None, None, module.Exit_status.ERROR),
(['grep'], 100, 'borg', None, module.Exit_status.ERROR),
(['borg'], 100, 'borg', None, module.Exit_status.WARNING),
(['borg1'], 100, 'borg1', None, module.Exit_status.WARNING),
(['grep'], 0, None, None, module.Exit_status.SUCCESS),
(['grep'], 0, 'borg', None, module.Exit_status.SUCCESS),
(['borg'], 0, 'borg', None, module.Exit_status.SUCCESS),
(['borg1'], 0, 'borg1', None, module.Exit_status.SUCCESS),
# -9 exit code occurs when child process get SIGKILLed.
(['grep'], -9, None, True),
(['grep'], -9, 'borg', True),
(['borg'], -9, 'borg', True),
(['borg1'], -9, 'borg1', True),
(['borg'], None, None, False),
(['grep'], -9, None, None, module.Exit_status.ERROR),
(['grep'], -9, 'borg', None, module.Exit_status.ERROR),
(['borg'], -9, 'borg', None, module.Exit_status.ERROR),
(['borg1'], -9, 'borg1', None, module.Exit_status.ERROR),
(['borg'], None, None, None, module.Exit_status.STILL_RUNNING),
(['borg'], 1, 'borg', [], module.Exit_status.WARNING),
(['borg'], 1, 'borg', [{}], module.Exit_status.WARNING),
(['borg'], 1, 'borg', [{'code': 1}], module.Exit_status.WARNING),
(['grep'], 1, 'borg', [{'code': 100, 'treat_as': 'error'}], module.Exit_status.ERROR),
(['borg'], 1, 'borg', [{'code': 100, 'treat_as': 'error'}], module.Exit_status.WARNING),
(['borg'], 1, 'borg', [{'code': 1, 'treat_as': 'error'}], module.Exit_status.ERROR),
(['borg'], 2, 'borg', [{'code': 99, 'treat_as': 'warning'}], module.Exit_status.ERROR),
(['borg'], 2, 'borg', [{'code': 2, 'treat_as': 'warning'}], module.Exit_status.WARNING),
(['borg'], 100, 'borg', [{'code': 1, 'treat_as': 'error'}], module.Exit_status.WARNING),
(['borg'], 100, 'borg', [{'code': 100, 'treat_as': 'error'}], module.Exit_status.ERROR),
),
)
def test_exit_code_indicates_error_respects_exit_code_and_borg_local_path(
command, exit_code, borg_local_path, expected_result
def test_interpret_exit_code_respects_exit_code_and_borg_local_path(
command, exit_code, borg_local_path, borg_exit_codes, expected_result
):
assert module.exit_code_indicates_error(command, exit_code, borg_local_path) is expected_result
assert (
module.interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
is expected_result
)
def test_command_for_process_converts_sequence_command_to_string():
@@ -178,7 +195,7 @@ def test_execute_command_calls_full_command_without_capturing_output():
flexmock(module.subprocess).should_receive('Popen').with_args(
full_command, stdin=None, stdout=None, stderr=None, shell=False, env=None, cwd=None
).and_return(flexmock(wait=lambda: 0)).once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('log_outputs')
output = module.execute_command(full_command, output_file=module.DO_NOT_CAPTURE)
@@ -323,7 +340,9 @@ def test_execute_command_and_capture_output_returns_output_when_process_error_is
flexmock(module.subprocess).should_receive('check_output').with_args(
full_command, stderr=None, shell=False, env=None, cwd=None
).and_raise(subprocess.CalledProcessError(1, full_command, err_output)).once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False).once()
flexmock(module).should_receive('interpret_exit_code').and_return(
module.Exit_status.SUCCESS
).once()
output = module.execute_command_and_capture_output(full_command)
@@ -338,7 +357,9 @@ def test_execute_command_and_capture_output_raises_when_command_errors():
flexmock(module.subprocess).should_receive('check_output').with_args(
full_command, stderr=None, shell=False, env=None, cwd=None
).and_raise(subprocess.CalledProcessError(2, full_command, expected_output)).once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(True).once()
flexmock(module).should_receive('interpret_exit_code').and_return(
module.Exit_status.ERROR
).once()
with pytest.raises(subprocess.CalledProcessError):
module.execute_command_and_capture_output(full_command)
@@ -467,7 +488,7 @@ def test_execute_command_with_processes_calls_full_command_without_capturing_out
flexmock(module.subprocess).should_receive('Popen').with_args(
full_command, stdin=None, stdout=None, stderr=None, shell=False, env=None, cwd=None
).and_return(flexmock(wait=lambda: 0)).once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('log_outputs')
output = module.execute_command_with_processes(

Some files were not shown because too many files have changed in this diff Show More