Compare commits

...
50 Commits
Author SHA1 Message Date
Dan Helfman f9998b50e8 Rephrase documentation and link to docs on exit codes feature (#798). 2024-01-21 14:47:21 -08:00
Dan Helfman 5f921a7f80 Add documentation heading (#798). 2024-01-21 11:48:23 -08:00
Dan Helfman abf2b3a8c7 Elevate specific Borg warnings to errors or squash errors to warnings (#798). 2024-01-21 11:34:40 -08:00
Dan Helfman 34f3c2bb16 Clarify "--override" command-line help (#814) 2024-01-19 11:55:00 -08:00
Dan Helfman 4d79f582df Fix a traceback when providing an invalid "--override" value for a list option (#814). 2024-01-18 10:39:40 -08:00
Dan Helfman 63198088c4 Store included configuration files within each backup archive in support of the "config bootstrap" action (#736). 2024-01-09 13:47:20 -08:00
Dan Helfman 3c22a8ec16 Prevent various shell injection attacks (#810). 2024-01-07 10:21:49 -08:00
Dan Helfman ca49109ce7 Bump version for release. 2024-01-03 10:08:05 -08:00
Dan Helfman 6a7f71f92f Clarify prune action help concerning running compact afterwards (#808). 2024-01-03 10:03:35 -08:00
Dan Helfman 5f3dc1cfb0 Stream SQLite databases directly to Borg instead of dumping to an intermediate file (#807). 2023-12-31 11:07:59 -08:00
Dan Helfman f2023aed22 Fix typo. 2023-12-30 15:48:55 -08:00
Dan Helfman a03c2744e5 Update docs/how-to/provide-your-passwords.md (#805).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/805
Reviewed-by: Dan Helfman <witten@torsion.org>
2023-12-30 23:48:32 +00:00
axel simon 4176532317 Update docs/how-to/provide-your-passwords.md
Provide an explanation of encryption_passcommand.
Also, adjust headers for consistency.
2023-12-30 23:45:56 +00:00
Dan Helfman 9d6025e902 Validate the configured action names in the "skip_actions" option (#804). 2023-12-28 20:07:57 -08:00
Dan Helfman cf739bc997 The "check --force" flag now runs checks even if "check" is in "skip_actions" (#802). 2023-12-28 10:22:48 -08:00
Dan Helfman 84823dfb91 Clarify constants/placeholders interaction and improve examples (#763). 2023-12-24 11:18:17 -08:00
Dan Helfman 20cf0f7089 Add an "--ssh-command" flag to the "config bootstrap" action (#767). 2023-12-24 10:33:55 -08:00
Dan Helfman 67af0f5734 Document limitation with constant interpolation at the start of a value (#741). 2023-12-22 21:39:44 -08:00
Dan Helfman e80e0a253c Add configured repository labels to the JSON output for all actions (#800). 2023-12-20 09:17:41 -08:00
Dan Helfman 72587a3b72 Merge branch 'main' of ssh://projects.torsion.org:3022/borgmatic-collective/borgmatic 2023-12-04 11:17:59 -08:00
Dan Helfman 8b49a59aff Fix a traceback when the "repositories" option contains both strings and key/value pairs (#794). 2023-12-04 11:17:13 -08:00
Dan Helfman e120dff9ff Add debug message that logs borg version for every config (#714).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/792
2023-11-25 03:59:40 +00:00
Tobias Hodapp 257678b66f Fixed borg -> Borg 2023-11-24 18:47:37 +01:00
Tobias Hodapp 422c5e32f4 Added debug message that logs borg version for every config 2023-11-23 11:46:10 +01:00
Dan Helfman c34ad7dde7 Update documentation about configuration includes and constants (#745). 2023-11-19 21:22:10 -08:00
Dan Helfman fdb353d358 Bump version for release. 2023-11-19 21:14:56 -08:00
Dan Helfman 3b99f7c75a Constants support includes and command-line overrides (#745, #782) 2023-11-19 21:13:35 -08:00
Dan Helfman 8b9abc6cf8 Documentation clarifications (#791). 2023-11-15 09:05:52 -08:00
Dan Helfman da034c316a Add another mention of "skip_actions" to the docs (#701). 2023-11-08 18:22:17 -08:00
Dan Helfman 08d01d8bcd Documentation formatting. 2023-11-08 17:57:31 -08:00
Dan Helfman eef69e23ee Document the possible units of times for a configured check frequency. 2023-11-08 17:53:59 -08:00
Dan Helfman 26bb54a9dd Remove broken link in documentation (#786). 2023-11-08 10:26:09 -08:00
Dan Helfman 715e2ac127 Add test support for Python 3.12. 2023-11-07 10:17:55 -08:00
Dan Helfman f39cea4abf Remove additional Python 3.7-isms (#784). 2023-11-07 10:17:16 -08:00
Dan Helfman 22101bdd49 Drop support for Python 3.7, which has been end-of-lifed (#784). 2023-11-07 10:11:29 -08:00
Dan Helfman 13cf863d89 Fix tests (#783). 2023-11-07 10:09:31 -08:00
Dan Helfman dcf25fa041 Upgrade ruamel.yaml dependency to support version 0.18.x (#783). 2023-11-07 10:00:13 -08:00
Dan Helfman 12b75f9075 Update documentation about logging changes from version 1.8.3 (#665). 2023-11-06 21:13:46 -08:00
Dan HelfmanandGitHub 9baf06a2f7 Fix typo.
Typo
2023-11-04 08:56:39 -07:00
tdltdc 56302e22cd Typo 2023-11-04 15:05:19 +01:00
Dan Helfman 6cc93c4eb9 Fix environment variable interpolation within configured repository paths (#782). 2023-11-03 21:16:04 -07:00
Dan Helfman 2da43239f6 Fix docs: minor typos (#781).
Reviewed-on: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/781
2023-11-03 00:59:29 +00:00
debuglevel 4beef36d3c Update docs/how-to/inspect-your-backups.md 2023-11-02 23:14:31 +00:00
debuglevel eacfbd742b Typo 2023-11-02 23:13:45 +00:00
debuglevel 82a85986b6 Typo 2023-11-02 22:57:46 +00:00
Dan Helfman ef448e2dd1 Add a "skip_actions" option to skip running particular actions (#701). 2023-10-31 21:54:41 -07:00
Dan Helfman c3efe1b90e Only parse "--override" values as complex data types when they're for options of those types (#779). 2023-10-29 19:02:28 -07:00
Dan Helfman d85c1ee216 Correct changelog addition (#779). 2023-10-29 16:25:40 -07:00
Dan Helfman b47088067c Add a "--match-archives" flag to the "check" action (#779). 2023-10-29 16:22:39 -07:00
Dan Helfman c5732aa4fc Fix home page CSS layout to prevent overflow at certain window widths (#777). 2023-10-27 14:12:35 -07:00
103 changed files with 3088 additions and 731 deletions
+41
View File
@@ -1,3 +1,44 @@
1.8.7
* #736: Store included configuration files within each backup archive in support of the "config
bootstrap" action. Previously, only top-level configuration files were stored.
* #798: Elevate specific Borg warnings to errors or squash errors to
* warnings. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/
* #810: SECURITY: Prevent shell injection attacks within the PostgreSQL hook, the MongoDB hook, the
SQLite hook, the "borgmatic borg" action, and command hook variable/constant interpolation.
* #814: Fix a traceback when providing an invalid "--override" value for a list option.
1.8.6
* #767: Add an "--ssh-command" flag to the "config bootstrap" action for setting a custom SSH
command, as no configuration is available (including the "ssh_command" option) until
bootstrapping completes.
* #794: Fix a traceback when the "repositories" option contains both strings and key/value pairs.
* #800: Add configured repository labels to the JSON output for all actions.
* #802: The "check --force" flag now runs checks even if "check" is in "skip_actions".
* #804: Validate the configured action names in the "skip_actions" option.
* #807: Stream SQLite databases directly to Borg instead of dumping to an intermediate file.
* When logging commands that borgmatic executes, log the environment variables that
borgmatic sets for those commands. (But don't log their values, since they often contain
passwords.)
1.8.5
* #701: Add a "skip_actions" option to skip running particular actions, handy for append-only or
checkless configurations. See the documentation for more information:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#skipping-actions
* #701: Deprecate the "disabled" value for the "checks" option in favor of the new "skip_actions"
option.
* #745: Constants now apply to included configuration, not just the file doing the includes. As a
side effect of this change, constants no longer apply to option names and only substitute into
configuration values.
* #779: Add a "--match-archives" flag to the "check" action for selecting the archives to check,
overriding the existing "archive_name_format" and "match_archives" options in configuration.
* #779: Only parse "--override" values as complex data types when they're for options of those
types.
* #782: Fix environment variable interpolation within configured repository paths.
* #782: Add configuration constant overriding via the existing "--override" flag.
* #783: Upgrade ruamel.yaml dependency to support version 0.18.x.
* #784: Drop support for Python 3.7, which has been end-of-lifed.
1.8.4
* #715: Add a monitoring hook for sending backup status to a variety of monitoring services via the
Apprise library. See the documentation for more information:
+1 -4
View File
@@ -39,13 +39,10 @@ def run_check(
repository['path'],
config,
local_borg_version,
check_arguments,
global_arguments,
local_path=local_path,
remote_path=remote_path,
progress=check_arguments.progress,
repair=check_arguments.repair,
only_checks=check_arguments.only,
force=check_arguments.force,
)
borgmatic.hooks.command.execute_hook(
config.get('after_check'),
+13 -13
View File
@@ -13,14 +13,11 @@ logger = logging.getLogger(__name__)
def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
'''
Given:
The bootstrap arguments, which include the repository and archive name, borgmatic source directory,
destination directory, and whether to strip components.
The global arguments, which include the dry run flag
and the local borg version,
Return:
The config paths from the manifest.json file in the borgmatic source directory after extracting it from the
repository.
Given the bootstrap arguments as an argparse.Namespace (containing the repository and archive
name, borgmatic source directory, destination directory, and whether to strip components), the
global arguments as an argparse.Namespace (containing the dry run flag and the local borg
version), return the config paths from the manifest.json file in the borgmatic source directory
after extracting it from the repository.
Raise ValueError if the manifest JSON is missing, can't be decoded, or doesn't contain the
expected configuration path data.
@@ -31,24 +28,26 @@ def get_config_paths(bootstrap_arguments, global_arguments, local_borg_version):
borgmatic_manifest_path = os.path.expanduser(
os.path.join(borgmatic_source_directory, 'bootstrap', 'manifest.json')
)
config = {'ssh_command': bootstrap_arguments.ssh_command}
extract_process = borgmatic.borg.extract.extract_archive(
global_arguments.dry_run,
bootstrap_arguments.repository,
borgmatic.borg.rlist.resolve_archive_name(
bootstrap_arguments.repository,
bootstrap_arguments.archive,
{},
config,
local_borg_version,
global_arguments,
),
[borgmatic_manifest_path],
{},
config,
local_borg_version,
global_arguments,
extract_to_stdout=True,
)
manifest_json = extract_process.stdout.read()
if not manifest_json:
raise ValueError(
'Cannot read configuration paths from archive due to missing bootstrap manifest'
@@ -79,6 +78,7 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
manifest_config_paths = get_config_paths(
bootstrap_arguments, global_arguments, local_borg_version
)
config = {'ssh_command': bootstrap_arguments.ssh_command}
logger.info(f"Bootstrapping config paths: {', '.join(manifest_config_paths)}")
@@ -88,12 +88,12 @@ def run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version):
borgmatic.borg.rlist.resolve_archive_name(
bootstrap_arguments.repository,
bootstrap_arguments.archive,
{},
config,
local_borg_version,
global_arguments,
),
[config_path.lstrip(os.path.sep) for config_path in manifest_config_paths],
{},
config,
local_borg_version,
global_arguments,
extract_to_stdout=False,
+10 -9
View File
@@ -1,12 +1,9 @@
import importlib.metadata
import json
import logging
import os
try:
import importlib_metadata
except ModuleNotFoundError: # pragma: nocover
import importlib.metadata as importlib_metadata
import borgmatic.actions.json
import borgmatic.borg.create
import borgmatic.borg.state
import borgmatic.config.validate
@@ -39,7 +36,7 @@ def create_borgmatic_manifest(config, config_paths, dry_run):
with open(borgmatic_manifest_path, 'w') as config_list_file:
json.dump(
{
'borgmatic_version': importlib_metadata.version('borgmatic'),
'borgmatic_version': importlib.metadata.version('borgmatic'),
'config_paths': config_paths,
},
config_list_file,
@@ -50,6 +47,7 @@ def run_create(
config_filename,
repository,
config,
config_paths,
hook_context,
local_borg_version,
create_arguments,
@@ -93,7 +91,9 @@ def run_create(
)
if config.get('store_config_files', True):
create_borgmatic_manifest(
config, global_arguments.used_config_paths, global_arguments.dry_run
config,
config_paths,
global_arguments.dry_run,
)
stream_processes = [process for processes in active_dumps.values() for process in processes]
@@ -101,6 +101,7 @@ def run_create(
global_arguments.dry_run,
repository['path'],
config,
config_paths,
local_borg_version,
global_arguments,
local_path=local_path,
@@ -111,8 +112,8 @@ def run_create(
list_files=create_arguments.list_files,
stream_processes=stream_processes,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
borgmatic.hooks.dispatch.call_hooks_even_if_unconfigured(
'remove_data_source_dumps',
+4 -4
View File
@@ -1,7 +1,7 @@
import json
import logging
import borgmatic.actions.arguments
import borgmatic.actions.json
import borgmatic.borg.info
import borgmatic.borg.rlist
import borgmatic.config.validate
@@ -26,7 +26,7 @@ def run_info(
if info_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, info_arguments.repository
):
if not info_arguments.json: # pragma: nocover
if not info_arguments.json:
logger.answer(
f'{repository.get("label", repository["path"])}: Displaying archive summary information'
)
@@ -48,5 +48,5 @@ def run_info(
local_path,
remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+16
View File
@@ -0,0 +1,16 @@
import json
def parse_json(borg_json_output, label):
'''
Given a Borg JSON output string, parse it as JSON into a dict. Inject the given borgmatic
repository label into it and return the dict.
'''
json_data = json.loads(borg_json_output)
if 'repository' not in json_data:
return json_data
json_data['repository']['label'] = label or ''
return json_data
+6 -6
View File
@@ -1,7 +1,7 @@
import json
import logging
import borgmatic.actions.arguments
import borgmatic.actions.json
import borgmatic.borg.list
import borgmatic.config.validate
@@ -25,10 +25,10 @@ def run_list(
if list_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, list_arguments.repository
):
if not list_arguments.json: # pragma: nocover
if list_arguments.find_paths:
if not list_arguments.json:
if list_arguments.find_paths: # pragma: no cover
logger.answer(f'{repository.get("label", repository["path"])}: Searching archives')
elif not list_arguments.archive:
elif not list_arguments.archive: # pragma: no cover
logger.answer(f'{repository.get("label", repository["path"])}: Listing archives')
archive_name = borgmatic.borg.rlist.resolve_archive_name(
@@ -49,5 +49,5 @@ def run_list(
local_path,
remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+4 -4
View File
@@ -1,6 +1,6 @@
import json
import logging
import borgmatic.actions.json
import borgmatic.borg.rinfo
import borgmatic.config.validate
@@ -24,7 +24,7 @@ def run_rinfo(
if rinfo_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, rinfo_arguments.repository
):
if not rinfo_arguments.json: # pragma: nocover
if not rinfo_arguments.json:
logger.answer(
f'{repository.get("label", repository["path"])}: Displaying repository summary information'
)
@@ -38,5 +38,5 @@ def run_rinfo(
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+4 -4
View File
@@ -1,6 +1,6 @@
import json
import logging
import borgmatic.actions.json
import borgmatic.borg.rlist
import borgmatic.config.validate
@@ -24,7 +24,7 @@ def run_rlist(
if rlist_arguments.repository is None or borgmatic.config.validate.repositories_match(
repository, rlist_arguments.repository
):
if not rlist_arguments.json: # pragma: nocover
if not rlist_arguments.json:
logger.answer(f'{repository.get("label", repository["path"])}: Listing repository')
json_output = borgmatic.borg.rlist.list_repository(
@@ -36,5 +36,5 @@ def run_rlist(
local_path=local_path,
remote_path=remote_path,
)
if json_output: # pragma: nocover
yield json.loads(json_output)
if json_output:
yield borgmatic.actions.json.parse_json(json_output, repository.get('label'))
+4 -2
View File
@@ -1,4 +1,5 @@
import logging
import shlex
import borgmatic.commands.arguments
import borgmatic.logger
@@ -56,9 +57,8 @@ def run_arbitrary_borg(
)
return execute_command(
full_command,
tuple(shlex.quote(part) for part in full_command),
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
shell=True,
extra_environment=dict(
(environment.make_environment(config) or {}),
@@ -67,4 +67,6 @@ def run_arbitrary_borg(
'ARCHIVE': archive if archive else '',
},
),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+6 -1
View File
@@ -34,4 +34,9 @@ def break_lock(
)
borg_environment = environment.make_environment(config)
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+36 -20
View File
@@ -39,7 +39,11 @@ def parse_checks(config, only_checks=None):
check_config['name'] for check_config in (config.get('checks', None) or DEFAULT_CHECKS)
)
checks = tuple(check.lower() for check in checks)
if 'disabled' in checks:
logger.warning(
'The "disabled" value for the "checks" option is deprecated and will be removed from a future release; use "skip_actions" instead'
)
if len(checks) > 1:
logger.warning(
'Multiple checks are configured, but one of them is "disabled"; not running any checks'
@@ -119,6 +123,9 @@ def filter_checks_on_frequency(
Raise ValueError if a frequency cannot be parsed.
'''
if not checks:
return checks
filtered_checks = list(checks)
if force:
@@ -149,11 +156,13 @@ def filter_checks_on_frequency(
return tuple(filtered_checks)
def make_archive_filter_flags(local_borg_version, config, checks, check_last=None, prefix=None):
def make_archive_filter_flags(
local_borg_version, config, checks, check_arguments, check_last=None, prefix=None
):
'''
Given the local Borg version, a configuration dict, a parsed sequence of checks, the check last
value, and a consistency check prefix, transform the checks into tuple of command-line flags for
filtering archives in a check command.
Given the local Borg version, a configuration dict, a parsed sequence of checks, check arguments
as an argparse.Namespace instance, the check last value, and a consistency check prefix,
transform the checks into tuple of command-line flags for filtering archives in a check command.
If a check_last value is given and "archives" is in checks, then include a "--last" flag. And if
a prefix value is given and "archives" is in checks, then include a "--match-archives" flag.
@@ -168,7 +177,7 @@ def make_archive_filter_flags(local_borg_version, config, checks, check_last=Non
if prefix
else (
flags.make_match_archives_flags(
config.get('match_archives'),
check_arguments.match_archives or config.get('match_archives'),
config.get('archive_name_format'),
local_borg_version,
)
@@ -353,18 +362,15 @@ def check_archives(
repository_path,
config,
local_borg_version,
check_arguments,
global_arguments,
local_path='borg',
remote_path=None,
progress=None,
repair=None,
only_checks=None,
force=None,
):
'''
Given a local or remote repository path, a configuration dict, local/remote commands to run,
whether to include progress information, whether to attempt a repair, and an optional list of
checks to use instead of configured checks, check the contained Borg archives for consistency.
Given a local or remote repository path, a configuration dict, the local Borg version, check
arguments as an argparse.Namespace instance, global arguments, and local/remote commands to run,
check the contained Borg archives for consistency.
If there are no consistency checks to run, skip running them.
@@ -389,11 +395,11 @@ def check_archives(
check_last = config.get('check_last', None)
prefix = config.get('prefix')
configured_checks = parse_checks(config, only_checks)
configured_checks = parse_checks(config, check_arguments.only_checks)
lock_wait = None
extra_borg_options = config.get('extra_borg_options', {}).get('check', '')
archive_filter_flags = make_archive_filter_flags(
local_borg_version, config, configured_checks, check_last, prefix
local_borg_version, config, configured_checks, check_arguments, check_last, prefix
)
archives_check_id = make_archives_check_id(archive_filter_flags)
@@ -401,7 +407,7 @@ def check_archives(
config,
borg_repository_id,
configured_checks,
force,
check_arguments.force,
archives_check_id,
)
@@ -416,27 +422,37 @@ def check_archives(
full_command = (
(local_path, 'check')
+ (('--repair',) if repair else ())
+ (('--repair',) if check_arguments.repair else ())
+ make_check_flags(checks, archive_filter_flags)
+ (('--remote-path', remote_path) if remote_path else ())
+ (('--log-json',) if global_arguments.log_json else ())
+ (('--lock-wait', str(lock_wait)) if lock_wait else ())
+ verbosity_flags
+ (('--progress',) if progress else ())
+ (('--progress',) if check_arguments.progress else ())
+ (tuple(extra_borg_options.split(' ')) if extra_borg_options else ())
+ flags.make_repository_flags(repository_path, local_borg_version)
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The Borg repair option triggers an interactive prompt, which won't work when output is
# captured. And progress messes with the terminal directly.
if repair or progress:
if check_arguments.repair or check_arguments.progress:
execute_command(
full_command, output_file=DO_NOT_CAPTURE, extra_environment=borg_environment
full_command,
output_file=DO_NOT_CAPTURE,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(full_command, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
for check in checks:
write_check_time(
+2 -1
View File
@@ -48,6 +48,7 @@ def compact_segments(
execute_command(
full_command,
output_log_level=logging.INFO,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+14 -10
View File
@@ -272,7 +272,7 @@ def any_parent_directories(path, candidate_parents):
def collect_special_file_paths(
create_command, local_path, working_directory, borg_environment, skip_directories
create_command, config, local_path, working_directory, borg_environment, skip_directories
):
'''
Given a Borg create command as a tuple, a local Borg path, a working directory, a dict of
@@ -290,6 +290,7 @@ def collect_special_file_paths(
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
paths = tuple(
@@ -323,6 +324,7 @@ def create_archive(
dry_run,
repository_path,
config,
config_paths,
local_borg_version,
global_arguments,
local_path='borg',
@@ -334,8 +336,9 @@ def create_archive(
stream_processes=None,
):
'''
Given vebosity/dry-run flags, a local or remote repository path, and a configuration dict,
create a Borg archive and return Borg's JSON output (if any).
Given vebosity/dry-run flags, a local or remote repository path, a configuration dict, a
sequence of loaded configuration paths, the local Borg version, and global arguments as an
argparse.Namespace instance, create a Borg archive and return Borg's JSON output (if any).
If a sequence of stream processes is given (instances of subprocess.Popen), then execute the
create command while also triggering the given processes to produce output.
@@ -351,11 +354,7 @@ def create_archive(
expand_directories(
tuple(config.get('source_directories', ()))
+ borgmatic_source_directories
+ tuple(
global_arguments.used_config_paths
if config.get('store_config_files', True)
else ()
)
+ tuple(config_paths if config.get('store_config_files', True) else ())
)
),
additional_directory_devices=map_directories_to_devices(
@@ -471,6 +470,7 @@ def create_archive(
logger.debug(f'{repository_path}: Collecting special file paths')
special_file_paths = collect_special_file_paths(
create_flags + create_positional_arguments,
config,
local_path,
working_directory,
borg_environment,
@@ -496,6 +496,7 @@ def create_archive(
+ (('--progress',) if progress else ())
+ (('--json',) if json else ())
)
borg_exit_codes = config.get('borg_exit_codes')
if stream_processes:
return execute_command_with_processes(
@@ -503,9 +504,10 @@ def create_archive(
stream_processes,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
elif output_log_level is None:
return execute_command_and_capture_output(
@@ -513,13 +515,15 @@ def create_archive(
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
create_flags + create_positional_arguments,
output_log_level,
output_file,
borg_local_path=local_path,
working_directory=working_directory,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+4
View File
@@ -50,4 +50,8 @@ def make_environment(config):
if value is not None:
environment[environment_variable_name] = 'YES' if value else 'NO'
# On Borg 1.4.0a1+, take advantage of more specific exit codes. No effect on
# older versions of Borg.
environment['BORG_EXIT_CODES'] = 'modern'
return environment
+2 -1
View File
@@ -65,6 +65,7 @@ def export_key(
full_command,
output_file=output_file,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+2 -1
View File
@@ -69,6 +69,7 @@ def export_tar_archive(
full_command,
output_file=DO_NOT_CAPTURE if destination_path == '-' else None,
output_log_level=output_log_level,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+15 -2
View File
@@ -57,7 +57,11 @@ def extract_last_archive_dry_run(
)
execute_command(
full_extract_command, working_directory=None, extra_environment=borg_environment
full_extract_command,
working_directory=None,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
@@ -127,6 +131,7 @@ def extract_archive(
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# The progress output isn't compatible with captured and logged output, as progress messes with
# the terminal directly.
@@ -136,6 +141,8 @@ def extract_archive(
output_file=DO_NOT_CAPTURE,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
return None
@@ -146,10 +153,16 @@ def extract_archive(
working_directory=destination_path,
run_to_completion=False,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
# Don't give Borg local path so as to error on warnings, as "borg extract" only gives a warning
# if the restore paths don't exist in the archive.
execute_command(
full_command, working_directory=destination_path, extra_environment=borg_environment
full_command,
working_directory=destination_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+4 -1
View File
@@ -95,11 +95,13 @@ def display_archives_info(
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_info = execute_command_and_capture_output(
json_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if info_arguments.json:
@@ -110,6 +112,7 @@ def display_archives_info(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+5 -1
View File
@@ -124,6 +124,7 @@ def capture_archive_listing(
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
.strip('\n')
.split('\n')
@@ -189,6 +190,7 @@ def list_archive(
)
borg_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
# If there are any paths to find (and there's not a single archive already selected), start by
# getting a list of archives to search.
@@ -219,6 +221,7 @@ def list_archive(
),
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
.strip('\n')
.split('\n')
@@ -251,6 +254,7 @@ def list_archive(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+8 -2
View File
@@ -65,9 +65,15 @@ def mount_archive(
execute_command(
full_command,
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
return
execute_command(full_command, borg_local_path=local_path, extra_environment=borg_environment)
execute_command(
full_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+2 -1
View File
@@ -94,6 +94,7 @@ def prune_archives(
execute_command(
full_command,
output_log_level=output_log_level,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+2 -1
View File
@@ -81,6 +81,7 @@ def create_repository(
execute_command(
rcreate_command,
output_file=DO_NOT_CAPTURE,
borg_local_path=local_path,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
+4 -1
View File
@@ -49,17 +49,20 @@ def display_repository_info(
)
extra_environment = environment.make_environment(config)
borg_exit_codes = config.get('borg_exit_codes')
if rinfo_arguments.json:
return execute_command_and_capture_output(
full_command,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
else:
execute_command(
full_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=extra_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+8 -2
View File
@@ -45,6 +45,7 @@ def resolve_archive_name(
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:
latest_archive = output.strip().splitlines()[-1]
@@ -147,9 +148,13 @@ def list_repository(
local_path,
remote_path,
)
borg_exit_codes = config.get('borg_exit_codes')
json_listing = execute_command_and_capture_output(
json_command, extra_environment=borg_environment, borg_local_path=local_path
json_command,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
if rlist_arguments.json:
@@ -160,6 +165,7 @@ def list_repository(
execute_command(
main_command,
output_log_level=logging.ANSWER,
borg_local_path=local_path,
extra_environment=borg_environment,
borg_local_path=local_path,
borg_exit_codes=borg_exit_codes,
)
+1
View File
@@ -56,5 +56,6 @@ def transfer_archives(
output_log_level=logging.ANSWER,
output_file=DO_NOT_CAPTURE if transfer_arguments.progress else None,
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
extra_environment=environment.make_environment(config),
)
+4 -2
View File
@@ -5,7 +5,7 @@ from borgmatic.execute import execute_command
logger = logging.getLogger(__name__)
def unmount_archive(mount_point, local_path='borg'):
def unmount_archive(config, mount_point, local_path='borg'):
'''
Given a mounted filesystem mount point, and an optional local Borg paths, umount the filesystem
from the mount point.
@@ -17,4 +17,6 @@ def unmount_archive(mount_point, local_path='borg'):
+ (mount_point,)
)
execute_command(full_command)
execute_command(
full_command, borg_local_path=local_path, borg_exit_codes=config.get('borg_exit_codes')
)
+1
View File
@@ -22,6 +22,7 @@ def local_borg_version(config, local_path='borg'):
full_command,
extra_environment=environment.make_environment(config),
borg_local_path=local_path,
borg_exit_codes=config.get('borg_exit_codes'),
)
try:
+16 -4
View File
@@ -302,7 +302,7 @@ def make_parsers():
metavar='OPTION.SUBOPTION=VALUE',
dest='overrides',
action='append',
help='Configuration file option to override with specified value, can specify flag multiple times',
help='Configuration file option to override with specified value, see documentation for overriding list or key/value options, can specify flag multiple times',
)
global_group.add_argument(
'--no-environment-interpolation',
@@ -467,8 +467,8 @@ def make_parsers():
prune_parser = action_parsers.add_parser(
'prune',
aliases=ACTION_ALIASES['prune'],
help='Prune archives according to the retention policy (with Borg 1.2+, run compact afterwards to actually free space)',
description='Prune archives according to the retention policy (with Borg 1.2+, run compact afterwards to actually free space)',
help='Prune archives according to the retention policy (with Borg 1.2+, you must run compact afterwards to actually free space)',
description='Prune archives according to the retention policy (with Borg 1.2+, you must run compact afterwards to actually free space)',
add_help=False,
)
prune_group = prune_parser.add_argument_group('prune arguments')
@@ -604,11 +604,18 @@ def make_parsers():
action='store_true',
help='Attempt to repair any inconsistencies found (for interactive use)',
)
check_group.add_argument(
'-a',
'--match-archives',
'--glob-archives',
metavar='PATTERN',
help='Only check archives with names matching this pattern',
)
check_group.add_argument(
'--only',
metavar='CHECK',
choices=('repository', 'archives', 'data', 'extract'),
dest='only',
dest='only_checks',
action='append',
help='Run a particular consistency check (repository, archives, data, or extract) instead of configured checks (subject to configured frequency, can specify flag multiple times)',
)
@@ -724,6 +731,11 @@ def make_parsers():
action='store_true',
help='Display progress for each file as it is extracted',
)
config_bootstrap_group.add_argument(
'--ssh-command',
metavar='COMMAND',
help='Command to use instead of "ssh"',
)
config_bootstrap_group.add_argument(
'-h', '--help', action='help', help='Show this help message and exit'
)
+66 -42
View File
@@ -1,4 +1,5 @@
import collections
import importlib.metadata
import json
import logging
import os
@@ -9,11 +10,6 @@ from subprocess import CalledProcessError
import colorama
try:
import importlib_metadata
except ModuleNotFoundError: # pragma: nocover
import importlib.metadata as importlib_metadata
import borgmatic.actions.borg
import borgmatic.actions.break_lock
import borgmatic.actions.check
@@ -48,11 +44,25 @@ from borgmatic.verbosity import verbosity_to_log_level
logger = logging.getLogger(__name__)
def run_configuration(config_filename, config, arguments):
def get_skip_actions(config, arguments):
'''
Given a config filename, the corresponding parsed config dict, and command-line arguments as a
dict from subparser name to a namespace of parsed arguments, execute the defined create, prune,
compact, check, and/or other actions.
Given a configuration dict and command-line arguments as an argparse.Namespace, return a list of
the configured action names to skip. Omit "check" from this list though if "check --force" is
part of the command-like arguments.
'''
skip_actions = config.get('skip_actions', [])
if 'check' in arguments and arguments['check'].force:
return [action for action in skip_actions if action != 'check']
return skip_actions
def run_configuration(config_filename, config, config_paths, arguments):
'''
Given a config filename, the corresponding parsed config dict, a sequence of loaded
configuration paths, and command-line arguments as a dict from subparser name to a namespace of
parsed arguments, execute the defined create, prune, compact, check, and/or other actions.
Yield a combination of:
@@ -70,9 +80,16 @@ def run_configuration(config_filename, config, arguments):
using_primary_action = {'create', 'prune', 'compact', 'check'}.intersection(arguments)
monitoring_log_level = verbosity_to_log_level(global_arguments.monitoring_verbosity)
monitoring_hooks_are_activated = using_primary_action and monitoring_log_level != DISABLED
skip_actions = get_skip_actions(config, arguments)
if skip_actions:
logger.debug(
f"{config_filename}: Skipping {'/'.join(skip_actions)} action{'s' if len(skip_actions) > 1 else ''} due to configured skip_actions"
)
try:
local_borg_version = borg_version.local_borg_version(config, local_path)
logger.debug(f'{config_filename}: Borg {local_borg_version}')
except (OSError, CalledProcessError, ValueError) as error:
yield from log_error_records(f'{config_filename}: Error getting local Borg version', error)
return
@@ -127,6 +144,7 @@ def run_configuration(config_filename, config, arguments):
arguments=arguments,
config_filename=config_filename,
config=config,
config_paths=config_paths,
local_path=local_path,
remote_path=remote_path,
local_borg_version=local_borg_version,
@@ -247,6 +265,7 @@ def run_actions(
arguments,
config_filename,
config,
config_paths,
local_path,
remote_path,
local_borg_version,
@@ -254,9 +273,9 @@ def run_actions(
):
'''
Given parsed command-line arguments as an argparse.ArgumentParser instance, the configuration
filename, several different configuration dicts, local and remote paths to Borg, a local Borg
version string, and a repository name, run all actions from the command-line arguments on the
given repository.
filename, a configuration dict, a sequence of loaded configuration paths, local and remote paths
to Borg, a local Borg version string, and a repository name, run all actions from the
command-line arguments on the given repository.
Yield JSON output strings from executing any actions that produce JSON.
@@ -274,6 +293,7 @@ def run_actions(
'repositories': ','.join([repo['path'] for repo in config['repositories']]),
'log_file': global_arguments.log_file if global_arguments.log_file else '',
}
skip_actions = set(get_skip_actions(config, arguments))
command.execute_hook(
config.get('before_actions'),
@@ -285,7 +305,7 @@ def run_actions(
)
for action_name, action_arguments in arguments.items():
if action_name == 'rcreate':
if action_name == 'rcreate' and action_name not in skip_actions:
borgmatic.actions.rcreate.run_rcreate(
repository,
config,
@@ -295,7 +315,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'transfer':
elif action_name == 'transfer' and action_name not in skip_actions:
borgmatic.actions.transfer.run_transfer(
repository,
config,
@@ -305,11 +325,12 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'create':
elif action_name == 'create' and action_name not in skip_actions:
yield from borgmatic.actions.create.run_create(
config_filename,
repository,
config,
config_paths,
hook_context,
local_borg_version,
action_arguments,
@@ -318,7 +339,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'prune':
elif action_name == 'prune' and action_name not in skip_actions:
borgmatic.actions.prune.run_prune(
config_filename,
repository,
@@ -331,7 +352,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'compact':
elif action_name == 'compact' and action_name not in skip_actions:
borgmatic.actions.compact.run_compact(
config_filename,
repository,
@@ -344,7 +365,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'check':
elif action_name == 'check' and action_name not in skip_actions:
if checks.repository_enabled_for_checks(repository, config):
borgmatic.actions.check.run_check(
config_filename,
@@ -357,7 +378,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'extract':
elif action_name == 'extract' and action_name not in skip_actions:
borgmatic.actions.extract.run_extract(
config_filename,
repository,
@@ -369,7 +390,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'export-tar':
elif action_name == 'export-tar' and action_name not in skip_actions:
borgmatic.actions.export_tar.run_export_tar(
repository,
config,
@@ -379,7 +400,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'mount':
elif action_name == 'mount' and action_name not in skip_actions:
borgmatic.actions.mount.run_mount(
repository,
config,
@@ -389,7 +410,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'restore':
elif action_name == 'restore' and action_name not in skip_actions:
borgmatic.actions.restore.run_restore(
repository,
config,
@@ -399,7 +420,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'rlist':
elif action_name == 'rlist' and action_name not in skip_actions:
yield from borgmatic.actions.rlist.run_rlist(
repository,
config,
@@ -409,7 +430,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'list':
elif action_name == 'list' and action_name not in skip_actions:
yield from borgmatic.actions.list.run_list(
repository,
config,
@@ -419,7 +440,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'rinfo':
elif action_name == 'rinfo' and action_name not in skip_actions:
yield from borgmatic.actions.rinfo.run_rinfo(
repository,
config,
@@ -429,7 +450,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'info':
elif action_name == 'info' and action_name not in skip_actions:
yield from borgmatic.actions.info.run_info(
repository,
config,
@@ -439,7 +460,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'break-lock':
elif action_name == 'break-lock' and action_name not in skip_actions:
borgmatic.actions.break_lock.run_break_lock(
repository,
config,
@@ -449,7 +470,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'export':
elif action_name == 'export' and action_name not in skip_actions:
borgmatic.actions.export_key.run_export_key(
repository,
config,
@@ -459,7 +480,7 @@ def run_actions(
local_path,
remote_path,
)
elif action_name == 'borg':
elif action_name == 'borg' and action_name not in skip_actions:
borgmatic.actions.borg.run_borg(
repository,
config,
@@ -484,13 +505,15 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
'''
Given a sequence of configuration filenames, load and validate each configuration file. Return
the results as a tuple of: dict of configuration filename to corresponding parsed configuration,
and sequence of logging.LogRecord instances containing any parse errors.
a sequence of paths for all loaded configuration files (including includes), and a sequence of
logging.LogRecord instances containing any parse errors.
Log records are returned here instead of being logged directly because logging isn't yet
initialized at this point!
'''
# Dict mapping from config filename to corresponding parsed config dict.
configs = collections.OrderedDict()
config_paths = set()
logs = []
# Parse and load each configuration file.
@@ -507,9 +530,10 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
]
)
try:
configs[config_filename], parse_logs = validate.parse_configuration(
configs[config_filename], paths, parse_logs = validate.parse_configuration(
config_filename, validate.schema_filename(), overrides, resolve_env
)
config_paths.update(paths)
logs.extend(parse_logs)
except PermissionError:
logs.extend(
@@ -539,7 +563,7 @@ def load_configurations(config_filenames, overrides=None, resolve_env=True):
]
)
return (configs, logs)
return (configs, sorted(config_paths), logs)
def log_record(suppress_log=False, **kwargs):
@@ -706,12 +730,12 @@ def collect_highlander_action_summary_logs(configs, arguments, configuration_par
return
def collect_configuration_run_summary_logs(configs, arguments):
def collect_configuration_run_summary_logs(configs, config_paths, arguments):
'''
Given a dict of configuration filename to corresponding parsed configuration and parsed
command-line arguments as a dict from subparser name to a parsed namespace of arguments, run
each configuration file and yield a series of logging.LogRecord instances containing summary
information about each run.
Given a dict of configuration filename to corresponding parsed configuration, a sequence of
loaded configuration paths, and parsed command-line arguments as a dict from subparser name to a
parsed namespace of arguments, run each configuration file and yield a series of
logging.LogRecord instances containing summary information about each run.
As a side effect of running through these configuration files, output their JSON results, if
any, to stdout.
@@ -756,7 +780,7 @@ def collect_configuration_run_summary_logs(configs, arguments):
# Execute the actions corresponding to each configuration file.
json_results = []
for config_filename, config in configs.items():
results = list(run_configuration(config_filename, config, arguments))
results = list(run_configuration(config_filename, config, config_paths, arguments))
error_logs = tuple(result for result in results if isinstance(result, logging.LogRecord))
if error_logs:
@@ -777,6 +801,7 @@ def collect_configuration_run_summary_logs(configs, arguments):
logger.info(f"Unmounting mount point {arguments['umount'].mount_point}")
try:
borg_umount.unmount_archive(
config,
mount_point=arguments['umount'].mount_point,
local_path=get_local_path(configs),
)
@@ -827,7 +852,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
global_arguments = arguments['global']
if global_arguments.version:
print(importlib_metadata.version('borgmatic'))
print(importlib.metadata.version('borgmatic'))
sys.exit(0)
if global_arguments.bash_completion:
print(borgmatic.commands.completion.bash.bash_completion())
@@ -837,8 +862,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
sys.exit(0)
config_filenames = tuple(collect.collect_config_filenames(global_arguments.config_paths))
global_arguments.used_config_paths = list(config_filenames)
configs, parse_logs = load_configurations(
configs, config_paths, parse_logs = load_configurations(
config_filenames, global_arguments.overrides, global_arguments.resolve_env
)
configuration_parse_errors = (
@@ -875,7 +899,7 @@ def main(extra_summary_logs=[]): # pragma: no cover
configs, arguments, configuration_parse_errors
)
)
or list(collect_configuration_run_summary_logs(configs, arguments))
or list(collect_configuration_run_summary_logs(configs, config_paths, arguments))
)
)
summary_logs_max_level = max(log.levelno for log in summary_logs)
+5 -5
View File
@@ -1,9 +1,9 @@
def repository_enabled_for_checks(repository, consistency):
def repository_enabled_for_checks(repository, config):
'''
Given a repository name and a consistency configuration dict, return whether the repository
is enabled to have consistency checks run.
Given a repository name and a configuration dict, return whether the
repository is enabled to have consistency checks run.
'''
if not consistency.get('check_repositories'):
if not config.get('check_repositories'):
return True
return repository in consistency['check_repositories']
return repository in config['check_repositories']
+61
View File
@@ -0,0 +1,61 @@
import shlex
def coerce_scalar(value):
'''
Given a configuration value, coerce it to an integer or a boolean as appropriate and return the
result.
'''
try:
return int(value)
except (TypeError, ValueError):
pass
if value == 'true' or value == 'True':
return True
if value == 'false' or value == 'False':
return False
return value
def apply_constants(value, constants, shell_escape=False):
'''
Given a configuration value (bool, dict, int, list, or string) and a dict of named constants,
replace any configuration string values of the form "{constant}" (or containing it) with the
value of the correspondingly named key from the constants. Recurse as necessary into nested
configuration to find values to replace.
For instance, if a configuration value contains "{foo}", replace it with the value of the "foo"
key found within the configuration's "constants".
If shell escape is True, then escape the constant's value before applying it.
Return the configuration value and modify the original.
'''
if not value or not constants:
return value
if isinstance(value, str):
for constant_name, constant_value in constants.items():
value = value.replace(
'{' + constant_name + '}',
shlex.quote(str(constant_value)) if shell_escape else str(constant_value),
)
# Support constants within non-string scalars by coercing the value to its appropriate type.
value = coerce_scalar(value)
elif isinstance(value, list):
for index, list_value in enumerate(value):
value[index] = apply_constants(list_value, constants, shell_escape)
elif isinstance(value, dict):
for option_name, option_value in value.items():
shell_escape = (
shell_escape
or option_name.startswith('before_')
or option_name.startswith('after_')
or option_name == 'on_error'
)
value[option_name] = apply_constants(option_value, constants, shell_escape)
return value
+20 -14
View File
@@ -1,21 +1,22 @@
import os
import re
_VARIABLE_PATTERN = re.compile(
VARIABLE_PATTERN = re.compile(
r'(?P<escape>\\)?(?P<variable>\$\{(?P<name>[A-Za-z0-9_]+)((:?-)(?P<default>[^}]+))?\})'
)
def _resolve_string(matcher):
def resolve_string(matcher):
'''
Get the value from environment given a matcher containing a name and an optional default value.
If the variable is not defined in environment and no default value is provided, an Error is raised.
Given a matcher containing a name and an optional default value, get the value from environment.
Raise ValueError if the variable is not defined in environment and no default value is provided.
'''
if matcher.group('escape') is not None:
# in case of escaped envvar, unescape it
# In the case of an escaped environment variable, unescape it.
return matcher.group('variable')
# resolve the env var
# Resolve the environment variable.
name, default = matcher.group('name'), matcher.group('default')
out = os.getenv(name, default=default)
@@ -27,19 +28,24 @@ def _resolve_string(matcher):
def resolve_env_variables(item):
'''
Resolves variables like or ${FOO} from given configuration with values from process environment
Supported formats:
- ${FOO} will return FOO env variable
- ${FOO-bar} or ${FOO:-bar} will return FOO env variable if it exists, else "bar"
Resolves variables like or ${FOO} from given configuration with values from process environment.
If any variable is missing in environment and no default value is provided, an Error is raised.
Supported formats:
* ${FOO} will return FOO env variable
* ${FOO-bar} or ${FOO:-bar} will return FOO env variable if it exists, else "bar"
Raise if any variable is missing in environment and no default value is provided.
'''
if isinstance(item, str):
return _VARIABLE_PATTERN.sub(_resolve_string, item)
return VARIABLE_PATTERN.sub(resolve_string, item)
if isinstance(item, list):
for i, subitem in enumerate(item):
item[i] = resolve_env_variables(subitem)
for index, subitem in enumerate(item):
item[index] = resolve_env_variables(subitem)
if isinstance(item, dict):
for key, value in item.items():
item[key] = resolve_env_variables(value)
return item
+11 -10
View File
@@ -3,7 +3,7 @@ import io
import os
import re
from ruamel import yaml
import ruamel.yaml
from borgmatic.config import load, normalize
@@ -17,7 +17,7 @@ def insert_newline_before_comment(config, field_name):
field and its comments.
'''
config.ca.items[field_name][1].insert(
0, yaml.tokens.CommentToken('\n', yaml.error.CommentMark(0), None)
0, ruamel.yaml.tokens.CommentToken('\n', ruamel.yaml.error.CommentMark(0), None)
)
@@ -32,12 +32,12 @@ def schema_to_sample_configuration(schema, level=0, parent_is_sequence=False):
return example
if schema_type == 'array':
config = yaml.comments.CommentedSeq(
config = ruamel.yaml.comments.CommentedSeq(
[schema_to_sample_configuration(schema['items'], level, parent_is_sequence=True)]
)
add_comments_to_configuration_sequence(config, schema, indent=(level * INDENT))
elif schema_type == 'object':
config = yaml.comments.CommentedMap(
config = ruamel.yaml.comments.CommentedMap(
[
(field_name, schema_to_sample_configuration(sub_schema, level + 1))
for field_name, sub_schema in schema['properties'].items()
@@ -101,7 +101,7 @@ def render_configuration(config):
'''
Given a config data structure of nested OrderedDicts, render the config as YAML and return it.
'''
dumper = yaml.YAML()
dumper = ruamel.yaml.YAML(typ='rt')
dumper.indent(mapping=INDENT, sequence=INDENT + SEQUENCE_INDENT, offset=INDENT)
rendered = io.StringIO()
dumper.dump(config, rendered)
@@ -225,8 +225,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
favoring values from the source when there are collisions.
The purpose of this is to upgrade configuration files from old versions of borgmatic by adding
new
configuration keys and comments.
new configuration keys and comments.
'''
if not source_config:
return destination_config
@@ -236,7 +235,9 @@ def merge_source_configuration_into_destination(destination_config, source_confi
for field_name, source_value in source_config.items():
# Since this key/value is from the source configuration, leave it uncommented and remove any
# sentinel that would cause it to get commented out.
remove_commented_out_sentinel(destination_config, field_name)
remove_commented_out_sentinel(
ruamel.yaml.comments.CommentedMap(destination_config), field_name
)
# This is a mapping. Recurse for this key/value.
if isinstance(source_value, collections.abc.Mapping):
@@ -248,7 +249,7 @@ def merge_source_configuration_into_destination(destination_config, source_confi
# This is a sequence. Recurse for each item in it.
if isinstance(source_value, collections.abc.Sequence) and not isinstance(source_value, str):
destination_value = destination_config[field_name]
destination_config[field_name] = yaml.comments.CommentedSeq(
destination_config[field_name] = ruamel.yaml.comments.CommentedSeq(
[
merge_source_configuration_into_destination(
destination_value[index] if index < len(destination_value) else None,
@@ -275,7 +276,7 @@ def generate_sample_configuration(
schema. If a source filename is provided, merge the parsed contents of that configuration into
the generated configuration.
'''
schema = yaml.round_trip_load(open(schema_filename))
schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
source_config = None
if source_filename:
+40 -44
View File
@@ -1,6 +1,5 @@
import functools
import itertools
import json
import logging
import operator
import os
@@ -10,18 +9,18 @@ import ruamel.yaml
logger = logging.getLogger(__name__)
def probe_and_include_file(filename, include_directories):
def probe_and_include_file(filename, include_directories, config_paths):
'''
Given a filename to include and a list of include directories to search for matching files,
probe for the file, load it, and return the loaded configuration as a data structure of nested
dicts, lists, etc.
Given a filename to include, a list of include directories to search for matching files, and a
set of configuration paths, probe for the file, load it, and return the loaded configuration as
a data structure of nested dicts, lists, etc. Add the filename to the given configuration paths.
Raise FileNotFoundError if the included file was not found.
'''
expanded_filename = os.path.expanduser(filename)
if os.path.isabs(expanded_filename):
return load_configuration(expanded_filename)
return load_configuration(expanded_filename, config_paths)
candidate_filenames = {
os.path.join(directory, expanded_filename) for directory in include_directories
@@ -29,32 +28,33 @@ def probe_and_include_file(filename, include_directories):
for candidate_filename in candidate_filenames:
if os.path.exists(candidate_filename):
return load_configuration(candidate_filename)
return load_configuration(candidate_filename, config_paths)
raise FileNotFoundError(
f'Could not find include {filename} at {" or ".join(candidate_filenames)}'
)
def include_configuration(loader, filename_node, include_directory):
def include_configuration(loader, filename_node, include_directory, config_paths):
'''
Given a ruamel.yaml.loader.Loader, a ruamel.yaml.nodes.ScalarNode containing the included
filename (or a list containing multiple such filenames), and an include directory path to search
for matching files, load the given YAML filenames (ignoring the given loader so we can use our
own) and return their contents as data structure of nested dicts, lists, etc. If the given
filename (or a list containing multiple such filenames), an include directory path to search for
matching files, and a set of configuration paths, load the given YAML filenames (ignoring the
given loader so we can use our own) and return their contents as data structure of nested dicts,
lists, etc. Add the names of included files to the given configuration paths. If the given
filename node's value is a scalar string, then the return value will be a single value. But if
the given node value is a list, then the return value will be a list of values, one per loaded
configuration file.
If a filename is relative, probe for it within 1. the current working directory and 2. the given
include directory.
If a filename is relative, probe for it within: 1. the current working directory and 2. the
given include directory.
Raise FileNotFoundError if an included file was not found.
'''
include_directories = [os.getcwd(), os.path.abspath(include_directory)]
if isinstance(filename_node.value, str):
return probe_and_include_file(filename_node.value, include_directories)
return probe_and_include_file(filename_node.value, include_directories, config_paths)
if (
isinstance(filename_node.value, list)
@@ -64,7 +64,7 @@ def include_configuration(loader, filename_node, include_directory):
# Reversing the values ensures the correct ordering if these includes are subsequently
# merged together.
return [
probe_and_include_file(node.value, include_directories)
probe_and_include_file(node.value, include_directories, config_paths)
for node in reversed(filename_node.value)
]
@@ -110,11 +110,17 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
separate YAML configuration files. Example syntax: `option: !include common.yaml`
'''
def __init__(self, preserve_quotes=None, loader=None, include_directory=None):
def __init__(
self, preserve_quotes=None, loader=None, include_directory=None, config_paths=None
):
super(Include_constructor, self).__init__(preserve_quotes, loader)
self.add_constructor(
'!include',
functools.partial(include_configuration, include_directory=include_directory),
functools.partial(
include_configuration,
include_directory=include_directory,
config_paths=config_paths,
),
)
# These are catch-all error handlers for tags that don't get applied and removed by
@@ -156,46 +162,36 @@ class Include_constructor(ruamel.yaml.SafeConstructor):
node.value = deep_merge_nodes(node.value)
def load_configuration(filename):
def load_configuration(filename, config_paths=None):
'''
Load the given configuration file and return its contents as a data structure of nested dicts
and lists. Also, replace any "{constant}" strings with the value of the "constant" key in the
"constants" option of the configuration file.
and lists. Add the filename to the given configuration paths set, and also add any included
configuration filenames.
Raise ruamel.yaml.error.YAMLError if something goes wrong parsing the YAML, or RecursionError
if there are too many recursive includes.
'''
if config_paths is None:
config_paths = set()
# Use an embedded derived class for the include constructor so as to capture the filename
# value. (functools.partial doesn't work for this use case because yaml.Constructor has to be
# an actual class.)
class Include_constructor_with_include_directory(Include_constructor):
# Use an embedded derived class for the include constructor so as to capture the include
# directory and configuration paths values. (functools.partial doesn't work for this use case
# because yaml.Constructor has to be an actual class.)
class Include_constructor_with_extras(Include_constructor):
def __init__(self, preserve_quotes=None, loader=None):
super(Include_constructor_with_include_directory, self).__init__(
preserve_quotes, loader, include_directory=os.path.dirname(filename)
super(Include_constructor_with_extras, self).__init__(
preserve_quotes,
loader,
include_directory=os.path.dirname(filename),
config_paths=config_paths,
)
yaml = ruamel.yaml.YAML(typ='safe')
yaml.Constructor = Include_constructor_with_include_directory
yaml.Constructor = Include_constructor_with_extras
config_paths.add(filename)
with open(filename) as file:
file_contents = file.read()
config = yaml.load(file_contents)
try:
has_constants = bool(config and 'constants' in config)
except TypeError:
has_constants = False
if has_constants:
for key, value in config['constants'].items():
value = json.dumps(value)
file_contents = file_contents.replace(f'{{{key}}}', value.strip('"'))
config = yaml.load(file_contents)
del config['constants']
return config
return yaml.load(file.read())
def filter_omitted_nodes(nodes, values):
+5 -2
View File
@@ -192,7 +192,7 @@ def normalize(config_filename, config):
# Upgrade remote repositories to ssh:// syntax, required in Borg 2.
repositories = config.get('repositories')
if repositories:
if isinstance(repositories[0], str):
if any(isinstance(repository, str) for repository in repositories):
logs.append(
logging.makeLogRecord(
dict(
@@ -202,7 +202,10 @@ def normalize(config_filename, config):
)
)
)
config['repositories'] = [{'path': repository} for repository in repositories]
config['repositories'] = [
{'path': repository} if isinstance(repository, str) else repository
for repository in repositories
]
repositories = config['repositories']
config['repositories'] = []
+49 -13
View File
@@ -13,6 +13,11 @@ def set_values(config, keys, value):
first_key = keys[0]
if len(keys) == 1:
if isinstance(config, list):
raise ValueError(
'When overriding a list option, the value must use list syntax (e.g., "[foo, bar]" or "[{key: value}]" as appropriate)'
)
config[first_key] = value
return
@@ -22,13 +27,19 @@ def set_values(config, keys, value):
set_values(config[first_key], keys[1:], value)
def convert_value_type(value):
def convert_value_type(value, option_type):
'''
Given a string value, determine its logical type (string, boolean, integer, etc.), and return it
converted to that type.
Given a string value and its schema type as a string, determine its logical type (string,
boolean, integer, etc.), and return it converted to that type.
If the option type is a string, leave the value as a string so that special characters in it
don't get interpreted as YAML during conversion.
Raise ruamel.yaml.error.YAMLError if there's a parse issue with the YAML.
'''
if option_type == 'string':
return value
return ruamel.yaml.YAML(typ='safe').load(io.StringIO(value))
@@ -46,11 +57,32 @@ def strip_section_names(parsed_override_key):
return parsed_override_key
def parse_overrides(raw_overrides):
def type_for_option(schema, option_keys):
'''
Given a sequence of configuration file override strings in the form of "option.suboption=value",
parse and return a sequence of tuples (keys, values), where keys is a sequence of strings. For
instance, given the following raw overrides:
Given a configuration schema and a sequence of keys identifying an option, e.g.
('extra_borg_options', 'init'), return the schema type of that option as a string.
Return None if the option or its type cannot be found in the schema.
'''
option_schema = schema
for key in option_keys:
try:
option_schema = option_schema['properties'][key]
except KeyError:
return None
try:
return option_schema['type']
except KeyError:
return None
def parse_overrides(raw_overrides, schema):
'''
Given a sequence of configuration file override strings in the form of "option.suboption=value"
and a configuration schema dict, parse and return a sequence of tuples (keys, values), where
keys is a sequence of strings. For instance, given the following raw overrides:
['my_option.suboption=value1', 'other_option=value2']
@@ -71,10 +103,13 @@ def parse_overrides(raw_overrides):
for raw_override in raw_overrides:
try:
raw_keys, value = raw_override.split('=', 1)
keys = strip_section_names(tuple(raw_keys.split('.')))
option_type = type_for_option(schema, keys)
parsed_overrides.append(
(
strip_section_names(tuple(raw_keys.split('.'))),
convert_value_type(value),
keys,
convert_value_type(value, option_type),
)
)
except ValueError:
@@ -87,12 +122,13 @@ def parse_overrides(raw_overrides):
return tuple(parsed_overrides)
def apply_overrides(config, raw_overrides):
def apply_overrides(config, schema, raw_overrides):
'''
Given a configuration dict and a sequence of configuration file override strings in the form of
"option.suboption=value", parse each override and set it the configuration dict.
Given a configuration dict, a corresponding configuration schema dict, and a sequence of
configuration file override strings in the form of "option.suboption=value", parse each override
and set it into the configuration dict.
'''
overrides = parse_overrides(raw_overrides)
overrides = parse_overrides(raw_overrides, schema)
for keys, value in overrides:
set_values(config, keys, value)
+90 -21
View File
@@ -6,14 +6,15 @@ properties:
constants:
type: object
description: |
Constants to use in the configuration file. All occurrences of the
constant name within curly braces will be replaced with the value.
For example, if you have a constant named "hostname" with the value
"myhostname", then the string "{hostname}" will be replaced with
"myhostname" in the configuration file.
Constants to use in the configuration file. Within option values,
all occurrences of the constant name in curly braces will be
replaced with the constant value. For example, if you have a
constant named "app_name" with the value "myapp", then the string
"{app_name}" will be replaced with "myapp" in the configuration
file.
example:
hostname: myhostname
prefix: myprefix
app_name: myapp
user: myuser
source_directories:
type: array
items:
@@ -216,7 +217,7 @@ properties:
Store configuration files used to create a backup in the backup
itself. Defaults to true. Changing this to false prevents "borgmatic
bootstrap" from extracting configuration files from the backup.
example: true
example: false
source_directories_must_exist:
type: boolean
description: |
@@ -287,14 +288,17 @@ properties:
retry_wait:
type: integer
description: |
Wait time between retries (in seconds) to allow transient issues to
pass. Increases after each retry as a form of backoff. Defaults to 0
(no wait).
Wait time between retries (in seconds) to allow transient issues
to pass. Increases after each retry by that same wait time as a
form of backoff. Defaults to 0 (no wait).
example: 10
temporary_directory:
type: string
description: |
Directory where temporary files are stored. Defaults to $TMPDIR.
Directory where temporary Borg files are stored. Defaults to
$TMPDIR. See "Resource Usage" at
https://borgbackup.readthedocs.io/en/stable/usage/general.html for
details.
example: /path/to/tmpdir
ssh_command:
type: string
@@ -337,6 +341,37 @@ properties:
Path for Borg encryption key files. Defaults to
$borg_base_directory/.config/borg/keys
example: /path/to/base/config/keys
borg_exit_codes:
type: array
items:
type: object
required: ['code', 'treat_as']
additionalProperties: false
properties:
code:
type: integer
not: {enum: [0]}
description: |
The exit code for an existing Borg warning or error.
example: 100
treat_as:
type: string
enum: ['error', 'warning']
description: |
Whether to consider the exit code as an error or as a
warning in borgmatic.
example: error
description: |
A list of Borg exit codes that should be elevated to errors or
squashed to warnings as indicated. By default, Borg error exit codes
(2 to 99) are treated as errors while warning exit codes (1 and
100+) are treated as warnings. Exit codes other than 1 and 2 are
only present in Borg 1.4.0+.
example:
- code: 13
treat_as: warning
- code: 100
treat_as: error
umask:
type: integer
description: |
@@ -423,7 +458,9 @@ properties:
command-line invocation.
keep_within:
type: string
description: Keep all archives within this time interval.
description: |
Keep all archives within this time interval. See "skip_actions" for
disabling pruning altogether.
example: 3H
keep_secondly:
type: integer
@@ -479,13 +516,13 @@ properties:
- disabled
description: |
Name of consistency check to run: "repository",
"archives", "data", and/or "extract". Set to "disabled"
to disable all consistency checks. "repository" checks
the consistency of the repository, "archives" checks all
of the archives, "data" verifies the integrity of the
data within the archives, and "extract" does an
extraction dry-run of the most recent archive. Note that
"data" implies "archives".
"archives", "data", and/or "extract". "repository"
checks the consistency of the repository, "archives"
checks all of the archives, "data" verifies the
integrity of the data within the archives, and "extract"
does an extraction dry-run of the most recent archive.
Note that "data" implies "archives". See "skip_actions"
for disabling checks altogether.
example: repository
frequency:
type: string
@@ -525,6 +562,38 @@ properties:
Apply color to console output. Can be overridden with --no-color
command-line flag. Defaults to true.
example: false
skip_actions:
type: array
items:
type: string
enum:
- rcreate
- transfer
- prune
- compact
- create
- check
- extract
- config
- export-tar
- mount
- umount
- restore
- rlist
- list
- rinfo
- info
- break-lock
- key
- borg
description: |
List of one or more actions to skip running for this configuration
file, even if specified on the command-line (explicitly or
implicitly). This is handy for append-only configurations where you
never want to run "compact" or checkless configuration where you
want to skip "check". Defaults to not skipping any actions.
example:
- compact
before_actions:
type: array
items:
@@ -1493,7 +1562,7 @@ properties:
ends, or errors.
example: https://cronhub.io/ping/1f5e3410-254c-5587
description: |
Configuration for a monitoring integration with Crunhub. Create an
Configuration for a monitoring integration with Cronhub. Create an
account at https://cronhub.io if you'd like to use this service. See
borgmatic monitoring documentation for details.
loki:
+12 -7
View File
@@ -4,7 +4,7 @@ import jsonschema
import ruamel.yaml
import borgmatic.config
from borgmatic.config import environment, load, normalize, override
from borgmatic.config import constants, environment, load, normalize, override
def schema_filename():
@@ -97,23 +97,28 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
'checks': ['repository', 'archives'],
}
Also return a sequence of logging.LogRecord instances containing any warnings about the
configuration.
Also return a set of loaded configuration paths and a sequence of logging.LogRecord instances
containing any warnings about the configuration.
Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
have permissions to read the file, or Validation_error if the config does not match the schema.
'''
config_paths = set()
try:
config = load.load_configuration(config_filename)
config = load.load_configuration(config_filename, config_paths)
schema = load.load_configuration(schema_filename)
except (ruamel.yaml.error.YAMLError, RecursionError) as error:
raise Validation_error(config_filename, (str(error),))
override.apply_overrides(config, overrides)
logs = normalize.normalize(config_filename, config)
override.apply_overrides(config, schema, overrides)
constants.apply_constants(config, config.get('constants') if config else {})
if resolve_env:
environment.resolve_env_variables(config)
logs = normalize.normalize(config_filename, config)
try:
validator = jsonschema.Draft7Validator(schema)
except AttributeError: # pragma: no cover
@@ -127,7 +132,7 @@ def parse_configuration(config_filename, schema_filename, overrides=None, resolv
apply_logical_validation(config_filename, config)
return config, logs
return config, config_paths, logs
def normalize_repository_path(repository):
+92 -29
View File
@@ -1,4 +1,5 @@
import collections
import enum
import logging
import os
import select
@@ -8,22 +9,61 @@ logger = logging.getLogger(__name__)
ERROR_OUTPUT_MAX_LINE_COUNT = 25
BORG_ERROR_EXIT_CODE = 2
BORG_ERROR_EXIT_CODE_START = 2
BORG_ERROR_EXIT_CODE_END = 99
def exit_code_indicates_error(command, exit_code, borg_local_path=None):
class Exit_status(enum.Enum):
STILL_RUNNING = 1
SUCCESS = 2
WARNING = 3
ERROR = 4
def interpret_exit_code(command, exit_code, borg_local_path=None, borg_exit_codes=None):
'''
Return True if the given exit code from running a command corresponds to an error. If a Borg
local path is given and matches the process' command, then treat exit code 1 as a warning
instead of an error.
Return an Exit_status value (e.g. SUCCESS, ERROR, or WARNING) based on interpreting the given
exit code. If a Borg local path is given and matches the process' command, then interpret the
exit code based on Borg's documented exit code semantics. And if Borg exit codes are given as a
sequence of exit code configuration dicts, then take those configured preferences into account.
'''
if exit_code is None:
return False
return Exit_status.STILL_RUNNING
if exit_code == 0:
return Exit_status.SUCCESS
if borg_local_path and command[0] == borg_local_path:
return bool(exit_code < 0 or exit_code >= BORG_ERROR_EXIT_CODE)
# First try looking for the exit code in the borg_exit_codes configuration.
for entry in borg_exit_codes or ():
if entry.get('code') == exit_code:
treat_as = entry.get('treat_as')
return bool(exit_code != 0)
if treat_as == 'error':
logger.error(
f'Treating exit code {exit_code} as an error, as per configuration'
)
return Exit_status.ERROR
elif treat_as == 'warning':
logger.warning(
f'Treating exit code {exit_code} as a warning, as per configuration'
)
return Exit_status.WARNING
# If the exit code doesn't have explicit configuration, then fall back to the default Borg
# behavior.
return (
Exit_status.ERROR
if (
exit_code < 0
or (
exit_code >= BORG_ERROR_EXIT_CODE_START
and exit_code <= BORG_ERROR_EXIT_CODE_END
)
)
else Exit_status.WARNING
)
return Exit_status.ERROR
def command_for_process(process):
@@ -60,7 +100,7 @@ def append_last_lines(last_lines, captured_output, line, output_log_level):
logger.log(output_log_level, line)
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path, borg_exit_codes):
'''
Given a sequence of subprocess.Popen() instances for multiple processes, log the output for each
process with the requested log level. Additionally, raise a CalledProcessError if a process
@@ -68,7 +108,8 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
path).
If output log level is None, then instead of logging, capture output for each process and return
it as a dict from the process to its output.
it as a dict from the process to its output. Use the given Borg local path and exit code
configuration to decide what's an error and what's a warning.
For simplicity, it's assumed that the output buffer for each process is its stdout. But if any
stdouts are given to exclude, then for any matching processes, log from their stderr instead.
@@ -132,11 +173,13 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
if exit_code is None:
still_running = True
command = process.args.split(' ') if isinstance(process.args, str) else process.args
continue
command = process.args.split(' ') if isinstance(process.args, str) else process.args
exit_status = interpret_exit_code(command, exit_code, borg_local_path, borg_exit_codes)
# If any process errors, then raise accordingly.
if exit_code_indicates_error(command, exit_code, borg_local_path):
if exit_status in (Exit_status.ERROR, Exit_status.WARNING):
# If an error occurs, include its output in the raised exception so that we don't
# inadvertently hide error output.
output_buffer = output_buffer_for_process(process, exclude_stdouts)
@@ -162,9 +205,13 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
other_process.stdout.read(0)
other_process.kill()
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
if exit_status == Exit_status.ERROR:
raise subprocess.CalledProcessError(
exit_code, command_for_process(process), '\n'.join(last_lines)
)
still_running = False
break
if captured_outputs:
return {
@@ -172,19 +219,19 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
}
def log_command(full_command, input_file=None, output_file=None):
def log_command(full_command, input_file=None, output_file=None, environment=None):
'''
Log the given command (a sequence of command/argument strings), along with its input/output file
paths.
paths and extra environment variables (with omitted values in case they contain passwords).
'''
logger.debug(
' '.join(full_command)
' '.join(tuple(f'{key}=***' for key in (environment or {}).keys()) + tuple(full_command))
+ (f" < {getattr(input_file, 'name', '')}" if input_file else '')
+ (f" > {getattr(output_file, 'name', '')}" if output_file else '')
)
# An sentinel passed as an output file to execute_command() to indicate that the command's output
# A sentinel passed as an output file to execute_command() to indicate that the command's output
# should be allowed to flow through to stdout without being captured for logging. Useful for
# commands with interactive prompts or those that mess directly with the console.
DO_NOT_CAPTURE = object()
@@ -199,6 +246,7 @@ def execute_command(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
run_to_completion=True,
):
'''
@@ -209,12 +257,13 @@ def execute_command(
augment the current environment, and pass the result into the command. If a working directory is
given, use that as the present working directory when running the command. If a Borg local path
is given, and the command matches it (regardless of arguments), treat exit code 1 as a warning
instead of an error. If run to completion is False, then return the process for the command
without executing it to completion.
instead of an error. But if Borg exit codes are given as a sequence of exit code configuration
dicts, then use that configuration to decide what's an error and what's a warning. If run to
completion is False, then return the process for the command without executing it to completion.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
log_command(full_command, input_file, output_file)
log_command(full_command, input_file, output_file, extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
@@ -232,7 +281,11 @@ def execute_command(
return process
log_outputs(
(process,), (input_file, output_file), output_log_level, borg_local_path=borg_local_path
(process,),
(input_file, output_file),
output_log_level,
borg_local_path,
borg_exit_codes,
)
@@ -243,6 +296,7 @@ def execute_command_and_capture_output(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings), capturing and returning its
@@ -251,11 +305,13 @@ def execute_command_and_capture_output(
given, then use it to augment the current environment, and pass the result into the command. If
a working directory is given, use that as the present working directory when running the
command. If a Borg local path is given, and the command matches it (regardless of arguments),
treat exit code 1 as a warning instead of an error.
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command.
'''
log_command(full_command)
log_command(full_command, environment=extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
command = ' '.join(full_command) if shell else full_command
@@ -268,7 +324,10 @@ def execute_command_and_capture_output(
cwd=working_directory,
)
except subprocess.CalledProcessError as error:
if exit_code_indicates_error(command, error.returncode, borg_local_path):
if (
interpret_exit_code(command, error.returncode, borg_local_path, borg_exit_codes)
== Exit_status.ERROR
):
raise
output = error.output
@@ -285,6 +344,7 @@ def execute_command_with_processes(
extra_environment=None,
working_directory=None,
borg_local_path=None,
borg_exit_codes=None,
):
'''
Execute the given command (a sequence of command/argument strings) and log its output at the
@@ -299,12 +359,14 @@ def execute_command_with_processes(
use it to augment the current environment, and pass the result into the command. If a working
directory is given, use that as the present working directory when running the command. If a
Borg local path is given, then for any matching command or process (regardless of arguments),
treat exit code 1 as a warning instead of an error.
treat exit code 1 as a warning instead of an error. But if Borg exit codes are given as a
sequence of exit code configuration dicts, then use that configuration to decide what's an error
and what's a warning.
Raise subprocesses.CalledProcessError if an error occurs while running the command or in the
upstream process.
'''
log_command(full_command, input_file, output_file)
log_command(full_command, input_file, output_file, extra_environment)
environment = {**os.environ, **extra_environment} if extra_environment else None
do_not_capture = bool(output_file is DO_NOT_CAPTURE)
command = ' '.join(full_command) if shell else full_command
@@ -334,7 +396,8 @@ def execute_command_with_processes(
tuple(processes) + (command_process,),
(input_file, output_file),
output_log_level,
borg_local_path=borg_local_path,
borg_local_path,
borg_exit_codes,
)
if output_log_level is None:
+2 -1
View File
@@ -1,6 +1,7 @@
import logging
import os
import re
import shlex
from borgmatic import execute
@@ -16,7 +17,7 @@ def interpolate_context(config_filename, hook_description, command, context):
names/values, interpolate the values by "{name}" into the command and return the result.
'''
for name, value in context.items():
command = command.replace(f'{{{name}}}', str(value))
command = command.replace(f'{{{name}}}', shlex.quote(str(value)))
for unsupported_variable in re.findall(r'{\w+}', command):
logger.warning(
+14 -9
View File
@@ -1,4 +1,5 @@
import logging
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
@@ -62,19 +63,23 @@ def build_dump_command(database, dump_filename, dump_format):
return (
('mongodump',)
+ (('--out', dump_filename) if dump_format == 'directory' else ())
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--password', database['password']) if 'password' in database else ())
+ (('--out', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (('--username', shlex.quote(database['username'])) if 'username' in database else ())
+ (('--password', shlex.quote(database['password'])) if 'password' in database else ())
+ (
('--authenticationDatabase', database['authentication_database'])
('--authenticationDatabase', shlex.quote(database['authentication_database']))
if 'authentication_database' in database
else ()
)
+ (('--db', database['name']) if not all_databases else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (('--archive', '>', dump_filename) if dump_format != 'directory' else ())
+ (('--db', shlex.quote(database['name'])) if not all_databases else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (('--archive', '>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
+17 -9
View File
@@ -137,23 +137,31 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
command = (
(
dump_command,
shlex.quote(dump_command),
'--no-password',
'--clean',
'--if-exists',
)
+ (('--host', database['hostname']) if 'hostname' in database else ())
+ (('--port', str(database['port'])) if 'port' in database else ())
+ (('--username', database['username']) if 'username' in database else ())
+ (('--host', shlex.quote(database['hostname'])) if 'hostname' in database else ())
+ (('--port', shlex.quote(str(database['port']))) if 'port' in database else ())
+ (
('--username', shlex.quote(database['username']))
if 'username' in database
else ()
)
+ (('--no-owner',) if database.get('no_owner', False) else ())
+ (('--format', dump_format) if dump_format else ())
+ (('--file', dump_filename) if dump_format == 'directory' else ())
+ (tuple(database['options'].split(' ')) if 'options' in database else ())
+ (() if database_name == 'all' else (database_name,))
+ (('--format', shlex.quote(dump_format)) if dump_format else ())
+ (('--file', shlex.quote(dump_filename)) if dump_format == 'directory' else ())
+ (
tuple(shlex.quote(option) for option in database['options'].split(' '))
if 'options' in database
else ()
)
+ (() if database_name == 'all' else (shlex.quote(database_name),))
# Use shell redirection rather than the --file flag to sidestep synchronization issues
# when pg_dump/pg_dumpall tries to write to a named pipe. But for the directory dump
# format in a particular, a named destination is required, and redirection doesn't work.
+ (('>', dump_filename) if dump_format != 'directory' else ())
+ (('>', shlex.quote(dump_filename)) if dump_format != 'directory' else ())
)
logger.debug(
+10 -6
View File
@@ -1,5 +1,6 @@
import logging
import os
import shlex
from borgmatic.execute import execute_command, execute_command_with_processes
from borgmatic.hooks import dump
@@ -18,10 +19,12 @@ def make_dump_path(config): # pragma: no cover
def dump_data_sources(databases, config, log_prefix, dry_run):
'''
Dump the given SQLite3 databases to a file. The databases are supplied as a sequence of
Dump the given SQLite3 databases to a named pipe. The databases are supplied as a sequence of
configuration dicts, as per the configuration schema. Use the given configuration dict to
construct the destination path and the given log prefix in any log entries. If this is a dry
run, then don't actually dump anything.
construct the destination path and the given log prefix in any log entries.
Return a sequence of subprocess.Popen instances for the dump processes ready to spew to a named
pipe. But if this is a dry run, then don't actually dump anything and return an empty sequence.
'''
dry_run_label = ' (dry run; not actually dumping anything)' if dry_run else ''
processes = []
@@ -40,6 +43,7 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
dump_path = make_dump_path(config)
dump_filename = dump.make_data_source_dump_filename(dump_path, database['name'])
if os.path.exists(dump_filename):
logger.warning(
f'{log_prefix}: Skipping duplicate dump of SQLite database at {database_path} to {dump_filename}'
@@ -48,10 +52,10 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
command = (
'sqlite3',
database_path,
shlex.quote(database_path),
'.dump',
'>',
dump_filename,
shlex.quote(dump_filename),
)
logger.debug(
f'{log_prefix}: Dumping SQLite database at {database_path} to {dump_filename}{dry_run_label}'
@@ -59,7 +63,7 @@ def dump_data_sources(databases, config, log_prefix, dry_run):
if dry_run:
continue
dump.create_parent_directory_for_dump(dump_filename)
dump.create_named_pipe_for_dump(dump_filename)
processes.append(execute_command(command, shell=True, run_to_completion=False))
return processes
+2 -2
View File
@@ -2,13 +2,13 @@
font-size: 1rem; /* Reset */
}
.elv-toc details {
--details-force-closed: (max-width: 63.9375em); /* 1023px */
--details-force-closed: (max-width: 79.9375em); /* 1023px */
}
.elv-toc details > summary {
font-size: 1.375rem; /* 22px /16 */
margin-bottom: .5em;
}
@media (min-width: 64em) { /* 1024px */
@media (min-width: 80em) {
.elv-toc {
position: absolute;
left: 3rem;
+3 -3
View File
@@ -121,7 +121,7 @@ main h1:first-child,
main .elv-toc + h1 {
border-bottom: 2px dotted #666;
}
@media (min-width: 64em) { /* 1024px */
@media (min-width: 80em) {
main .elv-toc + h1,
main .elv-toc + h2 {
margin-top: 0;
@@ -243,10 +243,10 @@ footer.elv-layout {
.elv-layout-full {
max-width: none;
}
@media (min-width: 64em) { /* 1024px */
@media (min-width: 80em) {
.elv-layout-toc {
padding-left: 15rem;
max-width: 60rem;
max-width: 76rem;
margin-right: 1rem;
position: relative;
}
@@ -126,7 +126,7 @@ for more information.
## Hook output
Any output produced by your hooks shows up both at the console and in syslog
(when run in a non-interactive console). For more information, read about <a
(when enabled). For more information, read about <a
href="https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/">inspecting
your backups</a>.
@@ -0,0 +1,86 @@
---
title: How to customize warnings and errors
eleventyNavigation:
key: 💥 Customize warnings/errors
parent: How-to guides
order: 12
---
## When things go wrong
After Borg runs, it indicates whether it succeeded via its exit code, a
numeric ID indicating success, warning, or error. borgmatic consumes this exit
code to decide how to respond. Normally, a Borg error results in a borgmatic
error, while a Borg warning or success doesn't.
But if that default behavior isn't sufficient for your needs, you can
customize how borgmatic interprets [Borg's exit
codes](https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes).
For instance, to elevate Borg warnings to errors, thereby causing borgmatic to
error on them, use the following borgmatic configuration:
```yaml
borg_exit_codes:
- exit_code: 1
treat_as: error
```
Be aware though that Borg exits with a warning code for a variety of benign
situations such as files changing while they're being read, so this example
may not meet your needs. Keep reading though for more granular exit code
configuration.
Here's an example that squashes Borg errors to warnings:
```yaml
borg_exit_codes:
- exit_code: 2
treat_as: warning
```
Be careful with this example though, because it prevents borgmatic from
erroring when Borg errors, which may not be desirable.
### More granular configuration
<span class="minilink minilink-addedin">New in Borg version 1.4</span> Borg
support for [more granular exit
codes](https://borgbackup.readthedocs.io/en/1.4-maint/usage/general.html#return-codes)
means that you can configure borgmatic to respond to specific Borg conditions.
See the full list of [Borg 1.4 error and warning exit
codes](https://borgbackup.readthedocs.io/en/1.4.0b1/internals/frontends.html#message-ids).
The `rc:` numeric value there tells you the exit code for each.
For instance, this borgmatic configuration elevates all Borg backup file
permission warnings (exit code `105`)—and only those warnings—to errors:
```yaml
borg_exit_codes:
- exit_code: 105
treat_as: error
```
The following configuration does that *and* elevates backup file not found
warnings (exit code `107`) to errors as well:
```yaml
borg_exit_codes:
- exit_code: 105
treat_as: error
- exit_code: 107
treat_as: error
```
If you don't know the exit code for a particular Borg error or warning you're
experiencing, you can usually find it in your borgmatic output when
`--verbosity 2` is enabled. For instance, here's a snippet of that output when
a backup file is not found:
```
/noexist: stat: [Errno 2] No such file or directory: '/noexist'
...
terminating with warning status, rc 107
```
So if you want to configure borgmatic to treat this as an error instead of a
warning, the exit status to use is `107`.
+33 -6
View File
@@ -51,6 +51,11 @@ cron job), while only running expensive consistency checks with `check` on a
much less frequent basis (e.g. with `borgmatic check` called from a separate
cron job).
<span class="minilink minilink-addedin">New in version 1.8.5</span> Instead of
(or in addition to) specifying actions on the command-line, you can configure
borgmatic to [skip particular
actions](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#skipping-actions).
### Consistency check configuration
@@ -116,8 +121,17 @@ this option in the `consistency:` section of your configuration.
This tells borgmatic to run the `repository` consistency check at most once
every two weeks for a given repository and the `archives` check at most once a
month. The `frequency` value is a number followed by a unit of time, e.g. "3
days", "1 week", "2 months", etc.
month. The `frequency` value is a number followed by a unit of time, e.g. `3
days`, `1 week`, `2 months`, etc. The set of possible time units is as
follows (singular or plural):
* `second`
* `minute`
* `hour`
* `day`
* `week` (7 days)
* `month` (30 days)
* `year` (365 days)
The `frequency` defaults to `always` for a check configured without a
`frequency`, which means run this check every time checks run. But if you omit
@@ -139,6 +153,10 @@ though—or the most frequently configured check will apply.
If you want to temporarily ignore your configured frequencies, you can invoke
`borgmatic check --force` to run checks unconditionally.
<span class="minilink minilink-addedin">New in version 1.8.6</span> `borgmatic
check --force` runs `check` even if it's specified in the `skip_actions`
option.
### Running only checks
@@ -162,7 +180,16 @@ location:
If that's still too slow, you can disable consistency checks entirely,
either for a single repository or for all repositories.
Disabling all consistency checks looks like this:
<span class="minilink minilink-addedin">New in version 1.8.5</span> Disabling
all consistency checks looks like this:
```yaml
skip_actions:
- check
```
<span class="minilink minilink-addedin">Prior to version 1.8.5</span> Use this
configuration instead:
```yaml
checks:
@@ -170,10 +197,10 @@ checks:
```
<span class="minilink minilink-addedin">Prior to version 1.8.0</span> Put
this option in the `consistency:` section of your configuration.
`checks:` in the `consistency:` section of your configuration.
<span class="minilink minilink-addedin">Prior to version 1.6.2</span> `checks`
was a plain list of strings without the `name:` part. For instance:
<span class="minilink minilink-addedin">Prior to version 1.6.2</span>
`checks:` was a plain list of strings without the `name:` part. For instance:
```yaml
checks:
+1 -1
View File
@@ -3,7 +3,7 @@ title: How to develop on borgmatic
eleventyNavigation:
key: 🏗️ Develop on borgmatic
parent: How-to guides
order: 13
order: 14
---
## Source code
+14 -5
View File
@@ -149,9 +149,10 @@ borgmatic umount --mount-point /mnt
<span class="minilink minilink-addedin">New in version 1.7.15</span> borgmatic
automatically stores all the configuration files used to create an archive
inside the archive itself. This is useful in cases where you've lost a
configuration file or you want to see what configurations were used to create a
particular archive.
inside the archive itself. They are stored in the archive using their full
paths from the machine being backed up. This is useful in cases where you've
lost a configuration file or you want to see what configurations were used to
create a particular archive.
To extract the configuration files from an archive, use the `config bootstrap`
action. For example:
@@ -166,8 +167,8 @@ configuration file used to create this archive was located at
`/etc/borgmatic/config.yaml` when the archive was created.
Note that to run the `config bootstrap` action, you don't need to have a
borgmatic configuration file. You only need to specify the repository to use via
the `--repository` flag; borgmatic will figure out the rest.
borgmatic configuration file. You only need to specify the repository to use
via the `--repository` flag; borgmatic will figure out the rest.
If a destination directory is not specified, the configuration files will be
extracted to their original locations, silently *overwriting* any configuration
@@ -182,8 +183,16 @@ If you want to extract the configuration file from a specific archive, use the
borgmatic config bootstrap --repository repo.borg --archive host-2023-01-02T04:06:07.080910 --destination /tmp
```
See the output of `config bootstrap --help` for additional flags you may need
for bootstrapping.
<span class="minilink minilink-addedin">New in version 1.8.1</span> Set the
`store_config_files` option to `false` to disable the automatic backup of
borgmatic configuration files, for instance if they contain sensitive
information you don't want to store even inside your encrypted backups. If you
do this though, the `config bootstrap` action will no longer work.
<span class="minilink minilink-addedin">New in version 1.8.7</span> Included
configuration files are stored in each backup archive. This means that the
`config bootstrap` action not only extracts the top-level configuration files
but also the includes they depend upon.
+16 -13
View File
@@ -116,27 +116,30 @@ archive, complete with file sizes.
## Logging
By default, borgmatic logs to a local syslog-compatible daemon if one is
present and borgmatic is running in a non-interactive console. Where those
logs show up depends on your particular system. If you're using systemd, try
running `journalctl -xe`. Otherwise, try viewing `/var/log/syslog` or
similar.
You can customize the log level used for syslog logging with the
`--syslog-verbosity` flag, and this is independent from the console logging
`--verbosity` flag described above. For instance, to get additional
information about the progress of the backup as it proceeds:
By default, borgmatic logs to the console. You can enable simultaneous syslog
logging and customize its log level with the `--syslog-verbosity` flag, which
is independent from the console logging `--verbosity` flag described above.
For instance, to enable syslog logging, run:
```bash
borgmatic --syslog-verbosity 1
```
Or to increase syslog logging to include debug spew:
To increase syslog logging further to include debugging information, run:
```bash
borgmatic --syslog-verbosity 2
```
See above for further details about the verbosity levels.
Where these logs show up depends on your particular system. If you're using
systemd, try running `journalctl -xe`. Otherwise, try viewing
`/var/log/syslog` or similar.
<span class="minilink minilink-addedin">Prior to version 1.8.3</span>borgmatic
logged to syslog by default whenever run at a non-interactive console.
### Rate limiting
If you are using rsyslog or systemd's journal, be aware that by default they
@@ -165,7 +168,7 @@ Note that if you use the `--log-file` flag, you are responsible for rotating
the log file so it doesn't grow too large, for example with
[logrotate](https://wiki.archlinux.org/index.php/Logrotate).
You can the `--log-file-verbosity` flag to customize the log file's log level:
You can use the `--log-file-verbosity` flag to customize the log file's log level:
```bash
borgmatic --log-file /path/to/file.log --log-file-verbosity 2
@@ -197,5 +200,5 @@ See the [Python logging
documentation](https://docs.python.org/3/library/logging.html#logrecord-attributes)
for additional placeholders.
Note that this `--log-file-format` flg only applies to the specified
Note that this `--log-file-format` flag only applies to the specified
`--log-file` and not to syslog or other logging.
+53 -26
View File
@@ -139,8 +139,8 @@ Some borgmatic command-line actions also have a `--match-archives` flag that
overrides both the auto-matching behavior and the `match_archives`
configuration option.
<span class="minilink minilink-addedin">Prior to 1.7.11</span> The way to
limit the archives used for the `prune` action was a `prefix` option in the
<span class="minilink minilink-addedin">Prior to version 1.7.11</span> The way
to limit the archives used for the `prune` action was a `prefix` option in the
`retention` section for matching against the start of archive names. And the
option for limiting the archives used for the `check` action was a separate
`prefix` in the `consistency` section. Both of these options are deprecated in
@@ -151,7 +151,7 @@ in newer versions of borgmatic.
## Configuration includes
Once you have multiple different configuration files, you might want to share
common configuration options across these files with having to copy and paste
common configuration options across these files without having to copy and paste
them. To achieve this, you can put fragments of common configuration options
into a file and then include or inline that file into one or more borgmatic
configuration files.
@@ -301,7 +301,7 @@ options via an include and then overrides one of them locally:
<<: !include /etc/borgmatic/common.yaml
constants:
hostname: myhostname
base_directory: /opt
repositories:
- path: repo.borg
@@ -311,13 +311,13 @@ This is what `common.yaml` might look like:
```yaml
constants:
prefix: myprefix
hostname: otherhost
app_name: myapp
base_directory: /var/lib
```
Once this include gets merged in, the resulting configuration would have a
`prefix` value of `myprefix` and an overridden `hostname` value of
`myhostname`.
Once this include gets merged in, the resulting configuration would have an
`app_name` value of `myapp` and an overridden `base_directory` value of
`/opt`.
When there's an option collision between the local file and the merged
include, the local file's option takes precedence.
@@ -495,21 +495,29 @@ borgmatic create --override parent_option.option1=value1 --override parent_optio
forget to specify the section that an option is in. That looks like a prefix
on the option name, e.g. `location.repositories`.
Note that each value is parsed as an actual YAML string, so you can even set
list values by using brackets. For instance:
Note that each value is parsed as an actual YAML string, so you can set list
values by using brackets. For instance:
```bash
borgmatic create --override repositories=[test1.borg,test2.borg]
```
Or even a single list element:
Or a single list element:
```bash
borgmatic create --override repositories=[/root/test.borg]
```
If your override value contains special YAML characters like colons, then
you'll need quotes for it to parse correctly:
Or a single list element that is a key/value pair:
```bash
borgmatic create --override repositories="[{path: test.borg, label: test}]"
```
If your override value contains characters like colons or spaces, then you'll
need to use quotes for it to parse correctly.
Another example:
```bash
borgmatic create --override repositories="['user@server:test.borg']"
@@ -518,16 +526,12 @@ borgmatic create --override repositories="['user@server:test.borg']"
There is not currently a way to override a single element of a list without
replacing the whole list.
Note that if you override an option of the list type (like
`location.repositories`), you do need to use the `[ ]` list syntax. See the
[configuration
Using the `[ ]` list syntax is required when overriding an option of the list
type (like `location.repositories`). See the [configuration
reference](https://torsion.org/borgmatic/docs/reference/configuration/) for
which options are list types. (YAML list values look like `- this` with an
indentation and a leading dash.)
Be sure to quote your overrides if they contain spaces or other characters
that your shell may interpret.
An alternate to command-line overrides is passing in your values via
[environment
variables](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
@@ -540,8 +544,7 @@ tool is borgmatic's support for defining custom constants. This is similar to
the [variable interpolation
feature](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation)
for command hooks, but the constants feature lets you substitute your own
custom values into anywhere in the entire configuration file. (Constants don't
work across includes or separate configuration files though.)
custom values into any option values in the entire configuration file.
Here's an example usage:
@@ -564,10 +567,15 @@ forget to specify the section (like `location:` or `storage:`) that any option
is in.
In this example, when borgmatic runs, all instances of `{user}` get replaced
with `foo` and all instances of `{archive_prefix}` get replaced with `bar-`.
(And in this particular example, `{now}` doesn't get replaced with anything,
but gets passed directly to Borg.) After substitution, the logical result
looks something like this:
with `foo` and all instances of `{archive_prefix}` get replaced with `bar`.
And `{now}` doesn't get replaced with anything, but gets passed directly to
Borg, which has its own
[placeholders](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-placeholders)
using the same syntax as borgmatic constants. So borgmatic options like
`archive_name_format` that get passed directly to Borg can use either Borg
placeholders or borgmatic constants or both!
After substitution, the logical result looks something like this:
```yaml
source_directories:
@@ -579,5 +587,24 @@ source_directories:
archive_name_format: 'bar-{now}'
```
Note that if you'd like to interpolate a constant into the beginning of a
value, you'll need to quote it. For instance, this won't work:
```yaml
source_directories:
- {my_home_directory}/.config # This will error!
```
Instead, do this:
```yaml
source_directories:
- "{my_home_directory}/.config"
```
<span class="minilink minilink-addedin">New in version 1.8.5</span> Constants
work across includes, meaning you can define a constant and then include a
separate configuration file that uses that constant.
An alternate to constants is passing in your values via [environment
variables](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/).
+16 -4
View File
@@ -5,7 +5,7 @@ eleventyNavigation:
parent: How-to guides
order: 2
---
## Environment variable interpolation
## Providing passwords and secrets to borgmatic
If you want to use a Borg repository passphrase or database passwords with
borgmatic, you can set them directly in your borgmatic configuration file,
@@ -19,6 +19,18 @@ encryption_passphrase: yourpassphrase
But if you'd rather store them outside of borgmatic, whether for convenience
or security reasons, read on.
### Delegating to another application
borgmatic supports calling another application such as a password manager to
obtain the Borg passphrase to a repository.
For example, to ask the *Pass* password manager to provide the passphrase:
```yaml
encryption_passcommand: pass path/to/borg-repokey
```
### Environment variable interpolation
<span class="minilink minilink-addedin">New in version 1.6.4</span> borgmatic
supports interpolating arbitrary environment variables directly into option
values in your configuration file. That means you can instruct borgmatic to
@@ -58,7 +70,7 @@ This uses the `YOUR_DATABASE_PASSWORD` environment variable as your database
password.
### Interpolation defaults
#### Interpolation defaults
If you'd like to set a default for your environment variables, you can do so
with the following syntax:
@@ -72,7 +84,7 @@ environment variable is not set. Without a default, if the environment
variable doesn't exist, borgmatic will error.
### Disabling interpolation
#### Disabling interpolation
To disable this environment variable interpolation feature entirely, you can
pass the `--no-environment-interpolation` flag on the command-line.
@@ -85,7 +97,7 @@ can escape it with a backslash. For instance, if your password is literally
encryption_passphrase: \${A}@!
```
### Related features
## Related features
Another way to override particular options within a borgmatic configuration
file is to use a [configuration
+15
View File
@@ -282,6 +282,21 @@ due to things like file damage. For instance:
sudo borgmatic --verbosity 1 --list --stats
```
### Skipping actions
<span class="minilink minilink-addedin">New in version 1.8.5</span> You can
configure borgmatic to skip running certain actions (default or otherwise).
For instance, to always skip the `compact` action when using [Borg's
append-only
mode](https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode-forbid-compaction),
set the `skip_actions` option:
```
skip_actions:
- compact
```
## Autopilot
Running backups manually is good for validating your configuration, but I'm
+1 -1
View File
@@ -3,7 +3,7 @@ title: How to upgrade borgmatic and Borg
eleventyNavigation:
key: 📦 Upgrade borgmatic/Borg
parent: How-to guides
order: 12
order: 13
---
## Upgrading borgmatic
-2
View File
@@ -21,5 +21,3 @@ version](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#configuration
```yaml
{% include borgmatic/config.yaml %}
```
Note that you can also [download this configuration
+3 -3
View File
@@ -1,6 +1,6 @@
from setuptools import find_packages, setup
VERSION = '1.8.4'
VERSION = '1.8.7'
setup(
@@ -33,10 +33,10 @@ setup(
'jsonschema',
'packaging',
'requests',
'ruamel.yaml>0.15.0,<0.18.0',
'ruamel.yaml>0.15.0',
'setuptools',
),
extras_require={"Apprise": ["apprise"]},
include_package_data=True,
python_requires='>=3.7',
python_requires='>=3.8',
)
+5 -8
View File
@@ -14,13 +14,12 @@ flake8-use-fstring==1.4
flake8-variables-names==0.0.5
flexmock==0.11.3
idna==3.4
importlib_metadata==6.3.0; python_version < '3.8'
isort==5.12.0
jsonschema==4.17.3
Markdown==3.4.1
mccabe==0.7.0
packaging==23.1
pathspec==0.11.1; python_version >= '3.8'
pathspec==0.11.1
pluggy==1.0.0
py==1.11.0
pycodestyle==2.10.0
@@ -28,10 +27,8 @@ pyflakes==3.0.1
pytest==7.3.0
pytest-cov==4.0.0
PyYAML>5.0.0
regex; python_version >= '3.8'
regex
requests==2.31.0
ruamel.yaml>0.15.0,<0.18.0
toml==0.10.2; python_version >= '3.8'
typed-ast; python_version >= '3.8'
typing-extensions==4.5.0; python_version < '3.8'
zipp==3.15.0; python_version < '3.8'
ruamel.yaml>0.15.0
toml==0.10.2
typed-ast
+22 -12
View File
@@ -10,7 +10,7 @@ from borgmatic.config import generate as module
def test_insert_newline_before_comment_does_not_raise():
field_name = 'foo'
config = module.yaml.comments.CommentedMap([(field_name, 33)])
config = module.ruamel.yaml.comments.CommentedMap([(field_name, 33)])
config.yaml_set_comment_before_after_key(key=field_name, before='Comment')
module.insert_newline_before_comment(config, field_name)
@@ -125,14 +125,16 @@ def test_write_configuration_with_already_existing_directory_does_not_raise():
def test_add_comments_to_configuration_sequence_of_strings_does_not_raise():
config = module.yaml.comments.CommentedSeq(['foo', 'bar'])
config = module.ruamel.yaml.comments.CommentedSeq(['foo', 'bar'])
schema = {'type': 'array', 'items': {'type': 'string'}}
module.add_comments_to_configuration_sequence(config, schema)
def test_add_comments_to_configuration_sequence_of_maps_does_not_raise():
config = module.yaml.comments.CommentedSeq([module.yaml.comments.CommentedMap([('foo', 'yo')])])
config = module.ruamel.yaml.comments.CommentedSeq(
[module.ruamel.yaml.comments.CommentedMap([('foo', 'yo')])]
)
schema = {
'type': 'array',
'items': {'type': 'object', 'properties': {'foo': {'description': 'yo'}}},
@@ -142,7 +144,9 @@ def test_add_comments_to_configuration_sequence_of_maps_does_not_raise():
def test_add_comments_to_configuration_sequence_of_maps_without_description_does_not_raise():
config = module.yaml.comments.CommentedSeq([module.yaml.comments.CommentedMap([('foo', 'yo')])])
config = module.ruamel.yaml.comments.CommentedSeq(
[module.ruamel.yaml.comments.CommentedMap([('foo', 'yo')])]
)
schema = {'type': 'array', 'items': {'type': 'object', 'properties': {'foo': {}}}}
module.add_comments_to_configuration_sequence(config, schema)
@@ -150,7 +154,7 @@ def test_add_comments_to_configuration_sequence_of_maps_without_description_does
def test_add_comments_to_configuration_object_does_not_raise():
# Ensure that it can deal with fields both in the schema and missing from the schema.
config = module.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
config = module.ruamel.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
schema = {
'type': 'object',
'properties': {'foo': {'description': 'Foo'}, 'bar': {'description': 'Bar'}},
@@ -160,7 +164,7 @@ def test_add_comments_to_configuration_object_does_not_raise():
def test_add_comments_to_configuration_object_with_skip_first_does_not_raise():
config = module.yaml.comments.CommentedMap([('foo', 33)])
config = module.ruamel.yaml.comments.CommentedMap([('foo', 33)])
schema = {'type': 'object', 'properties': {'foo': {'description': 'Foo'}}}
module.add_comments_to_configuration_object(config, schema, skip_first=True)
@@ -168,7 +172,7 @@ def test_add_comments_to_configuration_object_with_skip_first_does_not_raise():
def test_remove_commented_out_sentinel_keeps_other_comments():
field_name = 'foo'
config = module.yaml.comments.CommentedMap([(field_name, 33)])
config = module.ruamel.yaml.comments.CommentedMap([(field_name, 33)])
config.yaml_set_comment_before_after_key(key=field_name, before='Actual comment.\nCOMMENT_OUT')
module.remove_commented_out_sentinel(config, field_name)
@@ -180,7 +184,7 @@ def test_remove_commented_out_sentinel_keeps_other_comments():
def test_remove_commented_out_sentinel_without_sentinel_keeps_other_comments():
field_name = 'foo'
config = module.yaml.comments.CommentedMap([(field_name, 33)])
config = module.ruamel.yaml.comments.CommentedMap([(field_name, 33)])
config.yaml_set_comment_before_after_key(key=field_name, before='Actual comment.')
module.remove_commented_out_sentinel(config, field_name)
@@ -192,7 +196,7 @@ def test_remove_commented_out_sentinel_without_sentinel_keeps_other_comments():
def test_remove_commented_out_sentinel_on_unknown_field_does_not_raise():
field_name = 'foo'
config = module.yaml.comments.CommentedMap([(field_name, 33)])
config = module.ruamel.yaml.comments.CommentedMap([(field_name, 33)])
config.yaml_set_comment_before_after_key(key=field_name, before='Actual comment.')
module.remove_commented_out_sentinel(config, 'unknown')
@@ -201,7 +205,9 @@ def test_remove_commented_out_sentinel_on_unknown_field_does_not_raise():
def test_generate_sample_configuration_does_not_raise():
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('schema.yaml').and_return('')
flexmock(module.yaml).should_receive('round_trip_load')
flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
flexmock(load=lambda filename: {})
)
flexmock(module).should_receive('schema_to_sample_configuration')
flexmock(module).should_receive('merge_source_configuration_into_destination')
flexmock(module).should_receive('render_configuration')
@@ -214,7 +220,9 @@ def test_generate_sample_configuration_does_not_raise():
def test_generate_sample_configuration_with_source_filename_does_not_raise():
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('schema.yaml').and_return('')
flexmock(module.yaml).should_receive('round_trip_load')
flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
flexmock(load=lambda filename: {})
)
flexmock(module.load).should_receive('load_configuration')
flexmock(module.normalize).should_receive('normalize')
flexmock(module).should_receive('schema_to_sample_configuration')
@@ -229,7 +237,9 @@ def test_generate_sample_configuration_with_source_filename_does_not_raise():
def test_generate_sample_configuration_with_dry_run_does_not_write_file():
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('schema.yaml').and_return('')
flexmock(module.yaml).should_receive('round_trip_load')
flexmock(module.ruamel.yaml).should_receive('YAML').and_return(
flexmock(load=lambda filename: {})
)
flexmock(module).should_receive('schema_to_sample_configuration')
flexmock(module).should_receive('merge_source_configuration_into_destination')
flexmock(module).should_receive('render_configuration')
+62 -48
View File
@@ -12,36 +12,10 @@ def test_load_configuration_parses_contents():
config_file = io.StringIO('key: value')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == {'key': 'value'}
config_paths = {'other.yaml'}
def test_load_configuration_replaces_constants():
builtins = flexmock(sys.modules['builtins'])
config_file = io.StringIO(
'''
constants:
key: value
key: {key}
'''
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == {'key': 'value'}
def test_load_configuration_replaces_complex_constants():
builtins = flexmock(sys.modules['builtins'])
config_file = io.StringIO(
'''
constants:
key:
subkey: value
key: {key}
'''
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == {'key': {'subkey': 'value'}}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', 'other.yaml'}
def test_load_configuration_with_only_integer_value_does_not_raise():
@@ -49,7 +23,10 @@ def test_load_configuration_with_only_integer_value_does_not_raise():
config_file = io.StringIO('33')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
assert module.load_configuration('config.yaml') == 33
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml', config_paths) == 33
assert config_paths == {'config.yaml', 'other.yaml'}
def test_load_configuration_inlines_include_relative_to_current_directory():
@@ -63,8 +40,10 @@ def test_load_configuration_inlines_include_relative_to_current_directory():
config_file = io.StringIO('key: !include include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': 'value'}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_inlines_include_relative_to_config_parent_directory():
@@ -85,8 +64,10 @@ def test_load_configuration_inlines_include_relative_to_config_parent_directory(
config_file = io.StringIO('key: !include include.yaml')
config_file.name = '/etc/config.yaml'
builtins.should_receive('open').with_args('/etc/config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('/etc/config.yaml') == {'key': 'value'}
assert module.load_configuration('/etc/config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'/etc/config.yaml', '/etc/include.yaml', 'other.yaml'}
def test_load_configuration_raises_if_relative_include_does_not_exist():
@@ -99,9 +80,10 @@ def test_load_configuration_raises_if_relative_include_does_not_exist():
config_file = io.StringIO('key: !include include.yaml')
config_file.name = '/etc/config.yaml'
builtins.should_receive('open').with_args('/etc/config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(FileNotFoundError):
module.load_configuration('/etc/config.yaml')
module.load_configuration('/etc/config.yaml', config_paths)
def test_load_configuration_inlines_absolute_include():
@@ -115,8 +97,10 @@ def test_load_configuration_inlines_absolute_include():
config_file = io.StringIO('key: !include /root/include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': 'value'}
assert module.load_configuration('config.yaml', config_paths) == {'key': 'value'}
assert config_paths == {'config.yaml', '/root/include.yaml', 'other.yaml'}
def test_load_configuration_raises_if_absolute_include_does_not_exist():
@@ -127,9 +111,10 @@ def test_load_configuration_raises_if_absolute_include_does_not_exist():
config_file = io.StringIO('key: !include /root/include.yaml')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(FileNotFoundError):
assert module.load_configuration('config.yaml')
assert module.load_configuration('config.yaml', config_paths)
def test_load_configuration_inlines_multiple_file_include_as_list():
@@ -146,8 +131,15 @@ def test_load_configuration_inlines_multiple_file_include_as_list():
config_file = io.StringIO('key: !include [/root/include1.yaml, /root/include2.yaml]')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'key': ['value2', 'value1']}
assert module.load_configuration('config.yaml', config_paths) == {'key': ['value2', 'value1']}
assert config_paths == {
'config.yaml',
'/root/include1.yaml',
'/root/include2.yaml',
'other.yaml',
}
def test_load_configuration_include_with_unsupported_filename_type_raises():
@@ -158,9 +150,10 @@ def test_load_configuration_include_with_unsupported_filename_type_raises():
config_file = io.StringIO('key: !include {path: /root/include.yaml}')
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_merges_include():
@@ -184,8 +177,13 @@ def test_load_configuration_merges_include():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'foo': 'override', 'baz': 'quux'}
assert module.load_configuration('config.yaml', config_paths) == {
'foo': 'override',
'baz': 'quux',
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_merges_multiple_file_include():
@@ -217,12 +215,14 @@ def test_load_configuration_merges_multiple_file_include():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {
assert module.load_configuration('config.yaml', config_paths) == {
'foo': 'override',
'baz': 'second',
'original': 'yes',
}
assert config_paths == {'config.yaml', '/tmp/include1.yaml', '/tmp/include2.yaml', 'other.yaml'}
def test_load_configuration_with_retain_tag_merges_include_but_keeps_local_values():
@@ -255,11 +255,13 @@ def test_load_configuration_with_retain_tag_merges_include_but_keeps_local_value
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {
assert module.load_configuration('config.yaml', config_paths) == {
'stuff': {'foo': 'override'},
'other': {'a': 'override', 'c': 'd'},
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_retain_tag_but_without_merge_include_raises():
@@ -285,9 +287,10 @@ def test_load_configuration_with_retain_tag_but_without_merge_include_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_retain_tag_on_scalar_raises():
@@ -313,9 +316,10 @@ def test_load_configuration_with_retain_tag_on_scalar_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_merges_include_and_omits_requested_values():
@@ -344,8 +348,10 @@ def test_load_configuration_with_omit_tag_merges_include_and_omits_requested_val
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'stuff': ['a', 'c', 'x', 'y']}
assert module.load_configuration('config.yaml', config_paths) == {'stuff': ['a', 'c', 'x', 'y']}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_omit_tag_on_unknown_value_merges_include_and_does_not_raise():
@@ -374,8 +380,12 @@ def test_load_configuration_with_omit_tag_on_unknown_value_merges_include_and_do
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = {'other.yaml'}
assert module.load_configuration('config.yaml') == {'stuff': ['a', 'b', 'c', 'x', 'y']}
assert module.load_configuration('config.yaml', config_paths) == {
'stuff': ['a', 'b', 'c', 'x', 'y']
}
assert config_paths == {'config.yaml', '/tmp/include.yaml', 'other.yaml'}
def test_load_configuration_with_omit_tag_on_non_list_item_raises():
@@ -403,9 +413,10 @@ def test_load_configuration_with_omit_tag_on_non_list_item_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_on_non_scalar_list_item_raises():
@@ -432,9 +443,10 @@ def test_load_configuration_with_omit_tag_on_non_scalar_list_item_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_with_omit_tag_but_without_merge_raises():
@@ -462,9 +474,10 @@ def test_load_configuration_with_omit_tag_but_without_merge_raises():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(ValueError):
module.load_configuration('config.yaml')
module.load_configuration('config.yaml', config_paths)
def test_load_configuration_does_not_merge_include_list():
@@ -489,9 +502,10 @@ def test_load_configuration_does_not_merge_include_list():
)
config_file.name = 'config.yaml'
builtins.should_receive('open').with_args('config.yaml').and_return(config_file)
config_paths = set()
with pytest.raises(module.ruamel.yaml.error.YAMLError):
assert module.load_configuration('config.yaml')
assert module.load_configuration('config.yaml', config_paths)
@pytest.mark.parametrize(
+24 -12
View File
@@ -4,19 +4,24 @@ from borgmatic.config import override as module
@pytest.mark.parametrize(
'value,expected_result',
'value,expected_result,option_type',
(
('thing', 'thing'),
('33', 33),
('33b', '33b'),
('true', True),
('false', False),
('[foo]', ['foo']),
('[foo, bar]', ['foo', 'bar']),
('thing', 'thing', 'string'),
('33', 33, 'integer'),
('33', '33', 'string'),
('33b', '33b', 'integer'),
('33b', '33b', 'string'),
('true', True, 'boolean'),
('false', False, 'boolean'),
('true', 'true', 'string'),
('[foo]', ['foo'], 'array'),
('[foo]', '[foo]', 'string'),
('[foo, bar]', ['foo', 'bar'], 'array'),
('[foo, bar]', '[foo, bar]', 'string'),
),
)
def test_convert_value_type_coerces_values(value, expected_result):
assert module.convert_value_type(value) == expected_result
def test_convert_value_type_coerces_values(value, expected_result, option_type):
assert module.convert_value_type(value, option_type) == expected_result
def test_apply_overrides_updates_config():
@@ -25,16 +30,23 @@ def test_apply_overrides_updates_config():
'other_section.thing=value2',
'section.nested.key=value3',
'new.foo=bar',
'new.mylist=[baz]',
'new.nonlist=[quux]',
]
config = {
'section': {'key': 'value', 'other': 'other_value'},
'other_section': {'thing': 'thing_value'},
}
schema = {
'properties': {
'new': {'properties': {'mylist': {'type': 'array'}, 'nonlist': {'type': 'string'}}}
}
}
module.apply_overrides(config, raw_overrides)
module.apply_overrides(config, schema, raw_overrides)
assert config == {
'section': {'key': 'value1', 'other': 'other_value', 'nested': {'key': 'value3'}},
'other_section': {'thing': 'value2'},
'new': {'foo': 'bar'},
'new': {'foo': 'bar', 'mylist': ['baz'], 'nonlist': '[quux]'},
}
+26
View File
@@ -1,3 +1,9 @@
import pkgutil
import borgmatic.actions
import borgmatic.config.load
import borgmatic.config.validate
MAXIMUM_LINE_LENGTH = 80
@@ -6,3 +12,23 @@ def test_schema_line_length_stays_under_limit():
for line in schema_file.readlines():
assert len(line.rstrip('\n')) <= MAXIMUM_LINE_LENGTH
ACTIONS_MODULE_NAMES_TO_OMIT = {'arguments', 'export_key', 'json'}
ACTIONS_MODULE_NAMES_TO_ADD = {'key', 'umount'}
def test_schema_skip_actions_correspond_to_supported_actions():
'''
Ensure that the allowed actions in the schema's "skip_actions" option don't drift from
borgmatic's actual supported actions.
'''
schema = borgmatic.config.load.load_configuration(borgmatic.config.validate.schema_filename())
schema_skip_actions = set(schema['properties']['skip_actions']['items']['enum'])
supported_actions = {
module.name.replace('_', '-')
for module in pkgutil.iter_modules(borgmatic.actions.__path__)
if module.name not in ACTIONS_MODULE_NAMES_TO_OMIT
}.union(ACTIONS_MODULE_NAMES_TO_ADD)
assert schema_skip_actions == supported_actions
+20 -9
View File
@@ -1,4 +1,5 @@
import io
import os
import string
import sys
@@ -57,7 +58,7 @@ def test_parse_configuration_transforms_file_into_mapping():
'''
)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home', '/etc'],
@@ -67,6 +68,7 @@ def test_parse_configuration_transforms_file_into_mapping():
'keep_minutely': 60,
'checks': [{'name': 'repository'}, {'name': 'archives'}],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
@@ -83,12 +85,13 @@ def test_parse_configuration_passes_through_quoted_punctuation():
'''
)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': [f'/home/{string.punctuation}'],
'repositories': [{'path': 'test.borg'}],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
@@ -140,7 +143,7 @@ def test_parse_configuration_inlines_include_inside_deprecated_section():
include_file.name = 'include.yaml'
builtins.should_receive('open').with_args('/tmp/include.yaml').and_return(include_file)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
@@ -148,6 +151,7 @@ def test_parse_configuration_inlines_include_inside_deprecated_section():
'keep_daily': 7,
'keep_hourly': 24,
}
assert config_paths == {'/tmp/include.yaml', '/tmp/config.yaml'}
assert len(logs) == 1
@@ -174,7 +178,7 @@ def test_parse_configuration_merges_include():
include_file.name = 'include.yaml'
builtins.should_receive('open').with_args('/tmp/include.yaml').and_return(include_file)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
@@ -182,6 +186,7 @@ def test_parse_configuration_merges_include():
'keep_daily': 1,
'keep_hourly': 24,
}
assert config_paths == {'/tmp/include.yaml', '/tmp/config.yaml'}
assert logs == []
@@ -193,6 +198,9 @@ def test_parse_configuration_raises_for_missing_config_file():
def test_parse_configuration_raises_for_missing_schema_file():
mock_config_and_schema('')
builtins = flexmock(sys.modules['builtins'])
builtins.should_receive('open').with_args('/tmp/config.yaml').and_return(
io.StringIO('foo: bar')
)
builtins.should_receive('open').with_args('/tmp/schema.yaml').and_raise(FileNotFoundError)
with pytest.raises(FileNotFoundError):
@@ -232,7 +240,7 @@ def test_parse_configuration_applies_overrides():
'''
)
config, logs = module.parse_configuration(
config, config_paths, logs = module.parse_configuration(
'/tmp/config.yaml', '/tmp/schema.yaml', overrides=['location.local_path=borg2']
)
@@ -241,10 +249,11 @@ def test_parse_configuration_applies_overrides():
'repositories': [{'path': 'hostname.borg'}],
'local_path': 'borg2',
}
assert config_paths == {'/tmp/config.yaml'}
assert logs == []
def test_parse_configuration_applies_normalization():
def test_parse_configuration_applies_normalization_after_environment_variable_interpolation():
mock_config_and_schema(
'''
location:
@@ -252,17 +261,19 @@ def test_parse_configuration_applies_normalization():
- /home
repositories:
- path: hostname.borg
- ${NO_EXIST:-user@hostname:repo}
exclude_if_present: .nobackup
'''
)
flexmock(os).should_receive('getenv').replace_with(lambda variable_name, default: default)
config, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
config, config_paths, logs = module.parse_configuration('/tmp/config.yaml', '/tmp/schema.yaml')
assert config == {
'source_directories': ['/home'],
'repositories': [{'path': 'hostname.borg'}],
'repositories': [{'path': 'ssh://user@hostname/./repo'}],
'exclude_if_present': ['.nobackup'],
}
assert config_paths == {'/tmp/config.yaml'}
assert logs
+118 -30
View File
@@ -11,7 +11,7 @@ from borgmatic import execute as module
def test_log_outputs_logs_each_line_separately():
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'hi').once()
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'there').once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=subprocess.PIPE)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -28,13 +28,14 @@ def test_log_outputs_logs_each_line_separately():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_skips_logs_for_process_with_none_stdout():
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'hi').never()
flexmock(module.logger).should_receive('log').with_args(logging.INFO, 'there').once()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=None)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -51,12 +52,13 @@ def test_log_outputs_skips_logs_for_process_with_none_stdout():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
hi_process = subprocess.Popen(['echo', 'hi'], stdout=subprocess.PIPE)
flexmock(module).should_receive('output_buffer_for_process').with_args(
@@ -73,6 +75,7 @@ def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
exclude_stdouts=(),
output_log_level=None,
borg_local_path='borg',
borg_exit_codes=None,
)
assert captured_outputs == {hi_process: 'hi', there_process: 'there'}
@@ -80,7 +83,7 @@ def test_log_outputs_returns_output_without_logging_for_output_log_level_none():
def test_log_outputs_includes_error_output_in_exception():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
@@ -88,7 +91,11 @@ def test_log_outputs_includes_error_output_in_exception():
with pytest.raises(subprocess.CalledProcessError) as error:
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.output
@@ -100,7 +107,7 @@ def test_log_outputs_logs_multiline_error_output():
of a process' traceback.
'''
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
@@ -111,13 +118,17 @@ def test_log_outputs_logs_multiline_error_output():
with pytest.raises(subprocess.CalledProcessError):
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_skips_error_output_in_exception_for_process_with_none_stdout():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('exit_code_indicates_error').and_return(True)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=None)
@@ -125,30 +136,43 @@ def test_log_outputs_skips_error_output_in_exception_for_process_with_none_stdou
with pytest.raises(subprocess.CalledProcessError) as error:
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
assert not error.value.output
def test_log_outputs_kills_other_processes_when_one_errors():
def test_log_outputs_kills_other_processes_and_raises_when_one_errors():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], 2, 'borg'
).and_return(True)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.ERROR)
other_process = subprocess.Popen(
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['sleep', '2'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('interpret_exit_code').with_args(
['sleep', '2'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('output_buffer_for_process').with_args(process, ()).and_return(
process.stdout
)
@@ -163,12 +187,56 @@ def test_log_outputs_kills_other_processes_when_one_errors():
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
assert error.value.output
def test_log_outputs_kills_other_processes_and_returns_when_one_exits_with_warning():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.WARNING)
other_process = subprocess.Popen(
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('interpret_exit_code').with_args(
['sleep', '2'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('output_buffer_for_process').with_args(process, ()).and_return(
process.stdout
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, ()
).and_return(other_process.stdout)
flexmock(other_process).should_receive('kill').once()
module.log_outputs(
(process, other_process),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_vents_other_processes_when_one_exits():
'''
Execute a command to generate a longish random string and pipe it into another command that
@@ -204,6 +272,7 @@ def test_log_outputs_vents_other_processes_when_one_exits():
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
@@ -235,6 +304,7 @@ def test_log_outputs_does_not_error_when_one_process_exits():
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
@@ -243,17 +313,27 @@ def test_log_outputs_truncates_long_error_output():
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(['grep'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], None, 'borg'
).and_return(False)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
['grep'], 2, 'borg'
).and_return(True)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
None,
'borg',
None,
).and_return(module.Exit_status.SUCCESS)
flexmock(module).should_receive('interpret_exit_code').with_args(
['grep'],
2,
'borg',
None,
).and_return(module.Exit_status.ERROR)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
with pytest.raises(subprocess.CalledProcessError) as error:
flexmock(module, ERROR_OUTPUT_MAX_LINE_COUNT=0).log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
assert error.value.returncode == 2
@@ -262,24 +342,32 @@ def test_log_outputs_truncates_long_error_output():
def test_log_outputs_with_no_output_logs_nothing():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
process = subprocess.Popen(['true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
def test_log_outputs_with_unfinished_process_re_polls():
flexmock(module.logger).should_receive('log').never()
flexmock(module).should_receive('exit_code_indicates_error').and_return(False)
flexmock(module).should_receive('interpret_exit_code').and_return(module.Exit_status.SUCCESS)
process = subprocess.Popen(['true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
flexmock(process).should_receive('poll').and_return(None).and_return(0).times(3)
flexmock(module).should_receive('output_buffer_for_process').and_return(process.stdout)
module.log_outputs(
(process,), exclude_stdouts=(), output_log_level=logging.INFO, borg_local_path='borg'
(process,),
exclude_stdouts=(),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
)
+82 -2
View File
@@ -9,6 +9,7 @@ def test_get_config_paths_returns_list_of_config_paths():
borgmatic_source_directory=None,
repository='repo',
archive='archive',
ssh_command=None,
)
global_arguments = flexmock(
dry_run=False,
@@ -30,11 +31,46 @@ def test_get_config_paths_returns_list_of_config_paths():
]
def test_get_config_paths_translates_ssh_command_argument_to_config():
bootstrap_arguments = flexmock(
borgmatic_source_directory=None,
repository='repo',
archive='archive',
ssh_command='ssh -i key',
)
global_arguments = flexmock(
dry_run=False,
)
local_borg_version = flexmock()
extract_process = flexmock(
stdout=flexmock(
read=lambda: '{"config_paths": ["/borgmatic/config.yaml"]}',
),
)
flexmock(module.borgmatic.borg.extract).should_receive('extract_archive').with_args(
False,
'repo',
'archive',
object,
{'ssh_command': 'ssh -i key'},
object,
object,
extract_to_stdout=True,
).and_return(extract_process)
flexmock(module.borgmatic.borg.rlist).should_receive('resolve_archive_name').with_args(
'repo', 'archive', {'ssh_command': 'ssh -i key'}, object, object
).and_return('archive')
assert module.get_config_paths(bootstrap_arguments, global_arguments, local_borg_version) == [
'/borgmatic/config.yaml'
]
def test_get_config_paths_with_missing_manifest_raises_value_error():
bootstrap_arguments = flexmock(
borgmatic_source_directory=None,
repository='repo',
archive='archive',
ssh_command=None,
)
global_arguments = flexmock(
dry_run=False,
@@ -57,6 +93,7 @@ def test_get_config_paths_with_broken_json_raises_value_error():
borgmatic_source_directory=None,
repository='repo',
archive='archive',
ssh_command=None,
)
global_arguments = flexmock(
dry_run=False,
@@ -81,6 +118,7 @@ def test_get_config_paths_with_json_missing_key_raises_value_error():
borgmatic_source_directory=None,
repository='repo',
archive='archive',
ssh_command=None,
)
global_arguments = flexmock(
dry_run=False,
@@ -101,6 +139,7 @@ def test_get_config_paths_with_json_missing_key_raises_value_error():
def test_run_bootstrap_does_not_raise():
flexmock(module).should_receive('get_config_paths').and_return(['/borgmatic/config.yaml'])
bootstrap_arguments = flexmock(
repository='repo',
archive='archive',
@@ -108,6 +147,7 @@ def test_run_bootstrap_does_not_raise():
strip_components=1,
progress=False,
borgmatic_source_directory='/borgmatic',
ssh_command=None,
)
global_arguments = flexmock(
dry_run=False,
@@ -115,14 +155,54 @@ def test_run_bootstrap_does_not_raise():
local_borg_version = flexmock()
extract_process = flexmock(
stdout=flexmock(
read=lambda: '{"config_paths": ["/borgmatic/config.yaml"]}',
read=lambda: '{"config_paths": ["borgmatic/config.yaml"]}',
),
)
flexmock(module.borgmatic.borg.extract).should_receive('extract_archive').and_return(
extract_process
).twice()
).once()
flexmock(module.borgmatic.borg.rlist).should_receive('resolve_archive_name').and_return(
'archive'
)
module.run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version)
def test_run_bootstrap_translates_ssh_command_argument_to_config():
flexmock(module).should_receive('get_config_paths').and_return(['/borgmatic/config.yaml'])
bootstrap_arguments = flexmock(
repository='repo',
archive='archive',
destination='dest',
strip_components=1,
progress=False,
borgmatic_source_directory='/borgmatic',
ssh_command='ssh -i key',
)
global_arguments = flexmock(
dry_run=False,
)
local_borg_version = flexmock()
extract_process = flexmock(
stdout=flexmock(
read=lambda: '{"config_paths": ["borgmatic/config.yaml"]}',
),
)
flexmock(module.borgmatic.borg.extract).should_receive('extract_archive').with_args(
False,
'repo',
'archive',
object,
{'ssh_command': 'ssh -i key'},
object,
object,
extract_to_stdout=False,
destination_path='dest',
strip_components=1,
progress=False,
).and_return(extract_process).once()
flexmock(module.borgmatic.borg.rlist).should_receive('resolve_archive_name').with_args(
'repo', 'archive', {'ssh_command': 'ssh -i key'}, object, object
).and_return('archive')
module.run_bootstrap(bootstrap_arguments, global_arguments, local_borg_version)
+56 -10
View File
@@ -19,16 +19,17 @@ def test_run_create_executes_and_calls_hooks_for_configured_repository():
repository=None,
progress=flexmock(),
stats=flexmock(),
json=flexmock(),
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -54,16 +55,17 @@ def test_run_create_with_store_config_files_false_does_not_create_borgmatic_mani
repository=None,
progress=flexmock(),
stats=flexmock(),
json=flexmock(),
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={'store_config_files': False},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -91,16 +93,17 @@ def test_run_create_runs_with_selected_repository():
repository=flexmock(),
progress=flexmock(),
stats=flexmock(),
json=flexmock(),
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -123,16 +126,17 @@ def test_run_create_bails_if_repository_does_not_match():
repository=flexmock(),
progress=flexmock(),
stats=flexmock(),
json=flexmock(),
json=False,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False, used_config_paths=[])
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
list(
module.run_create(
config_filename='test.yaml',
repository='repo',
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
@@ -144,6 +148,48 @@ def test_run_create_bails_if_repository_does_not_match():
)
def test_run_create_produces_json():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive(
'repositories_match'
).once().and_return(True)
flexmock(module.borgmatic.borg.create).should_receive('create_archive').once().and_return(
flexmock()
)
parsed_json = flexmock()
flexmock(module.borgmatic.actions.json).should_receive('parse_json').and_return(parsed_json)
flexmock(module).should_receive('create_borgmatic_manifest').once()
flexmock(module.borgmatic.hooks.command).should_receive('execute_hook').times(2)
flexmock(module.borgmatic.hooks.dispatch).should_receive('call_hooks').and_return({})
flexmock(module.borgmatic.hooks.dispatch).should_receive(
'call_hooks_even_if_unconfigured'
).and_return({})
create_arguments = flexmock(
repository=flexmock(),
progress=flexmock(),
stats=flexmock(),
json=True,
list_files=flexmock(),
)
global_arguments = flexmock(monitoring_verbosity=1, dry_run=False)
assert list(
module.run_create(
config_filename='test.yaml',
repository={'path': 'repo'},
config={},
config_paths=['/tmp/test.yaml'],
hook_context={},
local_borg_version=None,
create_arguments=create_arguments,
global_arguments=global_arguments,
dry_run_label='',
local_path=None,
remote_path=None,
)
) == [parsed_json]
def test_create_borgmatic_manifest_creates_manifest_file():
flexmock(module.os.path).should_receive('join').with_args(
module.borgmatic.borg.state.DEFAULT_BORGMATIC_SOURCE_DIRECTORY, 'bootstrap', 'manifest.json'
@@ -151,7 +197,7 @@ def test_create_borgmatic_manifest_creates_manifest_file():
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.os).should_receive('makedirs').and_return(True)
flexmock(module.importlib_metadata).should_receive('version').and_return('1.0.0')
flexmock(module.importlib.metadata).should_receive('version').and_return('1.0.0')
flexmock(sys.modules['builtins']).should_receive('open').with_args(
'/home/user/.borgmatic/bootstrap/manifest.json', 'w'
).and_return(
@@ -172,7 +218,7 @@ def test_create_borgmatic_manifest_creates_manifest_file_with_custom_borgmatic_s
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.os).should_receive('makedirs').and_return(True)
flexmock(module.importlib_metadata).should_receive('version').and_return('1.0.0')
flexmock(module.importlib.metadata).should_receive('version').and_return('1.0.0')
flexmock(sys.modules['builtins']).should_receive('open').with_args(
'/borgmatic/bootstrap/manifest.json', 'w'
).and_return(
+30 -1
View File
@@ -13,7 +13,7 @@ def test_run_info_does_not_raise():
flexmock()
)
flexmock(module.borgmatic.borg.info).should_receive('display_archives_info')
info_arguments = flexmock(repository=flexmock(), archive=flexmock(), json=flexmock())
info_arguments = flexmock(repository=flexmock(), archive=flexmock(), json=False)
list(
module.run_info(
@@ -26,3 +26,32 @@ def test_run_info_does_not_raise():
remote_path=None,
)
)
def test_run_info_produces_json():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rlist).should_receive('resolve_archive_name').and_return(
flexmock()
)
flexmock(module.borgmatic.actions.arguments).should_receive('update_arguments').and_return(
flexmock()
)
flexmock(module.borgmatic.borg.info).should_receive('display_archives_info').and_return(
flexmock()
)
parsed_json = flexmock()
flexmock(module.borgmatic.actions.json).should_receive('parse_json').and_return(parsed_json)
info_arguments = flexmock(repository=flexmock(), archive=flexmock(), json=True)
assert list(
module.run_info(
repository={'path': 'repo'},
config={},
local_borg_version=None,
info_arguments=info_arguments,
global_arguments=flexmock(log_json=False),
local_path=None,
remote_path=None,
)
) == [parsed_json]
+25
View File
@@ -0,0 +1,25 @@
from flexmock import flexmock
from borgmatic.actions import json as module
def test_parse_json_loads_json_from_string():
flexmock(module.json).should_receive('loads').and_return({'repository': {'id': 'foo'}})
assert module.parse_json('{"repository": {"id": "foo"}}', label=None) == {
'repository': {'id': 'foo', 'label': ''}
}
def test_parse_json_injects_label_into_parsed_data():
flexmock(module.json).should_receive('loads').and_return({'repository': {'id': 'foo'}})
assert module.parse_json('{"repository": {"id": "foo"}}', label='bar') == {
'repository': {'id': 'foo', 'label': 'bar'}
}
def test_parse_json_injects_nothing_when_repository_missing():
flexmock(module.json).should_receive('loads').and_return({'stuff': {'id': 'foo'}})
assert module.parse_json('{"stuff": {"id": "foo"}}', label='bar') == {'stuff': {'id': 'foo'}}
+30 -1
View File
@@ -13,7 +13,9 @@ def test_run_list_does_not_raise():
flexmock()
)
flexmock(module.borgmatic.borg.list).should_receive('list_archive')
list_arguments = flexmock(repository=flexmock(), archive=flexmock(), json=flexmock())
list_arguments = flexmock(
repository=flexmock(), archive=flexmock(), json=False, find_paths=None
)
list(
module.run_list(
@@ -26,3 +28,30 @@ def test_run_list_does_not_raise():
remote_path=None,
)
)
def test_run_list_produces_json():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rlist).should_receive('resolve_archive_name').and_return(
flexmock()
)
flexmock(module.borgmatic.actions.arguments).should_receive('update_arguments').and_return(
flexmock()
)
flexmock(module.borgmatic.borg.list).should_receive('list_archive').and_return(flexmock())
parsed_json = flexmock()
flexmock(module.borgmatic.actions.json).should_receive('parse_json').and_return(parsed_json)
list_arguments = flexmock(repository=flexmock(), archive=flexmock(), json=True)
assert list(
module.run_list(
repository={'path': 'repo'},
config={},
local_borg_version=None,
list_arguments=list_arguments,
global_arguments=flexmock(log_json=False),
local_path=None,
remote_path=None,
)
) == [parsed_json]
+24 -1
View File
@@ -7,7 +7,7 @@ def test_run_rinfo_does_not_raise():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rinfo).should_receive('display_repository_info')
rinfo_arguments = flexmock(repository=flexmock(), json=flexmock())
rinfo_arguments = flexmock(repository=flexmock(), json=False)
list(
module.run_rinfo(
@@ -20,3 +20,26 @@ def test_run_rinfo_does_not_raise():
remote_path=None,
)
)
def test_run_rinfo_parses_json():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rinfo).should_receive('display_repository_info').and_return(
flexmock()
)
parsed_json = flexmock()
flexmock(module.borgmatic.actions.json).should_receive('parse_json').and_return(parsed_json)
rinfo_arguments = flexmock(repository=flexmock(), json=True)
list(
module.run_rinfo(
repository={'path': 'repo'},
config={},
local_borg_version=None,
rinfo_arguments=rinfo_arguments,
global_arguments=flexmock(log_json=False),
local_path=None,
remote_path=None,
)
) == [parsed_json]
+22 -1
View File
@@ -7,7 +7,7 @@ def test_run_rlist_does_not_raise():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rlist).should_receive('list_repository')
rlist_arguments = flexmock(repository=flexmock(), json=flexmock())
rlist_arguments = flexmock(repository=flexmock(), json=False)
list(
module.run_rlist(
@@ -20,3 +20,24 @@ def test_run_rlist_does_not_raise():
remote_path=None,
)
)
def test_run_rlist_produces_json():
flexmock(module.logger).answer = lambda message: None
flexmock(module.borgmatic.config.validate).should_receive('repositories_match').and_return(True)
flexmock(module.borgmatic.borg.rlist).should_receive('list_repository').and_return(flexmock())
parsed_json = flexmock()
flexmock(module.borgmatic.actions.json).should_receive('parse_json').and_return(parsed_json)
rlist_arguments = flexmock(repository=flexmock(), json=True)
assert list(
module.run_rlist(
repository={'path': 'repo'},
config={},
local_borg_version=None,
rlist_arguments=rlist_arguments,
global_arguments=flexmock(),
local_path=None,
remote_path=None,
)
) == [parsed_json]
+61 -1
View File
@@ -16,6 +16,7 @@ def test_run_arbitrary_borg_calls_borg_with_flags():
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -37,6 +38,7 @@ def test_run_arbitrary_borg_with_log_info_calls_borg_with_info_flag():
('borg', 'break-lock', '--info', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -59,6 +61,7 @@ def test_run_arbitrary_borg_with_log_debug_calls_borg_with_debug_flag():
('borg', 'break-lock', '--debug', '--show-rc', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -84,6 +87,7 @@ def test_run_arbitrary_borg_with_lock_wait_calls_borg_with_lock_wait_flags():
('borg', 'break-lock', '--lock-wait', '5', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -102,9 +106,10 @@ def test_run_arbitrary_borg_with_archive_calls_borg_with_archive_flag():
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '::$ARCHIVE'),
('borg', 'break-lock', "'::$ARCHIVE'"),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': 'archive'},
)
@@ -127,6 +132,7 @@ def test_run_arbitrary_borg_with_local_path_calls_borg_via_local_path():
('borg1', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg1',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -140,6 +146,29 @@ def test_run_arbitrary_borg_with_local_path_calls_borg_via_local_path():
)
def test_run_arbitrary_borg_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
module.run_arbitrary_borg(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
options=['break-lock', '::'],
)
def test_run_arbitrary_borg_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -151,6 +180,32 @@ def test_run_arbitrary_borg_with_remote_path_calls_borg_with_remote_path_flags()
('borg', 'break-lock', '--remote-path', 'borg1', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
module.run_arbitrary_borg(
repository_path='repo',
config={},
local_borg_version='1.2.3',
options=['break-lock', '::'],
remote_path='borg1',
)
def test_run_arbitrary_borg_with_remote_path_injection_attack_gets_escaped():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(
('--remote-path', 'borg1; naughty-command')
).and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'break-lock', '--remote-path', "'borg1; naughty-command'", '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -173,6 +228,7 @@ def test_run_arbitrary_borg_passes_borg_specific_flags_to_borg():
('borg', 'list', '--progress', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -194,6 +250,7 @@ def test_run_arbitrary_borg_omits_dash_dash_in_flags_passed_to_borg():
('borg', 'break-lock', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -215,6 +272,7 @@ def test_run_arbitrary_borg_without_borg_specific_flags_does_not_raise():
('borg',),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -236,6 +294,7 @@ def test_run_arbitrary_borg_passes_key_sub_command_to_borg_before_injected_flags
('borg', 'key', 'export', '--info', '::'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
@@ -258,6 +317,7 @@ def test_run_arbitrary_borg_passes_debug_sub_command_to_borg_before_injected_fla
('borg', 'debug', 'dump-manifest', '--info', '::', 'path'),
output_file=module.borgmatic.execute.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
shell=True,
extra_environment={'BORG_REPO': 'repo', 'ARCHIVE': ''},
)
+29 -2
View File
@@ -7,11 +7,12 @@ from borgmatic.borg import break_lock as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -28,6 +29,32 @@ def test_break_lock_calls_borg_with_required_flags():
)
def test_break_lock_calls_borg_with_local_path():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg1', 'break-lock', 'repo'))
module.break_lock(
repository_path='repo',
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_break_lock_calls_borg_using_exit_codes():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg1', 'break-lock', 'repo'))
module.break_lock(
repository_path='repo',
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_break_lock_calls_borg_with_remote_path_flags():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg', 'break-lock', '--remote-path', 'borg1', 'repo'))
+201 -22
View File
@@ -8,10 +8,13 @@ from borgmatic.borg import check as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command, extra_environment=None
command,
extra_environment=None,
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
).once()
@@ -193,6 +196,19 @@ def test_filter_checks_on_frequency_restains_check_with_unelapsed_frequency_and_
) == ('archives',)
def test_filter_checks_on_frequency_passes_through_empty_checks():
assert (
module.filter_checks_on_frequency(
config={'checks': [{'name': 'archives', 'frequency': '1 hour'}]},
borg_repository_id='repo',
checks=(),
force=False,
archives_check_id='1234',
)
== ()
)
def test_make_archive_filter_flags_with_default_checks_and_prefix_returns_default_flags():
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
@@ -201,6 +217,7 @@ def test_make_archive_filter_flags_with_default_checks_and_prefix_returns_defaul
'1.2.3',
{},
('repository', 'archives'),
check_arguments=flexmock(match_archives=None),
prefix='foo',
)
@@ -215,6 +232,7 @@ def test_make_archive_filter_flags_with_all_checks_and_prefix_returns_default_fl
'1.2.3',
{},
('repository', 'archives', 'extract'),
check_arguments=flexmock(match_archives=None),
prefix='foo',
)
@@ -229,6 +247,7 @@ def test_make_archive_filter_flags_with_all_checks_and_prefix_without_borg_featu
'1.2.3',
{},
('repository', 'archives', 'extract'),
check_arguments=flexmock(match_archives=None),
prefix='foo',
)
@@ -239,7 +258,9 @@ def test_make_archive_filter_flags_with_archives_check_and_last_includes_last_fl
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('archives',), check_last=3)
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('archives',), check_arguments=flexmock(match_archives=None), check_last=3
)
assert flags == ('--last', '3')
@@ -248,7 +269,9 @@ def test_make_archive_filter_flags_with_data_check_and_last_includes_last_flag()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('data',), check_last=3)
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('data',), check_arguments=flexmock(match_archives=None), check_last=3
)
assert flags == ('--last', '3')
@@ -257,7 +280,9 @@ def test_make_archive_filter_flags_with_repository_check_and_last_omits_last_fla
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('repository',), check_last=3)
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('repository',), check_arguments=flexmock(match_archives=None), check_last=3
)
assert flags == ()
@@ -266,7 +291,13 @@ def test_make_archive_filter_flags_with_default_checks_and_last_includes_last_fl
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('repository', 'archives'), check_last=3)
flags = module.make_archive_filter_flags(
'1.2.3',
{},
('repository', 'archives'),
check_arguments=flexmock(match_archives=None),
check_last=3,
)
assert flags == ('--last', '3')
@@ -275,7 +306,9 @@ def test_make_archive_filter_flags_with_archives_check_and_prefix_includes_match
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('archives',), prefix='foo-')
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('archives',), check_arguments=flexmock(match_archives=None), prefix='foo-'
)
assert flags == ('--match-archives', 'sh:foo-*')
@@ -284,11 +317,30 @@ def test_make_archive_filter_flags_with_data_check_and_prefix_includes_match_arc
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('data',), prefix='foo-')
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('data',), check_arguments=flexmock(match_archives=None), prefix='foo-'
)
assert flags == ('--match-archives', 'sh:foo-*')
def test_make_archive_filter_flags_prefers_check_arguments_match_archives_to_config_match_archives():
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').with_args(
'baz-*', None, '1.2.3'
).and_return(('--match-archives', 'sh:baz-*'))
flags = module.make_archive_filter_flags(
'1.2.3',
{'match_archives': 'bar-{now}'}, # noqa: FS003
('archives',),
check_arguments=flexmock(match_archives='baz-*'),
prefix='',
)
assert flags == ('--match-archives', 'sh:baz-*')
def test_make_archive_filter_flags_with_archives_check_and_empty_prefix_uses_archive_name_format_instead():
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').with_args(
@@ -296,7 +348,11 @@ def test_make_archive_filter_flags_with_archives_check_and_empty_prefix_uses_arc
).and_return(('--match-archives', 'sh:bar-*'))
flags = module.make_archive_filter_flags(
'1.2.3', {'archive_name_format': 'bar-{now}'}, ('archives',), prefix='' # noqa: FS003
'1.2.3',
{'archive_name_format': 'bar-{now}'}, # noqa: FS003
('archives',),
check_arguments=flexmock(match_archives=None),
prefix='',
)
assert flags == ('--match-archives', 'sh:bar-*')
@@ -306,7 +362,9 @@ def test_make_archive_filter_flags_with_archives_check_and_none_prefix_omits_mat
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('archives',), prefix=None)
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('archives',), check_arguments=flexmock(match_archives=None), prefix=None
)
assert flags == ()
@@ -315,7 +373,9 @@ def test_make_archive_filter_flags_with_repository_check_and_prefix_omits_match_
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('repository',), prefix='foo-')
flags = module.make_archive_filter_flags(
'1.2.3', {}, ('repository',), check_arguments=flexmock(match_archives=None), prefix='foo-'
)
assert flags == ()
@@ -324,7 +384,13 @@ def test_make_archive_filter_flags_with_default_checks_and_prefix_includes_match
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flags = module.make_archive_filter_flags('1.2.3', {}, ('repository', 'archives'), prefix='foo-')
flags = module.make_archive_filter_flags(
'1.2.3',
{},
('repository', 'archives'),
check_arguments=flexmock(match_archives=None),
prefix='foo-',
)
assert flags == ('--match-archives', 'sh:foo-*')
@@ -607,7 +673,7 @@ def test_upgrade_check_times_renames_stale_temporary_check_path():
module.upgrade_check_times(flexmock(), flexmock())
def test_check_archives_with_progress_calls_borg_with_progress_parameter():
def test_check_archives_with_progress_passes_through_to_borg():
checks = ('repository',)
config = {'check_last': None}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
@@ -626,6 +692,8 @@ def test_check_archives_with_progress_calls_borg_with_progress_parameter():
('borg', 'check', '--progress', 'repo'),
output_file=module.DO_NOT_CAPTURE,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
@@ -634,12 +702,14 @@ def test_check_archives_with_progress_calls_borg_with_progress_parameter():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=True, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
progress=True,
)
def test_check_archives_with_repair_calls_borg_with_repair_parameter():
def test_check_archives_with_repair_passes_through_to_borg():
checks = ('repository',)
config = {'check_last': None}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
@@ -658,6 +728,8 @@ def test_check_archives_with_repair_calls_borg_with_repair_parameter():
('borg', 'check', '--repair', 'repo'),
output_file=module.DO_NOT_CAPTURE,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
@@ -666,8 +738,10 @@ def test_check_archives_with_repair_calls_borg_with_repair_parameter():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=True, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
repair=True,
)
@@ -701,6 +775,9 @@ def test_check_archives_calls_borg_with_parameters(checks):
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -723,6 +800,9 @@ def test_check_archives_with_json_error_raises():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -743,6 +823,9 @@ def test_check_archives_with_missing_json_keys_raises():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -769,11 +852,14 @@ def test_check_archives_with_extract_check_calls_extract_only():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_log_info_calls_borg_with_info_parameter():
def test_check_archives_with_log_info_passes_through_to_borg():
checks = ('repository',)
config = {'check_last': None}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
@@ -795,11 +881,14 @@ def test_check_archives_with_log_info_calls_borg_with_info_parameter():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_log_debug_calls_borg_with_debug_parameter():
def test_check_archives_with_log_debug_passes_through_to_borg():
checks = ('repository',)
config = {'check_last': None}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
@@ -821,6 +910,9 @@ def test_check_archives_with_log_debug_calls_borg_with_debug_parameter():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -841,6 +933,9 @@ def test_check_archives_without_any_checks_bails():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -867,12 +962,45 @@ def test_check_archives_with_local_path_calls_borg_via_local_path():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_check_archives_with_remote_path_calls_borg_with_remote_path_parameters():
def test_check_archives_with_exit_codes_calls_borg_using_them():
checks = ('repository',)
check_last = flexmock()
borg_exit_codes = flexmock()
config = {'check_last': check_last, 'borg_exit_codes': borg_exit_codes}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
'{"repository": {"id": "repo"}}'
)
flexmock(module).should_receive('upgrade_check_times')
flexmock(module).should_receive('parse_checks')
flexmock(module).should_receive('make_archive_filter_flags').and_return(())
flexmock(module).should_receive('make_archives_check_id').and_return(None)
flexmock(module).should_receive('filter_checks_on_frequency').and_return(checks)
flexmock(module).should_receive('make_check_flags').with_args(checks, ()).and_return(())
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(('borg', 'check', 'repo'), borg_exit_codes=borg_exit_codes)
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
module.check_archives(
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_remote_path_passes_through_to_borg():
checks = ('repository',)
check_last = flexmock()
config = {'check_last': check_last}
@@ -894,12 +1022,15 @@ def test_check_archives_with_remote_path_calls_borg_with_remote_path_parameters(
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
remote_path='borg1',
)
def test_check_archives_with_log_json_calls_borg_with_log_json_parameters():
def test_check_archives_with_log_json_passes_through_to_borg():
checks = ('repository',)
check_last = flexmock()
config = {'check_last': check_last}
@@ -921,11 +1052,14 @@ def test_check_archives_with_log_json_calls_borg_with_log_json_parameters():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=True),
)
def test_check_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
def test_check_archives_with_lock_wait_passes_through_to_borg():
checks = ('repository',)
check_last = flexmock()
config = {'lock_wait': 5, 'check_last': check_last}
@@ -947,6 +1081,9 @@ def test_check_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
@@ -974,11 +1111,14 @@ def test_check_archives_with_retention_prefix():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_extra_borg_options_calls_borg_with_extra_options():
def test_check_archives_with_extra_borg_options_passes_through_to_borg():
checks = ('repository',)
config = {'check_last': None, 'extra_borg_options': {'check': '--extra --options'}}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
@@ -999,5 +1139,44 @@ def test_check_archives_with_extra_borg_options_calls_borg_with_extra_options():
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives=None
),
global_arguments=flexmock(log_json=False),
)
def test_check_archives_with_match_archives_passes_through_to_borg():
checks = ('archives',)
config = {'check_last': None}
flexmock(module.rinfo).should_receive('display_repository_info').and_return(
'{"repository": {"id": "repo"}}'
)
flexmock(module).should_receive('upgrade_check_times')
flexmock(module).should_receive('parse_checks')
flexmock(module).should_receive('make_archive_filter_flags').and_return(
('--match-archives', 'foo-*')
)
flexmock(module).should_receive('make_archives_check_id').and_return(None)
flexmock(module).should_receive('filter_checks_on_frequency').and_return(checks)
flexmock(module).should_receive('make_check_flags').and_return(('--match-archives', 'foo-*'))
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'check', '--match-archives', 'foo-*', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module).should_receive('make_check_time_path')
flexmock(module).should_receive('write_check_time')
module.check_archives(
repository_path='repo',
config=config,
local_borg_version='1.2.3',
check_arguments=flexmock(
progress=None, repair=None, only_checks=None, force=None, match_archives='foo-*'
),
global_arguments=flexmock(log_json=False),
)
+18 -1
View File
@@ -7,12 +7,13 @@ from borgmatic.borg import compact as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(compact_command, output_log_level):
def insert_execute_command_mock(compact_command, output_log_level, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
compact_command,
output_log_level=output_log_level,
borg_local_path=compact_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -87,6 +88,22 @@ def test_compact_segments_with_local_path_calls_borg_via_local_path():
)
def test_compact_segments_with_exit_codes_calls_borg_using_them():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
borg_exit_codes = flexmock()
insert_execute_command_mock(
COMPACT_COMMAND + ('repo',), logging.INFO, borg_exit_codes=borg_exit_codes
)
module.compact_segments(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_compact_segments_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
insert_execute_command_mock(COMPACT_COMMAND + ('--remote-path', 'borg1', 'repo'), logging.INFO)
+209 -55
View File
@@ -402,6 +402,7 @@ def test_collect_special_file_paths_parses_special_files_from_borg_dry_run_file_
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -420,6 +421,7 @@ def test_collect_special_file_paths_excludes_requested_directories():
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -438,6 +440,7 @@ def test_collect_special_file_paths_excludes_non_special_files():
assert module.collect_special_file_paths(
('borg', 'create'),
config={},
local_path=None,
working_directory=None,
borg_environment=None,
@@ -452,12 +455,14 @@ def test_collect_special_file_paths_omits_exclude_no_dump_flag_from_command():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('Processing files ...\n- /foo\n+ /bar\n- /baz').once()
flexmock(module).should_receive('special_file').and_return(True)
flexmock(module).should_receive('any_parent_directories').and_return(False)
module.collect_special_file_paths(
('borg', 'create', '--exclude-nodump'),
config={},
local_path='borg',
working_directory=None,
borg_environment=None,
@@ -494,6 +499,7 @@ def test_create_archive_calls_borg_with_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -506,8 +512,9 @@ def test_create_archive_calls_borg_with_parameters():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -537,6 +544,7 @@ def test_create_archive_calls_borg_with_environment():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -549,8 +557,9 @@ def test_create_archive_calls_borg_with_environment():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -582,6 +591,7 @@ def test_create_archive_with_patterns_calls_borg_with_patterns_including_convert
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -594,23 +604,24 @@ def test_create_archive_with_patterns_calls_borg_with_patterns_including_convert
'repositories': ['repo'],
'patterns': ['pattern'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sources_and_config_paths():
def test_create_archive_with_sources_and_config_paths_calls_borg_with_sources_and_config_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
flexmock(module).should_receive('deduplicate_directories').and_return(
('foo', 'bar', '/etc/borgmatic/config.yaml')
('foo', 'bar', '/tmp/test.yaml')
)
flexmock(module).should_receive('map_directories_to_devices').and_return({})
flexmock(module).should_receive('expand_directories').with_args([]).and_return(())
flexmock(module).should_receive('expand_directories').with_args(
('foo', 'bar', '/etc/borgmatic/config.yaml')
).and_return(('foo', 'bar', '/etc/borgmatic/config.yaml'))
('foo', 'bar', '/tmp/test.yaml')
).and_return(('foo', 'bar', '/tmp/test.yaml'))
flexmock(module).should_receive('expand_directories').with_args([]).and_return(())
flexmock(module).should_receive('pattern_root_directories').and_return([])
flexmock(module.os.path).should_receive('expanduser').and_raise(TypeError)
@@ -627,10 +638,11 @@ def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sourc
environment = {'BORG_THINGY': 'YUP'}
flexmock(module.environment).should_receive('make_environment').and_return(environment)
flexmock(module).should_receive('execute_command').with_args(
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS + ('/etc/borgmatic/config.yaml',),
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS + ('/tmp/test.yaml',),
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -642,12 +654,13 @@ def test_create_archive_with_sources_and_used_config_paths_calls_borg_with_sourc
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=['/etc/borgmatic/config.yaml']),
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_sources_and_used_config_paths_with_store_config_files_false_calls_borg_with_sources_and_no_config_paths():
def test_create_archive_with_sources_and_config_paths_with_store_config_files_false_calls_borg_with_sources_and_no_config_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
@@ -677,6 +690,7 @@ def test_create_archive_with_sources_and_used_config_paths_with_store_config_fil
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=environment,
)
@@ -689,8 +703,9 @@ def test_create_archive_with_sources_and_used_config_paths_with_store_config_fil
'repositories': ['repo'],
'store_config_files': False,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=['/etc/borgmatic/config.yaml']),
global_arguments=flexmock(log_json=False),
)
@@ -722,6 +737,7 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -734,8 +750,9 @@ def test_create_archive_with_exclude_patterns_calls_borg_with_excludes():
'repositories': ['repo'],
'exclude_patterns': ['exclude'],
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -764,6 +781,7 @@ def test_create_archive_with_log_info_calls_borg_with_info_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -777,8 +795,9 @@ def test_create_archive_with_log_info_calls_borg_with_info_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -807,6 +826,7 @@ def test_create_archive_with_log_info_and_json_suppresses_most_borg_output():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
)
insert_logging_mock(logging.INFO)
@@ -818,8 +838,9 @@ def test_create_archive_with_log_info_and_json_suppresses_most_borg_output():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -849,6 +870,7 @@ def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -862,8 +884,9 @@ def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -892,6 +915,7 @@ def test_create_archive_with_log_debug_and_json_suppresses_most_borg_output():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
)
insert_logging_mock(logging.DEBUG)
@@ -903,8 +927,9 @@ def test_create_archive_with_log_debug_and_json_suppresses_most_borg_output():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -934,6 +959,7 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -946,8 +972,9 @@ def test_create_archive_with_dry_run_calls_borg_with_dry_run_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -978,6 +1005,7 @@ def test_create_archive_with_stats_and_dry_run_calls_borg_without_stats_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -991,8 +1019,9 @@ def test_create_archive_with_stats_and_dry_run_calls_borg_without_stats_paramete
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stats=True,
)
@@ -1022,6 +1051,7 @@ def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_inte
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1035,8 +1065,9 @@ def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_inte
'exclude_patterns': None,
'checkpoint_interval': 600,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1065,6 +1096,7 @@ def test_create_archive_with_checkpoint_volume_calls_borg_with_checkpoint_volume
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1078,8 +1110,9 @@ def test_create_archive_with_checkpoint_volume_calls_borg_with_checkpoint_volume
'exclude_patterns': None,
'checkpoint_volume': 1024,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1108,6 +1141,7 @@ def test_create_archive_with_chunker_params_calls_borg_with_chunker_params_param
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1121,8 +1155,9 @@ def test_create_archive_with_chunker_params_calls_borg_with_chunker_params_param
'exclude_patterns': None,
'chunker_params': '1,2,3,4',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1151,6 +1186,7 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1164,8 +1200,9 @@ def test_create_archive_with_compression_calls_borg_with_compression_parameters(
'exclude_patterns': None,
'compression': 'rle',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1200,6 +1237,7 @@ def test_create_archive_with_upload_rate_limit_calls_borg_with_upload_ratelimit_
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1213,8 +1251,9 @@ def test_create_archive_with_upload_rate_limit_calls_borg_with_upload_ratelimit_
'exclude_patterns': None,
'upload_rate_limit': 100,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1245,6 +1284,7 @@ def test_create_archive_with_working_directory_calls_borg_with_working_directory
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory='/working/dir',
extra_environment=None,
)
@@ -1258,8 +1298,9 @@ def test_create_archive_with_working_directory_calls_borg_with_working_directory
'working_directory': '/working/dir',
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1288,6 +1329,7 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1301,8 +1343,9 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
'one_file_system': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1337,6 +1380,7 @@ def test_create_archive_with_numeric_ids_calls_borg_with_numeric_ids_parameter(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1350,8 +1394,9 @@ def test_create_archive_with_numeric_ids_calls_borg_with_numeric_ids_parameter(
'numeric_ids': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1382,6 +1427,7 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1390,6 +1436,7 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1403,8 +1450,9 @@ def test_create_archive_with_read_special_calls_borg_with_read_special_parameter
'read_special': True,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1445,6 +1493,7 @@ def test_create_archive_with_basic_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1458,8 +1507,9 @@ def test_create_archive_with_basic_option_calls_borg_with_corresponding_paramete
option_name: option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1499,6 +1549,7 @@ def test_create_archive_with_atime_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1512,8 +1563,9 @@ def test_create_archive_with_atime_option_calls_borg_with_corresponding_paramete
'atime': option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1553,6 +1605,7 @@ def test_create_archive_with_flags_option_calls_borg_with_corresponding_paramete
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1566,8 +1619,9 @@ def test_create_archive_with_flags_option_calls_borg_with_corresponding_paramete
'flags': option_value,
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1596,6 +1650,7 @@ def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1609,8 +1664,9 @@ def test_create_archive_with_files_cache_calls_borg_with_files_cache_parameters(
'files_cache': 'ctime,size',
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1639,6 +1695,7 @@ def test_create_archive_with_local_path_calls_borg_via_local_path():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg1',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1651,12 +1708,59 @@ def test_create_archive_with_local_path_calls_borg_via_local_path():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_create_archive_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('collect_borgmatic_source_directories').and_return([])
flexmock(module).should_receive('deduplicate_directories').and_return(('foo', 'bar'))
flexmock(module).should_receive('map_directories_to_devices').and_return({})
flexmock(module).should_receive('expand_directories').and_return(())
flexmock(module).should_receive('pattern_root_directories').and_return([])
flexmock(module.os.path).should_receive('expanduser').and_raise(TypeError)
flexmock(module).should_receive('expand_home_directories').and_return(())
flexmock(module).should_receive('write_pattern_file').and_return(None)
flexmock(module).should_receive('make_list_filter_flags').and_return('FOO')
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module).should_receive('ensure_files_readable')
flexmock(module).should_receive('make_pattern_flags').and_return(())
flexmock(module).should_receive('make_exclude_flags').and_return(())
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
(f'repo::{DEFAULT_ARCHIVE_NAME}',)
)
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'create') + REPO_ARCHIVE_WITH_PATHS,
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
working_directory=None,
extra_environment=None,
)
module.create_archive(
dry_run=False,
repository_path='repo',
config={
'source_directories': ['foo', 'bar'],
'repositories': ['repo'],
'exclude_patterns': None,
'borg_exit_codes': borg_exit_codes,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -1682,6 +1786,7 @@ def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters(
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1694,8 +1799,9 @@ def test_create_archive_with_remote_path_calls_borg_with_remote_path_parameters(
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
remote_path='borg1',
)
@@ -1725,6 +1831,7 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1738,8 +1845,9 @@ def test_create_archive_with_umask_calls_borg_with_umask_parameters():
'exclude_patterns': None,
'umask': 740,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1768,6 +1876,7 @@ def test_create_archive_with_log_json_calls_borg_with_log_json_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1780,8 +1889,9 @@ def test_create_archive_with_log_json_calls_borg_with_log_json_parameters():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=True, used_config_paths=[]),
global_arguments=flexmock(log_json=True),
)
@@ -1810,6 +1920,7 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1823,8 +1934,9 @@ def test_create_archive_with_lock_wait_calls_borg_with_lock_wait_parameters():
'exclude_patterns': None,
'lock_wait': 5,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -1853,6 +1965,7 @@ def test_create_archive_with_stats_calls_borg_with_stats_parameter_and_answer_ou
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1865,8 +1978,9 @@ def test_create_archive_with_stats_calls_borg_with_stats_parameter_and_answer_ou
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stats=True,
)
@@ -1896,6 +2010,7 @@ def test_create_archive_with_files_calls_borg_with_list_parameter_and_answer_out
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1908,8 +2023,9 @@ def test_create_archive_with_files_calls_borg_with_list_parameter_and_answer_out
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
list_files=True,
)
@@ -1939,6 +2055,7 @@ def test_create_archive_with_progress_and_log_info_calls_borg_with_progress_para
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1952,8 +2069,9 @@ def test_create_archive_with_progress_and_log_info_calls_borg_with_progress_para
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
)
@@ -1983,6 +2101,7 @@ def test_create_archive_with_progress_calls_borg_with_progress_parameter():
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -1995,8 +2114,9 @@ def test_create_archive_with_progress_calls_borg_with_progress_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
)
@@ -2036,6 +2156,7 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2045,6 +2166,7 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
output_log_level=logging.INFO,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2057,8 +2179,9 @@ def test_create_archive_with_progress_and_stream_processes_calls_borg_with_progr
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
progress=True,
stream_processes=processes,
)
@@ -2099,6 +2222,7 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2108,6 +2232,7 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2121,8 +2246,9 @@ def test_create_archive_with_stream_processes_ignores_read_special_false_and_log
'exclude_patterns': None,
'read_special': False,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2167,6 +2293,7 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2176,6 +2303,7 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2188,8 +2316,9 @@ def test_create_archive_with_stream_processes_adds_special_files_to_excludes():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2230,6 +2359,7 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2239,6 +2369,7 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2252,8 +2383,9 @@ def test_create_archive_with_stream_processes_and_read_special_does_not_add_spec
'exclude_patterns': None,
'read_special': True,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2283,6 +2415,7 @@ def test_create_archive_with_json_calls_borg_with_json_parameter():
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
json_output = module.create_archive(
@@ -2293,8 +2426,9 @@ def test_create_archive_with_json_calls_borg_with_json_parameter():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
)
@@ -2326,6 +2460,7 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_parameter()
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
json_output = module.create_archive(
@@ -2336,8 +2471,9 @@ def test_create_archive_with_stats_and_json_calls_borg_without_stats_parameter()
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
json=True,
stats=True,
)
@@ -2370,6 +2506,7 @@ def test_create_archive_with_source_directories_glob_expands():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2383,8 +2520,9 @@ def test_create_archive_with_source_directories_glob_expands():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2413,6 +2551,7 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2426,8 +2565,9 @@ def test_create_archive_with_non_matching_source_directories_glob_passes_through
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2456,6 +2596,7 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2468,8 +2609,9 @@ def test_create_archive_with_glob_calls_borg_with_expanded_directories():
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2498,6 +2640,7 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2511,8 +2654,9 @@ def test_create_archive_with_archive_name_format_calls_borg_with_archive_name():
'exclude_patterns': None,
'archive_name_format': 'ARCHIVE_NAME',
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2542,6 +2686,7 @@ def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2555,8 +2700,9 @@ def test_create_archive_with_archive_name_format_accepts_borg_placeholders():
'exclude_patterns': None,
'archive_name_format': 'Documents_{hostname}-{now}', # noqa: FS003
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2586,6 +2732,7 @@ def test_create_archive_with_repository_accepts_borg_placeholders():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2599,8 +2746,9 @@ def test_create_archive_with_repository_accepts_borg_placeholders():
'exclude_patterns': None,
'archive_name_format': 'Documents_{hostname}-{now}', # noqa: FS003
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2629,6 +2777,7 @@ def test_create_archive_with_extra_borg_options_calls_borg_with_extra_options():
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2642,8 +2791,9 @@ def test_create_archive_with_extra_borg_options_calls_borg_with_extra_options():
'exclude_patterns': None,
'extra_borg_options': {'create': '--extra --options'},
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
@@ -2681,6 +2831,7 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2690,6 +2841,7 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
output_log_level=logging.INFO,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
working_directory=None,
extra_environment=None,
)
@@ -2702,8 +2854,9 @@ def test_create_archive_with_stream_processes_calls_borg_with_processes_and_read
'repositories': ['repo'],
'exclude_patterns': None,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
stream_processes=processes,
)
@@ -2727,8 +2880,9 @@ def test_create_archive_with_non_existent_directory_and_source_directories_must_
'exclude_patterns': None,
'source_directories_must_exist': True,
},
config_paths=['/tmp/test.yaml'],
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False, used_config_paths=[]),
global_arguments=flexmock(log_json=False),
)
+2 -1
View File
@@ -22,7 +22,8 @@ def test_make_environment_with_ssh_command_should_set_environment():
def test_make_environment_without_configuration_should_not_set_environment():
environment = module.make_environment({})
assert environment == {}
# borgmatic always sets this Borg environment variable.
assert environment == {'BORG_EXIT_CODES': 'modern'}
def test_make_environment_with_relocated_repo_access_true_should_set_environment_yes():
+33 -2
View File
@@ -9,7 +9,7 @@ from borgmatic.borg import export_key as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE):
def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE, borg_exit_codes=None):
borgmatic.logger.add_custom_log_levels()
flexmock(module.environment).should_receive('make_environment')
@@ -17,7 +17,8 @@ def insert_execute_command_mock(command, output_file=module.DO_NOT_CAPTURE):
command,
output_file=output_file,
output_log_level=module.logging.ANSWER,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -36,6 +37,36 @@ def test_export_key_calls_borg_with_required_flags():
)
def test_export_key_calls_borg_with_local_path():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
insert_execute_command_mock(('borg1', 'key', 'export', 'repo'))
module.export_key(
repository_path='repo',
config={},
local_borg_version='1.2.3',
export_arguments=flexmock(paper=False, qr_html=False, path=None),
global_arguments=flexmock(dry_run=False, log_json=False),
local_path='borg1',
)
def test_export_key_calls_borg_using_exit_codes():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
borg_exit_codes = flexmock()
insert_execute_command_mock(('borg', 'key', 'export', 'repo'), borg_exit_codes=borg_exit_codes)
module.export_key(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
export_arguments=flexmock(paper=False, qr_html=False, path=None),
global_arguments=flexmock(dry_run=False, log_json=False),
)
def test_export_key_calls_borg_with_remote_path_flags():
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
flexmock(module.os.path).should_receive('exists').never()
+37 -8
View File
@@ -8,7 +8,11 @@ from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(
command, output_log_level=logging.INFO, borg_local_path='borg', capture=True
command,
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
capture=True,
):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
@@ -16,11 +20,12 @@ def insert_execute_command_mock(
output_file=None if capture else module.DO_NOT_CAPTURE,
output_log_level=output_log_level,
borg_local_path=borg_local_path,
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
def test_export_tar_archive_calls_borg_with_path_parameters():
def test_export_tar_archive_calls_borg_with_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -42,7 +47,7 @@ def test_export_tar_archive_calls_borg_with_path_parameters():
)
def test_export_tar_archive_calls_borg_with_local_path_parameters():
def test_export_tar_archive_calls_borg_with_local_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -65,7 +70,31 @@ def test_export_tar_archive_calls_borg_with_local_path_parameters():
)
def test_export_tar_archive_calls_borg_with_remote_path_parameters():
def test_export_tar_archive_calls_borg_using_exit_codes():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'export-tar', 'repo::archive', 'test.tar'),
borg_exit_codes=borg_exit_codes,
)
module.export_tar_archive(
dry_run=False,
repository_path='repo',
archive='archive',
paths=None,
destination_path='test.tar',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_export_tar_archive_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -88,7 +117,7 @@ def test_export_tar_archive_calls_borg_with_remote_path_parameters():
)
def test_export_tar_archive_calls_borg_with_umask_parameters():
def test_export_tar_archive_calls_borg_with_umask_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -130,7 +159,7 @@ def test_export_tar_archive_calls_borg_with_log_json_parameter():
)
def test_export_tar_archive_calls_borg_with_lock_wait_parameters():
def test_export_tar_archive_calls_borg_with_lock_wait_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -173,7 +202,7 @@ def test_export_tar_archive_with_log_info_calls_borg_with_info_parameter():
)
def test_export_tar_archive_with_log_debug_calls_borg_with_debug_parameters():
def test_export_tar_archive_with_log_debug_calls_borg_with_debug_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -216,7 +245,7 @@ def test_export_tar_archive_calls_borg_with_dry_run_parameter():
)
def test_export_tar_archive_calls_borg_with_tar_filter_parameters():
def test_export_tar_archive_calls_borg_with_tar_filter_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+76 -1
View File
@@ -8,12 +8,14 @@ from borgmatic.borg import extract as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command, working_directory=None):
def insert_execute_command_mock(command, working_directory=None, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
working_directory=working_directory,
extra_environment=None,
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
).once()
@@ -99,6 +101,25 @@ def test_extract_last_archive_dry_run_calls_borg_via_local_path():
)
def test_extract_last_archive_dry_run_calls_borg_using_exit_codes():
flexmock(module.rlist).should_receive('resolve_archive_name').and_return('archive')
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'extract', '--dry-run', 'repo::archive'), borg_exit_codes=borg_exit_codes
)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
module.extract_last_archive_dry_run(
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
repository_path='repo',
lock_wait=None,
)
def test_extract_last_archive_dry_run_calls_borg_with_remote_path_flags():
flexmock(module.rlist).should_receive('resolve_archive_name').and_return('archive')
insert_execute_command_mock(
@@ -174,6 +195,54 @@ def test_extract_archive_calls_borg_with_path_flags():
)
def test_extract_archive_calls_borg_with_local_path():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
insert_execute_command_mock(('borg1', 'extract', 'repo::archive'))
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
flexmock(module.borgmatic.config.validate).should_receive(
'normalize_repository_path'
).and_return('repo')
module.extract_archive(
dry_run=False,
repository='repo',
archive='archive',
paths=None,
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_extract_archive_calls_borg_with_exit_codes():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'extract', 'repo::archive'), borg_exit_codes=borg_exit_codes
)
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
flexmock(module.borgmatic.config.validate).should_receive(
'normalize_repository_path'
).and_return('repo')
module.extract_archive(
dry_run=False,
repository='repo',
archive='archive',
paths=None,
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_extract_archive_calls_borg_with_remote_path_flags():
flexmock(module.os.path).should_receive('abspath').and_return('repo')
insert_execute_command_mock(('borg', 'extract', '--remote-path', 'borg1', 'repo::archive'))
@@ -470,6 +539,8 @@ def test_extract_archive_calls_borg_with_progress_parameter():
output_file=module.DO_NOT_CAPTURE,
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -518,6 +589,8 @@ def test_extract_archive_calls_borg_with_stdout_parameter_and_returns_process():
working_directory=None,
run_to_completion=False,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(process).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -549,6 +622,8 @@ def test_extract_archive_skips_abspath_for_remote_repository():
('borg', 'extract', 'server:repo::archive'),
working_directory=None,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).once()
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
+58 -1
View File
@@ -293,7 +293,7 @@ def test_capture_archive_listing_does_not_raise():
module.capture_archive_listing(
repository_path='repo',
archive='archive',
config=flexmock(),
config={},
local_borg_version=flexmock(),
global_arguments=flexmock(log_json=False),
)
@@ -332,6 +332,7 @@ def test_list_archive_calls_borg_with_flags():
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -395,6 +396,7 @@ def test_list_archive_calls_borg_with_local_path():
('borg2', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg2',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -408,6 +410,53 @@ def test_list_archive_calls_borg_with_local_path():
)
def test_list_archive_calls_borg_using_exit_codes():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.logger).answer = lambda message: None
list_arguments = argparse.Namespace(
archive='archive',
paths=None,
json=False,
find_paths=None,
prefix=None,
match_archives=None,
sort_by=None,
first=None,
last=None,
)
global_arguments = flexmock(log_json=False)
flexmock(module.feature).should_receive('available').and_return(False)
borg_exit_codes = flexmock()
flexmock(module).should_receive('make_list_command').with_args(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
list_arguments=list_arguments,
global_arguments=global_arguments,
local_path='borg',
remote_path=None,
).and_return(('borg', 'list', 'repo::archive'))
flexmock(module).should_receive('make_find_paths').and_return(())
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
module.list_archive(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
list_arguments=list_arguments,
global_arguments=global_arguments,
)
def test_list_archive_calls_borg_multiple_times_with_find_paths():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -430,6 +479,7 @@ def test_list_archive_calls_borg_multiple_times_with_find_paths():
('borg', 'list', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('archive1\narchive2').once()
flexmock(module).should_receive('make_list_command').and_return(
('borg', 'list', 'repo::archive1')
@@ -440,12 +490,14 @@ def test_list_archive_calls_borg_multiple_times_with_find_paths():
('borg', 'list', 'repo::archive1') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', 'repo::archive2') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -491,6 +543,7 @@ def test_list_archive_calls_borg_with_archive():
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -611,6 +664,7 @@ def test_list_archive_with_archive_ignores_archive_filter_flag(
('borg', 'list', 'repo::archive'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -669,6 +723,7 @@ def test_list_archive_with_find_paths_allows_archive_filter_flag_but_only_passes
('borg', 'rlist', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('archive1\narchive2').once()
flexmock(module).should_receive('make_list_command').with_args(
@@ -715,12 +770,14 @@ def test_list_archive_with_find_paths_allows_archive_filter_flag_but_only_passes
('borg', 'list', '--repo', 'repo', 'archive1') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'list', '--repo', 'repo', 'archive2') + glob_paths,
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
+46 -2
View File
@@ -7,11 +7,12 @@ from borgmatic.borg import mount as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
def insert_execute_command_mock(command, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
command,
borg_local_path='borg',
borg_local_path=command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -93,6 +94,47 @@ def test_mount_archive_calls_borg_with_path_flags():
)
def test_mount_archive_calls_borg_with_local_path():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
insert_execute_command_mock(('borg1', 'mount', 'repo::archive', '/mnt'))
mount_arguments = flexmock(mount_point='/mnt', options=None, paths=None, foreground=False)
module.mount_archive(
repository_path='repo',
archive='archive',
mount_arguments=mount_arguments,
config={},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
local_path='borg1',
)
def test_mount_archive_calls_borg_using_exit_codes():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
('repo::archive',)
)
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg', 'mount', 'repo::archive', '/mnt'),
borg_exit_codes=borg_exit_codes,
)
mount_arguments = flexmock(mount_point='/mnt', options=None, paths=None, foreground=False)
module.mount_archive(
repository_path='repo',
archive='archive',
mount_arguments=mount_arguments,
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
def test_mount_archive_calls_borg_with_remote_path_flags():
flexmock(module.feature).should_receive('available').and_return(False)
flexmock(module.flags).should_receive('make_repository_archive_flags').and_return(
@@ -216,6 +258,7 @@ def test_mount_archive_calls_borg_with_foreground_parameter():
('borg', 'mount', '--foreground', 'repo::archive', '/mnt'),
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).once()
@@ -288,6 +331,7 @@ def test_mount_archive_with_date_based_matching_calls_borg_with_date_based_flags
'/mnt',
),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+24 -1
View File
@@ -7,12 +7,13 @@ from borgmatic.borg import prune as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(prune_command, output_log_level):
def insert_execute_command_mock(prune_command, output_log_level, borg_exit_codes=None):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
prune_command,
output_log_level=output_log_level,
borg_local_path=prune_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -227,6 +228,27 @@ def test_prune_archives_with_local_path_calls_borg_via_local_path():
)
def test_prune_archives_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module).should_receive('make_prune_flags').and_return(BASE_PRUNE_FLAGS)
flexmock(module.flags).should_receive('make_repository_flags').and_return(('repo',))
borg_exit_codes = flexmock()
insert_execute_command_mock(
('borg',) + PRUNE_COMMAND[1:] + ('repo',), logging.INFO, borg_exit_codes
)
prune_arguments = flexmock(stats=False, list_archives=False)
module.prune_archives(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
prune_arguments=prune_arguments,
)
def test_prune_archives_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -403,6 +425,7 @@ def test_prune_archives_with_date_based_matching_calls_borg_with_date_based_flag
),
output_log_level=logging.INFO,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+26 -1
View File
@@ -22,12 +22,13 @@ def insert_rinfo_command_not_found_mock():
)
def insert_rcreate_command_mock(rcreate_command, **kwargs):
def insert_rcreate_command_mock(rcreate_command, borg_exit_codes=None, **kwargs):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command').with_args(
rcreate_command,
output_file=module.DO_NOT_CAPTURE,
borg_local_path=rcreate_command[0],
borg_exit_codes=borg_exit_codes,
extra_environment=None,
).once()
@@ -353,6 +354,30 @@ def test_create_repository_with_local_path_calls_borg_via_local_path():
)
def test_create_repository_with_exit_codes_calls_borg_using_them():
borg_exit_codes = flexmock()
insert_rinfo_command_not_found_mock()
insert_rcreate_command_mock(
('borg',) + RCREATE_COMMAND[1:] + ('--repo', 'repo'), borg_exit_codes=borg_exit_codes
)
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_flags').and_return(
(
'--repo',
'repo',
)
)
module.create_repository(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
global_arguments=flexmock(log_json=False),
encryption_mode='repokey',
)
def test_create_repository_with_remote_path_calls_borg_with_remote_path_flag():
insert_rinfo_command_not_found_mock()
insert_rcreate_command_mock(RCREATE_COMMAND + ('--remote-path', 'borg1', '--repo', 'repo'))
+55
View File
@@ -21,6 +21,7 @@ def test_display_repository_info_calls_borg_with_flags():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -28,6 +29,7 @@ def test_display_repository_info_calls_borg_with_flags():
('borg', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -49,6 +51,7 @@ def test_display_repository_info_without_borg_features_calls_borg_with_info_sub_
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -56,6 +59,7 @@ def test_display_repository_info_without_borg_features_calls_borg_with_info_sub_
('borg', 'info', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -82,6 +86,7 @@ def test_display_repository_info_with_log_info_calls_borg_with_info_flag():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--info', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -89,6 +94,7 @@ def test_display_repository_info_with_log_info_calls_borg_with_info_flag():
('borg', 'rinfo', '--info', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.INFO)
@@ -116,6 +122,7 @@ def test_display_repository_info_with_log_info_and_json_suppresses_most_borg_out
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -145,6 +152,7 @@ def test_display_repository_info_with_log_debug_calls_borg_with_debug_flag():
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--debug', '--show-rc', '--json', '--repo', 'repo'),
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
@@ -152,6 +160,7 @@ def test_display_repository_info_with_log_debug_calls_borg_with_debug_flag():
('borg', 'rinfo', '--debug', '--show-rc', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.DEBUG)
@@ -180,6 +189,7 @@ def test_display_repository_info_with_log_debug_and_json_suppresses_most_borg_ou
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -210,6 +220,7 @@ def test_display_repository_info_with_json_calls_borg_with_json_flag():
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags').never()
@@ -239,12 +250,14 @@ def test_display_repository_info_with_local_path_calls_borg_via_local_path():
('borg1', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg1', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg1',
borg_exit_codes=None,
extra_environment=None,
)
@@ -258,6 +271,42 @@ def test_display_repository_info_with_local_path_calls_borg_via_local_path():
)
def test_display_repository_info_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.feature).should_receive('available').and_return(True)
flexmock(module.flags).should_receive('make_repository_flags').and_return(
(
'--repo',
'repo',
)
)
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'rinfo', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
)
module.display_repository_info(
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
rinfo_arguments=flexmock(json=False),
global_arguments=flexmock(log_json=False),
)
def test_display_repository_info_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -273,12 +322,14 @@ def test_display_repository_info_with_remote_path_calls_borg_with_remote_path_fl
('borg', 'rinfo', '--remote-path', 'borg1', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--remote-path', 'borg1', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -307,12 +358,14 @@ def test_display_repository_info_with_log_json_calls_borg_with_log_json_flags():
('borg', 'rinfo', '--log-json', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--log-json', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -341,12 +394,14 @@ def test_display_repository_info_with_lock_wait_calls_borg_with_lock_wait_flags(
('borg', 'rinfo', '--lock-wait', '5', '--json', '--repo', 'repo'),
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('[]')
flexmock(module.flags).should_receive('warn_for_aggressive_archive_flags')
flexmock(module).should_receive('execute_command').with_args(
('borg', 'rinfo', '--lock-wait', '5', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+31
View File
@@ -38,6 +38,7 @@ def test_resolve_archive_name_calls_borg_with_flags():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -59,6 +60,7 @@ def test_resolve_archive_name_with_log_info_calls_borg_without_info_flag():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
insert_logging_mock(logging.INFO)
@@ -81,6 +83,7 @@ def test_resolve_archive_name_with_log_debug_calls_borg_without_debug_flag():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
insert_logging_mock(logging.DEBUG)
@@ -103,6 +106,7 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
('borg1', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg1',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -118,6 +122,29 @@ def test_resolve_archive_name_with_local_path_calls_borg_via_local_path():
)
def test_resolve_archive_name_with_exit_codes_calls_borg_using_them():
expected_archive = 'archive-name'
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
).and_return(expected_archive + '\n')
assert (
module.resolve_archive_name(
'repo',
'latest',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='1.2.3',
global_arguments=flexmock(log_json=False),
)
== expected_archive
)
def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags():
expected_archive = 'archive-name'
flexmock(module.environment).should_receive('make_environment')
@@ -125,6 +152,7 @@ def test_resolve_archive_name_with_remote_path_calls_borg_with_remote_path_flags
('borg', 'list', '--remote-path', 'borg1') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -146,6 +174,7 @@ def test_resolve_archive_name_without_archives_raises():
('borg', 'list') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return('')
with pytest.raises(ValueError):
@@ -166,6 +195,7 @@ def test_resolve_archive_name_with_log_json_calls_borg_with_log_json_flags():
('borg', 'list', '--log-json') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
@@ -188,6 +218,7 @@ def test_resolve_archive_name_with_lock_wait_calls_borg_with_lock_wait_flags():
('borg', 'list', '--lock-wait', 'okay') + BORG_LIST_LATEST_ARGUMENTS,
extra_environment=None,
borg_local_path='borg',
borg_exit_codes=None,
).and_return(expected_archive + '\n')
assert (
+45
View File
@@ -21,6 +21,7 @@ def test_transfer_archives_calls_borg_with_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -52,6 +53,7 @@ def test_transfer_archives_with_dry_run_calls_borg_with_dry_run_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -80,6 +82,7 @@ def test_transfer_archives_with_log_info_calls_borg_with_info_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.INFO)
@@ -108,6 +111,7 @@ def test_transfer_archives_with_log_debug_calls_borg_with_debug_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
insert_logging_mock(logging.DEBUG)
@@ -139,6 +143,7 @@ def test_transfer_archives_with_archive_calls_borg_with_match_archives_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -169,6 +174,7 @@ def test_transfer_archives_with_match_archives_calls_borg_with_match_archives_fl
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -199,6 +205,7 @@ def test_transfer_archives_with_archive_name_format_calls_borg_with_match_archiv
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -227,6 +234,7 @@ def test_transfer_archives_with_local_path_calls_borg_via_local_path():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg2',
borg_exit_codes=None,
extra_environment=None,
)
@@ -243,6 +251,36 @@ def test_transfer_archives_with_local_path_calls_borg_via_local_path():
)
def test_transfer_archives_with_exit_codes_calls_borg_using_them():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
flexmock(module.flags).should_receive('make_flags').and_return(())
flexmock(module.flags).should_receive('make_match_archives_flags').and_return(())
flexmock(module.flags).should_receive('make_flags_from_arguments').and_return(())
flexmock(module.flags).should_receive('make_repository_flags').and_return(('--repo', 'repo'))
flexmock(module.environment).should_receive('make_environment')
borg_exit_codes = flexmock()
flexmock(module).should_receive('execute_command').with_args(
('borg', 'transfer', '--repo', 'repo'),
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=borg_exit_codes,
extra_environment=None,
)
module.transfer_archives(
dry_run=False,
repository_path='repo',
config={'borg_exit_codes': borg_exit_codes},
local_borg_version='2.3.4',
transfer_arguments=flexmock(
archive=None, progress=None, match_archives=None, source_repository=None
),
global_arguments=flexmock(log_json=False),
)
def test_transfer_archives_with_remote_path_calls_borg_with_remote_path_flags():
flexmock(module.borgmatic.logger).should_receive('add_custom_log_levels')
flexmock(module.logging).ANSWER = module.borgmatic.logger.ANSWER
@@ -259,6 +297,7 @@ def test_transfer_archives_with_remote_path_calls_borg_with_remote_path_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -291,6 +330,7 @@ def test_transfer_archives_with_log_json_calls_borg_with_log_json_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -323,6 +363,7 @@ def test_transfer_archives_with_lock_wait_calls_borg_with_lock_wait_flags():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -351,6 +392,7 @@ def test_transfer_archives_with_progress_calls_borg_with_progress_flag():
output_log_level=module.borgmatic.logger.ANSWER,
output_file=module.DO_NOT_CAPTURE,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -383,6 +425,7 @@ def test_transfer_archives_passes_through_arguments_to_borg(argument_name):
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -417,6 +460,7 @@ def test_transfer_archives_with_source_repository_calls_borg_with_other_repo_fla
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
@@ -459,6 +503,7 @@ def test_transfer_archives_with_date_based_matching_calls_borg_with_date_based_f
output_log_level=module.borgmatic.logger.ANSWER,
output_file=None,
borg_local_path='borg',
borg_exit_codes=None,
extra_environment=None,
)
+20 -5
View File
@@ -7,25 +7,40 @@ from borgmatic.borg import umount as module
from ..test_verbosity import insert_logging_mock
def insert_execute_command_mock(command):
flexmock(module).should_receive('execute_command').with_args(command).once()
def insert_execute_command_mock(command, borg_local_path='borg', borg_exit_codes=None):
flexmock(module).should_receive('execute_command').with_args(
command, borg_local_path=borg_local_path, borg_exit_codes=borg_exit_codes
).once()
def test_unmount_archive_calls_borg_with_required_parameters():
insert_execute_command_mock(('borg', 'umount', '/mnt'))
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_with_log_info_calls_borg_with_info_parameter():
insert_execute_command_mock(('borg', 'umount', '--info', '/mnt'))
insert_logging_mock(logging.INFO)
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_with_log_debug_calls_borg_with_debug_parameters():
insert_execute_command_mock(('borg', 'umount', '--debug', '--show-rc', '/mnt'))
insert_logging_mock(logging.DEBUG)
module.unmount_archive(mount_point='/mnt')
module.unmount_archive(config={}, mount_point='/mnt')
def test_unmount_archive_calls_borg_with_local_path():
insert_execute_command_mock(('borg1', 'umount', '/mnt'), borg_local_path='borg1')
module.unmount_archive(config={}, mount_point='/mnt', local_path='borg1')
def test_unmount_archive_calls_borg_with_exit_codes():
borg_exit_codes = flexmock()
insert_execute_command_mock(('borg', 'umount', '/mnt'), borg_exit_codes=borg_exit_codes)
module.unmount_archive(config={'borg_exit_codes': borg_exit_codes}, mount_point='/mnt')
+12 -1
View File
@@ -11,13 +11,14 @@ VERSION = '1.2.3'
def insert_execute_command_and_capture_output_mock(
command, borg_local_path='borg', version_output=f'borg {VERSION}'
command, borg_local_path='borg', borg_exit_codes=None, version_output=f'borg {VERSION}'
):
flexmock(module.environment).should_receive('make_environment')
flexmock(module).should_receive('execute_command_and_capture_output').with_args(
command,
extra_environment=None,
borg_local_path=borg_local_path,
borg_exit_codes=borg_exit_codes,
).once().and_return(version_output)
@@ -51,6 +52,16 @@ def test_local_borg_version_with_local_borg_path_calls_borg_with_it():
assert module.local_borg_version({}, 'borg1') == VERSION
def test_local_borg_version_with_borg_exit_codes_calls_using_with_them():
borg_exit_codes = flexmock()
insert_execute_command_and_capture_output_mock(
('borg', '--version'), borg_exit_codes=borg_exit_codes
)
flexmock(module.environment).should_receive('make_environment')
assert module.local_borg_version({'borg_exit_codes': borg_exit_codes}) == VERSION
def test_local_borg_version_with_invalid_version_raises():
insert_execute_command_and_capture_output_mock(('borg', '--version'), version_output='wtf')
flexmock(module.environment).should_receive('make_environment')
+242 -39
View File
@@ -2,14 +2,34 @@ import logging
import subprocess
import time
import pytest
from flexmock import flexmock
import borgmatic.hooks.command
from borgmatic.commands import borgmatic as module
@pytest.mark.parametrize(
'config,arguments,expected_actions',
(
({}, {}, []),
({'skip_actions': []}, {}, []),
({'skip_actions': ['prune', 'check']}, {}, ['prune', 'check']),
(
{'skip_actions': ['prune', 'check']},
{'check': flexmock(force=False)},
['prune', 'check'],
),
({'skip_actions': ['prune', 'check']}, {'check': flexmock(force=True)}, ['prune']),
),
)
def test_get_skip_actions_uses_config_and_arguments(config, arguments, expected_actions):
assert module.get_skip_actions(config, arguments) == expected_actions
def test_run_configuration_runs_actions_for_each_repository():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
expected_results = [flexmock(), flexmock()]
flexmock(module).should_receive('run_actions').and_return(expected_results[:1]).and_return(
@@ -18,13 +38,25 @@ def test_run_configuration_runs_actions_for_each_repository():
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
arguments = {'global': flexmock(monitoring_verbosity=1)}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_with_skip_actions_does_not_raise():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return(['compact'])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module).should_receive('run_actions').and_return(flexmock()).and_return(flexmock())
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}], 'skip_actions': ['compact']}
arguments = {'global': flexmock(monitoring_verbosity=1)}
list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
def test_run_configuration_with_invalid_borg_version_errors():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_raise(ValueError)
flexmock(module.command).should_receive('execute_hook').never()
flexmock(module.dispatch).should_receive('call_hooks').never()
@@ -32,11 +64,12 @@ def test_run_configuration_with_invalid_borg_version_errors():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'prune': flexmock()}
list(module.run_configuration('test.yaml', config, arguments))
list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
def test_run_configuration_logs_monitor_start_error():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.dispatch).should_receive('call_hooks').and_raise(OSError).and_return(
None
@@ -47,13 +80,14 @@ def test_run_configuration_logs_monitor_start_error():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_bails_for_monitor_start_soft_failure():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
flexmock(module.dispatch).should_receive('call_hooks').and_raise(error)
@@ -62,13 +96,14 @@ def test_run_configuration_bails_for_monitor_start_soft_failure():
config = {'repositories': ['foo']}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_logs_actions_error():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module.dispatch).should_receive('call_hooks')
@@ -78,13 +113,14 @@ def test_run_configuration_logs_actions_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False)}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_bails_for_actions_soft_failure():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.dispatch).should_receive('call_hooks')
error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
@@ -94,13 +130,14 @@ def test_run_configuration_bails_for_actions_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_logs_monitor_log_error():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.dispatch).should_receive('call_hooks').and_return(None).and_return(
None
@@ -111,13 +148,14 @@ def test_run_configuration_logs_monitor_log_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_bails_for_monitor_log_soft_failure():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
flexmock(module.dispatch).should_receive('call_hooks').and_return(None).and_return(
@@ -129,13 +167,14 @@ def test_run_configuration_bails_for_monitor_log_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_logs_monitor_finish_error():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.dispatch).should_receive('call_hooks').and_return(None).and_return(
None
@@ -146,13 +185,14 @@ def test_run_configuration_logs_monitor_finish_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_bails_for_monitor_finish_soft_failure():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
flexmock(module.dispatch).should_receive('call_hooks').and_return(None).and_return(
@@ -164,13 +204,14 @@ def test_run_configuration_bails_for_monitor_finish_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_does_not_call_monitoring_hooks_if_monitoring_hooks_are_disabled():
flexmock(module).should_receive('verbosity_to_log_level').and_return(module.DISABLED)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.dispatch).should_receive('call_hooks').never()
@@ -178,12 +219,13 @@ def test_run_configuration_does_not_call_monitoring_hooks_if_monitoring_hooks_ar
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=-2, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_logs_on_error_hook_error():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook').and_raise(OSError)
expected_results = [flexmock(), flexmock()]
@@ -194,13 +236,14 @@ def test_run_configuration_logs_on_error_hook_error():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_bails_for_on_error_hook_soft_failure():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
error = subprocess.CalledProcessError(borgmatic.hooks.command.SOFT_FAIL_EXIT_CODE, 'try again')
flexmock(module.command).should_receive('execute_hook').and_raise(error)
@@ -210,7 +253,7 @@ def test_run_configuration_bails_for_on_error_hook_soft_failure():
config = {'repositories': [{'path': 'foo'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
@@ -218,19 +261,21 @@ def test_run_configuration_bails_for_on_error_hook_soft_failure():
def test_run_configuration_retries_soft_error():
# Run action first fails, second passes
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).and_return([])
flexmock(module).should_receive('log_error_records').and_return([flexmock()]).once()
config = {'repositories': [{'path': 'foo'}], 'retries': 1}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == []
def test_run_configuration_retries_hard_error():
# Run action fails twice
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).times(2)
@@ -247,12 +292,13 @@ def test_run_configuration_retries_hard_error():
).and_return(error_logs)
config = {'repositories': [{'path': 'foo'}], 'retries': 1}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
def test_run_configuration_repos_ordered():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).times(2)
@@ -265,12 +311,13 @@ def test_run_configuration_repos_ordered():
).and_return(expected_results[1:]).ordered()
config = {'repositories': [{'path': 'foo'}, {'path': 'bar'}]}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == expected_results
def test_run_configuration_retries_round_robin():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).times(4)
@@ -299,12 +346,13 @@ def test_run_configuration_retries_round_robin():
'retries': 1,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == foo_error_logs + bar_error_logs
def test_run_configuration_retries_one_passes():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).and_raise(OSError).and_return(
@@ -331,12 +379,13 @@ def test_run_configuration_retries_one_passes():
'retries': 1,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
def test_run_configuration_retry_wait():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).times(4)
@@ -374,12 +423,13 @@ def test_run_configuration_retry_wait():
'retry_wait': 10,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
def test_run_configuration_retries_timeout_multiple_repos():
flexmock(module).should_receive('verbosity_to_log_level').and_return(logging.INFO)
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.borg_version).should_receive('local_borg_version').and_return(flexmock())
flexmock(module.command).should_receive('execute_hook')
flexmock(module).should_receive('run_actions').and_raise(OSError).and_raise(OSError).and_return(
@@ -413,12 +463,13 @@ def test_run_configuration_retries_timeout_multiple_repos():
'retry_wait': 10,
}
arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()}
results = list(module.run_configuration('test.yaml', config, arguments))
results = list(module.run_configuration('test.yaml', config, ['/tmp/test.yaml'], arguments))
assert results == error_logs
def test_run_actions_runs_rcreate():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.rcreate).should_receive('run_rcreate').once()
@@ -427,6 +478,7 @@ def test_run_actions_runs_rcreate():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rcreate': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -437,12 +489,14 @@ def test_run_actions_runs_rcreate():
def test_run_actions_adds_log_file_to_hook_context():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.create).should_receive('run_create').with_args(
config_filename=object,
repository={'path': 'repo'},
config={'repositories': []},
config_paths=[],
hook_context={'repository': 'repo', 'repositories': '', 'log_file': 'foo'},
local_borg_version=object,
create_arguments=object,
@@ -457,6 +511,7 @@ def test_run_actions_adds_log_file_to_hook_context():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -468,6 +523,7 @@ def test_run_actions_adds_log_file_to_hook_context():
def test_run_actions_runs_transfer():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.transfer).should_receive('run_transfer').once()
@@ -476,6 +532,7 @@ def test_run_actions_runs_transfer():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'transfer': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -486,6 +543,7 @@ def test_run_actions_runs_transfer():
def test_run_actions_runs_create():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.create).should_receive('run_create').and_yield(expected).once()
@@ -495,6 +553,7 @@ def test_run_actions_runs_create():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -504,8 +563,29 @@ def test_run_actions_runs_create():
assert result == (expected,)
def test_run_actions_with_skip_actions_skips_create():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return(['create'])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.create).should_receive('run_create').never()
tuple(
module.run_actions(
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'create': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['create']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
repository={'path': 'repo'},
)
)
def test_run_actions_runs_prune():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.prune).should_receive('run_prune').once()
@@ -514,6 +594,27 @@ def test_run_actions_runs_prune():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'prune': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
repository={'path': 'repo'},
)
)
def test_run_actions_with_skip_actions_skips_prune():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return(['prune'])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.prune).should_receive('run_prune').never()
tuple(
module.run_actions(
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'prune': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['prune']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -524,6 +625,7 @@ def test_run_actions_runs_prune():
def test_run_actions_runs_compact():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.compact).should_receive('run_compact').once()
@@ -532,6 +634,27 @@ def test_run_actions_runs_compact():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'compact': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
repository={'path': 'repo'},
)
)
def test_run_actions_with_skip_actions_skips_compact():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return(['compact'])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.compact).should_receive('run_compact').never()
tuple(
module.run_actions(
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'compact': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['compact']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -542,6 +665,7 @@ def test_run_actions_runs_compact():
def test_run_actions_runs_check_when_repository_enabled_for_checks():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(True)
flexmock(borgmatic.actions.check).should_receive('run_check').once()
@@ -551,6 +675,7 @@ def test_run_actions_runs_check_when_repository_enabled_for_checks():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -561,6 +686,7 @@ def test_run_actions_runs_check_when_repository_enabled_for_checks():
def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(False)
flexmock(borgmatic.actions.check).should_receive('run_check').never()
@@ -570,6 +696,28 @@ def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
repository={'path': 'repo'},
)
)
def test_run_actions_with_skip_actions_skips_check():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return(['check'])
flexmock(module.command).should_receive('execute_hook')
flexmock(module.checks).should_receive('repository_enabled_for_checks').and_return(True)
flexmock(borgmatic.actions.check).should_receive('run_check').never()
tuple(
module.run_actions(
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'check': flexmock()},
config_filename=flexmock(),
config={'repositories': [], 'skip_actions': ['check']},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -580,6 +728,7 @@ def test_run_actions_skips_check_when_repository_not_enabled_for_checks():
def test_run_actions_runs_extract():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.extract).should_receive('run_extract').once()
@@ -588,6 +737,7 @@ def test_run_actions_runs_extract():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'extract': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -598,6 +748,7 @@ def test_run_actions_runs_extract():
def test_run_actions_runs_export_tar():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.export_tar).should_receive('run_export_tar').once()
@@ -606,6 +757,7 @@ def test_run_actions_runs_export_tar():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'export-tar': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -616,6 +768,7 @@ def test_run_actions_runs_export_tar():
def test_run_actions_runs_mount():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.mount).should_receive('run_mount').once()
@@ -624,6 +777,7 @@ def test_run_actions_runs_mount():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'mount': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -634,6 +788,7 @@ def test_run_actions_runs_mount():
def test_run_actions_runs_restore():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.restore).should_receive('run_restore').once()
@@ -642,6 +797,7 @@ def test_run_actions_runs_restore():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'restore': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -652,6 +808,7 @@ def test_run_actions_runs_restore():
def test_run_actions_runs_rlist():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.rlist).should_receive('run_rlist').and_yield(expected).once()
@@ -661,6 +818,7 @@ def test_run_actions_runs_rlist():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rlist': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -672,6 +830,7 @@ def test_run_actions_runs_rlist():
def test_run_actions_runs_list():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.list).should_receive('run_list').and_yield(expected).once()
@@ -681,6 +840,7 @@ def test_run_actions_runs_list():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'list': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -692,6 +852,7 @@ def test_run_actions_runs_list():
def test_run_actions_runs_rinfo():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.rinfo).should_receive('run_rinfo').and_yield(expected).once()
@@ -701,6 +862,7 @@ def test_run_actions_runs_rinfo():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'rinfo': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -712,6 +874,7 @@ def test_run_actions_runs_rinfo():
def test_run_actions_runs_info():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
expected = flexmock()
flexmock(borgmatic.actions.info).should_receive('run_info').and_yield(expected).once()
@@ -721,6 +884,7 @@ def test_run_actions_runs_info():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'info': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -732,6 +896,7 @@ def test_run_actions_runs_info():
def test_run_actions_runs_break_lock():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.break_lock).should_receive('run_break_lock').once()
@@ -740,6 +905,7 @@ def test_run_actions_runs_break_lock():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'break-lock': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -750,6 +916,7 @@ def test_run_actions_runs_break_lock():
def test_run_actions_runs_export_key():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.export_key).should_receive('run_export_key').once()
@@ -758,6 +925,7 @@ def test_run_actions_runs_export_key():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'export': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -768,6 +936,7 @@ def test_run_actions_runs_export_key():
def test_run_actions_runs_borg():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.borg).should_receive('run_borg').once()
@@ -776,6 +945,7 @@ def test_run_actions_runs_borg():
arguments={'global': flexmock(dry_run=False, log_file='foo'), 'borg': flexmock()},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -786,6 +956,7 @@ def test_run_actions_runs_borg():
def test_run_actions_runs_multiple_actions_in_argument_order():
flexmock(module).should_receive('add_custom_log_levels')
flexmock(module).should_receive('get_skip_actions').and_return([])
flexmock(module.command).should_receive('execute_hook')
flexmock(borgmatic.actions.borg).should_receive('run_borg').once().ordered()
flexmock(borgmatic.actions.restore).should_receive('run_restore').once().ordered()
@@ -799,6 +970,7 @@ def test_run_actions_runs_multiple_actions_in_argument_order():
},
config_filename=flexmock(),
config={'repositories': []},
config_paths=[],
local_path=flexmock(),
remote_path=flexmock(),
local_borg_version=flexmock(),
@@ -813,30 +985,33 @@ def test_load_configurations_collects_parsed_configurations_and_logs():
test_expected_logs = [flexmock(), flexmock()]
other_expected_logs = [flexmock(), flexmock()]
flexmock(module.validate).should_receive('parse_configuration').and_return(
configuration, test_expected_logs
).and_return(other_configuration, other_expected_logs)
configuration, ['/tmp/test.yaml'], test_expected_logs
).and_return(other_configuration, ['/tmp/other.yaml'], other_expected_logs)
configs, logs = tuple(module.load_configurations(('test.yaml', 'other.yaml')))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml', 'other.yaml')))
assert configs == {'test.yaml': configuration, 'other.yaml': other_configuration}
assert config_paths == ['/tmp/other.yaml', '/tmp/test.yaml']
assert set(logs) >= set(test_expected_logs + other_expected_logs)
def test_load_configurations_logs_warning_for_permission_error():
flexmock(module.validate).should_receive('parse_configuration').and_raise(PermissionError)
configs, logs = tuple(module.load_configurations(('test.yaml',)))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml',)))
assert configs == {}
assert config_paths == []
assert max(log.levelno for log in logs) == logging.WARNING
def test_load_configurations_logs_critical_for_parse_error():
flexmock(module.validate).should_receive('parse_configuration').and_raise(ValueError)
configs, logs = tuple(module.load_configurations(('test.yaml',)))
configs, config_paths, logs = tuple(module.load_configurations(('test.yaml',)))
assert configs == {}
assert config_paths == []
assert max(log.levelno for log in logs) == logging.CRITICAL
@@ -1067,7 +1242,9 @@ def test_collect_configuration_run_summary_logs_info_for_success():
arguments = {}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1079,7 +1256,9 @@ def test_collect_configuration_run_summary_executes_hooks_for_create():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1092,7 +1271,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_extract():
arguments = {'extract': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1107,7 +1288,9 @@ def test_collect_configuration_run_summary_logs_extract_with_repository_error():
arguments = {'extract': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1120,7 +1303,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_mount():
arguments = {'mount': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1135,7 +1320,9 @@ def test_collect_configuration_run_summary_logs_mount_with_repository_error():
arguments = {'mount': flexmock(repository='repo')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1146,7 +1333,9 @@ def test_collect_configuration_run_summary_logs_missing_configs_error():
expected_logs = (flexmock(),)
flexmock(module).should_receive('log_error_records').and_return(expected_logs)
logs = tuple(module.collect_configuration_run_summary_logs({}, arguments=arguments))
logs = tuple(
module.collect_configuration_run_summary_logs({}, config_paths=[], arguments=arguments)
)
assert logs == expected_logs
@@ -1158,7 +1347,9 @@ def test_collect_configuration_run_summary_logs_pre_hook_error():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1173,7 +1364,9 @@ def test_collect_configuration_run_summary_logs_post_hook_error():
arguments = {'create': flexmock(), 'global': flexmock(monitoring_verbosity=1, dry_run=False)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert expected_logs[0] in logs
@@ -1188,7 +1381,9 @@ def test_collect_configuration_run_summary_logs_for_list_with_archive_and_reposi
arguments = {'list': flexmock(repository='repo', archive='test')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert logs == expected_logs
@@ -1200,7 +1395,9 @@ def test_collect_configuration_run_summary_logs_info_for_success_with_list():
arguments = {'list': flexmock(repository='repo', archive=None)}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO}
@@ -1215,7 +1412,9 @@ def test_collect_configuration_run_summary_logs_run_configuration_error():
arguments = {}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.CRITICAL}
@@ -1231,7 +1430,9 @@ def test_collect_configuration_run_summary_logs_run_umount_error():
arguments = {'umount': flexmock(mount_point='/mnt')}
logs = tuple(
module.collect_configuration_run_summary_logs({'test.yaml': {}}, arguments=arguments)
module.collect_configuration_run_summary_logs(
{'test.yaml': {}}, config_paths=['/tmp/test.yaml'], arguments=arguments
)
)
assert {log.levelno for log in logs} == {logging.INFO, logging.CRITICAL}
@@ -1249,6 +1450,8 @@ def test_collect_configuration_run_summary_logs_outputs_merged_json_results():
tuple(
module.collect_configuration_run_summary_logs(
{'test.yaml': {}, 'test2.yaml': {}}, arguments=arguments
{'test.yaml': {}, 'test2.yaml': {}},
config_paths=['/tmp/test.yaml', '/tmp/test2.yaml'],
arguments=arguments,
)
)
+3 -3
View File
@@ -2,14 +2,14 @@ from borgmatic.config import checks as module
def test_repository_enabled_for_checks_defaults_to_enabled_for_all_repositories():
enabled = module.repository_enabled_for_checks('repo.borg', consistency={})
enabled = module.repository_enabled_for_checks('repo.borg', config={})
assert enabled
def test_repository_enabled_for_checks_is_enabled_for_specified_repositories():
enabled = module.repository_enabled_for_checks(
'repo.borg', consistency={'check_repositories': ['repo.borg', 'other.borg']}
'repo.borg', config={'check_repositories': ['repo.borg', 'other.borg']}
)
assert enabled
@@ -17,7 +17,7 @@ def test_repository_enabled_for_checks_is_enabled_for_specified_repositories():
def test_repository_enabled_for_checks_is_disabled_for_other_repositories():
enabled = module.repository_enabled_for_checks(
'repo.borg', consistency={'check_repositories': ['other.borg']}
'repo.borg', config={'check_repositories': ['other.borg']}
)
assert not enabled
+68
View File
@@ -0,0 +1,68 @@
import pytest
from flexmock import flexmock
from borgmatic.config import constants as module
@pytest.mark.parametrize(
'value,expected_value',
(
('3', 3),
('0', 0),
('-3', -3),
('1234', 1234),
('true', True),
('True', True),
('false', False),
('False', False),
('thing', 'thing'),
({}, {}),
({'foo': 'bar'}, {'foo': 'bar'}),
([], []),
(['foo', 'bar'], ['foo', 'bar']),
),
)
def test_coerce_scalar_converts_value(value, expected_value):
assert module.coerce_scalar(value) == expected_value
def test_apply_constants_with_empty_constants_passes_through_value():
assert module.apply_constants(value='thing', constants={}) == 'thing'
@pytest.mark.parametrize(
'value,expected_value',
(
(None, None),
('thing', 'thing'),
('{foo}', 'bar'),
('abc{foo}', 'abcbar'),
('{foo}xyz', 'barxyz'),
('{foo}{baz}', 'barquux'),
('{int}', '3'),
('{bool}', 'True'),
(['thing', 'other'], ['thing', 'other']),
(['thing', '{foo}'], ['thing', 'bar']),
(['{foo}', '{baz}'], ['bar', 'quux']),
({'key': 'value'}, {'key': 'value'}),
({'key': '{foo}'}, {'key': 'bar'}),
({'key': '{inject}'}, {'key': 'echo hi; naughty-command'}),
({'before_backup': '{inject}'}, {'before_backup': "'echo hi; naughty-command'"}),
({'after_backup': '{inject}'}, {'after_backup': "'echo hi; naughty-command'"}),
({'on_error': '{inject}'}, {'on_error': "'echo hi; naughty-command'"}),
(3, 3),
(True, True),
(False, False),
),
)
def test_apply_constants_makes_string_substitutions(value, expected_value):
flexmock(module).should_receive('coerce_scalar').replace_with(lambda value: value)
constants = {
'foo': 'bar',
'baz': 'quux',
'int': 3,
'bool': True,
'inject': 'echo hi; naughty-command',
}
assert module.apply_constants(value, constants) == expected_value
+7 -7
View File
@@ -7,7 +7,7 @@ from borgmatic.config import generate as module
def test_schema_to_sample_configuration_generates_config_map_with_examples():
flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
flexmock(module.ruamel.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
flexmock(module).should_receive('add_comments_to_configuration_object')
schema = {
'type': 'object',
@@ -32,7 +32,7 @@ def test_schema_to_sample_configuration_generates_config_map_with_examples():
def test_schema_to_sample_configuration_generates_config_sequence_of_strings_with_example():
flexmock(module.yaml.comments).should_receive('CommentedSeq').replace_with(list)
flexmock(module.ruamel.yaml.comments).should_receive('CommentedSeq').replace_with(list)
flexmock(module).should_receive('add_comments_to_configuration_sequence')
schema = {'type': 'array', 'items': {'type': 'string'}, 'example': ['hi']}
@@ -42,7 +42,7 @@ def test_schema_to_sample_configuration_generates_config_sequence_of_strings_wit
def test_schema_to_sample_configuration_generates_config_sequence_of_maps_with_examples():
flexmock(module.yaml.comments).should_receive('CommentedSeq').replace_with(list)
flexmock(module.ruamel.yaml.comments).should_receive('CommentedSeq').replace_with(list)
flexmock(module).should_receive('add_comments_to_configuration_sequence')
flexmock(module).should_receive('add_comments_to_configuration_object')
schema = {
@@ -71,7 +71,7 @@ def test_merge_source_configuration_into_destination_inserts_map_fields():
destination_config = {'foo': 'dest1', 'bar': 'dest2'}
source_config = {'foo': 'source1', 'baz': 'source2'}
flexmock(module).should_receive('remove_commented_out_sentinel')
flexmock(module).should_receive('yaml.comments.CommentedSeq').replace_with(list)
flexmock(module).should_receive('ruamel.yaml.comments.CommentedSeq').replace_with(list)
module.merge_source_configuration_into_destination(destination_config, source_config)
@@ -82,7 +82,7 @@ def test_merge_source_configuration_into_destination_inserts_nested_map_fields()
destination_config = {'foo': {'first': 'dest1', 'second': 'dest2'}, 'bar': 'dest3'}
source_config = {'foo': {'first': 'source1'}}
flexmock(module).should_receive('remove_commented_out_sentinel')
flexmock(module).should_receive('yaml.comments.CommentedSeq').replace_with(list)
flexmock(module).should_receive('ruamel.yaml.comments.CommentedSeq').replace_with(list)
module.merge_source_configuration_into_destination(destination_config, source_config)
@@ -93,7 +93,7 @@ def test_merge_source_configuration_into_destination_inserts_sequence_fields():
destination_config = {'foo': ['dest1', 'dest2'], 'bar': ['dest3'], 'baz': ['dest4']}
source_config = {'foo': ['source1'], 'bar': ['source2', 'source3']}
flexmock(module).should_receive('remove_commented_out_sentinel')
flexmock(module).should_receive('yaml.comments.CommentedSeq').replace_with(list)
flexmock(module).should_receive('ruamel.yaml.comments.CommentedSeq').replace_with(list)
module.merge_source_configuration_into_destination(destination_config, source_config)
@@ -108,7 +108,7 @@ def test_merge_source_configuration_into_destination_inserts_sequence_of_maps():
destination_config = {'foo': [{'first': 'dest1', 'second': 'dest2'}], 'bar': 'dest3'}
source_config = {'foo': [{'first': 'source1'}, {'other': 'source2'}]}
flexmock(module).should_receive('remove_commented_out_sentinel')
flexmock(module).should_receive('yaml.comments.CommentedSeq').replace_with(list)
flexmock(module).should_receive('ruamel.yaml.comments.CommentedSeq').replace_with(list)
module.merge_source_configuration_into_destination(destination_config, source_config)
+19 -11
View File
@@ -6,27 +6,35 @@ from borgmatic.config import load as module
def test_probe_and_include_file_with_absolute_path_skips_probing():
config = flexmock()
flexmock(module).should_receive('load_configuration').with_args('/etc/include.yaml').and_return(
config
).once()
config_paths = set()
flexmock(module).should_receive('load_configuration').with_args(
'/etc/include.yaml', config_paths
).and_return(config).once()
assert module.probe_and_include_file('/etc/include.yaml', ['/etc', '/var']) == config
assert (
module.probe_and_include_file('/etc/include.yaml', ['/etc', '/var'], config_paths) == config
)
def test_probe_and_include_file_with_relative_path_probes_include_directories():
config = flexmock()
config = {'foo': 'bar'}
config_paths = set()
flexmock(module.os.path).should_receive('exists').with_args('/etc/include.yaml').and_return(
False
)
flexmock(module.os.path).should_receive('exists').with_args('/var/include.yaml').and_return(
True
)
flexmock(module).should_receive('load_configuration').with_args('/etc/include.yaml').never()
flexmock(module).should_receive('load_configuration').with_args('/var/include.yaml').and_return(
config
).once()
flexmock(module).should_receive('load_configuration').with_args(
'/etc/include.yaml', config_paths
).never()
flexmock(module).should_receive('load_configuration').with_args(
'/var/include.yaml', config_paths
).and_return(config).once()
assert module.probe_and_include_file('include.yaml', ['/etc', '/var']) == config
assert module.probe_and_include_file('include.yaml', ['/etc', '/var'], config_paths) == {
'foo': 'bar',
}
def test_probe_and_include_file_with_relative_path_and_missing_files_raises():
@@ -34,4 +42,4 @@ def test_probe_and_include_file_with_relative_path_and_missing_files_raises():
flexmock(module).should_receive('load_configuration').never()
with pytest.raises(FileNotFoundError):
module.probe_and_include_file('include.yaml', ['/etc', '/var'])
module.probe_and_include_file('include.yaml', ['/etc', '/var'], config_paths=set())
+5 -12
View File
@@ -216,6 +216,11 @@ def test_normalize_sections_with_only_scalar_raises():
{'repositories': [{'path': '/repo'}]},
True,
),
(
{'repositories': [{'path': 'first'}, 'file:///repo']},
{'repositories': [{'path': 'first'}, {'path': '/repo'}]},
True,
),
(
{'repositories': [{'path': 'foo@bar:/repo', 'label': 'foo'}]},
{'repositories': [{'path': 'ssh://foo@bar/repo', 'label': 'foo'}]},
@@ -251,15 +256,3 @@ def test_normalize_applies_hard_coded_normalization_to_config(
assert logs
else:
assert logs == []
def test_normalize_raises_error_if_repository_data_is_not_consistent():
flexmock(module).should_receive('normalize_sections').and_return([])
with pytest.raises(TypeError):
module.normalize(
'test.yaml',
{
'repositories': [{'path': 'foo@bar:/repo', 'label': 'foo'}, 'file:///repo'],
},
)
+48 -10
View File
@@ -44,6 +44,31 @@ def test_set_values_with_multiple_keys_updates_hierarchy():
assert config == {'option': {'key': 'value', 'other': 'other_value'}}
def test_set_values_with_key_when_list_index_expected_errors():
config = {'option': ['foo', 'bar', 'baz']}
with pytest.raises(ValueError):
module.set_values(config, keys=('option', 'key'), value='value')
@pytest.mark.parametrize(
'schema,option_keys,expected_type',
(
({'properties': {'foo': {'type': 'array'}}}, ('foo',), 'array'),
(
{'properties': {'foo': {'properties': {'bar': {'type': 'array'}}}}},
('foo', 'bar'),
'array',
),
({'properties': {'foo': {'type': 'array'}}}, ('other',), None),
({'properties': {'foo': {'description': 'stuff'}}}, ('foo',), None),
({}, ('foo',), None),
),
)
def test_type_for_option_grabs_type_if_found_in_schema(schema, option_keys, expected_type):
assert module.type_for_option(schema, option_keys) == expected_type
@pytest.mark.parametrize(
'key,expected_key',
(
@@ -63,51 +88,64 @@ def test_strip_section_names_passes_through_key_without_section_name(key, expect
def test_parse_overrides_splits_keys_and_values():
flexmock(module).should_receive('strip_section_names').replace_with(lambda value: value)
flexmock(module).should_receive('convert_value_type').replace_with(lambda value: value)
flexmock(module).should_receive('type_for_option').and_return('string')
flexmock(module).should_receive('convert_value_type').replace_with(
lambda value, option_type: value
)
raw_overrides = ['option.my_option=value1', 'other_option=value2']
expected_result = (
(('option', 'my_option'), 'value1'),
(('other_option'), 'value2'),
)
module.parse_overrides(raw_overrides) == expected_result
module.parse_overrides(raw_overrides, schema={}) == expected_result
def test_parse_overrides_allows_value_with_equal_sign():
flexmock(module).should_receive('strip_section_names').replace_with(lambda value: value)
flexmock(module).should_receive('convert_value_type').replace_with(lambda value: value)
flexmock(module).should_receive('type_for_option').and_return('string')
flexmock(module).should_receive('convert_value_type').replace_with(
lambda value, option_type: value
)
raw_overrides = ['option=this===value']
expected_result = ((('option',), 'this===value'),)
module.parse_overrides(raw_overrides) == expected_result
module.parse_overrides(raw_overrides, schema={}) == expected_result
def test_parse_overrides_raises_on_missing_equal_sign():
flexmock(module).should_receive('strip_section_names').replace_with(lambda value: value)
flexmock(module).should_receive('convert_value_type').replace_with(lambda value: value)
flexmock(module).should_receive('type_for_option').and_return('string')
flexmock(module).should_receive('convert_value_type').replace_with(
lambda value, option_type: value
)
raw_overrides = ['option']
with pytest.raises(ValueError):
module.parse_overrides(raw_overrides)
module.parse_overrides(raw_overrides, schema={})
def test_parse_overrides_raises_on_invalid_override_value():
flexmock(module).should_receive('strip_section_names').replace_with(lambda value: value)
flexmock(module).should_receive('type_for_option').and_return('string')
flexmock(module).should_receive('convert_value_type').and_raise(ruamel.yaml.parser.ParserError)
raw_overrides = ['option=[in valid]']
with pytest.raises(ValueError):
module.parse_overrides(raw_overrides)
module.parse_overrides(raw_overrides, schema={})
def test_parse_overrides_allows_value_with_single_key():
flexmock(module).should_receive('strip_section_names').replace_with(lambda value: value)
flexmock(module).should_receive('convert_value_type').replace_with(lambda value: value)
flexmock(module).should_receive('type_for_option').and_return('string')
flexmock(module).should_receive('convert_value_type').replace_with(
lambda value, option_type: value
)
raw_overrides = ['option=value']
expected_result = ((('option',), 'value'),)
module.parse_overrides(raw_overrides) == expected_result
module.parse_overrides(raw_overrides, schema={}) == expected_result
def test_parse_overrides_handles_empty_overrides():
module.parse_overrides(raw_overrides=None) == ()
module.parse_overrides(raw_overrides=None, schema={}) == ()
+10
View File
@@ -25,6 +25,16 @@ def test_interpolate_context_interpolates_variables():
)
def test_interpolate_context_escapes_interpolated_variables():
command = 'ls {foo} {inject}' # noqa: FS003
context = {'foo': 'bar', 'inject': 'hi; naughty-command'}
assert (
module.interpolate_context('test.yaml', 'pre-backup', command, context)
== "ls bar 'hi; naughty-command'"
)
def test_execute_hook_invokes_each_command():
flexmock(module).should_receive('interpolate_context').replace_with(
lambda config_file, hook_description, command, context: command
+8
View File
@@ -164,6 +164,14 @@ def test_dump_data_sources_runs_mongodumpall_for_all_databases():
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_build_dump_command_with_username_injection_attack_gets_escaped():
database = {'name': 'test', 'username': 'bob; naughty-command'}
command = module.build_dump_command(database, dump_filename='test', dump_format='archive')
assert "'bob; naughty-command'" in command
def test_restore_data_source_dump_runs_mongorestore():
hook_config = [{'name': 'foo', 'schemas': None}, {'name': 'bar'}]
extract_process = flexmock(stdout=flexmock())
+36
View File
@@ -345,6 +345,42 @@ def test_dump_data_sources_runs_pg_dump_with_username_and_password():
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_dump_data_sources_with_username_injection_attack_gets_escaped():
databases = [{'name': 'foo', 'username': 'postgres; naughty-command', 'password': 'trustsome1'}]
process = flexmock()
flexmock(module).should_receive('make_extra_environment').and_return(
{'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'}
)
flexmock(module).should_receive('make_dump_path').and_return('')
flexmock(module).should_receive('database_names_to_dump').and_return(('foo',))
flexmock(module.dump).should_receive('make_data_source_dump_filename').and_return(
'databases/localhost/foo'
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.dump).should_receive('create_named_pipe_for_dump')
flexmock(module).should_receive('execute_command').with_args(
(
'pg_dump',
'--no-password',
'--clean',
'--if-exists',
'--username',
"'postgres; naughty-command'",
'--format',
'custom',
'foo',
'>',
'databases/localhost/foo',
),
shell=True,
extra_environment={'PGPASSWORD': 'trustsome1', 'PGSSLMODE': 'disable'},
run_to_completion=False,
).and_return(process).once()
assert module.dump_data_sources(databases, {}, 'test.yaml', dry_run=False) == [process]
def test_dump_data_sources_runs_pg_dump_with_directory_format():
databases = [{'name': 'foo', 'format': 'directory'}]
flexmock(module).should_receive('make_extra_environment').and_return({'PGSSLMODE': 'disable'})

Some files were not shown because too many files have changed in this diff Show More