Compare commits

..
12 Commits
Author SHA1 Message Date
Dan Helfman 50f62d73b7 When using the Healthchecks monitoring hook, include borgmatic logs in the payloads for completion and failure pings (#241). 2019-11-17 16:54:27 -08:00
Dan Helfman 26a89de790 Higher resolution BorgBase logo. 2019-11-14 12:39:33 -08:00
Dan Helfman c2276b18c5 Add troubleshooting documentation for MySQL lock table errors (#244). 2019-11-14 10:33:47 -08:00
Dan Helfman 693434f8aa Icon spacing hack that'll show up on GitHub. 2019-11-14 09:45:38 -08:00
Dan Helfman 1e8edc05e9 Marketing. 2019-11-14 09:34:53 -08:00
Dan Helfman 1f166a47e9 Fix failing test due to dictionary order (or the lack thereof..) in Python 3.5. 2019-11-13 16:42:58 -08:00
Dan Helfman 9ee6151999 Add link to unofficial Fedora package of borgmatic that's more up-to-date. 2019-11-13 14:59:49 -08:00
Dan Helfman 6cdc92bd0c Fix for "borgmatic restore" showing success and incorrectly extracting archive files, even when no databases are configured to restore (#246). 2019-11-13 10:41:57 -08:00
Dan Helfman 612e1fea67 Add database version to issue template. 2019-11-13 08:38:58 -08:00
Dan Helfman 0a9f4e8708 Reopen the file given by "--log-file" flag if an external program rotates the log file while borgmatic is running. 2019-11-12 16:13:25 -08:00
Dan Helfman 781fac3266 Refactor ping monitors (Healthchecks, Cronitor, Cronhub) to share a common invocation and function signature. 2019-11-12 15:31:07 -08:00
Dan Helfman 4c38810a32 Add note about rsyslog rate limiting. 2019-11-12 12:32:32 -08:00
35 changed files with 485 additions and 189 deletions
+2
View File
@@ -32,6 +32,8 @@ module.exports = function(eleventyConfig) {
.use(markdownItReplaceLink)
);
eleventyConfig.addPassthroughCopy({"docs/static": "static"});
return {
templateFormats: [
"md",
+4
View File
@@ -28,4 +28,8 @@ Use `sudo borg --version`
Use `python3 --version`
**Database version (if applicable):** [version here]
Use `psql --version` or `mysql --version` on client and server.
**operating system and version:** [OS here]
+1 -1
View File
@@ -11,4 +11,4 @@ Robin `ypid` Schneider: Support additional options of Borg and add validate-borg
Scott Squires: Custom archive names
Thomas LÉVEIL: Support for a keep_minutely prune option. Support for the --json option
Any many others! See the output of "git log".
And many others! See the output of "git log".
+14
View File
@@ -1,3 +1,17 @@
1.4.11
* #241: When using the Healthchecks monitoring hook, include borgmatic logs in the payloads for
completion and failure pings.
* With --verbosity level 1 or 2, show error logs both inline when they occur and in the summary
logs at the bottom. With lower verbosity levels, suppress the summary and show error logs when
they occur.
1.4.10
* #246: Fix for "borgmatic restore" showing success and incorrectly extracting archive files, even
when no databases are configured to restore. As this can overwrite files from the archive and
lead to data loss, please upgrade to get the fix before using "borgmatic restore".
* Reopen the file given by "--log-file" flag if an external program rotates the log file while
borgmatic is running.
1.4.9
* #228: Database dump hooks for MySQL/MariaDB, so you can easily dump your databases before backups
run.
+33 -23
View File
@@ -2,68 +2,71 @@
title: borgmatic
permalink: index.html
---
<a href="https://build.torsion.org/witten/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/witten/borgmatic/status.svg?ref=refs/heads/master)</a>
## Overview
## It's your data. Keep it that way.
<img src="https://projects.torsion.org/witten/borgmatic/raw/branch/master/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
<img src="docs/static/borgmatic.png" alt="borgmatic logo" width="150px" style="float: right; padding-left: 1em;">
borgmatic is simple, configuration-driven backup software for servers and
workstations. Backup all of your machines from the command-line or scheduled
jobs. No GUI required. Built atop [Borg Backup](https://www.borgbackup.org/),
borgmatic initiates a backup, prunes any old backups according to a retention
policy, and validates backups for consistency. borgmatic supports specifying
your settings in a declarative configuration file, rather than having to put
them all on the command-line, and handles common errors.
workstations. Protect your files with client-side encryption. Backup your
databases too. Monitor it all with integrated third-party services.
Here's an example config file:
Here's an example configuration file:
```yaml
location:
# List of source directories to backup. Globs are expanded.
# List of source directories to backup.
source_directories:
- /home
- /etc
- /var/log/syslog*
# Paths to local or remote repositories.
repositories:
- user@backupserver:sourcehostname.borg
# Any paths matching these patterns are excluded from backups.
exclude_patterns:
- /home/*/.cache
retention:
# Retention policy for how many backups to keep in each category.
# Retention policy for how many backups to keep.
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
consistency:
# List of consistency checks to run: "repository", "archives", etc.
# List of checks to run to validate your backups.
checks:
- repository
- archives
hooks:
# Preparation scripts to run, databases to dump, and monitoring to perform.
# Custom preparation scripts to run.
before_backup:
- prepare-for-backup.sh
# Databases to dump and include in backups.
postgresql_databases:
- name: users
# Third-party services to notify you if backups aren't happening.
healthchecks: https://hc-ping.com/be067061-cf96-4412-8eae-62b0c50d6a8c
```
borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
available](https://projects.torsion.org/witten/borgmatic). It's also mirrored
on [GitHub](https://github.com/witten/borgmatic) for convenience.
Want to see borgmatic in action? Check out the <a
href="https://asciinema.org/a/203761" target="_blank">screencast</a>.
<script src="https://asciinema.org/a/203761.js" id="asciicast-203761" async></script>
borgmatic is powered by [Borg Backup](https://www.borgbackup.org/).
## Integrations
<a href="https://www.postgresql.org/"><img src="docs/static/postgresql.png" alt="PostgreSQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.mysql.com/"><img src="docs/static/mysql.png" alt="MySQL" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://mariadb.com/"><img src="docs/static/mariadb.png" alt="MariaDB" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://healthchecks.io/"><img src="docs/static/healthchecks.png" alt="Healthchecks" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronitor.io/"><img src="docs/static/cronitor.png" alt="Cronitor" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://cronhub.io/"><img src="docs/static/cronhub.png" alt="Cronhub" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.rsync.net/cgi-bin/borg.cgi?campaign=borg&adgroup=borgmatic"><img src="docs/static/rsyncnet.png" alt="rsync.net" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.borgbase.com/?utm_source=borgmatic"><img src="docs/static/borgbase.png" alt="BorgBase" height="60px" style="margin-bottom:20px;"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
## How-to guides
@@ -117,6 +120,10 @@ Other questions or comments? Contact <mailto:witten@torsion.org>.
### Contributing
borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
available](https://projects.torsion.org/witten/borgmatic). It's also mirrored
on [GitHub](https://github.com/witten/borgmatic) for convenience.
If you'd like to contribute to borgmatic development, please feel free to
submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
@@ -126,3 +133,6 @@ your thing. In general, contributions are very welcome. We don't bite!
Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/witten/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/witten/borgmatic/status.svg?ref=refs/heads/master)</a>
+50 -48
View File
@@ -18,7 +18,7 @@ from borgmatic.borg import list as borg_list
from borgmatic.borg import prune as borg_prune
from borgmatic.commands.arguments import parse_arguments
from borgmatic.config import checks, collect, convert, validate
from borgmatic.hooks import command, cronhub, cronitor, dispatch, dump, healthchecks
from borgmatic.hooks import command, dispatch, dump, monitor
from borgmatic.logger import configure_logging, should_do_markup
from borgmatic.signals import configure_signals
from borgmatic.verbosity import verbosity_to_log_level
@@ -53,14 +53,13 @@ def run_configuration(config_filename, config, arguments):
if 'create' in arguments:
try:
healthchecks.ping_healthchecks(
hooks.get('healthchecks'), config_filename, global_arguments.dry_run, 'start'
)
cronitor.ping_cronitor(
hooks.get('cronitor'), config_filename, global_arguments.dry_run, 'run'
)
cronhub.ping_cronhub(
hooks.get('cronhub'), config_filename, global_arguments.dry_run, 'start'
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.START,
global_arguments.dry_run,
)
command.execute_hook(
hooks.get('before_backup'),
@@ -119,14 +118,13 @@ def run_configuration(config_filename, config, arguments):
'post-backup',
global_arguments.dry_run,
)
healthchecks.ping_healthchecks(
hooks.get('healthchecks'), config_filename, global_arguments.dry_run
)
cronitor.ping_cronitor(
hooks.get('cronitor'), config_filename, global_arguments.dry_run, 'complete'
)
cronhub.ping_cronhub(
hooks.get('cronhub'), config_filename, global_arguments.dry_run, 'finish'
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.FINISH,
global_arguments.dry_run,
)
except (OSError, CalledProcessError) as error:
encountered_error = error
@@ -146,14 +144,13 @@ def run_configuration(config_filename, config, arguments):
error=encountered_error,
output=getattr(encountered_error, 'output', ''),
)
healthchecks.ping_healthchecks(
hooks.get('healthchecks'), config_filename, global_arguments.dry_run, 'fail'
)
cronitor.ping_cronitor(
hooks.get('cronitor'), config_filename, global_arguments.dry_run, 'fail'
)
cronhub.ping_cronhub(
hooks.get('cronhub'), config_filename, global_arguments.dry_run, 'fail'
dispatch.call_hooks(
'ping_monitor',
hooks,
config_filename,
monitor.MONITOR_HOOK_NAMES,
monitor.State.FAIL,
global_arguments.dry_run,
)
except (OSError, CalledProcessError) as error:
yield from make_error_log_records(
@@ -269,12 +266,13 @@ def run_actions(
dump.DATABASE_HOOK_NAMES,
restore_names,
)
borg_extract.extract_archive(
global_arguments.dry_run,
repository,
arguments['restore'].archive,
dump.convert_glob_patterns_to_borg_patterns(
[pattern for patterns in dump_patterns.values() for pattern in patterns]
dump.flatten_dump_patterns(dump_patterns, restore_names)
),
location,
storage,
@@ -367,33 +365,36 @@ def load_configurations(config_filenames):
return (configs, logs)
def log_record(**kwargs):
'''
Create a log record based on the given makeLogRecord() arguments, one of which must be
named "levelno". Log the record and return it.
'''
record = logging.makeLogRecord(kwargs)
logger.handle(record)
return record
def make_error_log_records(message, error=None):
'''
Given error message text and an optional exception object, yield a series of logging.LogRecord
instances with error summary information.
instances with error summary information. As a side effect, log each record.
'''
if not error:
yield logging.makeLogRecord(
dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
)
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
return
try:
raise error
except CalledProcessError as error:
yield logging.makeLogRecord(
dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
)
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
if error.output:
yield logging.makeLogRecord(
dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error.output)
)
yield logging.makeLogRecord(dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error))
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error.output)
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error)
except (ValueError, OSError) as error:
yield logging.makeLogRecord(
dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
)
yield logging.makeLogRecord(dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error))
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=message)
yield log_record(levelno=logging.CRITICAL, levelname='CRITICAL', msg=error)
except: # noqa: E722
# Raising above only as a means of determining the error type. Swallow the exception here
# because we don't want the exception to propagate out of this function.
@@ -537,13 +538,14 @@ def main(): # pragma: no cover
summary_logs = list(collect_configuration_run_summary_logs(configs, arguments))
logger.info('')
logger.info('summary:')
[
logger.handle(log)
for log in parse_logs + summary_logs
if log.levelno >= logger.getEffectiveLevel()
]
if logger.getEffectiveLevel() < logging.WARNING:
logger.info('')
logger.info('summary:')
[
logger.handle(log)
for log in parse_logs + summary_logs
if log.levelno >= logger.getEffectiveLevel()
]
if any(log.levelno == logging.CRITICAL for log in summary_logs):
exit_with_help_link()
+14 -8
View File
@@ -2,23 +2,29 @@ import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_CRONHUB = {
monitor.State.START: 'start',
monitor.State.FINISH: 'finish',
monitor.State.FAIL: 'fail',
}
def ping_cronhub(ping_url, config_filename, dry_run, state):
def ping_monitor(ping_url, config_filename, state, dry_run):
'''
Ping the given Cronhub URL, substituting in the state string. Use the given configuration
Ping the given Cronhub URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
if not ping_url:
logger.debug('{}: No Cronhub hook set'.format(config_filename))
return
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
formatted_state = '/{}/'.format(state)
formatted_state = '/{}/'.format(MONITOR_STATE_TO_CRONHUB[state])
ping_url = ping_url.replace('/start/', formatted_state).replace('/ping/', formatted_state)
logger.info('{}: Pinging Cronhub {}{}'.format(config_filename, state, dry_run_label))
logger.info(
'{}: Pinging Cronhub {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronhub ping URL {}'.format(config_filename, ping_url))
if not dry_run:
+16 -10
View File
@@ -2,22 +2,28 @@ import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_CRONITOR = {
monitor.State.START: 'run',
monitor.State.FINISH: 'complete',
monitor.State.FAIL: 'fail',
}
def ping_cronitor(ping_url, config_filename, dry_run, append):
'''
Ping the given Cronitor URL, appending the append string. Use the given configuration filename
in any log entries. If this is a dry run, then don't actually ping anything.
'''
if not ping_url:
logger.debug('{}: No Cronitor hook set'.format(config_filename))
return
def ping_monitor(ping_url, config_filename, state, dry_run):
'''
Ping the given Cronitor URL, modified with the monitor.State. Use the given configuration
filename in any log entries. If this is a dry run, then don't actually ping anything.
'''
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
ping_url = '{}/{}'.format(ping_url, append)
ping_url = '{}/{}'.format(ping_url, MONITOR_STATE_TO_CRONITOR[state])
logger.info('{}: Pinging Cronitor {}{}'.format(config_filename, append, dry_run_label))
logger.info(
'{}: Pinging Cronitor {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Cronitor ping URL {}'.format(config_filename, ping_url))
if not dry_run:
+8 -2
View File
@@ -1,10 +1,16 @@
import logging
from borgmatic.hooks import mysql, postgresql
from borgmatic.hooks import cronhub, cronitor, healthchecks, mysql, postgresql
logger = logging.getLogger(__name__)
HOOK_NAME_TO_MODULE = {'postgresql_databases': postgresql, 'mysql_databases': mysql}
HOOK_NAME_TO_MODULE = {
'healthchecks': healthchecks,
'cronitor': cronitor,
'cronhub': cronhub,
'postgresql_databases': postgresql,
'mysql_databases': mysql,
}
def call_hook(function_name, hooks, log_prefix, hook_name, *args, **kwargs):
+25 -1
View File
@@ -20,6 +20,26 @@ def make_database_dump_filename(dump_path, name, hostname=None):
return os.path.join(os.path.expanduser(dump_path), hostname or 'localhost', name)
def flatten_dump_patterns(dump_patterns, names):
'''
Given a dict from a database hook name to glob patterns matching the dumps for the named
databases, flatten out all the glob patterns into a single sequence, and return it.
Raise ValueError if there are no resulting glob patterns, which indicates that databases are not
configured in borgmatic's configuration.
'''
flattened = [pattern for patterns in dump_patterns.values() for pattern in patterns]
if not flattened:
raise ValueError(
'Cannot restore database(s) {} missing from borgmatic\'s configuration'.format(
', '.join(names) or '"all"'
)
)
return flattened
def remove_database_dumps(dump_path, databases, database_type_name, log_prefix, dry_run):
'''
Remove the database dumps for the given databases in the dump directory path. The databases are
@@ -109,7 +129,6 @@ def get_per_hook_database_configurations(hooks, names, dump_patterns):
Raise ValueError if one of the database names cannot be matched to a database in borgmatic's
database configuration.
'''
# TODO: Need to filter names by database type? Maybe take a database --type argument to disambiguate.
hook_databases = {
hook_name: list(
get_database_configurations(
@@ -122,6 +141,11 @@ def get_per_hook_database_configurations(hooks, names, dump_patterns):
}
if not names or 'all' in names:
if not any(hook_databases.values()):
raise ValueError(
'Cannot restore database "all", as there are no database dumps in the archive'
)
return hook_databases
found_names = {
+75 -11
View File
@@ -2,18 +2,80 @@ import logging
import requests
from borgmatic.hooks import monitor
logger = logging.getLogger(__name__)
MONITOR_STATE_TO_HEALTHCHECKS = {
monitor.State.START: 'start',
monitor.State.FINISH: None, # Healthchecks doesn't append to the URL for the finished state.
monitor.State.FAIL: 'fail',
}
def ping_healthchecks(ping_url_or_uuid, config_filename, dry_run, append=None):
PAYLOAD_TRUNCATION_INDICATOR = '...\n'
PAYLOAD_LIMIT_BYTES = 10 * 1024 - len(PAYLOAD_TRUNCATION_INDICATOR)
class Forgetful_buffering_handler(logging.Handler):
'''
Ping the given Healthchecks URL or UUID, appending the append string if any. Use the given
A buffering log handler that stores log messages in memory, and throws away messages (oldest
first) once a particular capacity in bytes is reached.
'''
def __init__(self, byte_capacity):
super().__init__()
self.byte_capacity = byte_capacity
self.byte_count = 0
self.buffer = []
self.forgot = False
def emit(self, record):
message = record.getMessage() + '\n'
self.byte_count += len(message)
self.buffer.append(message)
while self.byte_count > self.byte_capacity and self.buffer:
self.byte_count -= len(self.buffer[0])
self.buffer.pop(0)
self.forgot = True
def format_buffered_logs_for_payload():
'''
Get the handler previously added to the root logger, and slurp buffered logs out of it to
send to Healthchecks.
'''
try:
buffering_handler = next(
handler
for handler in logging.getLogger().handlers
if isinstance(handler, Forgetful_buffering_handler)
)
except StopIteration:
payload = 'Cannot find the log handler for sending logs to Healthchecks'
logger.warning(payload)
return payload
payload = ''.join(message for message in buffering_handler.buffer)
if buffering_handler.forgot:
return PAYLOAD_TRUNCATION_INDICATOR + payload
return payload
def ping_monitor(ping_url_or_uuid, config_filename, state, dry_run):
'''
Ping the given Healthchecks URL or UUID, modified with the monitor.State. Use the given
configuration filename in any log entries. If this is a dry run, then don't actually ping
anything.
'''
if not ping_url_or_uuid:
logger.debug('{}: No Healthchecks hook set'.format(config_filename))
return
if state is monitor.State.START:
# Add a handler to the root logger that stores in memory the most recent logs emitted. That
# way, we can send them all to Healthchecks upon a finish or failure state.
logging.getLogger().addHandler(Forgetful_buffering_handler(PAYLOAD_LIMIT_BYTES))
payload = ''
ping_url = (
ping_url_or_uuid
@@ -22,16 +84,18 @@ def ping_healthchecks(ping_url_or_uuid, config_filename, dry_run, append=None):
)
dry_run_label = ' (dry run; not actually pinging)' if dry_run else ''
if append:
ping_url = '{}/{}'.format(ping_url, append)
healthchecks_state = MONITOR_STATE_TO_HEALTHCHECKS.get(state)
if healthchecks_state:
ping_url = '{}/{}'.format(ping_url, healthchecks_state)
logger.info(
'{}: Pinging Healthchecks{}{}'.format(
config_filename, ' ' + append if append else '', dry_run_label
)
'{}: Pinging Healthchecks {}{}'.format(config_filename, state.name.lower(), dry_run_label)
)
logger.debug('{}: Using Healthchecks ping URL {}'.format(config_filename, ping_url))
if state in (monitor.State.FINISH, monitor.State.FAIL):
payload = format_buffered_logs_for_payload()
if not dry_run:
logging.getLogger('urllib3').setLevel(logging.ERROR)
requests.get(ping_url)
requests.post(ping_url, data=payload)
+9
View File
@@ -0,0 +1,9 @@
from enum import Enum
MONITOR_HOOK_NAMES = ('healthchecks', 'cronitor', 'cronhub')
class State(Enum):
START = 1
FINISH = 2
FAIL = 3
+1 -1
View File
@@ -104,7 +104,7 @@ def configure_logging(
syslog_handler.setLevel(syslog_log_level)
handlers = (console_handler, syslog_handler)
elif log_file:
file_handler = logging.FileHandler(log_file)
file_handler = logging.handlers.WatchedFileHandler(log_file)
file_handler.setFormatter(logging.Formatter('[%(asctime)s] %(levelname)s: %(message)s'))
file_handler.setLevel(log_file_log_level)
handlers = (console_handler, file_handler)
+18
View File
@@ -173,6 +173,24 @@ after backups. So if necessary, you can use these hooks to create database
dumps with any database system.
## Troubleshooting
### MySQL table lock errors
If you encounter table lock errors during a database dump with MySQL/MariaDB,
you may need to [use a
transaction](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html#option_mysqldump_single-transaction).
You can add any additional flags to the `options:` in your database
configuration. Here's an example:
```yaml
hooks:
mysql_databases:
- name: posts
options: "--single-transaction --quick"
```
## Related documentation
* [Set up backups with borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/)
+15 -14
View File
@@ -70,6 +70,21 @@ Or to increase syslog logging to include debug spew:
borgmatic --syslog-verbosity 2
```
### Rate limiting
If you are using rsyslog or systemd's journal, be aware that by default they
both throttle the rate at which logging occurs. So you may need to change
either [the global rate
limit](https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/) or
[the per-service rate
limit](https://www.freedesktop.org/software/systemd/man/journald.conf.html#RateLimitIntervalSec=)
if you're finding that borgmatic logs are missing.
Note that the [sample borgmatic systemd service
file](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#systemd)
already has this rate limit disabled for systemd's journal.
### Logging to file
If you don't want to use syslog, and you'd rather borgmatic log to a plain
@@ -84,20 +99,6 @@ the log file so it doesn't grow too large. Also, there is a
`--log-file-verbosity` flag to customize the log file's log level.
### systemd journal
If your local syslog daemon is systemd's journal, be aware that journald by
default throttles the rate at which a particular program can log. So you may
need to [change the journald rate
limit](https://www.freedesktop.org/software/systemd/man/journald.conf.html#RateLimitIntervalSec=)
in `/etc/systemd/journald.conf` if you're finding that borgmatic journald logs
are missing.
Note that the [sample borgmatic systemd service
file](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#systemd)
already has this rate limit disabled.
## Related documentation
* [Set up backups with borgmatic](https://torsion.org/borgmatic/docs/how-to/set-up-backups/)
+12 -2
View File
@@ -116,9 +116,19 @@ With this hook in place, borgmatic pings your Healthchecks project when a
backup begins, ends, or errors. Specifically, before the <a
href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup`
hooks</a> run, borgmatic lets Healthchecks know that a backup has started.
Then, if the backup completes successfully, borgmatic notifies Healthchecks of
the success after the `after_backup` hooks run. And if an error occurs during
the backup, borgmatic notifies Healthchecks after the `on_error` hooks run.
the success after the `after_backup` hooks run, and includes borgmatic logs in
the payload data sent to Healthchecks. This means that borgmatic logs show up
in the Healthchecks UI, although be aware that Healthchecks currently has a
10-kilobyte limit for the logs in each ping.
If an error occurs during the backup, borgmatic notifies Healthchecks after
the `on_error` hooks run, also tacking on logs including the error itself.
Note that borgmatic sends logs to Healthchecks by applying the maximum of any
other borgmatic verbosity level (`--verbosity`, `--syslog-verbosity`, etc.),
as there is not currently a dedicated Healthchecks verbosity setting.
You can configure Healthchecks to notify you by a [variety of
mechanisms](https://healthchecks.io/#welcome-integrations) when backups fail
+3 -2
View File
@@ -36,11 +36,12 @@ you're using Python 3, as borgmatic does not support Python 2.
Along with the above process, you have several other options for installing
borgmatic:
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/)
* [Docker base image](https://hub.docker.com/r/monachus/borgmatic/)
* [Docker image with support for scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/)
* [Debian](https://tracker.debian.org/pkg/borgmatic)
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
* [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
* [Fedora official](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
* [Fedora unofficial](https://copr.fedorainfracloud.org/coprs/heffer/borgmatic/)
* [Arch Linux](https://www.archlinux.org/packages/community/any/borgmatic/)
* [OpenBSD](http://ports.su/sysutils/borgmatic)
* [openSUSE](https://software.opensuse.org/package/borgmatic)
+3 -3
View File
@@ -39,9 +39,9 @@ generate-borgmatic-config --source config.yaml --destination config-new.yaml
New options start as commented out, so you can edit the file and decide
whether you want to use each one.
There are a few caveats to this process, however. First, when generating the
new configuration file, `generate-borgmatic-config` replaces any comments
you've written in your original configuration file with the newest generated
There are a few caveats to this process. First, when generating the new
configuration file, `generate-borgmatic-config` replaces any comments you've
written in your original configuration file with the newest generated
comments. Second, the script adds back any options you had originally deleted,
although it does so with the options commented out. And finally, any YAML
includes you've used in the source configuration get flattened out into a
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 5.7 KiB

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 3.5 KiB

Before

Width:  |  Height:  |  Size: 1.6 KiB

After

Width:  |  Height:  |  Size: 1.6 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 9.3 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.7 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 7.3 KiB

+1 -1
View File
@@ -1,6 +1,6 @@
from setuptools import find_packages, setup
VERSION = '1.4.9'
VERSION = '1.4.11'
setup(
+20 -14
View File
@@ -23,10 +23,7 @@ def test_run_configuration_runs_actions_for_each_repository():
def test_run_configuration_executes_hooks_for_create_action():
flexmock(module.borg_environment).should_receive('initialize')
flexmock(module.command).should_receive('execute_hook').twice()
flexmock(module.dispatch).should_receive('call_hooks').twice()
flexmock(module.healthchecks).should_receive('ping_healthchecks').twice()
flexmock(module.cronitor).should_receive('ping_cronitor').twice()
flexmock(module.cronhub).should_receive('ping_cronhub').twice()
flexmock(module.dispatch).should_receive('call_hooks').at_least().twice()
flexmock(module).should_receive('run_actions').and_return([])
config = {'location': {'repositories': ['foo']}}
arguments = {'global': flexmock(dry_run=False), 'create': flexmock()}
@@ -38,9 +35,6 @@ def test_run_configuration_logs_actions_error():
flexmock(module.borg_environment).should_receive('initialize')
flexmock(module.command).should_receive('execute_hook')
flexmock(module.dispatch).should_receive('call_hooks')
flexmock(module.healthchecks).should_receive('ping_healthchecks')
flexmock(module.cronitor).should_receive('ping_cronitor')
flexmock(module.cronhub).should_receive('ping_cronhub')
expected_results = [flexmock()]
flexmock(module).should_receive('make_error_log_records').and_return(expected_results)
flexmock(module).should_receive('run_actions').and_raise(OSError)
@@ -72,9 +66,6 @@ def test_run_configuration_logs_post_hook_error():
OSError
).and_return(None)
flexmock(module.dispatch).should_receive('call_hooks')
flexmock(module.healthchecks).should_receive('ping_healthchecks')
flexmock(module.cronitor).should_receive('ping_cronitor')
flexmock(module.cronhub).should_receive('ping_cronhub')
expected_results = [flexmock()]
flexmock(module).should_receive('make_error_log_records').and_return(expected_results)
flexmock(module).should_receive('run_actions').and_return([])
@@ -124,36 +115,50 @@ def test_load_configurations_logs_critical_for_parse_error():
assert {log.levelno for log in logs} == {logging.CRITICAL}
def test_log_record_does_not_raise():
module.log_record(levelno=1, foo='bar', baz='quux')
def test_make_error_log_records_generates_output_logs_for_message_only():
flexmock(module).should_receive('log_record').replace_with(dict)
logs = tuple(module.make_error_log_records('Error'))
assert {log.levelno for log in logs} == {logging.CRITICAL}
assert {log['levelno'] for log in logs} == {logging.CRITICAL}
def test_make_error_log_records_generates_output_logs_for_called_process_error():
flexmock(module).should_receive('log_record').replace_with(dict)
logs = tuple(
module.make_error_log_records(
'Error', subprocess.CalledProcessError(1, 'ls', 'error output')
)
)
assert {log.levelno for log in logs} == {logging.CRITICAL}
assert {log['levelno'] for log in logs} == {logging.CRITICAL}
assert any(log for log in logs if 'error output' in str(log))
def test_make_error_log_records_generates_logs_for_value_error():
flexmock(module).should_receive('log_record').replace_with(dict)
logs = tuple(module.make_error_log_records('Error', ValueError()))
assert {log.levelno for log in logs} == {logging.CRITICAL}
assert {log['levelno'] for log in logs} == {logging.CRITICAL}
def test_make_error_log_records_generates_logs_for_os_error():
flexmock(module).should_receive('log_record').replace_with(dict)
logs = tuple(module.make_error_log_records('Error', OSError()))
assert {log.levelno for log in logs} == {logging.CRITICAL}
assert {log['levelno'] for log in logs} == {logging.CRITICAL}
def test_make_error_log_records_generates_nothing_for_other_error():
flexmock(module).should_receive('log_record').replace_with(dict)
logs = tuple(module.make_error_log_records('Error', KeyError()))
assert logs == ()
@@ -277,6 +282,7 @@ def test_collect_configuration_run_summary_logs_run_configuration_error():
flexmock(module).should_receive('run_configuration').and_return(
[logging.makeLogRecord(dict(levelno=logging.CRITICAL, levelname='CRITICAL', msg='Error'))]
)
flexmock(module).should_receive('make_error_log_records').and_return([])
arguments = {}
logs = tuple(
+19 -13
View File
@@ -3,30 +3,36 @@ from flexmock import flexmock
from borgmatic.hooks import cronhub as module
def test_ping_cronhub_hits_ping_url_with_start_state():
def test_ping_monitor_rewrites_ping_url_for_start_state():
ping_url = 'https://example.com/start/abcdef'
state = 'bork'
flexmock(module.requests).should_receive('get').with_args('https://example.com/bork/abcdef')
flexmock(module.requests).should_receive('get').with_args('https://example.com/start/abcdef')
module.ping_cronhub(ping_url, 'config.yaml', dry_run=False, state=state)
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.START, dry_run=False)
def test_ping_cronhub_hits_ping_url_with_ping_state():
def test_ping_monitor_rewrites_ping_url_and_state_for_start_state():
ping_url = 'https://example.com/ping/abcdef'
state = 'bork'
flexmock(module.requests).should_receive('get').with_args('https://example.com/bork/abcdef')
flexmock(module.requests).should_receive('get').with_args('https://example.com/start/abcdef')
module.ping_cronhub(ping_url, 'config.yaml', dry_run=False, state=state)
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.START, dry_run=False)
def test_ping_cronhub_without_ping_url_does_not_raise():
flexmock(module.requests).should_receive('get').never()
def test_ping_monitor_rewrites_ping_url_for_finish_state():
ping_url = 'https://example.com/start/abcdef'
flexmock(module.requests).should_receive('get').with_args('https://example.com/finish/abcdef')
module.ping_cronhub(ping_url=None, config_filename='config.yaml', dry_run=False, state='oops')
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.FINISH, dry_run=False)
def test_ping_cronhub_dry_run_does_not_hit_ping_url():
def test_ping_monitor_rewrites_ping_url_for_fail_state():
ping_url = 'https://example.com/start/abcdef'
flexmock(module.requests).should_receive('get').with_args('https://example.com/fail/abcdef')
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.FAIL, dry_run=False)
def test_ping_monitor_dry_run_does_not_hit_ping_url():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').never()
module.ping_cronhub(ping_url, 'config.yaml', dry_run=True, state='yay')
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.START, dry_run=True)
+16 -9
View File
@@ -3,22 +3,29 @@ from flexmock import flexmock
from borgmatic.hooks import cronitor as module
def test_ping_cronitor_hits_ping_url():
def test_ping_monitor_hits_ping_url_for_start_state():
ping_url = 'https://example.com'
append = 'failed-so-hard'
flexmock(module.requests).should_receive('get').with_args('{}/{}'.format(ping_url, append))
flexmock(module.requests).should_receive('get').with_args('{}/{}'.format(ping_url, 'run'))
module.ping_cronitor(ping_url, 'config.yaml', dry_run=False, append=append)
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.START, dry_run=False)
def test_ping_cronitor_without_ping_url_does_not_raise():
flexmock(module.requests).should_receive('get').never()
def test_ping_monitor_hits_ping_url_for_finish_state():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').with_args('{}/{}'.format(ping_url, 'complete'))
module.ping_cronitor(ping_url=None, config_filename='config.yaml', dry_run=False, append='oops')
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.FINISH, dry_run=False)
def test_ping_cronitor_dry_run_does_not_hit_ping_url():
def test_ping_monitor_hits_ping_url_for_fail_state():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').with_args('{}/{}'.format(ping_url, 'fail'))
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.FAIL, dry_run=False)
def test_ping_monitor_dry_run_does_not_hit_ping_url():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').never()
module.ping_cronitor(ping_url, 'config.yaml', dry_run=True, append='yay')
module.ping_monitor(ping_url, 'config.yaml', module.monitor.State.START, dry_run=True)
+35
View File
@@ -26,6 +26,29 @@ def test_make_database_dump_filename_with_invalid_name_raises():
module.make_database_dump_filename('databases', 'invalid/name')
def test_flatten_dump_patterns_produces_list_of_all_patterns():
dump_patterns = {'postgresql_databases': ['*/glob', 'glob/*'], 'mysql_databases': ['*/*/*']}
expected_patterns = sorted(
dump_patterns['postgresql_databases'] + dump_patterns['mysql_databases']
)
assert sorted(module.flatten_dump_patterns(dump_patterns, ('bob',))) == expected_patterns
def test_flatten_dump_patterns_with_no_patterns_errors():
dump_patterns = {'postgresql_databases': [], 'mysql_databases': []}
with pytest.raises(ValueError):
assert module.flatten_dump_patterns(dump_patterns, ('bob',))
def test_flatten_dump_patterns_with_no_hooks_errors():
dump_patterns = {}
with pytest.raises(ValueError):
assert module.flatten_dump_patterns(dump_patterns, ('bob',))
def test_remove_database_dumps_removes_dump_for_each_database():
databases = [{'name': 'foo'}, {'name': 'bar'}]
flexmock(module).should_receive('make_database_dump_filename').and_return(
@@ -134,3 +157,15 @@ def test_get_per_hook_database_configurations_with_unknown_database_name_raises(
with pytest.raises(ValueError):
module.get_per_hook_database_configurations(hooks, names, dump_patterns)
def test_get_per_hook_database_configurations_with_all_and_no_archive_dumps_raises():
hooks = {'postgresql_databases': [flexmock()]}
names = ('foo', 'all')
dump_patterns = flexmock()
flexmock(module).should_receive('get_database_configurations').with_args(
hooks['postgresql_databases'], names
).and_return([])
with pytest.raises(ValueError):
module.get_per_hook_database_configurations(hooks, names, dump_patterns)
+85 -20
View File
@@ -3,38 +3,103 @@ from flexmock import flexmock
from borgmatic.hooks import healthchecks as module
def test_ping_healthchecks_hits_ping_url():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').with_args(ping_url)
def test_forgetful_buffering_handler_emit_collects_log_records():
handler = module.Forgetful_buffering_handler(byte_capacity=100)
handler.emit(flexmock(getMessage=lambda: 'foo'))
handler.emit(flexmock(getMessage=lambda: 'bar'))
module.ping_healthchecks(ping_url, 'config.yaml', dry_run=False)
assert handler.buffer == ['foo\n', 'bar\n']
assert not handler.forgot
def test_ping_healthchecks_without_ping_url_does_not_raise():
flexmock(module.requests).should_receive('get').never()
module.ping_healthchecks(ping_url_or_uuid=None, config_filename='config.yaml', dry_run=False)
def test_forgetful_buffering_handler_emit_forgets_log_records_when_capacity_reached():
handler = module.Forgetful_buffering_handler(byte_capacity=len('foo\nbar\n'))
handler.emit(flexmock(getMessage=lambda: 'foo'))
assert handler.buffer == ['foo\n']
handler.emit(flexmock(getMessage=lambda: 'bar'))
assert handler.buffer == ['foo\n', 'bar\n']
handler.emit(flexmock(getMessage=lambda: 'baz'))
assert handler.buffer == ['bar\n', 'baz\n']
handler.emit(flexmock(getMessage=lambda: 'quux'))
assert handler.buffer == ['quux\n']
assert handler.forgot
def test_ping_healthchecks_with_ping_uuid_hits_corresponding_url():
ping_uuid = 'abcd-efgh-ijkl-mnop'
flexmock(module.requests).should_receive('get').with_args(
'https://hc-ping.com/{}'.format(ping_uuid)
def test_format_buffered_logs_for_payload_flattens_log_buffer():
handler = module.Forgetful_buffering_handler(byte_capacity=100)
handler.buffer = ['foo\n', 'bar\n']
flexmock(module.logging).should_receive('getLogger').and_return(flexmock(handlers=[handler]))
payload = module.format_buffered_logs_for_payload()
assert payload == 'foo\nbar\n'
def test_format_buffered_logs_for_payload_inserts_truncation_indicator_when_logs_forgotten():
handler = module.Forgetful_buffering_handler(byte_capacity=100)
handler.buffer = ['foo\n', 'bar\n']
handler.forgot = True
flexmock(module.logging).should_receive('getLogger').and_return(flexmock(handlers=[handler]))
payload = module.format_buffered_logs_for_payload()
assert payload == '...\nfoo\nbar\n'
def test_format_buffered_logs_for_payload_without_handler_produces_payload_anyway():
flexmock(module.logging).should_receive('getLogger').and_return(
flexmock(handlers=[module.logging.Handler()])
)
module.ping_healthchecks(ping_uuid, 'config.yaml', dry_run=False)
payload = module.format_buffered_logs_for_payload()
assert payload
def test_ping_healthchecks_hits_ping_url_with_append():
def test_ping_monitor_hits_ping_url_for_start_state():
flexmock(module).should_receive('Forgetful_buffering_handler')
ping_url = 'https://example.com'
append = 'failed-so-hard'
flexmock(module.requests).should_receive('get').with_args('{}/{}'.format(ping_url, append))
flexmock(module.requests).should_receive('post').with_args(
'{}/{}'.format(ping_url, 'start'), data=''
)
module.ping_healthchecks(ping_url, 'config.yaml', dry_run=False, append=append)
module.ping_monitor(ping_url, 'config.yaml', state=module.monitor.State.START, dry_run=False)
def test_ping_healthchecks_dry_run_does_not_hit_ping_url():
def test_ping_monitor_hits_ping_url_for_finish_state():
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('get').never()
payload = flexmock()
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.requests).should_receive('post').with_args(ping_url, data=payload)
module.ping_healthchecks(ping_url, 'config.yaml', dry_run=True)
module.ping_monitor(ping_url, 'config.yaml', state=module.monitor.State.FINISH, dry_run=False)
def test_ping_monitor_hits_ping_url_for_fail_state():
ping_url = 'https://example.com'
payload = flexmock()
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'{}/{}'.format(ping_url, 'fail'), data=payload
)
module.ping_monitor(ping_url, 'config.yaml', state=module.monitor.State.FAIL, dry_run=False)
def test_ping_monitor_with_ping_uuid_hits_corresponding_url():
ping_uuid = 'abcd-efgh-ijkl-mnop'
payload = flexmock()
flexmock(module).should_receive('format_buffered_logs_for_payload').and_return(payload)
flexmock(module.requests).should_receive('post').with_args(
'https://hc-ping.com/{}'.format(ping_uuid), data=payload
)
module.ping_monitor(ping_uuid, 'config.yaml', state=module.monitor.State.FINISH, dry_run=False)
def test_ping_monitor_dry_run_does_not_hit_ping_url():
flexmock(module).should_receive('Forgetful_buffering_handler')
ping_url = 'https://example.com'
flexmock(module.requests).should_receive('post').never()
module.ping_monitor(ping_url, 'config.yaml', state=module.monitor.State.START, dry_run=True)
+6 -6
View File
@@ -203,10 +203,10 @@ def test_configure_logging_to_logfile_instead_of_syslog():
)
flexmock(module.os.path).should_receive('exists').with_args('/dev/log').and_return(True)
flexmock(module.logging.handlers).should_receive('SysLogHandler').never()
file_handler = logging.FileHandler('/tmp/logfile')
flexmock(module.logging).should_receive('FileHandler').with_args('/tmp/logfile').and_return(
file_handler
).once()
file_handler = logging.handlers.WatchedFileHandler('/tmp/logfile')
flexmock(module.logging.handlers).should_receive('WatchedFileHandler').with_args(
'/tmp/logfile'
).and_return(file_handler).once()
module.configure_logging(
console_log_level=logging.INFO, log_file_log_level=logging.DEBUG, log_file='/tmp/logfile'
@@ -214,12 +214,12 @@ def test_configure_logging_to_logfile_instead_of_syslog():
def test_configure_logging_skips_logfile_if_argument_is_none():
# No FileHandler added if argument --log-file is None
# No WatchedFileHandler added if argument --log-file is None
flexmock(module).should_receive('interactive_console').and_return(False)
flexmock(module.logging).should_receive('basicConfig').with_args(
level=logging.INFO, handlers=tuple
)
flexmock(module.os.path).should_receive('exists').and_return(False)
flexmock(module.logging).should_receive('FileHandler').never()
flexmock(module.logging.handlers).should_receive('WatchedFileHandler').never()
module.configure_logging(console_log_level=logging.INFO, log_file=None)