Configuration files includes and merging (#148 ).

Support for Borg create/extract --numeric-owner flag (#147 ).
Add note about uncommenting section names.
2026-07-22 18:13:02 +02:00 · 2019-03-06 12:06:27 -08:00 · 2019-03-05 09:11:35 -08:00 · 2019-03-04 15:15:49 -08:00 · 2019-02-23 23:02:17 -08:00 · 2019-02-18 22:43:32 -08:00
32 changed files with 1633 additions and 608 deletions
@@ -1,9 +1,27 @@
+1.3.0
+ * #148: Configuration file includes and merging via "!include" tag to support reuse of common
+   options across configuration files.
+
+1.2.18
+ * #147: Support for Borg create/extract --numeric-owner flag via "numeric_owner" option in
+   borgmatic's location section.
+
+1.2.17
+ * #140: List the files within an archive via --list --archive option.
+
+1.2.16
+ * #119: Include a sample borgmatic configuration file in the documentation.
+ * #123: Support for Borg archive restoration via borgmatic --extract command-line flag.
+ * Refactor documentation into multiple separate pages for clarity and findability.
+ * Organize options within command-line help into logical groups.
+ * Exclude tests from distribution packages.
+
 1.2.15
 * #127: Remove date echo from schema example, as it's not a substitute for real logging.
 * #132: Leave exclude_patterns glob expansion to Borg, since doing it in borgmatic leads to
   confusing behavior.
 * #136: Handle and format validation errors raised during argument parsing.
- * #139: Allow use of --stats flag when --create or --prune flags are implied.
+ * #138: Allow use of --stats flag when --create or --prune flags are implied.

 1.2.14
 * #103: When generating sample configuration with generate-borgmatic-config, document the defaults
@@ -29,8 +47,8 @@
 * #108: Support for Borg create --progress via borgmatic command-line flag.

 1.2.10
- * #105: Support for Borg --chunker-params create option via "chunker_params" in borgmatic's storage
-   section.
+ * #105: Support for Borg --chunker-params create option via "chunker_params" option in borgmatic's
+   storage section.

 1.2.9
 * #102: Fix for syntax error that occurred in Python 3.5 and below.
@@ -54,338 +54,22 @@ href="https://asciinema.org/a/203761" target="_blank">screencast</a>.
 <script src="https://asciinema.org/a/203761.js" id="asciicast-203761" async></script>


-## Installation
+## How-to guides

-To get up and running, first [install
-Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
-least version 1.1.
+ * [Set up backups with borgmatic](docs/how-to/set-up-backups.md) ⬅ *Start here!*
+ * [Make per-application backups](docs/how-to/make-per-application-backups.md)
+ * [Deal with very large backups](docs/how-to/deal-with-very-large-backups.md)
+ * [Inspect your backups](docs/how-to/inspect-your-backups.md)
+ * [Restore a backup](docs/how-to/restore-a-backup.md)
+ * [Run preparation steps before backups](docs/how-to/run-preparation-steps-before-backups.md)
+ * [Upgrade borgmatic](docs/how-to/upgrade.md)
+ * [Develop on borgmatic](docs/how-to/develop-on-borgmatic.md)

-Then, run the following command to download and install borgmatic:

-```bash
-sudo pip3 install --upgrade borgmatic
-```
+## Reference guides

-Note that your pip binary may have a different name than "pip3". Make sure
-you're using Python 3, as borgmatic does not support Python 2.
-
-
-### Other ways to install
-
- * [Docker image](https://hub.docker.com/r/monachus/borgmatic/)
- * [Another Docker image](https://hub.docker.com/r/b3vis/borgmatic/)
- * [Debian](https://tracker.debian.org/pkg/borgmatic)
- * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
- * [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
- * [Arch Linux](https://aur.archlinux.org/packages/borgmatic/)
- * [OpenBSD](http://ports.su/sysutils/borgmatic)
- * [openSUSE](https://software.opensuse.org/package/borgmatic)
-
-<br><br>
-
-
-## Configuration
-
-After you install borgmatic, generate a sample configuration file:
-
-```bash
-sudo generate-borgmatic-config
-```
-
-If that command is not found, then it may be installed in a location that's
-not in your system `PATH`. Try looking in `/usr/local/bin/`.
-
-This generates a sample configuration file at /etc/borgmatic/config.yaml (by
-default). You should edit the file to suit your needs, as the values are
-representative. All fields are optional except where indicated, so feel free
-to ignore anything you don't need.
-
-You can also have a look at the [full configuration
-schema](https://projects.torsion.org/witten/borgmatic/src/master/borgmatic/config/schema.yaml)
-for the authoritative set of all configuration options. This is handy if
-borgmatic has added new options since you originally created your
-configuration file.
-
-
-### Encryption
-
-Note that if you plan to run borgmatic on a schedule with cron, and you
-encrypt your Borg repository with a passphrase instead of a key file, you'll
-either need to set the borgmatic `encryption_passphrase` configuration
-variable or set the `BORG_PASSPHRASE` environment variable. See the
-[repository encryption
-section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
-of the Quick Start for more info.
-
-Alternatively, the passphrase can be specified programatically by setting
-either the borgmatic `encryption_passcommand` configuration variable or the
-`BORG_PASSCOMMAND` environment variable. See the [Borg Security
-FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
-for more info.
-
-
-## Usage
-
-### Initialization
-
-Before you can create backups with borgmatic, you first need to initialize a
-Borg repository so you have a destination for your backup archives. (But skip
-this step if you already have a Borg repository.) To create a repository, run
-a command like the following:
-
-```bash
-borgmatic --init --encryption repokey
-```
-
-This uses the borgmatic configuration file you created above to determine
-which local or remote repository to create, and encrypts it with the
-encryption passphrase specified there if one is provided. Read about [Borg
-encryption
-modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
-for the menu of available encryption modes.
-
-Also, optionally check out the [Borg Quick
-Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
-background about repository initialization.
-
-Note that borgmatic skips repository initialization if the repository already
-exists. This supports use cases like ensuring a repository exists prior to
-performing a backup.
-
-If the repository is on a remote host, make sure that your local user has
-key-based SSH access to the desired user account on the remote host.
-
-
-### Backups
-
-You can run borgmatic and start a backup by invoking it without arguments:
-
-```bash
-borgmatic
-```
-
-This will also prune any old backups as per the configured retention policy,
-and check backups for consistency problems due to things like file damage.
-
-If you'd like to see the available command-line arguments, view the help:
-
-```bash
-borgmatic --help
-```
-
-Note that borgmatic prunes archives *before* creating an archive, so as to
-free up space for archiving. This means that when a borgmatic run finishes,
-there may still be prune-able archives. Not to worry, as they will get cleaned
-up at the start of the next run.
-
-
-### Verbosity
-
-By default, the backup will proceed silently except in the case of errors. But
-if you'd like to to get additional information about the progress of the
-backup as it proceeds, use the verbosity option:
-
-```bash
-borgmatic --verbosity 1
-```
-
-Or, for even more progress spew:
-
-```bash
-borgmatic --verbosity 2
-```
-
-### À la carte
-
-If you want to run borgmatic with only pruning, creating, or checking enabled,
-the following optional flags are available:
-
-```bash
-borgmatic --prune
-borgmatic --create
-borgmatic --check
-```
-
-You can run with only one of these flags provided, or you can mix and match
-any number of them. This supports use cases like running consistency checks
-from a different cron job with a different frequency, or running pruning with
-a different verbosity level.
-
-Additionally, borgmatic provides convenient flags for Borg's
-[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
-[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
-functionality:
-
-
-```bash
-borgmatic --list
-borgmatic --info
-```
-
-You can include an optional `--json` flag with `--create`, `--list`, or
-`--info` to get the output formatted as JSON.
-
-
-## Autopilot
-
-If you want to run borgmatic automatically, say once a day, the you can
-configure a job runner to invoke it periodically.
-
-### cron
-
-If you're using cron, download the [sample cron
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
-Then, from the directory where you downloaded it:
-
-```bash
-sudo mv borgmatic /etc/cron.d/borgmatic
-sudo chmod +x /etc/cron.d/borgmatic
-```
-
-You can modify the cron file if you'd like to run borgmatic more or less frequently.
-
-### systemd
-
-If you're using systemd instead of cron to run jobs, download the [sample
-systemd service
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
-and the [sample systemd timer
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
-Then, from the directory where you downloaded them:
-
-```bash
-sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
-sudo systemctl enable borgmatic.timer
-sudo systemctl start borgmatic.timer
-```
-
-Feel free to modify the timer file based on how frequently you'd like
-borgmatic to run.
-
-
-## Advanced configuration
-
-### Multiple configuration files
-
-A more advanced usage is to create multiple separate configuration files and
-place each one in an /etc/borgmatic.d directory. For instance:
-
-```bash
-sudo mkdir /etc/borgmatic.d
-sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
-sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
-```
-
-With this approach, you can have entirely different backup policies for
-different applications on your system. For instance, you may want one backup
-configuration for your database data directory, and a different configuration
-for your user home directories.
-
-When you set up multiple configuration files like this, borgmatic will run
-each one in turn from a single borgmatic invocation. This includes, by
-default, the traditional /etc/borgmatic/config.yaml as well.
-
-And if you need even more customizability, you can specify alternate
-configuration paths on the command-line with borgmatic's `--config` option.
-See `borgmatic --help` for more information.
-
-
-### Hooks
-
-If you find yourself performing prepraration tasks before your backup runs, or
-cleanup work afterwards, borgmatic hooks may be of interest. Hooks are
-shell commands that borgmatic executes for you at various points, and they're
-configured in the `hooks` section of your configuration file.
-
-For instance, you can specify `before_backup` hooks to dump a database to file
-before backing it up, and specify `after_backup` hooks to delete the temporary
-file afterwards.
-
-borgmatic hooks run once per configuration file. `before_backup` hooks run
-prior to backups of all repositories. `after_backup` hooks run afterwards, but
-not if an error occurs in a previous hook or in the backups themselves. And
-borgmatic runs `on_error` hooks if an error occurs.
-
-An important security note about hooks: borgmatic executes all hook commands
-with the user permissions of borgmatic itself. So to prevent potential shell
-injection or privilege escalation, do not forget to set secure permissions
-(`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
-
-See the sample generated configuration file mentioned above for specifics
-about hook configuration syntax.
-
-
-## Upgrading
-
-In general, all you should need to do to upgrade borgmatic is run the
-following:
-
-```bash
-sudo pip3 install --upgrade borgmatic
-```
-
-See below about special cases.
-
-
-### Upgrading from borgmatic 1.0.x
-
-borgmatic changed its configuration file format in version 1.1.0 from
-INI-style to YAML. This better supports validation, and has a more natural way
-to express lists of values. To upgrade your existing configuration, first
-upgrade to the new version of borgmatic.
-
-As of version 1.1.0, borgmatic no longer supports Python 2. If you were
-already running borgmatic with Python 3, then you can upgrade borgmatic
-in-place:
-
-```bash
-sudo pip3 install --upgrade borgmatic
-```
-
-But if you were running borgmatic with Python 2, uninstall and reinstall instead:
-
-```bash
-sudo pip uninstall borgmatic
-sudo pip3 install borgmatic
-```
-
-The pip binary names for different versions of Python can differ, so the above
-commands may need some tweaking to work on your machine.
-
-
-Once borgmatic is upgraded, run:
-
-```bash
-sudo upgrade-borgmatic-config
-```
-
-That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
-(by default) using the values from both your existing configuration and
-excludes files. The new version of borgmatic will consume the YAML
-configuration file instead of the old one.
-
-
-### Upgrading from atticmatic
-
-You can ignore this section if you're not an atticmatic user (the former name
-of borgmatic).
-
-borgmatic only supports Borg now and no longer supports Attic. So if you're
-an Attic user, consider switching to Borg. See the [Borg upgrade
-command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
-for more information. Then, follow the instructions above about setting up
-your borgmatic configuration files.
-
-If you were already using Borg with atticmatic, then you can upgrade
-from atticmatic to borgmatic by running the following commands:
-
-```bash
-sudo pip3 uninstall atticmatic
-sudo pip3 install borgmatic
-```
-
-That's it! borgmatic will continue using your /etc/borgmatic configuration
-files.
+ * [borgmatic configuration reference](docs/reference/configuration.md)
+ * [borgmatic command-line reference](docs/reference/command-line.md)


 ## Support and contributing
@@ -409,127 +93,6 @@ or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
 to discuss your idea. We also accept Pull Requests on GitHub, if that's more
 your thing. In general, contributions are very welcome. We don't bite! 

-
-### Code style
-
-Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
-the following deviations from it:
-
- * For strings, prefer single quotes over double quotes.
- * Limit all lines to a maximum of 100 characters.
- * Use trailing commas within multiline values or argument lists.
- * For multiline constructs, put opening and closing delimeters on lines
-   separate from their contents.
- * Within multiline constructs, use standard four-space indentation. Don't align
-   indentation with an opening delimeter.
-
-borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
-formatter and [Flake8](http://flake8.pycqa.org/en/latest/) code checker, so
-certain code style requirements will be enforced when running automated tests.
-See the Black and Flake8 documentation for more information.
-
-
-### Development
-
-To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
-
-```bash
-git clone https://projects.torsion.org/witten/borgmatic.git
-```
-
-Or:
-
-```bash
-git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
-```
-
-Then, install borgmatic
-"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
-so that you can run borgmatic commands while you're hacking on them to
-make sure your changes work.
-
-```bash
-cd borgmatic/
-pip3 install --editable --user .
-```
-
-Note that this will typically install the borgmatic commands into
-`~/.local/bin`, which may or may not be on your PATH. There are other ways to
-install borgmatic editable as well, for instance into the system Python
-install (so without `--user`, as root), or even into a
-[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
-borgmatic is up to you, but generally an editable install makes development
-and testing easier.
-
-
-### Running tests
-
-Assuming you've cloned the borgmatic source code as described above, and
-you're in the `borgmatic/` working copy, install tox, which is used for
-setting up testing environments:
-
-```bash
-sudo pip3 install tox
-```
-
-Finally, to actually run tests, run:
-
-```bash
-cd borgmatic
-tox
-```
-
-If when running tests, you get an error from the
-[Black](https://black.readthedocs.io/en/stable/) code formatter about files
-that would be reformatted, you can ask Black to format them for you via the
-following:
-
-```bash
-tox -e black
-```
-
-### End-to-end tests
-
-borgmatic additionally includes some end-to-end tests that integration test
-with Borg for a few representative scenarios. These tests don't run by default
-because they're relatively slow and depend on Borg. If you would like to run
-them:
-
-```bash
-tox -e end-to-end
-```
-
-## Troubleshooting
-
-### Broken pipe with remote repository
-
-When running borgmatic on a large remote repository, you may receive errors
-like the following, particularly while "borg check" is validating backups for
-consistency:
-
-```text
-    Write failed: Broken pipe
-    borg: Error: Connection closed by remote host
-```
-
-This error can be caused by an ssh timeout, which you can rectify by adding
-the following to the `~/.ssh/config` file on the client:
-
-```text
-    Host *
-        ServerAliveInterval 120
-```
-
-This should make the client keep the connection alive while validating
-backups.
-
-
-### libyaml compilation errors
-
-borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
-use a C YAML library (libyaml) if present. But if it's not installed, then
-when installing or upgrading borgmatic, you may see errors about compiling the
-YAML library. If so, not to worry. borgmatic should install and function
-correctly even without the C YAML library. And borgmatic won't be any faster
-with the C library present, so you don't need to go out of your way to install
-it.
+Also, please check out the [borgmatic development
+how-to](docs/how-to/develop-on-borgmatic.md) for info on cloning source code,
+running tests, etc.
@@ -141,6 +141,7 @@ def create_archive(
        + (('--compression', compression) if compression else ())
        + (('--remote-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ())
        + (('--one-file-system',) if location_config.get('one_file_system') else ())
+        + (('--numeric-owner',) if location_config.get('numeric_owner') else ())
        + (('--read-special',) if location_config.get('read_special') else ())
        + (('--nobsdflags',) if location_config.get('bsd_flags') is False else ())
        + (('--files-cache', files_cache) if files_cache else ())
@@ -50,3 +50,39 @@ def extract_last_archive_dry_run(repository, lock_wait=None, local_path='borg',

    logger.debug(' '.join(full_extract_command))
    subprocess.check_call(full_extract_command)
+
+
+def extract_archive(
+    dry_run,
+    repository,
+    archive,
+    restore_paths,
+    location_config,
+    storage_config,
+    local_path='borg',
+    remote_path=None,
+    progress=False,
+):
+    '''
+    Given a dry-run flag, a local or remote repository path, an archive name, zero or more paths to
+    restore from the archive, and location/storage configuration dicts, extract the archive into the
+    current directory.
+    '''
+    umask = storage_config.get('umask', None)
+    lock_wait = storage_config.get('lock_wait', None)
+
+    full_command = (
+        (local_path, 'extract', '::'.join((repository, archive)))
+        + (tuple(restore_paths) if restore_paths else ())
+        + (('--remote-path', remote_path) if remote_path else ())
+        + (('--numeric-owner',) if location_config.get('numeric_owner') else ())
+        + (('--umask', str(umask)) if umask else ())
+        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
+        + (('--dry-run',) if dry_run else ())
+        + (('--progress',) if progress else ())
+    )
+
+    logger.debug(' '.join(full_command))
+    subprocess.check_call(full_command)
@@ -5,15 +5,17 @@ import subprocess
 logger = logging.getLogger(__name__)


-def list_archives(repository, storage_config, local_path='borg', remote_path=None, json=False):
+def list_archives(
+    repository, storage_config, archive=None, local_path='borg', remote_path=None, json=False
+):
    '''
-    Given a local or remote repository path, and a storage config dict,
-    list Borg archives in the repository.
+    Given a local or remote repository path and a storage config dict, list Borg archives in the
+    repository. Or, if an archive name is given, list the files in that archive.
    '''
    lock_wait = storage_config.get('lock_wait', None)

    full_command = (
-        (local_path, 'list', repository)
+        (local_path, 'list', '::'.join((repository, archive)) if archive else repository)
        + (('--remote-path', remote_path) if remote_path else ())
        + (('--lock-wait', str(lock_wait)) if lock_wait else ())
        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
@@ -1,4 +1,5 @@
 from argparse import ArgumentParser
+import collections
 import json
 import logging
 import os
@@ -12,6 +13,7 @@ from borgmatic.borg import (
    create as borg_create,
    environment as borg_environment,
    prune as borg_prune,
+    extract as borg_extract,
    list as borg_list,
    info as borg_info,
    init as borg_init,
@@ -38,11 +40,114 @@ def parse_arguments(*arguments):
    parser = ArgumentParser(
        description='''
            A simple wrapper script for the Borg backup software that creates and prunes backups.
-            If none of the --prune, --create, or --check options are given, then borgmatic defaults
-            to all three: prune, create, and check archives.
-            '''
+            If none of the action options are given, then borgmatic defaults to: prune, create, and
+            check archives.
+            ''',
+        add_help=False,
    )
-    parser.add_argument(
+
+    actions_group = parser.add_argument_group('actions')
+    actions_group.add_argument(
+        '-I', '--init', dest='init', action='store_true', help='Initialize an empty Borg repository'
+    )
+    actions_group.add_argument(
+        '-p',
+        '--prune',
+        dest='prune',
+        action='store_true',
+        help='Prune archives according to the retention policy',
+    )
+    actions_group.add_argument(
+        '-C',
+        '--create',
+        dest='create',
+        action='store_true',
+        help='Create archives (actually perform backups)',
+    )
+    actions_group.add_argument(
+        '-k', '--check', dest='check', action='store_true', help='Check archives for consistency'
+    )
+
+    actions_group.add_argument(
+        '-x',
+        '--extract',
+        dest='extract',
+        action='store_true',
+        help='Extract a named archive to the current directory',
+    )
+    actions_group.add_argument(
+        '-l', '--list', dest='list', action='store_true', help='List archives'
+    )
+    actions_group.add_argument(
+        '-i',
+        '--info',
+        dest='info',
+        action='store_true',
+        help='Display summary information on archives',
+    )
+
+    init_group = parser.add_argument_group('options for --init')
+    init_group.add_argument(
+        '-e', '--encryption', dest='encryption_mode', help='Borg repository encryption mode'
+    )
+    init_group.add_argument(
+        '--append-only',
+        dest='append_only',
+        action='store_true',
+        help='Create an append-only repository',
+    )
+    init_group.add_argument(
+        '--storage-quota',
+        dest='storage_quota',
+        help='Create a repository with a fixed storage quota',
+    )
+
+    prune_group = parser.add_argument_group('options for --prune')
+    stats_argument = prune_group.add_argument(
+        '--stats',
+        dest='stats',
+        default=False,
+        action='store_true',
+        help='Display statistics of archive',
+    )
+
+    create_group = parser.add_argument_group('options for --create')
+    progress_argument = create_group.add_argument(
+        '--progress',
+        dest='progress',
+        default=False,
+        action='store_true',
+        help='Display progress for each file as it is processed',
+    )
+    create_group._group_actions.append(stats_argument)
+    json_argument = create_group.add_argument(
+        '--json', dest='json', default=False, action='store_true', help='Output results as JSON'
+    )
+
+    extract_group = parser.add_argument_group('options for --extract')
+    repository_argument = extract_group.add_argument(
+        '--repository',
+        help='Path of repository to use, defaults to the configured repository if there is only one',
+    )
+    archive_argument = extract_group.add_argument('--archive', help='Name of archive to operate on')
+    extract_group.add_argument(
+        '--restore-path',
+        nargs='+',
+        dest='restore_paths',
+        help='Paths to restore from archive, defaults to the entire archive',
+    )
+    extract_group._group_actions.append(progress_argument)
+
+    list_group = parser.add_argument_group('options for --list')
+    list_group._group_actions.append(repository_argument)
+    list_group._group_actions.append(archive_argument)
+    list_group._group_actions.append(json_argument)
+
+    info_group = parser.add_argument_group('options for --info')
+    info_group._group_actions.append(json_argument)
+
+    common_group = parser.add_argument_group('common options')
+    common_group.add_argument(
        '-c',
        '--config',
        nargs='+',
@@ -52,85 +157,19 @@ def parse_arguments(*arguments):
            ' '.join(config_paths)
        ),
    )
-    parser.add_argument(
+    common_group.add_argument(
        '--excludes',
        dest='excludes_filename',
        help='Deprecated in favor of exclude_patterns within configuration',
    )
-    parser.add_argument(
-        '-I', '--init', dest='init', action='store_true', help='Initialize an empty Borg repository'
-    )
-    parser.add_argument(
-        '-e',
-        '--encryption',
-        dest='encryption_mode',
-        help='Borg repository encryption mode (for use with --init)',
-    )
-    parser.add_argument(
-        '--append-only',
-        dest='append_only',
-        action='store_true',
-        help='Create an append-only repository (for use with --init)',
-    )
-    parser.add_argument(
-        '--storage-quota',
-        dest='storage_quota',
-        help='Create a repository with a fixed storage quota (for use with --init)',
-    )
-    parser.add_argument(
-        '-p',
-        '--prune',
-        dest='prune',
-        action='store_true',
-        help='Prune archives according to the retention policy',
-    )
-    parser.add_argument(
-        '-C',
-        '--create',
-        dest='create',
-        action='store_true',
-        help='Create archives (actually perform backups)',
-    )
-    parser.add_argument(
-        '-k', '--check', dest='check', action='store_true', help='Check archives for consistency'
-    )
-    parser.add_argument('-l', '--list', dest='list', action='store_true', help='List archives')
-    parser.add_argument(
-        '-i',
-        '--info',
-        dest='info',
-        action='store_true',
-        help='Display summary information on archives',
-    )
-    parser.add_argument(
-        '--progress',
-        dest='progress',
-        default=False,
-        action='store_true',
-        help='Display progress with --create option for each file as it is backed up',
-    )
-    parser.add_argument(
-        '--stats',
-        dest='stats',
-        default=False,
-        action='store_true',
-        help='Display statistics of archive with --create or --prune option',
-    )
-    parser.add_argument(
-        '--json',
-        dest='json',
-        default=False,
-        action='store_true',
-        help='Output results from the --create, --list, or --info options as json',
-    )
-    parser.add_argument(
+    common_group.add_argument(
        '-n',
        '--dry-run',
        dest='dry_run',
        action='store_true',
        help='Go through the motions, but do not actually write to any repositories',
    )
-    parser.add_argument(
+    common_group.add_argument(
        '-v',
        '--verbosity',
        type=int,
@@ -138,13 +177,14 @@ def parse_arguments(*arguments):
        default=0,
        help='Display verbose progress (1 for some, 2 for lots)',
    )
-    parser.add_argument(
+    common_group.add_argument(
        '--version',
        dest='version',
        default=False,
        action='store_true',
        help='Display installed version number of borgmatic and exit',
    )
+    common_group.add_argument('--help', action='help', help='Show this help information and exit')

    args = parser.parse_args(arguments)

@@ -163,8 +203,25 @@ def parse_arguments(*arguments):
    if args.init and not args.encryption_mode:
        raise ValueError('The --encryption option is required with the --init option')

-    if args.progress and not args.create:
-        raise ValueError('The --progress option can only be used with the --create option')
+    if not args.extract:
+        if not args.list:
+            if args.repository:
+                raise ValueError(
+                    'The --repository option can only be used with the --extract and --list options'
+                )
+            if args.archive:
+                raise ValueError(
+                    'The --archive option can only be used with the --extract and --list options'
+                )
+        if args.restore_paths:
+            raise ValueError('The --restore-path option can only be used with the --extract option')
+    if args.extract and not args.archive:
+        raise ValueError('The --archive option is required with the --extract option')
+
+    if args.progress and not (args.create or args.extract):
+        raise ValueError(
+            'The --progress option can only be used with the --create and --extract options'
+        )

    if args.json and not (args.create or args.list or args.info):
        raise ValueError(
@@ -183,6 +240,7 @@ def parse_arguments(*arguments):
        and not args.prune
        and not args.create
        and not args.check
+        and not args.extract
        and not args.list
        and not args.info
    ):
@@ -196,13 +254,11 @@ def parse_arguments(*arguments):
    return args


-def run_configuration(config_filename, args):  # pragma: no cover
+def run_configuration(config_filename, config, args):  # pragma: no cover
    '''
-    Parse a single configuration file, and execute its defined pruning, backups, and/or consistency
-    checks.
+    Given a config filename and the corresponding parsed config dict, execute its defined pruning,
+    backups, consistency checks, and/or other actions.
    '''
-    logger.info('{}: Parsing configuration file'.format(config_filename))
-    config = validate.parse_configuration(config_filename, validate.schema_filename())
    (location, storage, retention, consistency, hooks) = (
        config.get(section_name, {})
        for section_name in ('location', 'storage', 'retention', 'consistency', 'hooks')
@@ -303,15 +359,35 @@ def _run_commands_on_repository(
        borg_check.check_archives(
            repository, storage, consistency, local_path=local_path, remote_path=remote_path
        )
+    if args.extract:
+        if args.repository is None or repository == args.repository:
+            logger.info('{}: Extracting archive {}'.format(repository, args.archive))
+            borg_extract.extract_archive(
+                args.dry_run,
+                repository,
+                args.archive,
+                args.restore_paths,
+                location,
+                storage,
+                local_path=local_path,
+                remote_path=remote_path,
+                progress=args.progress,
+            )
    if args.list:
-        logger.info('{}: Listing archives'.format(repository))
-        output = borg_list.list_archives(
-            repository, storage, local_path=local_path, remote_path=remote_path, json=args.json
-        )
-        if args.json:
-            json_results.append(json.loads(output))
-        else:
-            sys.stdout.write(output)
+        if args.repository is None or repository == args.repository:
+            logger.info('{}: Listing archives'.format(repository))
+            output = borg_list.list_archives(
+                repository,
+                storage,
+                args.archive,
+                local_path=local_path,
+                remote_path=remote_path,
+                json=args.json,
+            )
+            if args.json:
+                json_results.append(json.loads(output))
+            else:
+                sys.stdout.write(output)
    if args.info:
        logger.info('{}: Displaying summary info for archives'.format(repository))
        output = borg_info.display_archives_info(
@@ -329,9 +405,37 @@ def collect_configuration_run_summary_logs(config_filenames, args):
    argparse.ArgumentParser instance, run each configuration file and yield a series of
    logging.LogRecord instances containing summary information about each run.
    '''
+    # Dict mapping from config filename to corresponding parsed config dict.
+    configs = collections.OrderedDict()
+
+    # Parse and load each configuration file.
    for config_filename in config_filenames:
        try:
-            run_configuration(config_filename, args)
+            logger.info('{}: Parsing configuration file'.format(config_filename))
+            configs[config_filename] = validate.parse_configuration(
+                config_filename, validate.schema_filename()
+            )
+        except (ValueError, OSError, validate.Validation_error) as error:
+            yield logging.makeLogRecord(
+                dict(
+                    levelno=logging.CRITICAL,
+                    msg='{}: Error parsing configuration file'.format(config_filename),
+                )
+            )
+            yield logging.makeLogRecord(dict(levelno=logging.CRITICAL, msg=error))
+
+    # Run cross-file validation checks.
+    if args.extract or (args.list and args.archive):
+        try:
+            validate.guard_configuration_contains_repository(args.repository, configs)
+        except ValueError as error:
+            yield logging.makeLogRecord(dict(levelno=logging.CRITICAL, msg=error))
+            return
+
+    # Execute the actions corresponding to each configuration file.
+    for config_filename, config in configs.items():
+        try:
+            run_configuration(config_filename, config, args)
            yield logging.makeLogRecord(
                dict(
                    levelno=logging.INFO,
@@ -93,21 +93,3 @@ def guard_configuration_upgraded(source_config_filename, destination_config_file

    if os.path.exists(source_config_filename) and not destination_config_exists:
        raise LegacyConfigurationNotUpgraded()
-
-
-class LegacyExcludesFilenamePresent(FileNotFoundError):
-    def __init__(self):
-        super(LegacyExcludesFilenamePresent, self).__init__(
-            '''borgmatic changed its configuration file format in version 1.1.0 from INI-style
-to YAML. This better supports validation, and has a more natural way to express
-lists of values. The new configuration file incorporates excludes, so you no
-longer need to provide an excludes filename on the command-line with an
-"--excludes" argument.
-
-Please remove the "--excludes" argument and run borgmatic again.'''
-        )
-
-
-def guard_excludes_filename_omitted(excludes_filename):
-    if excludes_filename is not None:
-        raise LegacyExcludesFilenamePresent()
@@ -0,0 +1,62 @@
+import os
+import logging
+
+import ruamel.yaml
+
+
+logger = logging.getLogger(__name__)
+
+
+def load_configuration(filename):
+    '''
+    Load the given configuration file and return its contents as a data structure of nested dicts
+    and lists.
+
+    Raise ruamel.yaml.error.YAMLError if something goes wrong parsing the YAML, or RecursionError
+    if there are too many recursive includes.
+    '''
+    yaml = ruamel.yaml.YAML(typ='safe')
+    yaml.Constructor = Include_constructor
+
+    return yaml.load(open(filename))
+
+
+def include_configuration(loader, filename_node):
+    '''
+    Load the given YAML filename (ignoring the given loader so we can use our own), and return its
+    contents as a data structure of nested dicts and lists.
+    '''
+    return load_configuration(os.path.expanduser(filename_node.value))
+
+
+class Include_constructor(ruamel.yaml.SafeConstructor):
+    '''
+    A YAML "constructor" (a ruamel.yaml concept) that supports a custom "!include" tag for including
+    separate YAML configuration files. Example syntax: `retention: !include common.yaml`
+    '''
+
+    def __init__(self, preserve_quotes=None, loader=None):
+        super(Include_constructor, self).__init__(preserve_quotes, loader)
+        self.add_constructor('!include', include_configuration)
+
+    def flatten_mapping(self, node):
+        '''
+        Support the special case of shallow merging included configuration into an existing mapping
+        using the YAML '<<' merge key. Example syntax:
+
+        ```
+        retention:
+            keep_daily: 1
+            <<: !include common.yaml
+        ```
+        '''
+        representer = ruamel.yaml.representer.SafeRepresenter()
+
+        for index, (key_node, value_node) in enumerate(node.value):
+            if key_node.tag == u'tag:yaml.org,2002:merge' and value_node.tag == '!include':
+                included_value = representer.represent_mapping(
+                    tag='tag:yaml.org,2002:map', mapping=self.construct_object(value_node)
+                )
+                node.value[index] = (key_node, included_value)
+
+        super(Include_constructor, self).flatten_mapping(node)
@@ -32,6 +32,10 @@ map:
                type: bool
                desc: Stay in same file system (do not cross mount points). Defaults to false.
                example: true
+            numeric_owner:
+                type: bool
+                desc: Only store/extract numeric user and group identifiers. Defaults to false.
+                example: true
            read_special:
                type: bool
                desc: |
@@ -3,7 +3,9 @@ import logging
 import pkg_resources
 import pykwalify.core
 import pykwalify.errors
-from ruamel import yaml
+import ruamel.yaml
+
+from borgmatic.config import load


 logger = logging.getLogger(__name__)
@@ -87,9 +89,9 @@ def parse_configuration(config_filename, schema_filename):
    logging.getLogger('pykwalify').setLevel(logging.ERROR)

    try:
-        config = yaml.safe_load(open(config_filename))
-        schema = yaml.safe_load(open(schema_filename))
-    except yaml.error.YAMLError as error:
+        config = load.load_configuration(config_filename)
+        schema = load.load_configuration(schema_filename)
+    except (ruamel.yaml.error.YAMLError, RecursionError) as error:
        raise Validation_error(config_filename, (str(error),))

    # pykwalify gets angry if the example field is not a string. So rather than bend to its will,
@@ -107,3 +109,46 @@ def parse_configuration(config_filename, schema_filename):
    apply_logical_validation(config_filename, parsed_result)

    return parsed_result
+
+
+def guard_configuration_contains_repository(repository, configurations):
+    '''
+    Given a repository path and a dict mapping from config filename to corresponding parsed config
+    dict, ensure that the repository is declared exactly once in all of the configurations.
+
+    If no repository is given, then error if there are multiple configured repositories.
+
+    Raise ValueError if the repository is not found in a configuration, or is declared multiple
+    times.
+    '''
+    if not repository:
+        count = len(
+            tuple(
+                config_repository
+                for config in configurations.values()
+                for config_repository in config['location']['repositories']
+            )
+        )
+
+        if count > 1:
+            raise ValueError(
+                'Can\'t determine which repository to use. Use --repository option to disambiguate'.format(
+                    repository
+                )
+            )
+
+        return
+
+    count = len(
+        tuple(
+            config_repository
+            for config in configurations.values()
+            for config_repository in config['location']['repositories']
+            if repository == config_repository
+        )
+    )
+
+    if count == 0:
+        raise ValueError('Repository {} not found in configuration files'.format(repository))
+    if count > 1:
+        raise ValueError('Repository {} found in multiple configuration files'.format(repository))
@@ -0,0 +1,96 @@
+---
+title: How to deal with very large backups
+---
+## Biggish data
+
+Borg itself is great for efficiently de-duplicating data across successive
+backup archives, even when dealing with very large repositories. But you may
+find that while borgmatic's default mode of "prune, create, and check" works
+well on small repositories, it's not so great on larger ones. That's because
+running the default consistency checks takes a long time on large
+repositories.
+
+### A la carte actions
+
+If you find yourself in this situation, you have some options. First, you can
+run borgmatic's pruning, creating, or checking actions separately. For
+instance, the the following optional flags are available:
+
+```bash
+borgmatic --prune
+borgmatic --create
+borgmatic --check
+```
+
+You can run with only one of these flags provided, or you can mix and match
+any number of them in a single borgmatic run. This supports approaches like
+making backups with `--create` on a frequent schedule, while only running
+expensive consistency checks with `--check` on a much less frequent basis from
+a separate cron job.
+
+### Consistency check configuration
+
+Another option is to customize your consistency checks. The default
+consistency checks run both full-repository checks and per-archive checks
+within each repository.
+
+But if you find that archive checks are too slow, for example, you can
+configure borgmatic to run repository checks only. Configure this in the
+`consistency` section of borgmatic configuration:
+
+```yaml
+consistency:
+    checks:
+        - repository
+```
+
+If that's still too slow, you can disable consistency checks entirely,
+either for a single repository or for all repositories.
+
+Disabling all consistency checks looks like this:
+
+```yaml
+consistency:
+    checks:
+        - disabled
+```
+
+Or, if you have multiple repositories in your borgmatic configuration file,
+you can keep running consistency checks, but only against a subset of the
+repositories:
+
+```yaml
+consistency:
+    check_repositories:
+        - path/of/repository_to_check.borg
+```
+
+
+## Troubleshooting
+
+### Broken pipe with remote repository
+
+When running borgmatic on a large remote repository, you may receive errors
+like the following, particularly while "borg check" is validating backups for
+consistency:
+
+```text
+    Write failed: Broken pipe
+    borg: Error: Connection closed by remote host
+```
+
+This error can be caused by an ssh timeout, which you can rectify by adding
+the following to the `~/.ssh/config` file on the client:
+
+```text
+    Host *
+        ServerAliveInterval 120
+```
+
+This should make the client keep the connection alive while validating
+backups.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
@@ -0,0 +1,97 @@
+---
+title: How to develop on borgmatic
+---
+## Source code
+
+To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
+
+```bash
+git clone https://projects.torsion.org/witten/borgmatic.git
+```
+
+Or:
+
+```bash
+git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
+```
+
+Then, install borgmatic
+"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
+so that you can run borgmatic commands while you're hacking on them to
+make sure your changes work.
+
+```bash
+cd borgmatic/
+pip3 install --editable --user .
+```
+
+Note that this will typically install the borgmatic commands into
+`~/.local/bin`, which may or may not be on your PATH. There are other ways to
+install borgmatic editable as well, for instance into the system Python
+install (so without `--user`, as root), or even into a
+[virtualenv](https://virtualenv.pypa.io/en/stable/). How or where you install
+borgmatic is up to you, but generally an editable install makes development
+and testing easier.
+
+
+## Automated tests
+
+Assuming you've cloned the borgmatic source code as described above, and
+you're in the `borgmatic/` working copy, install tox, which is used for
+setting up testing environments:
+
+```bash
+sudo pip3 install tox
+```
+
+Finally, to actually run tests, run:
+
+```bash
+cd borgmatic
+tox
+```
+
+### Code formatting
+
+If when running tests, you get an error from the
+[Black](https://black.readthedocs.io/en/stable/) code formatter about files
+that would be reformatted, you can ask Black to format them for you via the
+following:
+
+```bash
+tox -e black
+```
+
+### End-to-end tests
+
+borgmatic additionally includes some end-to-end tests that integration test
+with Borg for a few representative scenarios. These tests don't run by default
+because they're relatively slow and depend on Borg. If you would like to run
+them:
+
+```bash
+tox -e end-to-end
+```
+
+## Code style
+
+Start with [PEP 8](https://www.python.org/dev/peps/pep-0008/). But then, apply
+the following deviations from it:
+
+ * For strings, prefer single quotes over double quotes.
+ * Limit all lines to a maximum of 100 characters.
+ * Use trailing commas within multiline values or argument lists.
+ * For multiline constructs, put opening and closing delimeters on lines
+   separate from their contents.
+ * Within multiline constructs, use standard four-space indentation. Don't align
+   indentation with an opening delimeter.
+
+borgmatic code uses the [Black](https://black.readthedocs.io/en/stable/) code
+formatter and [Flake8](http://flake8.pycqa.org/en/latest/) code checker, so
+certain code style requirements will be enforced when running automated tests.
+See the Black and Flake8 documentation for more information.
+
+
+## Related documentation
+
+ * [Inspect your backups](../../docs/how-to/inspect-your-backups.md)
@@ -0,0 +1,46 @@
+---
+title: How to inspect your backups
+---
+## Backup progress
+
+By default, borgmatic runs proceed silently except in the case of errors. But
+if you'd like to to get additional information about the progress of the
+backup as it proceeds, use the verbosity option:
+
+```bash
+borgmatic --verbosity 1
+```
+
+This lists the files that borgmatic is archiving, which are those that are new
+or changed since the last backup.
+
+Or, for even more progress and debug spew:
+
+```bash
+borgmatic --verbosity 2
+```
+
+## Existing backups
+
+Borgmatic provides convenient flags for Borg's
+[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
+[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
+functionality:
+
+
+```bash
+borgmatic --list
+borgmatic --info
+```
+
+## Scripting borgmatic
+
+To consume the output of borgmatic in other software, you can include an
+optional `--json` flag with `--create`, `--list`, or `--info` to get the
+output formatted as JSON.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
+ * [Develop on borgmatic](../../docs/how-to/develop-on-borgmatic.md)
@@ -0,0 +1,111 @@
+---
+title: How to make per-application backups
+---
+## Multiple backup configurations
+
+You may find yourself wanting to create different backup policies for
+different applications on your system. For instance, you may want one backup
+configuration for your database data directory, and a different configuration
+for your user home directories.
+
+The way to accomplish that is pretty simple: Create multiple separate
+configuration files and place each one in a `/etc/borgmatic.d/` directory. For
+instance:
+
+```bash
+sudo mkdir /etc/borgmatic.d
+sudo generate-borgmatic-config --destination /etc/borgmatic.d/app1.yaml
+sudo generate-borgmatic-config --destination /etc/borgmatic.d/app2.yaml
+```
+
+When you set up multiple configuration files like this, borgmatic will run
+each one in turn from a single borgmatic invocation. This includes, by
+default, the traditional `/etc/borgmatic/config.yaml` as well.
+
+And if you need even more customizability, you can specify alternate
+configuration paths on the command-line with borgmatic's `--config` option.
+See `borgmatic --help` for more information.
+
+
+## Configuration includes
+
+Once you have multiple different configuration files, you might want to share
+common configuration options across these files with having to copy and paste
+them. To achieve this, you can put fragments of common configuration options
+into a file, and then include or inline that file into one or more borgmatic
+configuration files.
+
+Let's say that you want to include common retention configuration across all
+of your configuration files. You could do that in each configuration file with
+the following:
+
+```yaml
+location:
+   ...
+
+retention:
+    !include /etc/borgmatic/common_retention.yaml
+```
+
+And then the contents of `common_retention.yaml` could be:
+
+```yaml
+keep_hourly: 24
+keep_daily: 7
+```
+
+To prevent borgmatic from trying to load these configuration fragments by
+themselves and complaining that they are not valid configuration files, you
+should put them in a directory other than `/etc/borgmatic.d/`. (A subdirectory
+is fine.)
+
+Note that this form of include must be a YAML value rather than a key. For
+example, this will not work:
+
+```yaml
+location:
+   ...
+
+# Don't do this. It won't work!
+!include /etc/borgmatic/common_retention.yaml
+```
+
+But if you do want to merge in a YAML key and its values, keep reading!
+
+
+## Include merging
+
+If you need to get even fancier and pull in common configuration options while
+potentially overriding individual options, you can perform a YAML merge of
+included configuration using the YAML `<<` key. For instance, here's an
+example of a main configuration file that pulls in two retention options via
+an include, and then overrides one of them locally:
+
+```yaml
+location:
+   ...
+
+retention:
+    keep_daily: 5
+    <<: !include /etc/borgmatic/common_retention.yaml
+```
+
+This is what `common_retention.yaml` might look like:
+
+```yaml
+keep_hourly: 24
+keep_daily: 7
+```
+
+Once this include gets merged in, the resulting configuration would have a
+`keep_hourly` value of `24` and an overridden `keep_daily` value of `5`.
+
+When there is a collision of an option between the local file and the merged
+include, the local file's option takes precedent. And note that this is a
+shallow merge rather than a deep merge, so the merging does not descend into
+nested values.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
@@ -0,0 +1,61 @@
+---
+title: How to restore a backup
+---
+## Extract
+
+When the worst happens—or you want to test your backups—the first step is
+to figure out which archive to restore. A good way to do that is to use the
+`--list` action:
+
+```bash
+borgmatic --list
+```
+
+That should yield output looking something like:
+
+```text
+host-2019-01-01T04:05:06.070809      Tue, 2019-01-01 04:05:06 [...]
+host-2019-01-02T04:06:07.080910      Wed, 2019-01-02 04:06:07 [...]
+```
+
+Assuming that you want to restore the archive with the most up-to-date files
+and therefore the latest timestamp, run a command like:
+
+```bash
+borgmatic --extract --archive host-2019-01-02T04:06:07.080910
+```
+
+The `--archive` value is the name of the archive to restore. This extracts the
+entire contents of the archive to the current directory, so make sure you're
+in the right place before running the command.
+
+
+## Repository selection
+
+If you have a single repository in your borgmatic configuration file(s), no
+problem: the `--extract` action figures out which repository to use.
+
+But if you have multiple repositories configured, then you'll need to specify
+the repository path containing the archive to extract. Here's an example:
+
+```bash
+borgmatic --extract --repository repo.borg --archive host-2019-...
+```
+
+## Restore particular files
+
+Sometimes, you want to restore a single deleted file, rather than restoring
+everything from an archive. To do that, tack on one or more `--restore-path`
+values. For instance:
+
+```bash
+borgmatic --extract --archive host-2019-... --restore-path /path/1 /path/2
+```
+
+Like a whole-archive restore, this also restores into the current directory.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
+ * [Inspect your backups](../../docs/how-to/inspect-your-backups.md)
@@ -0,0 +1,51 @@
+---
+title: How to run preparation steps before backups
+---
+## Preparation and cleanup hooks
+
+If you find yourself performing prepraration tasks before your backup runs, or
+cleanup work afterwards, borgmatic hooks may be of interest. Hooks are
+shell commands that borgmatic executes for you at various points, and they're
+configured in the `hooks` section of your configuration file.
+
+For instance, you can specify `before_backup` hooks to dump a database to file
+before backing it up, and specify `after_backup` hooks to delete the temporary
+file afterwards. Here's an example:
+
+```yaml
+hooks:
+    before_backup:
+        - dump-a-database /to/file.sql
+    after_backup:
+        - rm /to/file.sql
+```
+
+borgmatic hooks run once per configuration file. `before_backup` hooks run
+prior to backups of all repositories. `after_backup` hooks run afterwards, but
+not if an error occurs in a previous hook or in the backups themselves.
+
+
+## Error hooks
+
+borgmatic also runs `on_error` hooks if an error occurs. Here's an example
+configuration:
+
+```yaml
+hooks:
+    on_error:
+        - echo "Error while creating a backup."
+```
+
+
+## Security
+
+An important security note about hooks: borgmatic executes all hook commands
+with the user permissions of borgmatic itself. So to prevent potential shell
+injection or privilege escalation, do not forget to set secure permissions
+(`chmod 0700`) on borgmatic configuration files and scripts invoked by hooks.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
+ * [Make per-application backups](../../docs/how-to/make-per-application-backups.md)
@@ -0,0 +1,188 @@
+---
+title: How to set up backups with borgmatic
+---
+## Installation
+
+To get up and running, first [install
+Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
+least version 1.1.
+
+Then, run the following command to download and install borgmatic:
+
+```bash
+sudo pip3 install --upgrade borgmatic
+```
+
+Note that your pip binary may have a different name than "pip3". Make sure
+you're using Python 3, as borgmatic does not support Python 2.
+
+
+### Other ways to install
+
+Along with the above process, you have several other options for installing
+borgmatic:
+
+ * [Docker image](https://hub.docker.com/r/monachus/borgmatic/)
+ * [Another Docker image](https://hub.docker.com/r/b3vis/borgmatic/)
+ * [Debian](https://tracker.debian.org/pkg/borgmatic)
+ * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
+ * [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
+ * [Arch Linux](https://aur.archlinux.org/packages/borgmatic/)
+ * [OpenBSD](http://ports.su/sysutils/borgmatic)
+ * [openSUSE](https://software.opensuse.org/package/borgmatic)
+
+
+## Configuration
+
+After you install borgmatic, generate a sample configuration file:
+
+```bash
+sudo generate-borgmatic-config
+```
+
+If that command is not found, then it may be installed in a location that's
+not in your system `PATH`. Try looking in `/usr/local/bin/`.
+
+This generates a sample configuration file at /etc/borgmatic/config.yaml (by
+default). You should edit the file to suit your needs, as the values are
+representative. All options are optional except where indicated, so feel free
+to ignore anything you don't need.
+
+Note that the configuration file is organized into distinct sections, each
+with a section name like `location:` or `storage:`. So take care that if you
+uncomment a particular option, also uncomment its containing section name, or
+else borgmatic won't recognize the option.
+
+You can also get the same sample configuration file from the [configuration
+reference](../../docs/reference/configuration.md), the authoritative set of
+all configuration options. This is handy if borgmatic has added new options
+since you originally created your configuration file.
+
+
+### Encryption
+
+Note that if you plan to run borgmatic on a schedule with cron, and you
+encrypt your Borg repository with a passphrase instead of a key file, you'll
+either need to set the borgmatic `encryption_passphrase` configuration
+variable or set the `BORG_PASSPHRASE` environment variable. See the
+[repository encryption
+section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
+of the Borg Quick Start for more info.
+
+Alternatively, you can specify the passphrase programatically by setting
+either the borgmatic `encryption_passcommand` configuration variable or the
+`BORG_PASSCOMMAND` environment variable. See the [Borg Security
+FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
+for more info.
+
+
+###
+
+
+## Initialization
+
+Before you can create backups with borgmatic, you first need to initialize a
+Borg repository so you have a destination for your backup archives. (But skip
+this step if you already have a Borg repository.) To create a repository, run
+a command like the following:
+
+```bash
+borgmatic --init --encryption repokey
+```
+
+This uses the borgmatic configuration file you created above to determine
+which local or remote repository to create, and encrypts it with the
+encryption passphrase specified there if one is provided. Read about [Borg
+encryption
+modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
+for the menu of available encryption modes.
+
+Also, optionally check out the [Borg Quick
+Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
+background about repository initialization.
+
+Note that borgmatic skips repository initialization if the repository already
+exists. This supports use cases like ensuring a repository exists prior to
+performing a backup.
+
+If the repository is on a remote host, make sure that your local user has
+key-based SSH access to the desired user account on the remote host.
+
+
+## Backups
+
+Now that you've configured borgmatic and initialized a repository, it's a
+good idea to test that borgmatic is working. So to run borgmatic and start a
+backup, you can invoke it like this:
+
+```bash
+borgmatic --verbosity 1
+```
+
+By default, this will also prune any old backups as per the configured
+retention policy, and check backups for consistency problems due to things
+like file damage.
+
+The verbosity flag makes borgmatic list the files that it's archiving, which
+are those that are new or changed since the last backup. Eyeball the list and
+see if it matches your expectations based on the configuration.
+
+
+## Autopilot
+
+Running backups manually is good for validating your configuration, but I'm
+guessing that you want to run borgmatic automatically, say once a day. To do
+that, you can configure a separate job runner to invoke it periodically.
+
+### cron
+
+If you're using cron, download the [sample cron
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
+Then, from the directory where you downloaded it:
+
+```bash
+sudo mv borgmatic /etc/cron.d/borgmatic
+sudo chmod +x /etc/cron.d/borgmatic
+```
+
+You can modify the cron file if you'd like to run borgmatic more or less frequently.
+
+### systemd
+
+If you're using systemd instead of cron to run jobs, download the [sample
+systemd service
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
+and the [sample systemd timer
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
+Then, from the directory where you downloaded them:
+
+```bash
+sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
+sudo systemctl enable borgmatic.timer
+sudo systemctl start borgmatic.timer
+```
+
+Feel free to modify the timer file based on how frequently you'd like
+borgmatic to run.
+
+
+## Troubleshooting
+
+### libyaml compilation errors
+
+borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
+use a C YAML library (libyaml) if present. But if it's not installed, then
+when installing or upgrading borgmatic, you may see errors about compiling the
+YAML library. If so, not to worry. borgmatic should install and function
+correctly even without the C YAML library. And borgmatic won't be any faster
+with the C library present, so you don't need to go out of your way to install
+it.
+
+
+## Related documentation
+
+ * [Make per-application backups](../../docs/how-to/make-per-application-backups.md)
+ * [Deal with very large backups](../../docs/how-to/deal-with-very-large-backups.md)
+ * [Inspect your backups](../../docs/how-to/inspect-your-backups.md)
+ * [borgmatic configuration reference](../../docs/reference/configuration.md)
+ * [borgmatic command-line reference](../../docs/reference/command-line.md)
@@ -0,0 +1,79 @@
+---
+title: How to upgrade borgmatic
+---
+## Upgrading
+
+In general, all you should need to do to upgrade borgmatic is run the
+following:
+
+```bash
+sudo pip3 install --upgrade borgmatic
+```
+
+See below about special cases.
+
+
+### Upgrading from borgmatic 1.0.x
+
+borgmatic changed its configuration file format in version 1.1.0 from
+INI-style to YAML. This better supports validation, and has a more natural way
+to express lists of values. To upgrade your existing configuration, first
+upgrade to the new version of borgmatic.
+
+As of version 1.1.0, borgmatic no longer supports Python 2. If you were
+already running borgmatic with Python 3, then you can upgrade borgmatic
+in-place:
+
+```bash
+sudo pip3 install --upgrade borgmatic
+```
+
+But if you were running borgmatic with Python 2, uninstall and reinstall instead:
+
+```bash
+sudo pip uninstall borgmatic
+sudo pip3 install borgmatic
+```
+
+The pip binary names for different versions of Python can differ, so the above
+commands may need some tweaking to work on your machine.
+
+
+Once borgmatic is upgraded, run:
+
+```bash
+sudo upgrade-borgmatic-config
+```
+
+That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
+(by default) using the values from both your existing configuration and
+excludes files. The new version of borgmatic will consume the YAML
+configuration file instead of the old one.
+
+
+### Upgrading from atticmatic
+
+You can ignore this section if you're not an atticmatic user (the former name
+of borgmatic).
+
+borgmatic only supports Borg now and no longer supports Attic. So if you're
+an Attic user, consider switching to Borg. See the [Borg upgrade
+command](https://borgbackup.readthedocs.io/en/stable/usage.html#borg-upgrade)
+for more information. Then, follow the instructions above about setting up
+your borgmatic configuration files.
+
+If you were already using Borg with atticmatic, then you can upgrade
+from atticmatic to borgmatic by running the following commands:
+
+```bash
+sudo pip3 uninstall atticmatic
+sudo pip3 install borgmatic
+```
+
+That's it! borgmatic will continue using your /etc/borgmatic configuration
+files.
+
+
+## Related documentation
+
+ * [Develop on borgmatic](../../docs/how-to/develop-on-borgmatic.md)
@@ -0,0 +1,16 @@
+---
+title: borgmatic command-line reference
+---
+## borgmatic options
+
+Here are all of the available borgmatic command-line options:
+
+```
+{% include borgmatic/command-line.txt %}
+```
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
+ * [borgmatic configuration reference](../../docs/reference/configuration.md)
@@ -0,0 +1,19 @@
+---
+title: borgmatic configuration reference
+---
+## Configuration file
+
+Here is a full sample borgmatic configuration file including all available options: 
+
+```yaml
+{% include borgmatic/config.yaml %}
+```
+
+Note that you can also [download this configuration
+file](../../docs/reference/config.yaml) for use locally.
+
+
+## Related documentation
+
+ * [Set up backups with borgmatic](../../docs/how-to/set-up-backups.md)
+ * [borgmatic command-line reference](../../docs/reference/command-line.md)
@@ -2,5 +2,7 @@

 set -e

-git push -u origin master
-git push -u github master
+branch_name=$(git rev-parse --abbrev-ref HEAD)
+
+git push -u github "$branch_name"
+git push -u origin "$branch_name"
@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages


-VERSION = '1.2.15'
+VERSION = '1.3.0'


 setup(
@@ -20,7 +20,7 @@ setup(
        'Topic :: Security :: Cryptography',
        'Topic :: System :: Archiving :: Backup',
    ],
-    packages=find_packages(),
+    packages=find_packages(exclude=['tests*']),
    entry_points={
        'console_scripts': [
            'borgmatic = borgmatic.commands.borgmatic:main',
@@ -33,6 +33,11 @@ def test_borgmatic_command():
    # Create a Borg repository.
    temporary_directory = tempfile.mkdtemp()
    repository_path = os.path.join(temporary_directory, 'test.borg')
+    extract_path = os.path.join(temporary_directory, 'extract')
+
+    original_working_directory = os.getcwd()
+    os.mkdir(extract_path)
+    os.chdir(extract_path)

    try:
        config_path = os.path.join(temporary_directory, 'test.yaml')
@@ -51,8 +56,19 @@ def test_borgmatic_command():

        assert len(parsed_output) == 1
        assert len(parsed_output[0]['archives']) == 1
+        archive_name = parsed_output[0]['archives'][0]['archive']

-        # Also exercise the info flag.
+        # Extract the created archive into the current (temporary) directory, and confirm that the
+        # extracted file looks right.
+        output = subprocess.check_output(
+            'borgmatic --config {} --extract --archive {}'.format(config_path, archive_name).split(
+                ' '
+            )
+        ).decode(sys.stdout.encoding)
+        extracted_config_path = os.path.join(extract_path, config_path)
+        assert open(extracted_config_path).read() == open(config_path).read()
+
+        # Exercise the info flag.
        output = subprocess.check_output(
            'borgmatic --config {} --info --json'.format(config_path).split(' ')
        ).decode(sys.stdout.encoding)
@@ -61,4 +77,5 @@ def test_borgmatic_command():
        assert len(parsed_output) == 1
        assert 'repository' in parsed_output[0]
    finally:
+        os.chdir(original_working_directory)
        shutil.rmtree(temporary_directory)
@@ -94,6 +94,12 @@ def test_parse_arguments_disallows_encryption_mode_without_init():
        module.parse_arguments('--config', 'myconfig', '--encryption', 'repokey')


+def test_parse_arguments_allows_encryption_mode_with_init():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--init', '--encryption', 'repokey')
+
+
 def test_parse_arguments_requires_encryption_mode_with_init():
    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])

@@ -136,48 +142,128 @@ def test_parse_arguments_disallows_init_and_dry_run():
        )


+def test_parse_arguments_disallows_repository_without_extract_or_list():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--repository', 'test.borg')
+
+
+def test_parse_arguments_allows_repository_with_extract():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments(
+        '--config', 'myconfig', '--extract', '--repository', 'test.borg', '--archive', 'test'
+    )
+
+
+def test_parse_arguments_allows_repository_with_list():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--list', '--repository', 'test.borg')
+
+
+def test_parse_arguments_disallows_archive_without_extract_or_list():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--archive', 'test')
+
+
+def test_parse_arguments_disallows_restore_paths_without_extract():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--restore-path', 'test')
+
+
+def test_parse_arguments_allows_archive_with_extract():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--extract', '--archive', 'test')
+
+
+def test_parse_arguments_allows_archive_with_list():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--list', '--archive', 'test')
+
+
+def test_parse_arguments_requires_archive_with_extract():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--extract')
+
+
 def test_parse_arguments_allows_progress_and_create():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    module.parse_arguments('--progress', '--create', '--list')


+def test_parse_arguments_allows_progress_and_extract():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--progress', '--extract', '--archive', 'test', '--list')
+
+
 def test_parse_arguments_disallows_progress_without_create():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    with pytest.raises(ValueError):
        module.parse_arguments('--progress', '--list')


 def test_parse_arguments_with_stats_and_create_flags_does_not_raise():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    module.parse_arguments('--stats', '--create', '--list')


 def test_parse_arguments_with_stats_and_prune_flags_does_not_raise():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    module.parse_arguments('--stats', '--prune', '--list')


 def test_parse_arguments_with_stats_flag_but_no_create_or_prune_flag_raises_value_error():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    with pytest.raises(ValueError):
        module.parse_arguments('--stats', '--list')


 def test_parse_arguments_with_just_stats_flag_does_not_raise():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    module.parse_arguments('--stats')


 def test_parse_arguments_allows_json_with_list_or_info():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    module.parse_arguments('--list', '--json')
    module.parse_arguments('--info', '--json')


 def test_parse_arguments_disallows_json_without_list_or_info():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    with pytest.raises(ValueError):
        module.parse_arguments('--json')


 def test_parse_arguments_disallows_json_with_both_list_and_info():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    with pytest.raises(ValueError):
        module.parse_arguments('--list', '--info', '--json')


 def test_borgmatic_version_matches_news_version():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
    borgmatic_version = subprocess.check_output(('borgmatic', '--version')).decode('ascii')
    news_version = open('NEWS').readline()

@@ -0,0 +1,40 @@
+import sys
+
+from flexmock import flexmock
+
+from borgmatic.config import load as module
+
+
+def test_load_configuration_parses_contents():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('config.yaml').and_return('key: value')
+
+    assert module.load_configuration('config.yaml') == {'key': 'value'}
+
+
+def test_load_configuration_inlines_include():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('include.yaml').and_return('value')
+    builtins.should_receive('open').with_args('config.yaml').and_return(
+        'key: !include include.yaml'
+    )
+
+    assert module.load_configuration('config.yaml') == {'key': 'value'}
+
+
+def test_load_configuration_merges_include():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('include.yaml').and_return(
+        '''
+        foo: bar
+        baz: quux
+        '''
+    )
+    builtins.should_receive('open').with_args('config.yaml').and_return(
+        '''
+        foo: override
+        <<: !include include.yaml
+        '''
+    )
+
+    assert module.load_configuration('config.yaml') == {'foo': 'override', 'baz': 'quux'}
@@ -118,6 +118,67 @@ def test_parse_configuration_with_schema_lacking_examples_does_not_raise():
    module.parse_configuration('config.yaml', 'schema.yaml')


+def test_parse_configuration_inlines_include():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+
+            repositories:
+                - hostname.borg
+
+        retention:
+            !include include.yaml
+        '''
+    )
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('include.yaml').and_return(
+        '''
+        keep_daily: 7
+        keep_hourly: 24
+        '''
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {'source_directories': ['/home'], 'repositories': ['hostname.borg']},
+        'retention': {'keep_daily': 7, 'keep_hourly': 24},
+    }
+
+
+def test_parse_configuration_merges_include():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+
+            repositories:
+                - hostname.borg
+
+        retention:
+            keep_daily: 1
+            <<: !include include.yaml
+        '''
+    )
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('include.yaml').and_return(
+        '''
+        keep_daily: 7
+        keep_hourly: 24
+        '''
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {'source_directories': ['/home'], 'repositories': ['hostname.borg']},
+        'retention': {'keep_daily': 1, 'keep_hourly': 24},
+    }
+
+
 def test_parse_configuration_raises_for_missing_config_file():
    with pytest.raises(FileNotFoundError):
        module.parse_configuration('config.yaml', 'schema.yaml')
@@ -425,7 +425,7 @@ def test_create_archive_with_remote_rate_limit_calls_borg_with_remote_ratelimit_
    )


-def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_parameters():
+def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_parameter():
    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar'))
    flexmock(module).should_receive('_expand_home_directories').and_return(())
    flexmock(module).should_receive('_write_pattern_file').and_return(None)
@@ -446,6 +446,27 @@ def test_create_archive_with_one_file_system_calls_borg_with_one_file_system_par
    )


+def test_create_archive_with_numeric_owner_calls_borg_with_numeric_owner_parameter():
+    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar'))
+    flexmock(module).should_receive('_expand_home_directories').and_return(())
+    flexmock(module).should_receive('_write_pattern_file').and_return(None)
+    flexmock(module).should_receive('_make_pattern_flags').and_return(())
+    flexmock(module).should_receive('_make_exclude_flags').and_return(())
+    insert_subprocess_mock(CREATE_COMMAND + ('--numeric-owner',))
+
+    module.create_archive(
+        dry_run=False,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'numeric_owner': True,
+            'exclude_patterns': None,
+        },
+        storage_config={},
+    )
+
+
 def test_create_archive_with_read_special_calls_borg_with_read_special_parameter():
    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar'))
    flexmock(module).should_receive('_expand_home_directories').and_return(())
@@ -24,7 +24,7 @@ def insert_subprocess_check_output_mock(check_output_command, result, **kwargs):
    ).once()


-def test_extract_last_archive_dry_run_should_call_borg_with_last_archive():
+def test_extract_last_archive_dry_run_calls_borg_with_last_archive():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg', 'list', '--short', 'repo'), result='archive1\narchive2\n'.encode('utf-8')
@@ -44,7 +44,7 @@ def test_extract_last_archive_dry_run_without_any_archives_should_bail():
    module.extract_last_archive_dry_run(repository='repo', lock_wait=None)


-def test_extract_last_archive_dry_run_with_log_info_should_call_borg_with_info_parameter():
+def test_extract_last_archive_dry_run_with_log_info_calls_borg_with_info_parameter():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg', 'list', '--short', 'repo', '--info'), result='archive1\narchive2\n'.encode('utf-8')
@@ -55,7 +55,7 @@ def test_extract_last_archive_dry_run_with_log_info_should_call_borg_with_info_p
    module.extract_last_archive_dry_run(repository='repo', lock_wait=None)


-def test_extract_last_archive_dry_run_with_log_debug_should_call_borg_with_debug_parameter():
+def test_extract_last_archive_dry_run_with_log_debug_calls_borg_with_debug_parameter():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg', 'list', '--short', 'repo', '--debug', '--show-rc'),
@@ -69,7 +69,7 @@ def test_extract_last_archive_dry_run_with_log_debug_should_call_borg_with_debug
    module.extract_last_archive_dry_run(repository='repo', lock_wait=None)


-def test_extract_last_archive_dry_run_should_call_borg_via_local_path():
+def test_extract_last_archive_dry_run_calls_borg_via_local_path():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg1', 'list', '--short', 'repo'), result='archive1\narchive2\n'.encode('utf-8')
@@ -79,7 +79,7 @@ def test_extract_last_archive_dry_run_should_call_borg_via_local_path():
    module.extract_last_archive_dry_run(repository='repo', lock_wait=None, local_path='borg1')


-def test_extract_last_archive_dry_run_should_call_borg_with_remote_path_parameters():
+def test_extract_last_archive_dry_run_calls_borg_with_remote_path_parameters():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg', 'list', '--short', 'repo', '--remote-path', 'borg1'),
@@ -92,7 +92,7 @@ def test_extract_last_archive_dry_run_should_call_borg_with_remote_path_paramete
    module.extract_last_archive_dry_run(repository='repo', lock_wait=None, remote_path='borg1')


-def test_extract_last_archive_dry_run_should_call_borg_with_lock_wait_parameters():
+def test_extract_last_archive_dry_run_calls_borg_with_lock_wait_parameters():
    flexmock(sys.stdout).encoding = 'utf-8'
    insert_subprocess_check_output_mock(
        ('borg', 'list', '--short', 'repo', '--lock-wait', '5'),
@@ -101,3 +101,124 @@ def test_extract_last_archive_dry_run_should_call_borg_with_lock_wait_parameters
    insert_subprocess_mock(('borg', 'extract', '--dry-run', 'repo::archive2', '--lock-wait', '5'))

    module.extract_last_archive_dry_run(repository='repo', lock_wait=5)
+
+
+def test_extract_archive_calls_borg_with_restore_path_parameters():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', 'path1', 'path2'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=['path1', 'path2'],
+        location_config={},
+        storage_config={},
+    )
+
+
+def test_extract_archive_calls_borg_with_remote_path_parameters():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--remote-path', 'borg1'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={},
+        remote_path='borg1',
+    )
+
+
+def test_extract_archive_calls_borg_with_numeric_owner_parameter():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--numeric-owner'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={'numeric_owner': True},
+        storage_config={},
+    )
+
+
+def test_extract_archive_calls_borg_with_umask_parameters():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--umask', '0770'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={'umask': '0770'},
+    )
+
+
+def test_extract_archive_calls_borg_with_lock_wait_parameters():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--lock-wait', '5'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={'lock_wait': '5'},
+    )
+
+
+def test_extract_archive_with_log_info_calls_borg_with_info_parameter():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--info'))
+    insert_logging_mock(logging.INFO)
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={},
+    )
+
+
+def test_extract_archive_with_log_debug_calls_borg_with_debug_parameters():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--debug', '--list', '--show-rc'))
+    insert_logging_mock(logging.DEBUG)
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={},
+    )
+
+
+def test_extract_archive_calls_borg_with_dry_run_parameter():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--dry-run'))
+
+    module.extract_archive(
+        dry_run=True,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={},
+    )
+
+
+def test_extract_archive_calls_borg_with_progress_parameter():
+    insert_subprocess_mock(('borg', 'extract', 'repo::archive', '--progress'))
+
+    module.extract_archive(
+        dry_run=False,
+        repository='repo',
+        archive='archive',
+        restore_paths=None,
+        location_config={},
+        storage_config={},
+        progress=True,
+    )
@@ -34,10 +34,18 @@ def test_list_archives_with_log_debug_calls_borg_with_debug_parameter():
    module.list_archives(repository='repo', storage_config={})


-def test_list_archives_with_json_calls_borg_with_json_parameter():
-    insert_subprocess_mock(LIST_COMMAND + ('--json',))
+def test_list_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
+    storage_config = {'lock_wait': 5}
+    insert_subprocess_mock(LIST_COMMAND + ('--lock-wait', '5'))

-    module.list_archives(repository='repo', storage_config={}, json=True)
+    module.list_archives(repository='repo', storage_config=storage_config)
+
+
+def test_list_archives_with_archive_calls_borg_with_archive_parameter():
+    storage_config = {}
+    insert_subprocess_mock(('borg', 'list', 'repo::archive'))
+
+    module.list_archives(repository='repo', storage_config=storage_config, archive='archive')


 def test_list_archives_with_local_path_calls_borg_via_local_path():
@@ -52,8 +60,7 @@ def test_list_archives_with_remote_path_calls_borg_with_remote_path_parameters()
    module.list_archives(repository='repo', storage_config={}, remote_path='borg1')


-def test_list_archives_with_lock_wait_calls_borg_with_lock_wait_parameters():
-    storage_config = {'lock_wait': 5}
-    insert_subprocess_mock(LIST_COMMAND + ('--lock-wait', '5'))
+def test_list_archives_with_json_calls_borg_with_json_parameter():
+    insert_subprocess_mock(LIST_COMMAND + ('--json',))

-    module.list_archives(repository='repo', storage_config=storage_config)
+    module.list_archives(repository='repo', storage_config={}, json=True)
@@ -48,26 +48,85 @@ def test_run_commands_handles_multiple_json_outputs_in_array():


 def test_collect_configuration_run_summary_logs_info_for_success():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
    flexmock(module).should_receive('run_configuration')
+    args = flexmock(extract=False, list=False)

-    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=()))
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))

-    assert any(log for log in logs if log.levelno == module.logging.INFO)
+    assert all(log for log in logs if log.levelno == module.logging.INFO)


-def test_collect_configuration_run_summary_logs_critical_for_error():
+def test_collect_configuration_run_summary_logs_info_for_success_with_extract():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository')
+    flexmock(module).should_receive('run_configuration')
+    args = flexmock(extract=True, list=False, repository='repo')
+
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))
+
+    assert all(log for log in logs if log.levelno == module.logging.INFO)
+
+
+def test_collect_configuration_run_summary_logs_critical_for_extract_with_repository_error():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository').and_raise(
+        ValueError
+    )
+    args = flexmock(extract=True, list=False, repository='repo')
+
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))
+
+    assert any(log for log in logs if log.levelno == module.logging.CRITICAL)
+
+
+def test_collect_configuration_run_summary_logs_critical_for_list_with_archive_and_repository_error():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository').and_raise(
+        ValueError
+    )
+    args = flexmock(extract=False, list=True, repository='repo', archive='test')
+
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))
+
+    assert any(log for log in logs if log.levelno == module.logging.CRITICAL)
+
+
+def test_collect_configuration_run_summary_logs_info_for_success_with_list():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module).should_receive('run_configuration')
+    args = flexmock(extract=False, list=True, repository='repo', archive=None)
+
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))
+
+    assert all(log for log in logs if log.levelno == module.logging.INFO)
+
+
+def test_collect_configuration_run_summary_logs_critical_for_parse_error():
+    flexmock(module.validate).should_receive('parse_configuration').and_raise(ValueError)
+    args = flexmock(extract=False, list=False)
+
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))
+
+    assert any(log for log in logs if log.levelno == module.logging.CRITICAL)
+
+
+def test_collect_configuration_run_summary_logs_critical_for_run_error():
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module.validate).should_receive('guard_configuration_contains_repository')
    flexmock(module).should_receive('run_configuration').and_raise(ValueError)
+    args = flexmock(extract=False, list=False)

-    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=()))
+    logs = tuple(module.collect_configuration_run_summary_logs(('test.yaml',), args=args))

    assert any(log for log in logs if log.levelno == module.logging.CRITICAL)


 def test_collect_configuration_run_summary_logs_critical_for_missing_configs():
-    logs = tuple(
-        module.collect_configuration_run_summary_logs(
-            config_filenames=(), args=flexmock(config_paths=())
-        )
-    )
+    flexmock(module.validate).should_receive('parse_configuration').and_return({'test.yaml': {}})
+    flexmock(module).should_receive('run_configuration')
+    args = flexmock(config_paths=(), extract=False, list=False)
+
+    logs = tuple(module.collect_configuration_run_summary_logs(config_filenames=(), args=args))

    assert any(log for log in logs if log.levelno == module.logging.CRITICAL)
@@ -117,12 +117,3 @@ def test_guard_configuration_upgraded_does_not_raise_when_neither_config_present
    flexmock(os.path).should_receive('exists').with_args('other.yaml').and_return(False)

    module.guard_configuration_upgraded('config', ('config.yaml', 'other.yaml'))
-
-
-def test_guard_excludes_filename_omitted_raises_when_filename_provided():
-    with pytest.raises(module.LegacyExcludesFilenamePresent):
-        module.guard_excludes_filename_omitted(excludes_filename='/etc/borgmatic/excludes')
-
-
-def test_guard_excludes_filename_omitted_does_not_raise_when_filename_not_provided():
-    module.guard_excludes_filename_omitted(excludes_filename=None)
@@ -90,3 +90,42 @@ def test_apply_logical_validation_does_not_raise_or_warn_if_archive_name_format_

 def test_apply_logical_validation_does_not_raise_otherwise():
    module.apply_logical_validation('config.yaml', {'retention': {'keep_secondly': 1000}})
+
+
+def test_guard_configuration_contains_repository_does_not_raise_when_repository_in_config():
+    module.guard_configuration_contains_repository(
+        repository='repo', configurations={'config.yaml': {'location': {'repositories': ['repo']}}}
+    )
+
+
+def test_guard_configuration_contains_repository_does_not_raise_when_repository_not_given():
+    module.guard_configuration_contains_repository(
+        repository=None, configurations={'config.yaml': {'location': {'repositories': ['repo']}}}
+    )
+
+
+def test_guard_configuration_contains_repository_errors_when_repository_assumed_to_match_config_twice():
+    with pytest.raises(ValueError):
+        module.guard_configuration_contains_repository(
+            repository=None,
+            configurations={'config.yaml': {'location': {'repositories': ['repo', 'repo2']}}},
+        )
+
+
+def test_guard_configuration_contains_repository_errors_when_repository_missing_from_config():
+    with pytest.raises(ValueError):
+        module.guard_configuration_contains_repository(
+            repository='nope',
+            configurations={'config.yaml': {'location': {'repositories': ['repo', 'repo2']}}},
+        )
+
+
+def test_guard_configuration_contains_repository_errors_when_repository_matches_config_twice():
+    with pytest.raises(ValueError):
+        module.guard_configuration_contains_repository(
+            repository='repo',
+            configurations={
+                'config.yaml': {'location': {'repositories': ['repo', 'repo2']}},
+                'other.yaml': {'location': {'repositories': ['repo']}},
+            },
+        )
Author	SHA1	Message	Date
Dan Helfman	6ff1867312	Configuration files includes and merging (#148 ).	2019-03-06 12:06:27 -08:00
Dan Helfman	3cb52423d2	Support for Borg create/extract --numeric-owner flag (#147 ).	2019-03-05 09:11:35 -08:00
Dan Helfman	5a5b6491ac	Add note about uncommenting section names.	2019-03-04 15:15:49 -08:00
Dan Helfman	4272c6b077	List the files within an archive via --list --archive option (#140 ).	2019-02-23 23:02:17 -08:00
Dan Helfman	26071de2e7	Update extraction docs.	2019-02-18 22:43:32 -08:00
Dan Helfman	fe92d9e838	Fix restore paths list to tuple conversion.	2019-02-18 21:59:09 -08:00
Dan Helfman	5ea2d644a2	Fix error handling when --extract repository guard fails.	2019-02-18 21:52:56 -08:00
Dan Helfman	c35f90154f	Only guard repository when --extract is used.	2019-02-18 21:43:30 -08:00
Dan Helfman	36305c50b1	Update push script to support branches.	2019-02-18 13:51:33 -08:00
Dan Helfman	2b3b8eab71	Add archive extract to end-to-end test.	2019-02-18 13:47:18 -08:00
Dan Helfman	aa7c7651e5	Fix config repositories consumption.	2019-02-18 13:27:35 -08:00
Dan Helfman	c41ffb5ceb	If no extract repository is given, then error if there are multiple configured repositories.	2019-02-18 13:22:14 -08:00
Dan Helfman	766a03375a	Guard that the given repository occurs in config exactly once.	2019-02-18 12:58:39 -08:00
Dan Helfman	2a4d4247e3	Tests for extract_archive().	2019-02-18 10:31:52 -08:00
Dan Helfman	9de5083a7e	Additional test coverage for extract options in borgmatic command.	2019-02-18 09:52:56 -08:00
Dan Helfman	d0557b2bcd	Initial work on #123 : Support for Borg extract.	2019-02-18 09:30:34 -08:00
Dan Helfman	1a980d6321	Organize options within command-line help into logical groups.	2019-02-12 22:27:04 -08:00
Dan Helfman	fb21d4e645	Remove dead code.	2019-02-09 21:17:55 -08:00
Dan Helfman	5933a4d778	Note tests exclusion in changelog.	2019-02-08 20:59:50 -08:00
Dan HelfmanandGitea	8cf16c7831	Exclude tests from distribution packages.	2019-02-09 05:02:19 +00:00
Julien Nicoulaud	fcf4e03c2f	exclude tests from distribution packages	2019-02-08 19:22:56 +01:00
Dan Helfman	d1b29e82da	borgmatic command-line reference.	2019-02-04 22:27:40 -08:00
Dan Helfman	290e969a22	Include a sample borgmatic configuration file in the documentation (#119 ).	2019-02-04 22:12:33 -08:00
Dan Helfman	18ae91ea6e	Strike some unnecessary words from docs.	2019-02-04 20:58:27 -08:00
Dan Helfman	0bce77a2ac	Distribute troubleshooting among relevant how-to guides.	2019-02-04 20:53:47 -08:00
Dan Helfman	19155607af	Include link to development how-to.	2019-02-03 22:42:33 -08:00
Dan Helfman	f357c37e2c	Fix/remove some documentation links.	2019-02-03 22:35:38 -08:00
Dan Helfman	2980c14728	Fix README links on GitHub.	2019-02-03 22:26:39 -08:00
Dan Helfman	7e0e00d45d	Refactor documentation into multiple separate pages for clarity and findability.	2019-02-03 22:20:59 -08:00
Dan Helfman	8b4ac0017b	Fix ticket number in changelog.	2019-01-27 14:00:24 -08:00