#32: Fix for passing check_last as integer to subprocess when calling Borg.

.hgignore

@@ -2,6 +2,8 @@ syntax: glob
 *.egg-info
 *.pyc
 *.swp
+.cache
+.coverage
 .tox
 build
 dist

.hgtags

@@ -26,3 +26,9 @@ e58246fc92bb22c2b2fd8b86a1227de69d2d0315 0.1.4
 0000000000000000000000000000000000000000 github/yaml_config_files
 28434dd0440cc8da44c2f3e9bd7e9402a59c3b40 github/master
 dbc96d3f83bd5570b6826537616d4160b3374836 0.1.8
+0e1fbee9358de4f062fa9539e1355db83db70caa 1.0.0
+de2d7721cdec93a52d20222a9ddd579ed93c1017 1.0.1
+9603d13910b32d57a887765cab694ac5d0acc1f4 1.0.2
+32c6341dda9fad77a3982641bce8a3a45821842e 1.0.3
+5a003056a8ff4709c5bd4d6d33354199423f8a1d 1.1.0
+7d3d11eff6c0773883c48f221431f157bc7995eb 1.1.1

MANIFEST.in

@@ -0,0 +1 @@
+include borgmatic/config/schema.yaml

NEWS

@@ -1,3 +1,38 @@
+1.1.2
+
+ * #32: Fix for passing check_last as integer to subprocess when calling Borg.
+
+1.1.1
+
+ * Part of #32: Fix for upgrade-borgmatic-config converting check_last option as a string instead of
+   an integer.
+ * Fix for upgrade-borgmatic-config erroring when consistency checks option is not present.
+
+1.1.0
+
+ * Switched config file format to YAML. Run upgrade-borgmatic-config to upgrade.
+ * Added generate-borgmatic-config command for initial config creation.
+ * Dropped Python 2 support. Now Python 3 only.
+ * #18: Fix for README mention of sample files not included in package.
+ * #22: Sample files for triggering borgmatic from a systemd timer.
+ * Support for backing up to multiple repositories.
+ * To free up space, now pruning backups prior to creating a new backup.
+ * Enabled test coverage output during tox runs.
+ * Added logo.
+
+1.0.3
+
+ * #21: Fix for verbosity flag not actually causing verbose output.
+
+1.0.2
+
+ * #20: Fix for traceback when remote_path option is missing.
+
+1.0.1
+
+ * #19: Support for Borg's --remote-path option to use an alternate Borg
+   executable. See sample/config.
+
 1.0.0
 
  * Attic is no longer supported, as there hasn't been any recent development on

README.md

@@ -1,4 +1,6 @@
-title: Borgmatic
+title: borgmatic
+
+<img src="static/borgmatic.svg" alt="borgmatic logo" style="width: 8em; float: right; padding-left: 1em;" />
 
 ## Overview
 
@@ -11,36 +13,42 @@ all on the command-line, and handles common errors.
 
 Here's an example config file:
 
-```INI
+```yaml
-[location]
+location:
-# Space-separated list of source directories to backup.
+    # List of source directories to backup. Globs are expanded.
-# Globs are expanded.
+    source_directories:
-source_directories: /home /etc /var/log/syslog*
+        - /home
+        - /etc
+        - /var/log/syslog*
 
-# Path to local or remote backup repository.
+    # Paths to local or remote repositories.
-repository: user@backupserver:sourcehostname.borg
+    repositories:
+        - user@backupserver:sourcehostname.borg
 
-[retention]
+    # Any paths matching these patterns are excluded from backups.
-# Retention policy for how many backups to keep in each category.
+    exclude_patterns:
-keep_daily: 7
+        - /home/*/.cache
-keep_weekly: 4
-keep_monthly: 6
 
-[consistency]
+retention:
-# Consistency checks to run, or "disabled" to prevent checks.
+    # Retention policy for how many backups to keep in each category.
-checks: repository archives
+    keep_daily: 7
+    keep_weekly: 4
+    keep_monthly: 6
+
+consistency:
+    # List of consistency checks to run: "repository", "archives", or both.
+    checks:
+        - repository
+        - archives
 ```
 
-Additionally, exclude patterns can be specified in a separate excludes config
-file, one pattern per line.
-
 borgmatic is hosted at <https://torsion.org/borgmatic> with [source code
 available](https://torsion.org/hg/borgmatic). It's also mirrored on
 [GitHub](https://github.com/witten/borgmatic) and
 [BitBucket](https://bitbucket.org/dhelfman/borgmatic) for convenience.
 
 
-## Setup
+## Installation
 
 To get up and running, follow the [Borg Quick
 Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create
@@ -57,16 +65,65 @@ To install borgmatic, run the following command to download and install it:
 
     sudo pip install --upgrade borgmatic
 
-Then, copy the following configuration files:
+Make sure you're using Python 3, as borgmatic does not support Python 2. (You
+may have to use "pip3" or similar instead of "pip".)
 
-    sudo cp sample/borgmatic.cron /etc/cron.d/borgmatic
+Then, generate a sample configuration file:
-    sudo mkdir /etc/borgmatic/
-    sudo cp sample/config sample/excludes /etc/borgmatic/
 
-Lastly, modify the /etc files with your desired configuration.
+    sudo generate-borgmatic-config
+
+This generates a sample configuration file at /etc/borgmatic/config.yaml (by
+default). You should edit the file to suit your needs, as the values are just
+representative. All fields are optional except where indicated, so feel free
+to remove anything you don't need.
 
 
-## Upgrading from atticmatic
+## Upgrading
+
+In general, all you should need to do to upgrade borgmatic is run the
+following:
+
+    sudo pip install --upgrade borgmatic
+
+(You may have to use "pip3" or similar instead of "pip", so Python 3 gets
+used.)
+
+However, see below about special cases.
+
+
+### Upgrading from borgmatic 1.0.x
+
+borgmatic changed its configuration file format in version 1.1.0 from
+INI-style to YAML. This better supports validation, and has a more natural way
+to express lists of values. To upgrade your existing configuration, first
+upgrade to the new version of borgmatic:
+
+As of version 1.1.0, borgmatic no longer supports Python 2. If you were
+already running borgmatic with Python 3, then you can simply upgrade borgmatic
+in-place:
+
+    sudo pip install --upgrade borgmatic
+
+But if you were running borgmatic with Python 2, uninstall and reinstall instead:
+
+    sudo pip uninstall borgmatic
+    sudo pip3 install borgmatic
+
+The pip binary names for different versions of Python can differ, so the above
+commands may need some tweaking to work on your machine.
+
+
+Once borgmatic is upgraded, run:
+
+    sudo upgrade-borgmatic-config
+
+That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
+(by default) using the values from both your existing configuration and
+excludes files. The new version of borgmatic will consume the YAML
+configuration file instead of the old one.
+
+
+### Upgrading from atticmatic
 
 You can ignore this section if you're not an atticmatic user (the former name
 of borgmatic).
@@ -86,6 +143,7 @@ from atticmatic to borgmatic. Simply run the following commands:
 That's it! borgmatic will continue using your /etc/borgmatic configuration
 files.
 
+
 ## Usage
 
 You can run borgmatic and start a backup simply by invoking it without
@@ -111,6 +169,39 @@ If you'd like to see the available command-line arguments, view the help:
     borgmatic --help
 
 
+## Autopilot
+
+If you want to run borgmatic automatically, say once a day, the you can
+configure a job runner to invoke it periodically.
+
+### cron
+
+If you're using cron, download the [sample cron
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/cron/borgmatic).
+Then, from the directory where you downloaded it:
+
+    sudo mv borgmatic /etc/cron.d/borgmatic
+    sudo chmod +x /etc/cron.d/borgmatic
+
+You can modify the cron file if you'd like to run borgmatic more or less frequently.
+
+### systemd
+
+If you're using systemd instead of cron to run jobs, download the [sample
+systemd service
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.service)
+and the [sample systemd timer
+file](https://torsion.org/hg/borgmatic/raw-file/tip/sample/systemd/borgmatic.timer).
+Then, from the directory where you downloaded them:
+
+    sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
+    sudo systemctl enable borgmatic.timer
+    sudo systemctl start borgmatic.timer
+
+Feel free to modify the timer file based on how frequently you'd like
+borgmatic to run.
+
+
 ## Running tests
 
 First install tox, which is used for setting up testing environments:
@@ -143,10 +234,21 @@ This should make the client keep the connection alive while validating
 backups.
 
 
+### libyaml compilation errors
+
+borgmatic depends on a Python YAML library (ruamel.yaml) that will optionally
+use a C YAML library (libyaml) if present. But if it's not installed, then
+when installing or upgrading borgmatic, you may see errors about compiling the
+YAML library. If so, not to worry. borgmatic should install and function
+correctly even without the C YAML library. And borgmatic won't be any faster
+with the C library present, so you don't need to go out of your way to install
+it.
+
+
 ## Issues and feedback
 
 Got an issue or an idea for a feature enhancement? Check out the [borgmatic
-issue tracker](https://tree.taiga.io/project/witten-borgmatic/issues). In
+issue tracker](https://tree.taiga.io/project/witten-borgmatic/issues?page=1&status=399951,399952,399955). In
 order to create a new issue or comment on an issue, you'll need to [login
 first](https://tree.taiga.io/login).
 

borgmatic/borg.py

@@ -1,10 +1,11 @@
 from datetime import datetime
+import glob
+import itertools
 import os
-import re
 import platform
+import re
 import subprocess
-from glob import glob
+import tempfile
-from itertools import chain
 
 from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
@@ -22,26 +23,47 @@ def initialize(storage_config, command=COMMAND):
         os.environ['{}_PASSPHRASE'.format(command.upper())] = passphrase
 
 
+def _write_exclude_file(exclude_patterns=None):
+    '''
+    Given a sequence of exclude patterns, write them to a named temporary file and return it. Return
+    None if no patterns are provided.
+    '''
+    if not exclude_patterns:
+        return None
+
+    exclude_file = tempfile.NamedTemporaryFile('w')
+    exclude_file.write('\n'.join(exclude_patterns))
+    exclude_file.flush()
+
+    return exclude_file
+
+
 def create_archive(
-    excludes_filename, verbosity, storage_config, source_directories, repository, command=COMMAND,
+    verbosity, repository, location_config, storage_config, command=COMMAND,
-    one_file_system=None
 ):
     '''
-    Given an excludes filename (or None), a vebosity flag, a storage config dict, a space-separated
+    Given a vebosity flag, a storage config dict, a list of source directories, a local or remote
-    list of source directories, a local or remote repository path, and a command to run, create an
+    repository path, a list of exclude patterns, and a command to run, create an attic archive.
-    attic archive.
     '''
-    sources = re.split('\s+', source_directories)
+    sources = tuple(
-    sources = tuple(chain.from_iterable(glob(x) or [x] for x in sources))
+        itertools.chain.from_iterable(
-    exclude_flags = ('--exclude-from', excludes_filename) if excludes_filename else ()
+            glob.glob(directory) or [directory]
+            for directory in location_config['source_directories']
+        )
+    )
+
+    exclude_file = _write_exclude_file(location_config.get('exclude_patterns'))
+    exclude_flags = ('--exclude-from', exclude_file.name) if exclude_file else ()
     compression = storage_config.get('compression', None)
     compression_flags = ('--compression', compression) if compression else ()
     umask = storage_config.get('umask', None)
     umask_flags = ('--umask', str(umask)) if umask else ()
-    one_file_system_flags = ('--one-file-system',) if one_file_system else ()
+    one_file_system_flags = ('--one-file-system',) if location_config.get('one_file_system') else ()
+    remote_path = location_config.get('remote_path')
+    remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
     verbosity_flags = {
-        VERBOSITY_SOME: ('--stats',),
+        VERBOSITY_SOME: ('--info', '--stats',),
-        VERBOSITY_LOTS: ('--verbose', '--stats'),
+        VERBOSITY_LOTS: ('--debug', '--list', '--stats'),
     }.get(verbosity, ())
 
     full_command = (
@@ -52,7 +74,7 @@ def create_archive(
             timestamp=datetime.now().isoformat(),
         ),
     ) + sources + exclude_flags + compression_flags + one_file_system_flags + \
-        umask_flags + verbosity_flags
+        remote_path_flags + umask_flags + verbosity_flags
 
     subprocess.check_call(full_command)
 
@@ -79,15 +101,16 @@ def _make_prune_flags(retention_config):
     )
 
 
-def prune_archives(verbosity, repository, retention_config, command=COMMAND):
+def prune_archives(verbosity, repository, retention_config, command=COMMAND, remote_path=None):
     '''
     Given a verbosity flag, a local or remote repository path, a retention config dict, and a
     command to run, prune attic archives according the the retention policy specified in that
     configuration.
     '''
+    remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
     verbosity_flags = {
-        VERBOSITY_SOME: ('--stats',),
+        VERBOSITY_SOME: ('--info', '--stats',),
-        VERBOSITY_LOTS: ('--verbose', '--stats'),
+        VERBOSITY_LOTS: ('--debug', '--stats'),
     }.get(verbosity, ())
 
     full_command = (
@@ -97,7 +120,7 @@ def prune_archives(verbosity, repository, retention_config, command=COMMAND):
         element
         for pair in _make_prune_flags(retention_config)
         for element in pair
-    ) + verbosity_flags
+    ) + remote_path_flags + verbosity_flags
 
     subprocess.check_call(full_command)
 
@@ -107,12 +130,11 @@ DEFAULT_CHECKS = ('repository', 'archives')
 
 def _parse_checks(consistency_config):
     '''
-    Given a consistency config with a space-separated "checks" option, transform it to a tuple of
+    Given a consistency config with a "checks" list, transform it to a tuple of named checks to run.
-    named checks to run.
 
     For example, given a retention config of:
 
-        {'checks': 'repository archives'}
+        {'checks': ['repository', 'archives']}
 
     This will be returned as:
 
@@ -121,14 +143,11 @@ def _parse_checks(consistency_config):
     If no "checks" option is present, return the DEFAULT_CHECKS. If the checks value is the string
     "disabled", return an empty tuple, meaning that no checks should be run.
     '''
-    checks = consistency_config.get('checks', '').strip()
+    checks = consistency_config.get('checks', [])
-    if not checks:
+    if checks == ['disabled']:
-        return DEFAULT_CHECKS
+        return ()
 
-    return tuple(
+    return tuple(check for check in checks if check.lower() not in ('disabled', '')) or DEFAULT_CHECKS
-        check for check in consistency_config['checks'].split(' ')
-        if check.lower() not in ('disabled', '')
-    )
 
 
 def _make_check_flags(checks, check_last=None):
@@ -143,10 +162,9 @@ def _make_check_flags(checks, check_last=None):
 
         ('--repository-only',)
 
-    Additionally, if a check_last value is given, a "--last" flag will be added. Note that only
+    Additionally, if a check_last value is given, a "--last" flag will be added.
-    Borg supports this flag.
     '''
-    last_flag = ('--last', check_last) if check_last else ()
+    last_flag = ('--last', str(check_last)) if check_last else ()
     if checks == DEFAULT_CHECKS:
         return last_flag
 
@@ -155,7 +173,7 @@ def _make_check_flags(checks, check_last=None):
     ) + last_flag
 
 
-def check_archives(verbosity, repository, consistency_config, command=COMMAND):
+def check_archives(verbosity, repository, consistency_config, command=COMMAND, remote_path=None):
     '''
     Given a verbosity flag, a local or remote repository path, a consistency config dict, and a
     command to run, check the contained attic archives for consistency.
@@ -167,15 +185,16 @@ def check_archives(verbosity, repository, consistency_config, command=COMMAND):
     if not checks:
         return
 
+    remote_path_flags = ('--remote-path', remote_path) if remote_path else ()
     verbosity_flags = {
-        VERBOSITY_SOME: ('--verbose',),
+        VERBOSITY_SOME: ('--info',),
-        VERBOSITY_LOTS: ('--verbose',),
+        VERBOSITY_LOTS: ('--debug',),
     }.get(verbosity, ())
 
     full_command = (
         command, 'check',
         repository,
-    ) + _make_check_flags(checks, check_last) + verbosity_flags
+    ) + _make_check_flags(checks, check_last) + remote_path_flags + verbosity_flags
 
     # The check command spews to stdout/stderr even without the verbose flag. Suppress it.
     stdout = None if verbosity_flags else open(os.devnull, 'w')

borgmatic/command.py

@@ -1,57 +0,0 @@
-from __future__ import print_function
-from argparse import ArgumentParser
-import os
-from subprocess import CalledProcessError
-import sys
-
-from borgmatic import borg
-from borgmatic.config import parse_configuration, CONFIG_FORMAT
-
-
-DEFAULT_CONFIG_FILENAME = '/etc/borgmatic/config'
-DEFAULT_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
-
-
-def parse_arguments(*arguments):
-    '''
-    Given the name of the command with which this script was invoked and command-line arguments,
-    parse the arguments and return them as an ArgumentParser instance. Use the command name to
-    determine the default configuration and excludes paths.
-    '''
-    parser = ArgumentParser()
-    parser.add_argument(
-        '-c', '--config',
-        dest='config_filename',
-        default=DEFAULT_CONFIG_FILENAME,
-        help='Configuration filename',
-    )
-    parser.add_argument(
-        '--excludes',
-        dest='excludes_filename',
-        default=DEFAULT_EXCLUDES_FILENAME if os.path.exists(DEFAULT_EXCLUDES_FILENAME) else None,
-        help='Excludes filename',
-    )
-    parser.add_argument(
-        '-v', '--verbosity',
-        type=int,
-        help='Display verbose progress (1 for some, 2 for lots)',
-    )
-
-    return parser.parse_args(arguments)
-
-
-def main():
-    try:
-        args = parse_arguments(*sys.argv[1:])
-        config = parse_configuration(args.config_filename, CONFIG_FORMAT)
-        repository = config.location['repository']
-
-        borg.initialize(config.storage)
-        borg.create_archive(
-            args.excludes_filename, args.verbosity, config.storage, **config.location
-        )
-        borg.prune_archives(args.verbosity, repository, config.retention)
-        borg.check_archives(args.verbosity, repository, config.consistency)
-    except (ValueError, IOError, CalledProcessError) as error:
-        print(error, file=sys.stderr)
-        sys.exit(1)

borgmatic/commands/borgmatic.py

@@ -0,0 +1,66 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+
+from borgmatic import borg
+from borgmatic.config import convert, validate
+
+
+LEGACY_CONFIG_FILENAME = '/etc/borgmatic/config'
+DEFAULT_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
+DEFAULT_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser()
+    parser.add_argument(
+        '-c', '--config',
+        dest='config_filename',
+        default=DEFAULT_CONFIG_FILENAME,
+        help='Configuration filename',
+    )
+    parser.add_argument(
+        '--excludes',
+        dest='excludes_filename',
+        help='Excludes filename, deprecated in favor of exclude_patterns within configuration',
+    )
+    parser.add_argument(
+        '-v', '--verbosity',
+        type=int,
+        help='Display verbose progress (1 for some, 2 for lots)',
+    )
+
+    return parser.parse_args(arguments)
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+        convert.guard_configuration_upgraded(LEGACY_CONFIG_FILENAME, args.config_filename)
+        config = validate.parse_configuration(args.config_filename, validate.schema_filename())
+        (location, storage, retention, consistency) = (
+            config.get(section_name, {})
+            for section_name in ('location', 'storage', 'retention', 'consistency')
+        )
+        remote_path = location.get('remote_path')
+
+        borg.initialize(storage)
+
+        for repository in location['repositories']:
+            borg.prune_archives(args.verbosity, repository, retention, remote_path=remote_path)
+            borg.create_archive(
+                args.verbosity,
+                repository,
+                location,
+                storage,
+            )
+            borg.check_archives(args.verbosity, repository, consistency, remote_path=remote_path)
+    except (ValueError, OSError, CalledProcessError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/commands/convert_config.py

@@ -0,0 +1,99 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+import textwrap
+
+from ruamel import yaml
+
+from borgmatic import borg
+from borgmatic.config import convert, generate, legacy, validate
+
+
+DEFAULT_SOURCE_CONFIG_FILENAME = '/etc/borgmatic/config'
+DEFAULT_SOURCE_EXCLUDES_FILENAME = '/etc/borgmatic/excludes'
+DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser(
+        description='''
+            Convert legacy INI-style borgmatic configuration and excludes files to a single YAML
+            configuration file. Note that this replaces any comments from the source files.
+        '''
+    )
+    parser.add_argument(
+        '-s', '--source-config',
+        dest='source_config_filename',
+        default=DEFAULT_SOURCE_CONFIG_FILENAME,
+        help='Source INI-style configuration filename. Default: {}'.format(DEFAULT_SOURCE_CONFIG_FILENAME),
+    )
+    parser.add_argument(
+        '-e', '--source-excludes',
+        dest='source_excludes_filename',
+        default=DEFAULT_SOURCE_EXCLUDES_FILENAME if os.path.exists(DEFAULT_SOURCE_EXCLUDES_FILENAME) else None,
+        help='Excludes filename',
+    )
+    parser.add_argument(
+        '-d', '--destination-config',
+        dest='destination_config_filename',
+        default=DEFAULT_DESTINATION_CONFIG_FILENAME,
+        help='Destination YAML configuration filename. Default: {}'.format(DEFAULT_DESTINATION_CONFIG_FILENAME),
+    )
+
+    return parser.parse_args(arguments)
+
+
+TEXT_WRAP_CHARACTERS = 80
+
+
+def display_result(args):  # pragma: no cover
+    result_lines = textwrap.wrap(
+        'Your borgmatic configuration has been upgraded. Please review the result in {}.'.format(
+            args.destination_config_filename
+        ),
+        TEXT_WRAP_CHARACTERS,
+    )
+    
+    delete_lines = textwrap.wrap(
+        'Once you are satisfied, you can safely delete {}{}.'.format(
+            args.source_config_filename,
+            ' and {}'.format(args.source_excludes_filename) if args.source_excludes_filename else '',
+        ),
+        TEXT_WRAP_CHARACTERS,
+    )
+
+    print('\n'.join(result_lines))
+    print()
+    print('\n'.join(delete_lines))
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+        schema = yaml.round_trip_load(open(validate.schema_filename()).read())
+        source_config = legacy.parse_configuration(args.source_config_filename, legacy.CONFIG_FORMAT)
+        source_config_file_mode = os.stat(args.source_config_filename).st_mode
+        source_excludes = (
+            open(args.source_excludes_filename).read().splitlines()
+            if args.source_excludes_filename
+            else []
+        )
+
+        destination_config = convert.convert_legacy_parsed_config(source_config, source_excludes, schema)
+
+        generate.write_configuration(
+            args.destination_config_filename,
+            destination_config,
+            mode=source_config_file_mode,
+        )
+
+        display_result(args)
+    except (ValueError, OSError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/commands/generate_config.py

@@ -0,0 +1,44 @@
+from __future__ import print_function
+from argparse import ArgumentParser
+import os
+from subprocess import CalledProcessError
+import sys
+
+from ruamel import yaml
+
+from borgmatic import borg
+from borgmatic.config import convert, generate, validate
+
+
+DEFAULT_DESTINATION_CONFIG_FILENAME = '/etc/borgmatic/config.yaml'
+
+
+def parse_arguments(*arguments):
+    '''
+    Given command-line arguments with which this script was invoked, parse the arguments and return
+    them as an ArgumentParser instance.
+    '''
+    parser = ArgumentParser(description='Generate a sample borgmatic YAML configuration file.')
+    parser.add_argument(
+        '-d', '--destination',
+        dest='destination_filename',
+        default=DEFAULT_DESTINATION_CONFIG_FILENAME,
+        help='Destination YAML configuration filename. Default: {}'.format(DEFAULT_DESTINATION_CONFIG_FILENAME),
+    )
+
+    return parser.parse_args(arguments)
+
+
+def main():  # pragma: no cover
+    try:
+        args = parse_arguments(*sys.argv[1:])
+
+        generate.generate_sample_configuration(args.destination_filename, validate.schema_filename())
+
+        print('Generated a sample configuration file at {}.'.format(args.destination_filename))
+        print()
+        print('Please edit the file to suit your needs. The values are just representative.')
+        print('All fields are optional except where indicated.')
+    except (ValueError, OSError) as error:
+        print(error, file=sys.stderr)
+        sys.exit(1)

borgmatic/config/convert.py

@@ -0,0 +1,106 @@
+import os
+
+from ruamel import yaml
+
+from borgmatic.config import generate
+
+
+def _convert_section(source_section_config, section_schema):
+    '''
+    Given a legacy Parsed_config instance for a single section, convert it to its corresponding
+    yaml.comments.CommentedMap representation in preparation for actual serialization to YAML.
+
+    Where integer types exist in the given section schema, convert their values to integers.
+    '''
+    destination_section_config = yaml.comments.CommentedMap([
+        (
+            option_name,
+            int(option_value)
+                if section_schema['map'].get(option_name, {}).get('type') == 'int' else option_value
+        )
+        for option_name, option_value in source_section_config.items()
+    ])
+
+    return destination_section_config
+
+
+def convert_legacy_parsed_config(source_config, source_excludes, schema):
+    '''
+    Given a legacy Parsed_config instance loaded from an INI-style config file and a list of exclude
+    patterns, convert them to a corresponding yaml.comments.CommentedMap representation in
+    preparation for serialization to a single YAML config file.
+
+    Additionally, use the given schema as a source of helpful comments to include within the
+    returned CommentedMap.
+    '''
+    destination_config = yaml.comments.CommentedMap([
+        (section_name, _convert_section(section_config, schema['map'][section_name]))
+        for section_name, section_config in source_config._asdict().items()
+    ])
+
+    # Split space-seperated values into actual lists, make "repository" into a list, and merge in
+    # excludes.
+    location = destination_config['location']
+    location['source_directories'] = source_config.location['source_directories'].split(' ')
+    location['repositories'] = [location.pop('repository')]
+    location['exclude_patterns'] = source_excludes
+
+    if source_config.consistency.get('checks'):
+        destination_config['consistency']['checks'] = source_config.consistency['checks'].split(' ')
+
+    # Add comments to each section, and then add comments to the fields in each section.
+    generate.add_comments_to_configuration(destination_config, schema)
+
+    for section_name, section_config in destination_config.items():
+        generate.add_comments_to_configuration(
+            section_config,
+            schema['map'][section_name],
+            indent=generate.INDENT,
+        )
+
+    return destination_config
+
+
+class LegacyConfigurationNotUpgraded(FileNotFoundError):
+    def __init__(self):
+        super(LegacyConfigurationNotUpgraded, self).__init__(
+            '''borgmatic changed its configuration file format in version 1.1.0 from INI-style
+to YAML. This better supports validation, and has a more natural way to express
+lists of values. To upgrade your existing configuration, run:
+
+    sudo upgrade-borgmatic-config
+
+That will generate a new YAML configuration file at /etc/borgmatic/config.yaml
+(by default) using the values from both your existing configuration and excludes
+files. The new version of borgmatic will consume the YAML configuration file
+instead of the old one.'''
+        )
+
+
+def guard_configuration_upgraded(source_config_filename, destination_config_filename):
+    '''
+    If legacy souce configuration exists but destination upgraded config doesn't, raise
+    LegacyConfigurationNotUpgraded.
+
+    The idea is that we want to alert the user about upgrading their config if they haven't already.
+    '''
+    if os.path.exists(source_config_filename) and not os.path.exists(destination_config_filename):
+        raise LegacyConfigurationNotUpgraded()
+
+
+class LegacyExcludesFilenamePresent(FileNotFoundError):
+    def __init__(self):
+        super(LegacyExcludesFilenamePresent, self).__init__(
+            '''borgmatic changed its configuration file format in version 1.1.0 from INI-style
+to YAML. This better supports validation, and has a more natural way to express
+lists of values. The new configuration file incorporates excludes, so you no
+longer need to provide an excludes filename on the command-line with an
+"--excludes" argument.
+
+Please remove the "--excludes" argument and run borgmatic again.'''
+        )
+
+
+def guard_excludes_filename_omitted(excludes_filename):
+    if excludes_filename != None:
+        raise LegacyExcludesFilenamePresent()

borgmatic/config/generate.py

@@ -0,0 +1,93 @@
+from collections import OrderedDict
+import os
+
+from ruamel import yaml
+
+
+INDENT = 4
+
+
+def _insert_newline_before_comment(config, field_name):
+    '''
+    Using some ruamel.yaml black magic, insert a blank line in the config right befor the given
+    field and its comments.
+    '''
+    config.ca.items[field_name][1].insert(
+        0,
+        yaml.tokens.CommentToken('\n', yaml.error.CommentMark(0), None),
+    )
+
+
+def _schema_to_sample_configuration(schema, level=0):
+    '''
+    Given a loaded configuration schema, generate and return sample config for it. Include comments
+    for each section based on the schema "desc" description.
+    '''
+    example = schema.get('example')
+    if example:
+        return example
+
+    config = yaml.comments.CommentedMap([
+        (
+            section_name,
+            _schema_to_sample_configuration(section_schema, level + 1),
+        )
+        for section_name, section_schema in schema['map'].items()
+    ])
+
+    add_comments_to_configuration(config, schema, indent=(level * INDENT))
+
+    return config
+
+
+def write_configuration(config_filename, config, mode=0o600):
+    '''
+    Given a target config filename and a config data structure of nested OrderedDicts, write out the
+    config to file as YAML. Create any containing directories as needed.
+    '''
+    if os.path.exists(config_filename):
+        raise FileExistsError('{} already exists. Aborting.'.format(config_filename))
+
+    try:
+        os.makedirs(os.path.dirname(config_filename), mode=0o700)
+    except (FileExistsError, FileNotFoundError):
+        pass
+
+    with open(config_filename, 'w') as config_file:
+        config_file.write(yaml.round_trip_dump(config, indent=INDENT, block_seq_indent=INDENT))
+
+    os.chmod(config_filename, mode)
+
+
+def add_comments_to_configuration(config, schema, indent=0):
+    '''
+    Using descriptions from a schema as a source, add those descriptions as comments to the given
+    config before each field. This function only adds comments for the top-most config map level.
+    Indent the comment the given number of characters.
+    '''
+    for index, field_name in enumerate(config.keys()):
+        field_schema = schema['map'].get(field_name, {})
+        description = field_schema.get('desc')
+
+        # No description to use? Skip it.
+        if not field_schema or not description:
+            continue
+
+        config.yaml_set_comment_before_after_key(
+            key=field_name,
+            before=description,
+            indent=indent,
+        )
+        if index > 0:
+            _insert_newline_before_comment(config, field_name)
+
+
+def generate_sample_configuration(config_filename, schema_filename):
+    '''
+    Given a target config filename and the path to a schema filename in pykwalify YAML schema
+    format, write out a sample configuration file based on that schema.
+    '''
+    schema = yaml.round_trip_load(open(schema_filename))
+    config = _schema_to_sample_configuration(schema)
+
+    write_configuration(config_filename, config)

borgmatic/config/legacy.py

@@ -1,11 +1,5 @@
 from collections import OrderedDict, namedtuple
-
+from configparser import RawConfigParser
-try:
-    # Python 2
-    from ConfigParser import RawConfigParser
-except ImportError:
-    # Python 3
-    from configparser import RawConfigParser
 
 
 Section_format = namedtuple('Section_format', ('name', 'options'))
@@ -26,6 +20,7 @@ CONFIG_FORMAT = (
         (
             option('source_directories'),
             option('one_file_system', value_type=bool, required=False),
+            option('remote_path', required=False),
             option('repository'),
         ),
     ),

borgmatic/config/schema.yaml

@@ -0,0 +1,126 @@
+name: Borgmatic configuration file schema
+version: 1
+map:
+    location:
+        desc: |
+            Where to look for files to backup, and where to store those backups. See
+            https://borgbackup.readthedocs.io/en/stable/quickstart.html and
+            https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
+        required: True
+        map:
+            source_directories:
+                required: True
+                seq:
+                    - type: scalar
+                desc: List of source directories to backup (required). Globs are expanded.
+                example:
+                    - /home
+                    - /etc
+                    - /var/log/syslog*
+            one_file_system:
+                type: bool
+                desc: Stay in same file system (do not cross mount points).
+                example: yes
+            remote_path:
+                type: scalar
+                desc: Alternate Borg remote executable. Defaults to "borg".
+                example: borg1
+            repositories:
+                required: True
+                seq:
+                    - type: scalar
+                desc: |
+                    Paths to local or remote repositories (required). Multiple repositories are
+                    backed up to in sequence.
+                example:
+                    - user@backupserver:sourcehostname.borg
+            exclude_patterns:
+                seq:
+                    - type: scalar
+                desc: |
+                    Any paths matching these patterns are excluded from backups. Globs are expanded.
+                    See https://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-patterns
+                    for details.
+                example:
+                    - '*.pyc'
+                    - /home/*/.cache
+                    - /etc/ssl
+    storage:
+        desc: |
+            Repository storage options. See
+            https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
+            https://borgbackup.readthedocs.io/en/stable/usage.html#environment-variables for details.
+        map:
+            encryption_passphrase:
+                type: scalar
+                desc: |
+                    Passphrase to unlock the encryption key with. Only use on repositories that were
+                    initialized with passphrase/repokey encryption. Quote the value if it contains
+                    punctuation, so it parses correctly. And backslash any quote or backslash
+                    literals as well.
+                example: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
+            compression:
+                type: scalar
+                desc: |
+                    Type of compression to use when creating archives. See
+                    https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.
+                    Defaults to no compression.
+                example: lz4
+            umask:
+                type: scalar
+                desc: Umask to be used for borg create.
+                example: 0077
+    retention:
+        desc: |
+            Retention policy for how many backups to keep in each category. See
+            https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
+        map:
+            keep_within:
+                type: scalar
+                desc: Keep all archives within this time interval.
+                example: 3H
+            keep_hourly:
+                type: int
+                desc: Number of hourly archives to keep.
+                example: 24
+            keep_daily:
+                type: int
+                desc: Number of daily archives to keep.
+                example: 7
+            keep_weekly:
+                type: int
+                desc: Number of weekly archives to keep.
+                example: 4
+            keep_monthly:
+                type: int
+                desc: Number of monthly archives to keep.
+                example: 6
+            keep_yearly:
+                type: int
+                desc: Number of yearly archives to keep.
+                example: 1
+            prefix:
+                type: scalar
+                desc: When pruning, only consider archive names starting with this prefix.
+                example: sourcehostname
+    consistency:
+        desc: |
+            Consistency checks to run after backups. See
+            https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check for details.
+        map:
+            checks:
+                seq:
+                    - type: str
+                      enum: ['repository', 'archives', 'disabled']
+                      unique: True
+                desc: |
+                    List of consistency checks to run: "repository", "archives", or both. Defaults
+                    to both. Set to "disabled" to disable all consistency checks. See
+                    https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check for details.
+                example:
+                    - repository
+                    - archives
+            check_last:
+                type: int
+                desc: Restrict the number of checked archives to the last n.
+                example: 3

borgmatic/config/validate.py

@@ -0,0 +1,74 @@
+import logging
+import sys
+import warnings
+
+import pkg_resources
+import pykwalify.core
+import pykwalify.errors
+from ruamel import yaml
+
+
+def schema_filename():
+    '''
+    Path to the installed YAML configuration schema file, used to validate and parse the
+    configuration.
+    '''
+    return pkg_resources.resource_filename('borgmatic', 'config/schema.yaml')
+
+
+class Validation_error(ValueError):
+    '''
+    A collection of error message strings generated when attempting to validate a particular
+    configurartion file.
+    '''
+    def __init__(self, config_filename, error_messages):
+        self.config_filename = config_filename
+        self.error_messages = error_messages
+
+
+def parse_configuration(config_filename, schema_filename):
+    '''
+    Given the path to a config filename in YAML format and the path to a schema filename in
+    pykwalify YAML schema format, return the parsed configuration as a data structure of nested
+    dicts and lists corresponding to the schema. Example return value:
+
+       {'location': {'source_directories': ['/home', '/etc'], 'repository': 'hostname.borg'},
+       'retention': {'keep_daily': 7}, 'consistency': {'checks': ['repository', 'archives']}}
+
+    Raise FileNotFoundError if the file does not exist, PermissionError if the user does not
+    have permissions to read the file, or Validation_error if the config does not match the schema.
+    '''
+    try:
+        config = yaml.round_trip_load(open(config_filename))
+        schema = yaml.round_trip_load(open(schema_filename))
+    except yaml.error.YAMLError as error:
+        raise Validation_error(config_filename, (str(error),))
+
+    # pykwalify gets angry if the example field is not a string. So rather than bend to its will,
+    # simply remove all examples before passing the schema to pykwalify.
+    for section_name, section_schema in schema['map'].items():
+        for field_name, field_schema in section_schema['map'].items():
+            field_schema.pop('example')
+
+    validator = pykwalify.core.Core(source_data=config, schema_data=schema)
+    parsed_result = validator.validate(raise_exception=False)
+
+    if validator.validation_errors:
+        raise Validation_error(config_filename, validator.validation_errors)
+
+    return parsed_result
+
+
+def display_validation_error(validation_error):
+    '''
+    Given a Validation_error, display its error messages to stderr.
+    '''
+    print(
+        'An error occurred while parsing a configuration file at {}:'.format(
+            validation_error.config_filename
+        ),
+        file=sys.stderr,
+    )
+
+    for error in validation_error.error_messages:
+        print(error, file=sys.stderr)

borgmatic/tests/builtins.py

@@ -1,11 +0,0 @@
-from flexmock import flexmock
-import sys
-
-
-def builtins_mock():
-    try:
-        # Python 2
-        return flexmock(sys.modules['__builtin__'])
-    except KeyError:
-        # Python 3
-        return flexmock(sys.modules['builtins'])

borgmatic/tests/integration/commands/test_borgmatic.py

@@ -0,0 +1,35 @@
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.commands import borgmatic as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    parser = module.parse_arguments()
+
+    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
+    assert parser.excludes_filename == None
+    assert parser.verbosity is None
+
+
+def test_parse_arguments_with_filename_arguments_overrides_defaults():
+    parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
+
+    assert parser.config_filename == 'myconfig'
+    assert parser.excludes_filename == 'myexcludes'
+    assert parser.verbosity is None
+
+
+def test_parse_arguments_with_verbosity_flag_overrides_default():
+    parser = module.parse_arguments('--verbosity', '1')
+
+    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
+    assert parser.excludes_filename == None
+    assert parser.verbosity == 1
+
+
+def test_parse_arguments_with_invalid_arguments_exits():
+    with pytest.raises(SystemExit):
+        module.parse_arguments('--posix-me-harder')

borgmatic/tests/integration/commands/test_convert_config.py

@@ -0,0 +1,47 @@
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.commands import convert_config as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    parser = module.parse_arguments()
+
+    assert parser.source_config_filename == module.DEFAULT_SOURCE_CONFIG_FILENAME
+    assert parser.source_excludes_filename == module.DEFAULT_SOURCE_EXCLUDES_FILENAME
+    assert parser.destination_config_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+
+def test_parse_arguments_with_filename_arguments_overrides_defaults():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    parser = module.parse_arguments(
+        '--source-config', 'config',
+        '--source-excludes', 'excludes',
+        '--destination-config', 'config.yaml',
+    )
+
+    assert parser.source_config_filename == 'config'
+    assert parser.source_excludes_filename == 'excludes'
+    assert parser.destination_config_filename == 'config.yaml'
+
+
+def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_none():
+    flexmock(os.path).should_receive('exists').and_return(False)
+
+    parser = module.parse_arguments()
+
+    assert parser.source_config_filename == module.DEFAULT_SOURCE_CONFIG_FILENAME
+    assert parser.source_excludes_filename is None
+    assert parser.destination_config_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+
+def test_parse_arguments_with_invalid_arguments_exits():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    with pytest.raises(SystemExit):
+        module.parse_arguments('--posix-me-harder')

borgmatic/tests/integration/commands/test_generate_config.py

@@ -0,0 +1,12 @@
+from borgmatic.commands import generate_config as module
+
+
+def test_parse_arguments_with_no_arguments_uses_defaults():
+    parser = module.parse_arguments()
+
+    assert parser.destination_filename == module.DEFAULT_DESTINATION_CONFIG_FILENAME
+
+def test_parse_arguments_with_filename_argument_overrides_defaults():
+    parser = module.parse_arguments('--destination', 'config.yaml')
+
+    assert parser.destination_filename == 'config.yaml'

borgmatic/tests/integration/config/test_generate.py

@@ -0,0 +1,65 @@
+from io import StringIO
+import os
+import sys
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import generate as module
+
+
+def test_insert_newline_before_comment_does_not_raise():
+    field_name = 'foo'
+    config = module.yaml.comments.CommentedMap([(field_name, 33)])
+    config.yaml_set_comment_before_after_key(key=field_name, before='Comment',)
+
+    module._insert_newline_before_comment(config, field_name)
+
+
+def test_write_configuration_does_not_raise():
+    flexmock(os.path).should_receive('exists').and_return(False)
+    flexmock(os).should_receive('makedirs')
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').and_return(StringIO())
+    flexmock(os).should_receive('chmod')
+
+    module.write_configuration('config.yaml', {})
+
+
+def test_write_configuration_with_already_existing_file_raises():
+    flexmock(os.path).should_receive('exists').and_return(True)
+
+    with pytest.raises(FileExistsError):
+        module.write_configuration('config.yaml', {})
+
+
+def test_write_configuration_with_already_existing_directory_does_not_raise():
+    flexmock(os.path).should_receive('exists').and_return(False)
+    flexmock(os).should_receive('makedirs').and_raise(FileExistsError)
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').and_return(StringIO())
+    flexmock(os).should_receive('chmod')
+
+    module.write_configuration('config.yaml', {})
+
+
+def test_add_comments_to_configuration_does_not_raise():
+    # Ensure that it can deal with fields both in the schema and missing from the schema.
+    config = module.yaml.comments.CommentedMap([('foo', 33), ('bar', 44), ('baz', 55)])
+    schema = {
+        'map': {
+            'foo': {'desc': 'Foo'},
+            'bar': {'desc': 'Bar'},
+        }
+    }
+
+    module.add_comments_to_configuration(config, schema)
+
+
+def test_generate_sample_configuration_does_not_raise():
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_return('')
+    flexmock(module).should_receive('write_configuration')
+    flexmock(module).should_receive('_schema_to_sample_configuration')
+
+    module.generate_sample_configuration('config.yaml', 'schema.yaml')

borgmatic/tests/integration/config/test_legacy.py

@@ -1,14 +1,9 @@
-try:
+from io import StringIO
-    # Python 2
-    from cStringIO import StringIO
-except ImportError:
-    # Python 3
-    from io import StringIO
 
 from collections import OrderedDict
 import string
 
-from borgmatic import config as module
+from borgmatic.config import legacy as module
 
 
 def test_parse_section_options_with_punctuation_should_return_section_options():

borgmatic/tests/integration/config/test_validate.py

@@ -0,0 +1,123 @@
+import io
+import string
+import sys
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import validate as module
+
+
+def test_schema_filename_returns_plausable_path():
+    schema_path = module.schema_filename()    
+
+    assert schema_path.endswith('/schema.yaml')
+
+
+def mock_config_and_schema(config_yaml):
+    '''
+    Set up mocks for the config config YAML string and the default schema so that the code under
+    test consumes them when parsing the configuration.
+    '''
+    config_stream = io.StringIO(config_yaml)
+    schema_stream = open(module.schema_filename())
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('config.yaml').and_return(config_stream)
+    builtins.should_receive('open').with_args('schema.yaml').and_return(schema_stream)
+
+
+def test_parse_configuration_transforms_file_into_mapping():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+                - /etc
+
+            repositories:
+                - hostname.borg
+
+        retention:
+            keep_daily: 7
+
+        consistency:
+            checks:
+                - repository
+                - archives
+        '''
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {'source_directories': ['/home', '/etc'], 'repositories': ['hostname.borg']},
+        'retention': {'keep_daily': 7},
+        'consistency': {'checks': ['repository', 'archives']},
+    }
+
+
+def test_parse_configuration_passes_through_quoted_punctuation():
+    escaped_punctuation = string.punctuation.replace('\\', r'\\').replace('"', r'\"')
+
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories:
+                - /home
+
+            repositories:
+                - "{}.borg"
+        '''.format(escaped_punctuation)
+    )
+
+    result = module.parse_configuration('config.yaml', 'schema.yaml')
+
+    assert result == {
+        'location': {
+            'source_directories': ['/home'],
+            'repositories': ['{}.borg'.format(string.punctuation)],
+        },
+    }
+
+
+def test_parse_configuration_raises_for_missing_config_file():
+    with pytest.raises(FileNotFoundError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_missing_schema_file():
+    mock_config_and_schema('')
+    builtins = flexmock(sys.modules['builtins'])
+    builtins.should_receive('open').with_args('schema.yaml').and_raise(FileNotFoundError)
+
+    with pytest.raises(FileNotFoundError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_syntax_error():
+    mock_config_and_schema('foo:\nbar')
+
+    with pytest.raises(ValueError):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_parse_configuration_raises_for_validation_error():
+    mock_config_and_schema(
+        '''
+        location:
+            source_directories: yes
+            repositories:
+                - hostname.borg
+        '''
+    )
+
+    with pytest.raises(module.Validation_error):
+        module.parse_configuration('config.yaml', 'schema.yaml')
+
+
+def test_display_validation_error_does_not_raise():
+    flexmock(sys.modules['builtins']).should_receive('print')
+    error = module.Validation_error('config.yaml', ('oops', 'uh oh'))
+
+    module.display_validation_error(error)

borgmatic/tests/integration/test_command.py

@@ -1,69 +0,0 @@
-import os
-import sys
-
-from flexmock import flexmock
-import pytest
-
-from borgmatic import command as module
-
-
-def test_parse_arguments_with_no_arguments_uses_defaults():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments()
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_filename_arguments_overrides_defaults():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
-
-    assert parser.config_filename == 'myconfig'
-    assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_missing_default_excludes_file_sets_filename_to_none():
-    flexmock(os.path).should_receive('exists').and_return(False)
-
-    parser = module.parse_arguments()
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == None
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_missing_overridden_excludes_file_retains_filename():
-    flexmock(os.path).should_receive('exists').and_return(False)
-
-    parser = module.parse_arguments('--excludes', 'myexcludes')
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity == None
-
-
-def test_parse_arguments_with_verbosity_flag_overrides_default():
-    flexmock(os.path).should_receive('exists').and_return(True)
-
-    parser = module.parse_arguments('--verbosity', '1')
-
-    assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
-    assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME
-    assert parser.verbosity == 1
-
-
-def test_parse_arguments_with_invalid_arguments_exits():
-    flexmock(os.path).should_receive('exists').and_return(True)
-    original_stderr = sys.stderr
-    sys.stderr = sys.stdout
-
-    try:
-        with pytest.raises(SystemExit):
-            module.parse_arguments('--posix-me-harder')
-    finally:
-        sys.stderr = original_stderr

borgmatic/tests/unit/config/test_convert.py

@@ -0,0 +1,114 @@
+from collections import defaultdict, OrderedDict, namedtuple
+import os
+
+from flexmock import flexmock
+import pytest
+
+from borgmatic.config import convert as module
+
+
+Parsed_config = namedtuple('Parsed_config', ('location', 'storage', 'retention', 'consistency'))
+
+
+def test_convert_section_generates_integer_value_for_integer_type_in_schema():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_section_config = OrderedDict([('check_last', '3')])
+    section_schema = {'map': {'check_last': {'type': 'int'}}}
+
+    destination_config = module._convert_section(source_section_config, section_schema)
+
+    assert destination_config == OrderedDict([('check_last', 3)])
+
+
+def test_convert_legacy_parsed_config_transforms_source_config_to_mapping():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_config = Parsed_config(
+        location=OrderedDict([('source_directories', '/home'), ('repository', 'hostname.borg')]),
+        storage=OrderedDict([('encryption_passphrase', 'supersecret')]),
+        retention=OrderedDict([('keep_daily', 7)]),
+        consistency=OrderedDict([('checks', 'repository')]),
+    )
+    source_excludes = ['/var']
+    schema = {'map': defaultdict(lambda: {'map': {}})}
+
+    destination_config = module.convert_legacy_parsed_config(source_config, source_excludes, schema)
+
+    assert destination_config == OrderedDict([
+        (
+            'location',
+            OrderedDict([
+                ('source_directories', ['/home']),
+                ('repositories', ['hostname.borg']),
+                ('exclude_patterns', ['/var']),
+            ]),
+        ),
+        ('storage', OrderedDict([('encryption_passphrase', 'supersecret')])),
+        ('retention', OrderedDict([('keep_daily', 7)])),
+        ('consistency', OrderedDict([('checks', ['repository'])])),
+    ])
+
+
+def test_convert_legacy_parsed_config_splits_space_separated_values():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    source_config = Parsed_config(
+        location=OrderedDict([('source_directories', '/home /etc'), ('repository', 'hostname.borg')]),
+        storage=OrderedDict(),
+        retention=OrderedDict(),
+        consistency=OrderedDict([('checks', 'repository archives')]),
+    )
+    source_excludes = ['/var']
+    schema = {'map': defaultdict(lambda: {'map': {}})}
+
+    destination_config = module.convert_legacy_parsed_config(source_config, source_excludes, schema) 
+
+    assert destination_config == OrderedDict([
+        (
+            'location',
+            OrderedDict([
+                ('source_directories', ['/home', '/etc']),
+                ('repositories', ['hostname.borg']),
+                ('exclude_patterns', ['/var']),
+            ]),
+        ),
+        ('storage', OrderedDict()),
+        ('retention', OrderedDict()),
+        ('consistency', OrderedDict([('checks', ['repository', 'archives'])])),
+    ])
+
+
+def test_guard_configuration_upgraded_raises_when_only_source_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(True)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+
+    with pytest.raises(module.LegacyConfigurationNotUpgraded):
+        module.guard_configuration_upgraded('config', 'config.yaml')
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_only_destination_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(True)
+
+    module.guard_configuration_upgraded('config', 'config.yaml')
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_both_configs_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(True)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(True)
+
+    module.guard_configuration_upgraded('config', 'config.yaml')
+
+
+def test_guard_configuration_upgraded_does_not_raise_when_neither_config_present():
+    flexmock(os.path).should_receive('exists').with_args('config').and_return(False)
+    flexmock(os.path).should_receive('exists').with_args('config.yaml').and_return(False)
+
+    module.guard_configuration_upgraded('config', 'config.yaml')
+
+
+def test_guard_excludes_filename_omitted_raises_when_filename_provided():
+    with pytest.raises(module.LegacyExcludesFilenamePresent):
+        module.guard_excludes_filename_omitted(excludes_filename='/etc/borgmatic/excludes')
+
+
+def test_guard_excludes_filename_omitted_does_not_raise_when_filename_not_provided():
+    module.guard_excludes_filename_omitted(excludes_filename=None)

borgmatic/tests/unit/config/test_generate.py

@@ -0,0 +1,49 @@
+from collections import OrderedDict
+
+from flexmock import flexmock
+
+from borgmatic.config import generate as module
+
+
+def test_schema_to_sample_configuration_generates_config_with_examples():
+    flexmock(module.yaml.comments).should_receive('CommentedMap').replace_with(OrderedDict)
+    flexmock(module).should_receive('add_comments_to_configuration')
+    schema = {
+        'map': OrderedDict([
+            (
+                'section1', {
+                    'map': {
+                        'field1': OrderedDict([
+                            ('example', 'Example 1')
+                        ]),
+                    },
+                },
+            ),
+            (
+                'section2', {
+                    'map': OrderedDict([
+                        ('field2', {'example': 'Example 2'}),
+                        ('field3', {'example': 'Example 3'}),
+                    ]),
+                }
+            ),
+        ])
+    }
+
+    config = module._schema_to_sample_configuration(schema)
+
+    assert config == OrderedDict([
+        (
+            'section1',
+            OrderedDict([
+                ('field1', 'Example 1'),
+            ]),
+        ),
+        (
+            'section2',
+            OrderedDict([
+                ('field2', 'Example 2'),
+                ('field3', 'Example 3'),
+            ]),
+        )
+    ])

borgmatic/tests/unit/config/test_legacy.py

@@ -3,7 +3,7 @@ from collections import OrderedDict
 from flexmock import flexmock
 import pytest
 
-from borgmatic import config as module
+from borgmatic.config import legacy as module
 
 
 def test_option_should_create_config_option():

borgmatic/tests/unit/test_borg.py

@@ -1,11 +1,11 @@
 from collections import OrderedDict
 from subprocess import STDOUT
+import sys
 import os
 
 from flexmock import flexmock
 
 from borgmatic import borg as module
-from borgmatic.tests.builtins import builtins_mock
 from borgmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
 
 
@@ -14,8 +14,8 @@ def test_initialize_with_passphrase_should_set_environment():
 
     try:
         os.environ = {}
-        module.initialize({'encryption_passphrase': 'pass'}, command='attic')
+        module.initialize({'encryption_passphrase': 'pass'}, command='borg')
-        assert os.environ.get('ATTIC_PASSPHRASE') == 'pass'
+        assert os.environ.get('BORG_PASSPHRASE') == 'pass'
     finally:
         os.environ = orig_environ
 
@@ -25,11 +25,25 @@ def test_initialize_without_passphrase_should_not_set_environment():
 
     try:
         os.environ = {}
-        module.initialize({}, command='attic')
+        module.initialize({}, command='borg')
-        assert os.environ.get('ATTIC_PASSPHRASE') == None
+        assert os.environ.get('BORG_PASSPHRASE') == None
     finally:
         os.environ = orig_environ
 
+def test_write_exclude_file_does_not_raise():
+    temporary_file = flexmock(
+        name='filename',
+        write=lambda mode: None,
+        flush=lambda: None,
+    )
+    flexmock(module.tempfile).should_receive('NamedTemporaryFile').and_return(temporary_file)
+
+    module._write_exclude_file(['exclude'])
+
+
+def test_write_exclude_file_with_empty_exclude_patterns_does_not_raise():
+    module._write_exclude_file([])
+
 
 def insert_subprocess_mock(check_call_command, **kwargs):
     subprocess = flexmock(STDOUT=STDOUT)
@@ -53,176 +67,220 @@ def insert_datetime_mock():
     ).mock
 
 
-CREATE_COMMAND_WITHOUT_EXCLUDES = ('attic', 'create', 'repo::host-now', 'foo', 'bar')
+CREATE_COMMAND = ('borg', 'create', 'repo::host-now', 'foo', 'bar')
-CREATE_COMMAND = CREATE_COMMAND_WITHOUT_EXCLUDES + ('--exclude-from', 'excludes')
 
 
-def test_create_archive_should_call_attic_with_parameters():
+def test_create_archive_should_call_borg_with_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND)
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_two_spaces_in_source_directories():
+def test_create_archive_with_exclude_patterns_should_call_borg_with_excludes():
-    insert_subprocess_mock(CREATE_COMMAND)
+    flexmock(module).should_receive('_write_exclude_file').and_return(flexmock(name='excludes'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--exclude-from', 'excludes'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo  bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': ['exclude'],
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_none_excludes_filename_should_call_attic_without_excludes():
+def test_create_archive_with_verbosity_some_should_call_borg_with_info_parameter():
-    insert_subprocess_mock(CREATE_COMMAND_WITHOUT_EXCLUDES)
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(CREATE_COMMAND + ('--info', '--stats',))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename=None,
-        verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
-        repository='repo',
-        command='attic',
-    )
-
-
-def test_create_archive_with_verbosity_some_should_call_attic_with_stats_parameter():
-    insert_subprocess_mock(CREATE_COMMAND + ('--stats',))
-    insert_platform_mock()
-    insert_datetime_mock()
-
-    module.create_archive(
-        excludes_filename='excludes',
         verbosity=VERBOSITY_SOME,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_verbosity_lots_should_call_attic_with_verbose_parameter():
+def test_create_archive_with_verbosity_lots_should_call_borg_with_debug_parameter():
-    insert_subprocess_mock(CREATE_COMMAND + ('--verbose', '--stats'))
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(CREATE_COMMAND + ('--debug', '--list', '--stats'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=VERBOSITY_LOTS,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_compression_should_call_attic_with_compression_parameters():
+def test_create_archive_with_compression_should_call_borg_with_compression_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--compression', 'rle'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={'compression': 'rle'},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'compression': 'rle'},
+        command='borg',
     )
 
 
-def test_create_archive_with_one_file_system_should_call_attic_with_one_file_system_parameters():
+def test_create_archive_with_one_file_system_should_call_borg_with_one_file_system_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--one-file-system',))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
-        one_file_system=True,
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'one_file_system': True,
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_umask_should_call_attic_with_umask_parameters():
+def test_create_archive_with_remote_path_should_call_borg_with_remote_path_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(CREATE_COMMAND + ('--remote-path', 'borg1'))
+    insert_platform_mock()
+    insert_datetime_mock()
+
+    module.create_archive(
+        verbosity=None,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'remote_path': 'borg1',
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
+    )
+
+
+def test_create_archive_with_umask_should_call_borg_with_umask_parameters():
+    flexmock(module).should_receive('_write_exclude_file')
     insert_subprocess_mock(CREATE_COMMAND + ('--umask', '740'))
     insert_platform_mock()
     insert_datetime_mock()
 
     module.create_archive(
-        excludes_filename='excludes',
         verbosity=None,
-        storage_config={'umask': 740},
-        source_directories='foo bar',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'umask': 740},
+        command='borg',
     )
 
 
 def test_create_archive_with_source_directories_glob_expands():
-    insert_subprocess_mock(('attic', 'create', 'repo::host-now', 'foo', 'food'))
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo', 'food'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
 def test_create_archive_with_non_matching_source_directories_glob_passes_through():
-    insert_subprocess_mock(('attic', 'create', 'repo::host-now', 'foo*'))
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo*'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return([])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return([])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
-def test_create_archive_with_glob_should_call_attic_with_expanded_directories():
+def test_create_archive_with_glob_should_call_borg_with_expanded_directories():
-    insert_subprocess_mock(('attic', 'create', 'repo::host-now', 'foo', 'food'))
+    flexmock(module).should_receive('_write_exclude_file')
+    insert_subprocess_mock(('borg', 'create', 'repo::host-now', 'foo', 'food'))
     insert_platform_mock()
     insert_datetime_mock()
-    flexmock(module).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
+    flexmock(module.glob).should_receive('glob').with_args('foo*').and_return(['foo', 'food'])
 
     module.create_archive(
-        excludes_filename=None,
         verbosity=None,
-        storage_config={},
-        source_directories='foo*',
         repository='repo',
-        command='attic',
+        location_config={
+            'source_directories': ['foo*'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={},
+        command='borg',
     )
 
 
@@ -248,11 +306,11 @@ def test_make_prune_flags_should_return_flags_from_config():
 
 
 PRUNE_COMMAND = (
-    'attic', 'prune', 'repo', '--keep-daily', '1', '--keep-weekly', '2', '--keep-monthly', '3',
+    'borg', 'prune', 'repo', '--keep-daily', '1', '--keep-weekly', '2', '--keep-monthly', '3',
 )
 
 
-def test_prune_archives_should_call_attic_with_parameters():
+def test_prune_archives_should_call_borg_with_parameters():
     retention_config = flexmock()
     flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
         BASE_PRUNE_FLAGS,
@@ -263,42 +321,57 @@ def test_prune_archives_should_call_attic_with_parameters():
         verbosity=None,
         repository='repo',
         retention_config=retention_config,
-        command='attic',
+        command='borg',
     )
 
 
-def test_prune_archives_with_verbosity_some_should_call_attic_with_stats_parameter():
+def test_prune_archives_with_verbosity_some_should_call_borg_with_info_parameter():
     retention_config = flexmock()
     flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
         BASE_PRUNE_FLAGS,
     )
-    insert_subprocess_mock(PRUNE_COMMAND + ('--stats',))
+    insert_subprocess_mock(PRUNE_COMMAND + ('--info', '--stats',))
 
     module.prune_archives(
         repository='repo',
         verbosity=VERBOSITY_SOME,
         retention_config=retention_config,
-        command='attic',
+        command='borg',
     )
 
 
-def test_prune_archives_with_verbosity_lots_should_call_attic_with_verbose_parameter():
+def test_prune_archives_with_verbosity_lots_should_call_borg_with_debug_parameter():
     retention_config = flexmock()
     flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
         BASE_PRUNE_FLAGS,
     )
-    insert_subprocess_mock(PRUNE_COMMAND + ('--verbose', '--stats',))
+    insert_subprocess_mock(PRUNE_COMMAND + ('--debug', '--stats',))
 
     module.prune_archives(
         repository='repo',
         verbosity=VERBOSITY_LOTS,
         retention_config=retention_config,
-        command='attic',
+        command='borg',
+    )
+
+def test_prune_archive_with_remote_path_should_call_borg_with_remote_path_parameters():
+    retention_config = flexmock()
+    flexmock(module).should_receive('_make_prune_flags').with_args(retention_config).and_return(
+        BASE_PRUNE_FLAGS,
+    )
+    insert_subprocess_mock(PRUNE_COMMAND + ('--remote-path', 'borg1'))
+
+    module.prune_archives(
+        verbosity=None,
+        repository='repo',
+        retention_config=retention_config,
+        command='borg',
+        remote_path='borg1',
     )
 
 
 def test_parse_checks_returns_them_as_tuple():
-    checks = module._parse_checks({'checks': 'foo disabled bar'})
+    checks = module._parse_checks({'checks': ['foo', 'disabled', 'bar']})
 
     assert checks == ('foo', 'bar')
 
@@ -310,13 +383,13 @@ def test_parse_checks_with_missing_value_returns_defaults():
 
 
 def test_parse_checks_with_blank_value_returns_defaults():
-    checks = module._parse_checks({'checks': ''})
+    checks = module._parse_checks({'checks': []})
 
     assert checks == module.DEFAULT_CHECKS
 
 
 def test_parse_checks_with_disabled_returns_no_checks():
-    checks = module._parse_checks({'checks': 'disabled'})
+    checks = module._parse_checks({'checks': ['disabled']})
 
     assert checks == ()
 
@@ -336,16 +409,16 @@ def test_make_check_flags_with_default_checks_returns_no_flags():
 def test_make_check_flags_with_checks_and_last_returns_flags_including_last():
     flags = module._make_check_flags(('foo', 'bar'), check_last=3)
 
-    assert flags == ('--foo-only', '--bar-only', '--last', 3)
+    assert flags == ('--foo-only', '--bar-only', '--last', '3')
 
 
 def test_make_check_flags_with_last_returns_last_flag():
     flags = module._make_check_flags(module.DEFAULT_CHECKS, check_last=3)
 
-    assert flags == ('--last', 3)
+    assert flags == ('--last', '3')
 
 
-def test_check_archives_should_call_attic_with_parameters():
+def test_check_archives_should_call_borg_with_parameters():
     checks = flexmock()
     check_last = flexmock()
     consistency_config = flexmock().should_receive('get').and_return(check_last).mock
@@ -353,28 +426,28 @@ def test_check_archives_should_call_attic_with_parameters():
     flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last).and_return(())
     stdout = flexmock()
     insert_subprocess_mock(
-        ('attic', 'check', 'repo'),
+        ('borg', 'check', 'repo'),
         stdout=stdout, stderr=STDOUT,
     )
     insert_platform_mock()
     insert_datetime_mock()
-    builtins_mock().should_receive('open').and_return(stdout)
+    flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
     flexmock(module.os).should_receive('devnull')
 
     module.check_archives(
         verbosity=None,
         repository='repo',
         consistency_config=consistency_config,
-        command='attic',
+        command='borg',
     )
 
 
-def test_check_archives_with_verbosity_some_should_call_attic_with_verbose_parameter():
+def test_check_archives_with_verbosity_some_should_call_borg_with_info_parameter():
     consistency_config = flexmock().should_receive('get').and_return(None).mock
     flexmock(module).should_receive('_parse_checks').and_return(flexmock())
     flexmock(module).should_receive('_make_check_flags').and_return(())
     insert_subprocess_mock(
-        ('attic', 'check', 'repo', '--verbose'),
+        ('borg', 'check', 'repo', '--info'),
         stdout=None, stderr=STDOUT,
     )
     insert_platform_mock()
@@ -384,16 +457,16 @@ def test_check_archives_with_verbosity_some_should_call_attic_with_verbose_param
         verbosity=VERBOSITY_SOME,
         repository='repo',
         consistency_config=consistency_config,
-        command='attic',
+        command='borg',
     )
 
 
-def test_check_archives_with_verbosity_lots_should_call_attic_with_verbose_parameter():
+def test_check_archives_with_verbosity_lots_should_call_borg_with_debug_parameter():
     consistency_config = flexmock().should_receive('get').and_return(None).mock
     flexmock(module).should_receive('_parse_checks').and_return(flexmock())
     flexmock(module).should_receive('_make_check_flags').and_return(())
     insert_subprocess_mock(
-        ('attic', 'check', 'repo', '--verbose'),
+        ('borg', 'check', 'repo', '--debug'),
         stdout=None, stderr=STDOUT,
     )
     insert_platform_mock()
@@ -403,7 +476,7 @@ def test_check_archives_with_verbosity_lots_should_call_attic_with_verbose_param
         verbosity=VERBOSITY_LOTS,
         repository='repo',
         consistency_config=consistency_config,
-        command='attic',
+        command='borg',
     )
 
 
@@ -416,5 +489,30 @@ def test_check_archives_without_any_checks_should_bail():
         verbosity=None,
         repository='repo',
         consistency_config=consistency_config,
-        command='attic',
+        command='borg',
+    )
+
+
+def test_check_archives_with_remote_path_should_call_borg_with_remote_path_parameters():
+    checks = flexmock()
+    check_last = flexmock()
+    consistency_config = flexmock().should_receive('get').and_return(check_last).mock
+    flexmock(module).should_receive('_parse_checks').and_return(checks)
+    flexmock(module).should_receive('_make_check_flags').with_args(checks, check_last).and_return(())
+    stdout = flexmock()
+    insert_subprocess_mock(
+        ('borg', 'check', 'repo', '--remote-path', 'borg1'),
+        stdout=stdout, stderr=STDOUT,
+    )
+    insert_platform_mock()
+    insert_datetime_mock()
+    flexmock(sys.modules['builtins']).should_receive('open').and_return(stdout)
+    flexmock(module.os).should_receive('devnull')
+
+    module.check_archives(
+        verbosity=None,
+        repository='repo',
+        consistency_config=consistency_config,
+        command='borg',
+        remote_path='borg1',
     )

sample/config

@@ -1,42 +0,0 @@
-[location]
-# Space-separated list of source directories to backup.
-# Globs are expanded.
-source_directories: /home /etc /var/log/syslog*
-
-# For Borg only, you can specify to stay in same file system (do not cross
-# mount points).
-#one_file_system: True
-
-# Path to local or remote repository.
-repository: user@backupserver:sourcehostname.borg
-
-[storage]
-# Passphrase to unlock the encryption key with. Only use on repositories that
-# were initialized with passphrase/repokey encryption.
-#encryption_passphrase: foo
-# Type of compression to use when creating archives. See
-# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create
-# for details. Defaults to no compression.
-#compression: lz4
-# Umask to be used for borg create.
-#umask: 0740
-
-[retention]
-# Retention policy for how many backups to keep in each category. See
-# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
-#keep_within: 3H
-#keep_hourly: 24
-keep_daily: 7
-keep_weekly: 4
-keep_monthly: 6
-keep_yearly: 1
-#prefix: sourcehostname
-
-[consistency]
-# Space-separated list of consistency checks to run: "repository", "archives",
-# or both. Defaults to both. Set to "disabled" to disable all consistency
-# checks. See https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check
-# for details.
-checks: repository archives
-# Restrict the number of checked archives to the last n.
-#check_last: 3

sample/excludes

@@ -1,3 +0,0 @@
-*.pyc
-/home/*/.cache
-/etc/ssl

sample/systemd/borgmatic.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=borgmatic backup
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/borgmatic

sample/systemd/borgmatic.timer

@@ -0,0 +1,8 @@
+[Unit]
+Description=Run borgmatic backup
+
+[Timer]
+OnCalendar=daily
+
+[Install]
+WantedBy=timers.target

setup.py

@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 
 
-VERSION = '1.0.0'
+VERSION = '1.1.2'
 
 
 setup(
@@ -24,14 +24,22 @@ setup(
     packages=find_packages(),
     entry_points={
         'console_scripts': [
-            'borgmatic = borgmatic.command:main',
+            'borgmatic = borgmatic.commands.borgmatic:main',
+            'upgrade-borgmatic-config = borgmatic.commands.convert_config:main',
+            'generate-borgmatic-config = borgmatic.commands.generate_config:main',
         ]
     },
-    obsoletes=(
+    obsoletes=[
         'atticmatic',
+    ],
+    install_requires=(
+        'pykwalify',
+        'ruamel.yaml<=0.15',
+        'setuptools',
     ),
     tests_require=(
         'flexmock',
         'pytest',
-    )
+    ),
+    include_package_data=True,
 )

static/borgmatic.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" x="0px" y="0px" viewBox="0 0 100 100" enable-background="new 0 0 100 100" xml:space="preserve"><g><circle cx="63.461" cy="59.615" r="5.626"></circle><path d="M81.432,48.015c0.779-10.849-1.758-19.894-7.407-26.293C68.447,15.403,59.915,11.923,50,11.923   c-6.729,0-12.834,1.645-17.81,4.724c-0.405-0.06-0.823-0.073-1.248,0.005C8.128,20.817,0,34.623,0,46.154   c0,6.155,1.856,11.005,5.519,14.417c2.082,1.941,4.461,3.223,6.659,3.921C13.041,74.256,18.345,81,23.077,81h1.438   C30.168,94,44.847,99.614,50,99.614c6.708,0,24.493-7.875,27.38-25.392c4.758-1.888,10.697-7.373,10.697-13.711   C88.077,54.639,85.395,49.855,81.432,48.015z M7.691,46.154c0-3.397,1.238-13.399,14.747-19.083   c-2.043,3.821-3.31,8.234-3.771,13.136c-3.06,2.521-5.577,7.792-6.366,15.873c-0.534-0.316-1.056-0.686-1.538-1.135   C8.725,53.045,7.691,50.088,7.691,46.154z M74.23,67.947c-1.806,0-3.309,1.389-3.451,3.188C69.608,85.928,53.914,92.575,50,92.575   c-3.275,0-14.607-4.474-19.03-14.325c1.001-0.709,2.168-1.25,3.646-1.25c3.809,0,3.847,0,3.847,0c2.126,0,7.692-4.023,7.692-19.494   C46.154,46.241,40.588,42,38.462,42c0,0-0.195,0-3.847,0c-3.65,0-5.566-4-7.692-4h-1.046c0.813-4,2.646-8.756,5.473-11.94   c4.321-4.87,10.771-7.329,18.65-7.329c8.016,0,14.528,2.636,18.835,7.515c4.821,5.462,6.697,13.762,5.424,23.95   c-0.123,0.984,0.183,1.99,0.84,2.734s1.602,1.179,2.595,1.179c1.671,0,3.46,2.562,3.46,6.371   C81.153,64.016,75.754,67.812,74.23,67.947z M30.432,59.615c0-3.377,2.734-6.107,6.106-6.107c3.373,0,6.107,2.73,6.107,6.107   s-2.734,6.107-6.107,6.107C33.166,65.723,30.432,62.992,30.432,59.615z"></path></g></svg>

test_requirements.txt

@@ -1,2 +1,5 @@
 flexmock==0.10.2
+pykwalify==1.6.0
 pytest==2.9.1
+pytest-cov==2.5.1
+ruamel.yaml==0.15.18

tox.ini

@@ -1,8 +1,8 @@
 [tox]
-envlist=py27,py34
+envlist=py34
 skipsdist=True
 
 [testenv]
 usedevelop=True
 deps=-rtest_requirements.txt
-commands = py.test []
+commands = py.test --cov-report term-missing:skip-covered --cov=borgmatic borgmatic []