Fix duplicate issue number.

NEWS

@@ -1,3 +1,23 @@
+1.2.13
+ * #100: Support for --stats command-line flag independent of --verbosity.
+ * #117: With borgmatic --init command-line flag, proceed without erroring if a repository already
+   exists.
+
+1.2.12
+ * #110: Support for Borg repository initialization via borgmatic --init command-line flag.
+ * #111: Update Borg create --filter values so a dry run lists files to back up.
+ * #113: Update README with link to a new/forked Docker image.
+ * Prevent deprecated --excludes command-line option from being used.
+ * Refactor README a bit to flow better for first-time users.
+ * Update README with a few additional borgmatic packages (Debian and Ubuntu).
+
+1.2.11
+ * #108: Support for Borg create --progress via borgmatic command-line flag.
+
+1.2.10
+ * #105: Support for Borg --chunker-params create option via "chunker_params" in borgmatic's storage
+   section.
+
 1.2.9
  * #102: Fix for syntax error that occurred in Python 3.5 and below.
  * Make automated tests support running in Python 3.5.

README.md

@@ -58,28 +58,9 @@ href="https://asciinema.org/a/203761" target="_blank">screencast</a>.
 
 To get up and running, first [install
 Borg](https://borgbackup.readthedocs.io/en/latest/installation.html), at
-least version 1.1. Then, follow the [Borg Quick
-Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) to create
-a repository on a local or remote host.
+least version 1.1.
 
-Note that if you plan to run borgmatic on a schedule with cron, and you
-encrypt your Borg repository with a passphrase instead of a key file, you'll
-either need to set the borgmatic `encryption_passphrase` configuration
-variable or set the `BORG_PASSPHRASE` environment variable. See the
-[repository encryption
-section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
-of the Quick Start for more info.
-
-Alternatively, the passphrase can be specified programatically by setting
-either the borgmatic `encryption_passcommand` configuration variable or the
-`BORG_PASSCOMMAND` environment variable. See the [Borg Security
-FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
-for more info.
-
-If the repository is on a remote host, make sure that your local root user has
-key-based ssh access to the desired user account on the remote host.
-
-To install borgmatic, run the following command to download and install it:
+Then, run the following command to download and install borgmatic:
 
 ```bash
 sudo pip3 install --upgrade borgmatic
@@ -88,19 +69,19 @@ sudo pip3 install --upgrade borgmatic
 Note that your pip binary may have a different name than "pip3". Make sure
 you're using Python 3, as borgmatic does not support Python 2.
 
+
 ### Other ways to install
 
- * [A borgmatic Docker image](https://hub.docker.com/r/b3vis/borgmatic/) based
-   on Alpine Linux.
- * [Another borgmatic Docker image](https://hub.docker.com/r/coaxial/borgmatic/) based
-   on Alpine Linux, updated automatically whenever the Alpine image updates.
- * [A borgmatic package for
-   Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic).
- * [A borgmatic package for Arch
-   Linux](https://aur.archlinux.org/packages/borgmatic/).
- * [A borgmatic package for OpenBSD](http://ports.su/sysutils/borgmatic).
+ * [Docker image](https://hub.docker.com/r/monachus/borgmatic/)
+ * [Another Docker image](https://hub.docker.com/r/b3vis/borgmatic/)
+ * [Debian](https://tracker.debian.org/pkg/borgmatic)
+ * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
+ * [Fedora](https://bodhi.fedoraproject.org/updates/?search=borgmatic)
+ * [Arch Linux](https://aur.archlinux.org/packages/borgmatic/)
+ * [OpenBSD](http://ports.su/sysutils/borgmatic)
 <br><br>
 
+
 ## Configuration
 
 After you install borgmatic, generate a sample configuration file:
@@ -124,6 +105,165 @@ borgmatic has added new options since you originally created your
 configuration file.
 
 
+### Encryption
+
+Note that if you plan to run borgmatic on a schedule with cron, and you
+encrypt your Borg repository with a passphrase instead of a key file, you'll
+either need to set the borgmatic `encryption_passphrase` configuration
+variable or set the `BORG_PASSPHRASE` environment variable. See the
+[repository encryption
+section](https://borgbackup.readthedocs.io/en/latest/quickstart.html#repository-encryption)
+of the Quick Start for more info.
+
+Alternatively, the passphrase can be specified programatically by setting
+either the borgmatic `encryption_passcommand` configuration variable or the
+`BORG_PASSCOMMAND` environment variable. See the [Borg Security
+FAQ](http://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-specify-the-encryption-passphrase-programmatically)
+for more info.
+
+
+## Usage
+
+### Initialization
+
+Before you can create backups with borgmatic, you first need to initialize a
+Borg repository so you have a destination for your backup archives. (But skip
+this step if you already have a Borg repository.) To create a repository, run
+a command like the following:
+
+```bash
+borgmatic --init --encryption repokey
+```
+
+This uses the borgmatic configuration file you created above to determine
+which local or remote repository to create, and encrypts it with the
+encryption passphrase specified there if one is provided. Read about [Borg
+encryption
+modes](https://borgbackup.readthedocs.io/en/latest/usage/init.html#encryption-modes)
+for the menu of available encryption modes.
+
+Also, optionally check out the [Borg Quick
+Start](https://borgbackup.readthedocs.org/en/latest/quickstart.html) for more
+background about repository initialization.
+
+Note that borgmatic skips repository initialization if the repository already
+exists. This supports use cases like ensuring a repository exists prior to
+performing a backup.
+
+If the repository is on a remote host, make sure that your local user has
+key-based SSH access to the desired user account on the remote host.
+
+
+### Backups
+
+You can run borgmatic and start a backup simply by invoking it without
+arguments:
+
+```bash
+borgmatic
+```
+
+This will also prune any old backups as per the configured retention policy,
+and check backups for consistency problems due to things like file damage.
+
+If you'd like to see the available command-line arguments, view the help:
+
+```bash
+borgmatic --help
+```
+
+Note that borgmatic prunes archives *before* creating an archive, so as to
+free up space for archiving. This means that when a borgmatic run finishes,
+there may still be prune-able archives. Not to worry, as they will get cleaned
+up at the start of the next run.
+
+
+### Verbosity
+
+By default, the backup will proceed silently except in the case of errors. But
+if you'd like to to get additional information about the progress of the
+backup as it proceeds, use the verbosity option:
+
+```bash
+borgmatic --verbosity 1
+```
+
+Or, for even more progress spew:
+
+```bash
+borgmatic --verbosity 2
+```
+
+### À la carte
+
+If you want to run borgmatic with only pruning, creating, or checking enabled,
+the following optional flags are available:
+
+```bash
+borgmatic --prune
+borgmatic --create
+borgmatic --check
+```
+
+You can run with only one of these flags provided, or you can mix and match
+any number of them. This supports use cases like running consistency checks
+from a different cron job with a different frequency, or running pruning with
+a different verbosity level.
+
+Additionally, borgmatic provides convenient flags for Borg's
+[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
+[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
+functionality:
+
+
+```bash
+borgmatic --list
+borgmatic --info
+```
+
+You can include an optional `--json` flag with `--create`, `--list`, or
+`--info` to get the output formatted as JSON.
+
+
+## Autopilot
+
+If you want to run borgmatic automatically, say once a day, the you can
+configure a job runner to invoke it periodically.
+
+### cron
+
+If you're using cron, download the [sample cron
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
+Then, from the directory where you downloaded it:
+
+```bash
+sudo mv borgmatic /etc/cron.d/borgmatic
+sudo chmod +x /etc/cron.d/borgmatic
+```
+
+You can modify the cron file if you'd like to run borgmatic more or less frequently.
+
+### systemd
+
+If you're using systemd instead of cron to run jobs, download the [sample
+systemd service
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
+and the [sample systemd timer
+file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
+Then, from the directory where you downloaded them:
+
+```bash
+sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
+sudo systemctl enable borgmatic.timer
+sudo systemctl start borgmatic.timer
+```
+
+Feel free to modify the timer file based on how frequently you'd like
+borgmatic to run.
+
+
+## Advanced configuration
+
 ### Multiple configuration files
 
 A more advanced usage is to create multiple separate configuration files and
@@ -247,113 +387,6 @@ That's it! borgmatic will continue using your /etc/borgmatic configuration
 files.
 
 
-## Usage
-
-You can run borgmatic and start a backup simply by invoking it without
-arguments:
-
-```bash
-borgmatic
-```
-
-This will also prune any old backups as per the configured retention policy,
-and check backups for consistency problems due to things like file damage.
-
-If you'd like to see the available command-line arguments, view the help:
-
-```bash
-borgmatic --help
-```
-
-Note that borgmatic prunes archives *before* creating an archive, so as to
-free up space for archiving. This means that when a borgmatic run finishes,
-there may still be prune-able archives. Not to worry, as they will get cleaned
-up at the start of the next run.
-
-### Verbosity
-
-By default, the backup will proceed silently except in the case of errors. But
-if you'd like to to get additional information about the progress of the
-backup as it proceeds, use the verbosity option:
-
-```bash
-borgmatic --verbosity 1
-```
-
-Or, for even more progress spew:
-
-```bash
-borgmatic --verbosity 2
-```
-
-### À la carte
-
-If you want to run borgmatic with only pruning, creating, or checking enabled,
-the following optional flags are available:
-
-```bash
-borgmatic --prune
-borgmatic --create
-borgmatic --check
-```
-
-You can run with only one of these flags provided, or you can mix and match
-any number of them. This supports use cases like running consistency checks
-from a different cron job with a different frequency, or running pruning with
-a different verbosity level.
-
-Additionally, borgmatic provides convenient flags for Borg's
-[list](https://borgbackup.readthedocs.io/en/stable/usage/list.html) and
-[info](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
-functionality:
-
-
-```bash
-borgmatic --list
-borgmatic --info
-```
-
-You can include an optional `--json` flag with `--create`, `--list`, or
-`--info` to get the output formatted as JSON.
-
-
-## Autopilot
-
-If you want to run borgmatic automatically, say once a day, the you can
-configure a job runner to invoke it periodically.
-
-### cron
-
-If you're using cron, download the [sample cron
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
-Then, from the directory where you downloaded it:
-
-```bash
-sudo mv borgmatic /etc/cron.d/borgmatic
-sudo chmod +x /etc/cron.d/borgmatic
-```
-
-You can modify the cron file if you'd like to run borgmatic more or less frequently.
-
-### systemd
-
-If you're using systemd instead of cron to run jobs, download the [sample
-systemd service
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.service)
-and the [sample systemd timer
-file](https://projects.torsion.org/witten/borgmatic/src/master/sample/systemd/borgmatic.timer).
-Then, from the directory where you downloaded them:
-
-```bash
-sudo mv borgmatic.service borgmatic.timer /etc/systemd/system/
-sudo systemctl enable borgmatic.timer
-sudo systemctl start borgmatic.timer
-```
-
-Feel free to modify the timer file based on how frequently you'd like
-borgmatic to run.
-
-
 ## Support and contributing
 
 ### Issues

borgmatic/borg/create.py

@@ -9,20 +9,6 @@ import tempfile
 logger = logging.getLogger(__name__)
 
 
-def initialize_environment(storage_config):
-    passcommand = storage_config.get('encryption_passcommand')
-    if passcommand:
-        os.environ['BORG_PASSCOMMAND'] = passcommand
-
-    passphrase = storage_config.get('encryption_passphrase')
-    if passphrase:
-        os.environ['BORG_PASSPHRASE'] = passphrase
-
-    ssh_command = storage_config.get('ssh_command')
-    if ssh_command:
-        os.environ['BORG_RSH'] = ssh_command
-
-
 def _expand_directory(directory):
     '''
     Given a directory path, expand any tilde (representing a user's home directory) and any globs
@@ -104,6 +90,8 @@ def create_archive(
     storage_config,
     local_path='borg',
     remote_path=None,
+    progress=False,
+    stats=False,
     json=False,
 ):
     '''
@@ -115,6 +103,7 @@ def create_archive(
     pattern_file = _write_pattern_file(location_config.get('patterns'))
     exclude_file = _write_pattern_file(_expand_directories(location_config.get('exclude_patterns')))
     checkpoint_interval = storage_config.get('checkpoint_interval', None)
+    chunker_params = storage_config.get('chunker_params', None)
     compression = storage_config.get('compression', None)
     remote_rate_limit = storage_config.get('remote_rate_limit', None)
     umask = storage_config.get('umask', None)
@@ -135,6 +124,7 @@ def create_archive(
         + _make_pattern_flags(location_config, pattern_file.name if pattern_file else None)
         + _make_exclude_flags(location_config, exclude_file.name if exclude_file else None)
         + (('--checkpoint-interval', str(checkpoint_interval)) if checkpoint_interval else ())
+        + (('--chunker-params', chunker_params) if chunker_params else ())
         + (('--compression', compression) if compression else ())
         + (('--remote-ratelimit', str(remote_rate_limit)) if remote_rate_limit else ())
         + (('--one-file-system',) if location_config.get('one_file_system') else ())
@@ -144,11 +134,12 @@ def create_archive(
         + (('--remote-path', remote_path) if remote_path else ())
         + (('--umask', str(umask)) if umask else ())
         + (('--lock-wait', str(lock_wait)) if lock_wait else ())
-        + (('--list', '--filter', 'AME') if logger.isEnabledFor(logging.INFO) else ())
+        + (('--list', '--filter', 'AME-') if logger.isEnabledFor(logging.INFO) else ())
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
-        + (('--stats',) if not dry_run and logger.isEnabledFor(logging.INFO) else ())
+        + (('--stats',) if not dry_run and (logger.isEnabledFor(logging.INFO) or stats) else ())
         + (('--debug', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
+        + (('--progress',) if progress else ())
         + (('--json',) if json else ())
     )
 

borgmatic/borg/environment.py

@@ -0,0 +1,15 @@
+import os
+
+
+def initialize(storage_config):
+    passcommand = storage_config.get('encryption_passcommand')
+    if passcommand:
+        os.environ['BORG_PASSCOMMAND'] = passcommand
+
+    passphrase = storage_config.get('encryption_passphrase')
+    if passphrase:
+        os.environ['BORG_PASSPHRASE'] = passphrase
+
+    ssh_command = storage_config.get('ssh_command')
+    if ssh_command:
+        os.environ['BORG_RSH'] = ssh_command

borgmatic/borg/init.py

@@ -0,0 +1,39 @@
+import logging
+import subprocess
+
+
+logger = logging.getLogger(__name__)
+
+
+def initialize_repository(
+    repository,
+    encryption_mode,
+    append_only=None,
+    storage_quota=None,
+    local_path='borg',
+    remote_path=None,
+):
+    '''
+    Given a local or remote repository path, a Borg encryption mode, whether the repository should
+    be append-only, and the storage quota to use, initialize the repository. If the repository
+    already exists, then log and skip initialization.
+    '''
+    info_command = (local_path, 'info', repository)
+    logger.debug(' '.join(info_command))
+
+    if subprocess.call(info_command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:
+        logger.info('Repository already exists. Skipping initialization.')
+        return
+
+    init_command = (
+        (local_path, 'init', repository)
+        + (('--encryption', encryption_mode) if encryption_mode else ())
+        + (('--append-only',) if append_only else ())
+        + (('--storage-quota', storage_quota) if storage_quota else ())
+        + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
+        + (('--debug',) if logger.isEnabledFor(logging.DEBUG) else ())
+        + (('--remote-path', remote_path) if remote_path else ())
+    )
+
+    logger.debug(' '.join(init_command))
+    subprocess.check_call(init_command)

borgmatic/borg/prune.py

@@ -31,7 +31,13 @@ def _make_prune_flags(retention_config):
 
 
 def prune_archives(
-    dry_run, repository, storage_config, retention_config, local_path='borg', remote_path=None
+    dry_run,
+    repository,
+    storage_config,
+    retention_config,
+    local_path='borg',
+    remote_path=None,
+    stats=False,
 ):
     '''
     Given dry-run flag, a local or remote repository path, a storage config dict, and a
@@ -51,6 +57,7 @@ def prune_archives(
         + (('--info',) if logger.getEffectiveLevel() == logging.INFO else ())
         + (('--debug', '--list', '--show-rc') if logger.isEnabledFor(logging.DEBUG) else ())
         + (('--dry-run',) if dry_run else ())
+        + (('--stats',) if stats else ())
     )
 
     logger.debug(' '.join(full_command))

borgmatic/commands/borgmatic.py

@@ -8,9 +8,11 @@ import sys
 from borgmatic.borg import (
     check as borg_check,
     create as borg_create,
+    environment as borg_environment,
     prune as borg_prune,
     list as borg_list,
     info as borg_info,
+    init as borg_init,
 )
 from borgmatic.commands import hook
 from borgmatic.config import checks, collect, convert, validate
@@ -53,6 +55,26 @@ def parse_arguments(*arguments):
         dest='excludes_filename',
         help='Deprecated in favor of exclude_patterns within configuration',
     )
+    parser.add_argument(
+        '-I', '--init', dest='init', action='store_true', help='Initialize an empty Borg repository'
+    )
+    parser.add_argument(
+        '-e',
+        '--encryption',
+        dest='encryption_mode',
+        help='Borg repository encryption mode (for use with --init)',
+    )
+    parser.add_argument(
+        '--append-only',
+        dest='append_only',
+        action='store_true',
+        help='Create an append-only repository (for use with --init)',
+    )
+    parser.add_argument(
+        '--storage-quota',
+        dest='storage_quota',
+        help='Create a repository with a fixed storage quota (for use with --init)',
+    )
     parser.add_argument(
         '-p',
         '--prune',
@@ -78,6 +100,20 @@ def parse_arguments(*arguments):
         action='store_true',
         help='Display summary information on archives',
     )
+    parser.add_argument(
+        '--progress',
+        dest='progress',
+        default=False,
+        action='store_true',
+        help='Display progress with --create option for each file as it is backed up',
+    )
+    parser.add_argument(
+        '--stats',
+        dest='stats',
+        default=False,
+        action='store_true',
+        help='Display statistics of archive with --create or --prune option',
+    )
     parser.add_argument(
         '--json',
         dest='json',
@@ -103,6 +139,27 @@ def parse_arguments(*arguments):
 
     args = parser.parse_args(arguments)
 
+    if args.excludes_filename:
+        raise ValueError(
+            'The --excludes option has been replaced with exclude_patterns in configuration'
+        )
+
+    if (args.encryption_mode or args.append_only or args.storage_quota) and not args.init:
+        raise ValueError(
+            'The --encryption, --append-only, and --storage-quota options can only be used with the --init option'
+        )
+
+    if args.init and args.dry_run:
+        raise ValueError('The --init option cannot be used with the --dry-run option')
+    if args.init and not args.encryption_mode:
+        raise ValueError('The --encryption option is required with the --init option')
+
+    if args.progress and not args.create:
+        raise ValueError('The --progress option can only be used with the --create option')
+
+    if args.stats and not (args.create or args.prune):
+        raise ValueError('The --stats option can only be used with the --create or --prune options')
+
     if args.json and not (args.create or args.list or args.info):
         raise ValueError(
             'The --json option can only be used with the --create, --list, or --info options'
@@ -115,7 +172,7 @@ def parse_arguments(*arguments):
 
     # If any of the action flags are explicitly requested, leave them as-is. Otherwise, assume
     # defaults: Mutate the given arguments to enable the default actions.
-    if args.prune or args.create or args.check or args.list or args.info:
+    if args.init or args.prune or args.create or args.check or args.list or args.info:
         return args
 
     args.prune = True
@@ -139,7 +196,7 @@ def run_configuration(config_filename, args):  # pragma: no cover
     try:
         local_path = location.get('local_path', 'borg')
         remote_path = location.get('remote_path')
-        borg_create.initialize_environment(storage)
+        borg_environment.initialize(storage)
 
         if args.create:
             hook.execute_hook(hooks.get('before_backup'), config_filename, 'pre-backup')
@@ -193,6 +250,16 @@ def _run_commands_on_repository(
 ):  # pragma: no cover
     repository = os.path.expanduser(unexpanded_repository)
     dry_run_label = ' (dry run; not making any changes)' if args.dry_run else ''
+    if args.init:
+        logger.info('{}: Initializing repository'.format(repository))
+        borg_init.initialize_repository(
+            repository,
+            args.encryption_mode,
+            args.append_only,
+            args.storage_quota,
+            local_path=local_path,
+            remote_path=remote_path,
+        )
     if args.prune:
         logger.info('{}: Pruning archives{}'.format(repository, dry_run_label))
         borg_prune.prune_archives(
@@ -202,6 +269,7 @@ def _run_commands_on_repository(
             retention,
             local_path=local_path,
             remote_path=remote_path,
+            stats=args.stats,
         )
     if args.create:
         logger.info('{}: Creating archive{}'.format(repository, dry_run_label))
@@ -212,6 +280,8 @@ def _run_commands_on_repository(
             storage,
             local_path=local_path,
             remote_path=remote_path,
+            progress=args.progress,
+            stats=args.stats,
         )
     if args.check and checks.repository_enabled_for_checks(repository, consistency):
         logger.info('{}: Running consistency checks'.format(repository))

borgmatic/config/schema.yaml

@@ -138,6 +138,13 @@ map:
                     https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there
                     for details. Defaults to checkpoints every 1800 seconds (30 minutes).
                 example: 1800
+            chunker_params:
+                type: scalar
+                desc: |
+                    Specify the parameters passed to then chunker (CHUNK_MIN_EXP, CHUNK_MAX_EXP,
+                    HASH_MASK_BITS, HASH_WINDOW_SIZE). See https://borgbackup.readthedocs.io/en/stable/internals.html
+                    for details.
+                example: 19,23,21,4095
             compression:
                 type: scalar
                 desc: |

scripts/find-unsupported-borg-options

@@ -40,6 +40,8 @@ for sub_command in prune create check list info; do
             | grep -v '^--list$' \
             | grep -v '^--nobsdflags$' \
             | grep -v '^--pattern$' \
+            | grep -v '^--progress$' \
+            | grep -v '^--stats$' \
             | grep -v '^--read-special$' \
             | grep -v '^--repository-only$' \
             | grep -v '^--show-rc$' \

scripts/release

@@ -29,7 +29,7 @@ twine upload -r pypi dist/borgmatic-*-py3-none-any.whl
 # Set release changelogs on projects.evoworx.org and GitHub.
 release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
 escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
-curl --request POST \
+curl --silent --request POST \
     "https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases?access_token=$projects_token" \
     --header "Accept: application/json" \
     --header "Content-Type: application/json" \

setup.py

@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 
 
-VERSION = '1.2.9'
+VERSION = '1.2.13'
 
 
 setup(
@@ -11,7 +11,7 @@ setup(
     author='Dan Helfman',
     author_email='witten@torsion.org',
     url='https://torsion.org/borgmatic',
-    classifiers=(
+    classifiers=[
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
         'Intended Audience :: System Administrators',
@@ -19,7 +19,7 @@ setup(
         'Programming Language :: Python',
         'Topic :: Security :: Cryptography',
         'Topic :: System :: Archiving :: Backup',
-    ),
+    ],
     packages=find_packages(),
     entry_points={
         'console_scripts': [

tests/end-to-end/test_borgmatic.py

@@ -9,7 +9,8 @@ import tempfile
 def generate_configuration(config_path, repository_path):
     '''
     Generate borgmatic configuration into a file at the config path, and update the defaults so as
-    to work for testing (including injecting the given repository path).
+    to work for testing (including injecting the given repository path and tacking on an encryption
+    passphrase).
     '''
     subprocess.check_call(
         'generate-borgmatic-config --destination {}'.format(config_path).split(' ')
@@ -21,6 +22,7 @@ def generate_configuration(config_path, repository_path):
         .replace('- /home', '- {}'.format(config_path))
         .replace('- /etc', '')
         .replace('- /var/log/syslog*', '')
+        + 'storage:\n    encryption_passphrase: "test"'
     )
     config_file = open(config_path, 'w')
     config_file.write(config)
@@ -33,14 +35,13 @@ def test_borgmatic_command():
     repository_path = os.path.join(temporary_directory, 'test.borg')
 
     try:
-        subprocess.check_call(
-            'borg init --encryption repokey {}'.format(repository_path).split(' '),
-            env={'BORG_PASSPHRASE': '', **os.environ},
-        )
-
         config_path = os.path.join(temporary_directory, 'test.yaml')
         generate_configuration(config_path, repository_path)
 
+        subprocess.check_call(
+            'borgmatic -v 2 --config {} --init --encryption repokey'.format(config_path).split(' ')
+        )
+
         # Run borgmatic to generate a backup archive, and then list it to make sure it exists.
         subprocess.check_call('borgmatic --config {}'.format(config_path).split(' '))
         output = subprocess.check_output(

tests/integration/commands/test_borgmatic.py

@@ -16,16 +16,6 @@ def test_parse_arguments_with_no_arguments_uses_defaults():
     assert parser.json is False
 
 
-def test_parse_arguments_with_path_arguments_overrides_defaults():
-    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
-
-    parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
-
-    assert parser.config_paths == ['myconfig']
-    assert parser.excludes_filename == 'myexcludes'
-    assert parser.verbosity is 0
-
-
 def test_parse_arguments_with_multiple_config_paths_parses_as_list():
     flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
 
@@ -35,7 +25,7 @@ def test_parse_arguments_with_multiple_config_paths_parses_as_list():
     assert parser.verbosity is 0
 
 
-def test_parse_arguments_with_verbosity_flag_overrides_default():
+def test_parse_arguments_with_verbosity_overrides_default():
     config_paths = ['default']
     flexmock(module.collect).should_receive('get_default_config_paths').and_return(config_paths)
 
@@ -46,7 +36,7 @@ def test_parse_arguments_with_verbosity_flag_overrides_default():
     assert parser.verbosity == 1
 
 
-def test_parse_arguments_with_json_flag_overrides_default():
+def test_parse_arguments_with_json_overrides_default():
     parser = module.parse_arguments('--list', '--json')
     assert parser.json is True
 
@@ -88,16 +78,94 @@ def test_parse_arguments_with_invalid_arguments_exits():
         module.parse_arguments('--posix-me-harder')
 
 
-def test_parse_arguments_with_json_flag_with_list_or_info_flag_does_not_raise_any_error():
+def test_parse_arguments_disallows_deprecated_excludes_option():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes')
+
+
+def test_parse_arguments_disallows_encryption_mode_without_init():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--encryption', 'repokey')
+
+
+def test_parse_arguments_requires_encryption_mode_with_init():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--init')
+
+
+def test_parse_arguments_disallows_append_only_without_init():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--append-only')
+
+
+def test_parse_arguments_disallows_storage_quota_without_init():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments('--config', 'myconfig', '--storage-quota', '5G')
+
+
+def test_parse_arguments_allows_init_and_prune():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--init', '--encryption', 'repokey', '--prune')
+
+
+def test_parse_arguments_allows_init_and_create():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    module.parse_arguments('--config', 'myconfig', '--init', '--encryption', 'repokey', '--create')
+
+
+def test_parse_arguments_disallows_init_and_dry_run():
+    flexmock(module.collect).should_receive('get_default_config_paths').and_return(['default'])
+
+    with pytest.raises(ValueError):
+        module.parse_arguments(
+            '--config', 'myconfig', '--init', '--encryption', 'repokey', '--dry-run'
+        )
+
+
+def test_parse_arguments_allows_progress_and_create():
+    module.parse_arguments('--progress', '--create', '--list')
+
+
+def test_parse_arguments_disallows_progress_without_create():
+    with pytest.raises(ValueError):
+        module.parse_arguments('--progress', '--list')
+
+
+def test_parse_arguments_with_stats_and_create_flags_does_not_raise():
+    module.parse_arguments('--stats', '--create', '--list')
+
+
+def test_parse_arguments_with_stats_and_prune_flags_does_not_raise():
+    module.parse_arguments('--stats', '--prune', '--list')
+
+
+def test_parse_arguments_with_stats_flag_but_no_create_or_prune_flag_raises_value_error():
+    with pytest.raises(ValueError):
+        module.parse_arguments('--stats', '--list')
+
+
+def test_parse_arguments_allows_json_with_list_or_info():
     module.parse_arguments('--list', '--json')
     module.parse_arguments('--info', '--json')
 
 
-def test_parse_arguments_with_json_flag_but_no_list_or_info_flag_raises_value_error():
+def test_parse_arguments_disallows_json_without_list_or_info():
     with pytest.raises(ValueError):
         module.parse_arguments('--json')
 
 
-def test_parse_arguments_with_json_flag_and_both_list_and_info_flag_raises_value_error():
+def test_parse_arguments_disallows_json_with_both_list_and_info():
     with pytest.raises(ValueError):
         module.parse_arguments('--list', '--info', '--json')

tests/unit/borg/test_create.py

@@ -1,5 +1,4 @@
 import logging
-import os
 
 from flexmock import flexmock
 
@@ -7,52 +6,6 @@ from borgmatic.borg import create as module
 from ..test_verbosity import insert_logging_mock
 
 
-def test_initialize_environment_with_passcommand_should_set_environment():
-    orig_environ = os.environ
-
-    try:
-        os.environ = {}
-        module.initialize_environment({'encryption_passcommand': 'command'})
-        assert os.environ.get('BORG_PASSCOMMAND') == 'command'
-    finally:
-        os.environ = orig_environ
-
-
-def test_initialize_environment_with_passphrase_should_set_environment():
-    orig_environ = os.environ
-
-    try:
-        os.environ = {}
-        module.initialize_environment({'encryption_passphrase': 'pass'})
-        assert os.environ.get('BORG_PASSPHRASE') == 'pass'
-    finally:
-        os.environ = orig_environ
-
-
-def test_initialize_environment_with_ssh_command_should_set_environment():
-    orig_environ = os.environ
-
-    try:
-        os.environ = {}
-        module.initialize_environment({'ssh_command': 'ssh -C'})
-        assert os.environ.get('BORG_RSH') == 'ssh -C'
-    finally:
-        os.environ = orig_environ
-
-
-def test_initialize_environment_without_configuration_should_not_set_environment():
-    orig_environ = os.environ
-
-    try:
-        os.environ = {}
-        module.initialize_environment({})
-        assert os.environ.get('BORG_PASSCOMMAND') is None
-        assert os.environ.get('BORG_PASSPHRASE') is None
-        assert os.environ.get('BORG_RSH') is None
-    finally:
-        os.environ = orig_environ
-
-
 def test_expand_directory_with_basic_path_passes_it_through():
     flexmock(module.os.path).should_receive('expanduser').and_return('foo')
     flexmock(module.glob).should_receive('glob').and_return([])
@@ -266,7 +219,7 @@ def test_create_archive_with_log_info_calls_borg_with_info_parameter():
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME', '--info', '--stats'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME-', '--info', '--stats'))
     insert_logging_mock(logging.INFO)
 
     module.create_archive(
@@ -287,7 +240,7 @@ def test_create_archive_with_log_debug_calls_borg_with_debug_parameter():
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(
-        CREATE_COMMAND + ('--list', '--filter', 'AME', '--stats', '--debug', '--show-rc')
+        CREATE_COMMAND + ('--list', '--filter', 'AME-', '--stats', '--debug', '--show-rc')
     )
     insert_logging_mock(logging.DEBUG)
 
@@ -331,7 +284,7 @@ def test_create_archive_with_dry_run_and_log_info_calls_borg_without_stats_param
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
-    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME', '--info', '--dry-run'))
+    insert_subprocess_mock(CREATE_COMMAND + ('--list', '--filter', 'AME-', '--info', '--dry-run'))
     insert_logging_mock(logging.INFO)
 
     module.create_archive(
@@ -355,7 +308,7 @@ def test_create_archive_with_dry_run_and_log_debug_calls_borg_without_stats_para
     flexmock(module).should_receive('_make_pattern_flags').and_return(())
     flexmock(module).should_receive('_make_exclude_flags').and_return(())
     insert_subprocess_mock(
-        CREATE_COMMAND + ('--list', '--filter', 'AME', '--debug', '--show-rc', '--dry-run')
+        CREATE_COMMAND + ('--list', '--filter', 'AME-', '--debug', '--show-rc', '--dry-run')
     )
     insert_logging_mock(logging.DEBUG)
 
@@ -390,6 +343,25 @@ def test_create_archive_with_checkpoint_interval_calls_borg_with_checkpoint_inte
     )
 
 
+def test_create_archive_with_chunker_params_calls_borg_with_chunker_params_parameters():
+    flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
+    flexmock(module).should_receive('_write_pattern_file').and_return(None)
+    flexmock(module).should_receive('_make_pattern_flags').and_return(())
+    flexmock(module).should_receive('_make_exclude_flags').and_return(())
+    insert_subprocess_mock(CREATE_COMMAND + ('--chunker-params', '1,2,3,4'))
+
+    module.create_archive(
+        dry_run=False,
+        repository='repo',
+        location_config={
+            'source_directories': ['foo', 'bar'],
+            'repositories': ['repo'],
+            'exclude_patterns': None,
+        },
+        storage_config={'chunker_params': '1,2,3,4'},
+    )
+
+
 def test_create_archive_with_compression_calls_borg_with_compression_parameters():
     flexmock(module).should_receive('_expand_directories').and_return(('foo', 'bar')).and_return(())
     flexmock(module).should_receive('_write_pattern_file').and_return(None)

tests/unit/borg/test_environment.py

@@ -0,0 +1,49 @@
+import os
+
+from borgmatic.borg import environment as module
+
+
+def test_initialize_with_passcommand_should_set_environment():
+    orig_environ = os.environ
+
+    try:
+        os.environ = {}
+        module.initialize({'encryption_passcommand': 'command'})
+        assert os.environ.get('BORG_PASSCOMMAND') == 'command'
+    finally:
+        os.environ = orig_environ
+
+
+def test_initialize_with_passphrase_should_set_environment():
+    orig_environ = os.environ
+
+    try:
+        os.environ = {}
+        module.initialize({'encryption_passphrase': 'pass'})
+        assert os.environ.get('BORG_PASSPHRASE') == 'pass'
+    finally:
+        os.environ = orig_environ
+
+
+def test_initialize_with_ssh_command_should_set_environment():
+    orig_environ = os.environ
+
+    try:
+        os.environ = {}
+        module.initialize({'ssh_command': 'ssh -C'})
+        assert os.environ.get('BORG_RSH') == 'ssh -C'
+    finally:
+        os.environ = orig_environ
+
+
+def test_initialize_without_configuration_should_not_set_environment():
+    orig_environ = os.environ
+
+    try:
+        os.environ = {}
+        module.initialize({})
+        assert os.environ.get('BORG_PASSCOMMAND') is None
+        assert os.environ.get('BORG_PASSPHRASE') is None
+        assert os.environ.get('BORG_RSH') is None
+    finally:
+        os.environ = orig_environ

tests/unit/borg/test_init.py

@@ -0,0 +1,79 @@
+import logging
+
+from flexmock import flexmock
+
+from borgmatic.borg import init as module
+from ..test_verbosity import insert_logging_mock
+
+
+INFO_REPOSITORY_EXISTS_RESPONSE_CODE = 0
+INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE = 2
+INIT_COMMAND = ('borg', 'init', 'repo', '--encryption', 'repokey')
+
+
+def insert_info_command_mock(info_response):
+    subprocess = flexmock(module.subprocess)
+    subprocess.should_receive('call').and_return(info_response)
+
+
+def insert_init_command_mock(init_command, **kwargs):
+    subprocess = flexmock(module.subprocess)
+    subprocess.should_receive('check_call').with_args(init_command, **kwargs).once()
+
+
+def test_initialize_repository_calls_borg_with_parameters():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND)
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey')
+
+
+def test_initialize_repository_skips_initialization_when_repository_already_exists():
+    insert_info_command_mock(INFO_REPOSITORY_EXISTS_RESPONSE_CODE)
+    flexmock(module.subprocess).should_receive('check_call').never()
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey')
+
+
+def test_initialize_repository_with_append_only_calls_borg_with_append_only_parameter():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND + ('--append-only',))
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey', append_only=True)
+
+
+def test_initialize_repository_with_storage_quota_calls_borg_with_storage_quota_parameter():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND + ('--storage-quota', '5G'))
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey', storage_quota='5G')
+
+
+def test_initialize_repository_with_log_info_calls_borg_with_info_parameter():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND + ('--info',))
+    insert_logging_mock(logging.INFO)
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey')
+
+
+def test_initialize_repository_with_log_debug_calls_borg_with_debug_parameter():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND + ('--debug',))
+    insert_logging_mock(logging.DEBUG)
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey')
+
+
+def test_initialize_repository_with_local_path_calls_borg_via_local_path():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(('borg1',) + INIT_COMMAND[1:])
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey', local_path='borg1')
+
+
+def test_initialize_repository_with_remote_path_calls_borg_with_remote_path_parameter():
+    insert_info_command_mock(INFO_REPOSITORY_NOT_FOUND_RESPONSE_CODE)
+    insert_init_command_mock(INIT_COMMAND + ('--remote-path', 'borg1'))
+
+    module.initialize_repository(repository='repo', encryption_mode='repokey', remote_path='borg1')

tests/unit/test_verbosity.py

@@ -6,9 +6,11 @@ from borgmatic import verbosity as module
 
 
 def insert_logging_mock(log_level):
-    """ Mocks the isEnabledFor from python logging. """
+    '''
+    Mock the isEnabledFor from Python logging.
+    '''
     logging = flexmock(module.logging.Logger)
-    logging.should_receive('isEnabledFor').replace_with(lambda lvl: lvl >= log_level)
+    logging.should_receive('isEnabledFor').replace_with(lambda level: level >= log_level)
     logging.should_receive('getEffectiveLevel').replace_with(lambda: log_level)