Home

Awesome

build status pre-commit.ci status

all-repos

Clone all your repositories and apply sweeping changes.

Installation

pip install all-repos

CLI

All command line interfaces provided by all-repos provide the following options:

all-repos-complete [options]

Add git clone tab completion for all-repos repositories.

Requires jq to function.

Add to .bash_profile:

eval "$(all-repos-complete -C ~/.../all-repos.json --bash)"

all-repos-clone [options]

Clone all the repositories into the output_dir. If run again, this command will update existing repositories.

Options:

Sample invocations:

all-repos-find-files [options] PATTERN

Similar to a distributed git ls-files | grep -P PATTERN.

Arguments:

Options:

Sample invocations:

all-repos-grep [options] [GIT_GREP_OPTIONS]

Similar to a distributed git grep ....

Options:

Sample invocations:

all-repos-list-repos [options]

List all cloned repository names.

all-repos-manual [options]

Interactively apply a manual change across repos.

note: all-repos-manual will always run in --interactive autofixing mode.

note: all-repos-manual requires the --repos autofixer option.

Options:

all-repos-sed [options] EXPRESSION FILENAMES

Similar to a distributed git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION.

note: this assumes GNU sed. If you're on macOS, install gnu-sed with Homebrew:

brew install gnu-sed

# Add to .bashrc / .zshrc
export PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH"

Arguments:

Options:

Sample invocations:

Configuring

A configuration file looks roughly like this:

{
    "output_dir": "output",
    "source": "all_repos.source.github",
    "source_settings":  {
        "api_key": "...",
        "username": "asottile"
    },
    "push": "all_repos.push.github_pull_request",
    "push_settings": {
        "api_key": "...",
        "username": "asottile"
    }
}

Source modules

all_repos.source.json_file

Clones all repositories listed in a file. The file must be formatted as follows:

{
    "example/repo1": "https://git.example.com/example/repo1",
    "repo2": "https://git.example.com/repo2"
}

Required source_settings

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_key1}/
+--- {repo_key2}/
+--- {repo_key3}/

all_repos.source.github

Clones all repositories available to a user on github.

Required source_settings

Optional source_settings

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
    +--- {repo1}/
    +--- {repo2}/
+--- {username2}/
    +--- {repo3}/

all_repos.source.github_forks

Clones all repositories forked from a repository on github.

Required source_settings

Optional source_settings

Directory location

See the directory structure for all_repos.source.github.

all_repos.source.github_org

Clones all repositories from an organization on github.

Required source_settings

Optional source_settings

Directory location

See the directory structure for all_repos.source.github.

all_repos.source.gitolite

Clones all repositories available to a user on a gitolite host.

Required source_settings

The gitolite API is served over SSH. It is assumed that when all-repos-clone is called, it's possible to make SSH connections with the username and hostname configured here in order to query that API.

Optional source_settings

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_name1}.git/
+--- {repo_name2}.git/
+--- {repo_name3}.git/

all_repos.source.bitbucket

Clones all repositories available to a user on Bitbucket Cloud.

Required source_settings

all_repos.source.bitbucket_server

Clones all repositories available to a user on Bitbucket Server.

Required source_settings

Optional source_settings

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
    +--- {repo1}/
    +--- {repo2}/
+--- {username2}/
    +--- {repo3}/

all_repos.source.gitlab_org

Clones all repositories from an organization on gitlab.

Required source_settings

Optional source_settings

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {org}/
    +--- {subpgroup1}/
        +--- {subpgroup2}/
            +--- {repo1}/
        +--- {repo2}/
    +--- {repo3}/
    +--- {repo4}/

Writing your own source

First create a module. This module must have the following api:

A Settings class

This class will receive keyword arguments for all values in the source_settings dictionary.

An easy way to implement the Settings class is by using a namedtuple:

Settings = collections.namedtuple('Settings', ('required_thing', 'optional'))
Settings.__new__.__defaults__ = ('optional default value',)

In this example, the required_thing setting is a required setting whereas optional may be omitted (and will get a default value of 'optional default value').

def list_repos(settings: Settings) -> Dict[str, str]: callable

This callable will be passed an instance of your Settings class. It must return a mapping from {repo_name: repository_url}. The repo_name is the directory name inside the output_dir.

Push modules

all_repos.push.merge_to_master

Merges the branch directly to the default branch and pushes. The commands it runs look roughly like this:

git checkout main
git pull
git merge --no-ff $BRANCH
git push origin HEAD

Optional push_settings

all_repos.push.github_pull_request

Pushes the branch to origin and then creates a github pull request for the branch.

Required push_settings

Optional push_settings

all_repos.push.bitbucket_server_pull_request

Pushes the branch to origin and then creates a Bitbucket pull request for the branch.

Required push_settings

all_repos.push.gitlab_pull_request

Pushes the branch to origin and then creates a GitLab pull request for the branch.

Required push_settings

all_repos.push.readonly

Does nothing.

push_settings

There are no configurable settings for readonly.

Writing your own push module

First create a module. This module must have the following api:

A Settings class

This class will receive keyword arguments for all values in the push_settings dictionary.

def push(settings: Settings, branch_name: str) -> None:

This callable will be passed an instance of your Settings class. It should deploy the branch. The function will be called with the root of the repository as the cwd.

Writing an autofixer

An autofixer applies a change over all repositories.

all-repos provides several api functions to write your autofixers with:

all_repos.autofix_lib.add_fixer_args

def add_fixer_args(parser):

Adds the autofixer cli options.

Options:

all_repos.autofix_lib.from_cli

def from_cli(args, *, find_repos, msg, branch_name):

Parse cli arguments and produce autofix_lib primitives. Returns (repos, config, commit, autofix_settings). This is handled separately from fix to allow for fixers to adjust arguments.

all_repos.autofix_lib.fix

def fix(
        repos, *,
        apply_fix,
        check_fix=_noop_check_fix,
        config: Config,
        commit: Commit,
        autofix_settings: AutofixSettings,
):

Apply the fix.

all_repos.autofix_lib.run

def run(*cmd, **kwargs):

Wrapper around subprocess.run which prints the command it will run. Unlike subprocess.run, this defaults check=True unless explicitly disabled.

Example autofixer

The trivial autofixer is as follows:

import argparse

from all_repos import autofix_lib

def find_repos(config):
    return []

def apply_fix():
    pass

def main(argv=None):
    parser = argparse.ArgumentParser()
    autofix_lib.add_fixer_args(parser)
    args = parser.parse_args(argv)

    repos, config, commit, autofix_settings = autofix_lib.from_cli(
        args, find_repos=find_repos, msg='msg', branch_name='branch-name',
    )
    autofix_lib.fix(
        repos, apply_fix=apply_fix, config=config, commit=commit,
        autofix_settings=autofix_settings,
    )

if __name__ == '__main__':
    raise SystemExit(main())

You can find some more involved examples in all_repos/autofix: