Home

Awesome

ghorg

Go Report Card <a href="https://godoc.org/github.com/gabrie30/ghorg"><img src="https://godoc.org/github.com/gabrie30/ghorg?status.svg" alt="GoDoc"></a> Mentioned in Awesome Go License WakeMeOps

Pronounced [gore-guh]; similar to gorge. You can use ghorg to gorge on orgs.

Use ghorg to quickly clone all of an orgs, or users repos into a single directory. This can be useful in many situations including

  1. Searching an orgs/users codebase with ack, silver searcher, grep etc..
  2. Bash scripting
  3. Creating backups
  4. Onboarding new team members (cloning all team repos)
  5. Performing Audits

With default configuration ghorg performs two actions.

  1. Will clone a repo if its not inside the clone directory.
  2. If repo does exists locally in the clone directory it will perform a git pull and git clean on the repo.

So when running ghorg a second time on the same org/user, all local changes in the cloned directory by default will be overwritten by what's on GitHub. If you want to work out of this directory, make sure you either rename the directory or set the --no-clean flag on all future clones to prevent losing your changes locally.

<p align="center"> <img width="648" alt="ghorg cli example" src="https://user-images.githubusercontent.com/1512282/63229247-5459f880-c1b3-11e9-9e5d-d20723046946.png"> </p>

Supported Providers

The terminology used in ghorg is that of GitHub, mainly orgs/repos. GitLab and BitBucket use different terminology. There is a handy chart thanks to GitLab that translates terminology here. Note, some features may be different for certain providers.

Installation

There are a installation methods available, please choose the one that suits your fancy:

For each installation method, optionally create a ghorg configuration file. See the configuration section for more details.

mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration

Prebuilt Binaries

See latest release to download directly for

If you don't know which to choose its likely going to be the x86_64 version for your operating system.

Homebrew

brew install gabrie30/utils/ghorg

Golang

# ensure $HOME/go/bin is in your path ($ echo $PATH | grep $HOME/go/bin)

# if using go 1.16+ locally
go install github.com/gabrie30/ghorg@latest

# older go versions can run
go get github.com/gabrie30/ghorg

Configuration

Precedence for configuration is first given to the flags set on the command-line, then to what's set in your $HOME/.config/ghorg/conf.yaml. This file comes from the sample-conf.yaml and can be installed by performing the following.

mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration

If no configuration file is found ghorg will use its defaults and try to clone a GitHub Org, however an api token is always required.

You can have multiple configuration files which is useful if you clone from multiple SCM providers with different tokens and settings. Alternative configuration files can only be referenced as a command-line flag --config.

If you have multiple different orgs/users/configurations to clone see the ghorg reclone command as a way to manage them.

Note: ghorg will respect the XDG_CONFIG_HOME environment variable if set.

SCM Provider Setup

Note: if you are running into issues, read the troubleshooting and known issues section below

GitHub Setup

  1. Create Personal Access Token with all repo scopes. Update GHORG_GITHUB_TOKEN in your ghorg/conf.yaml or as a cli flag or place it in a file and add the path to GHORG_GITHUB_TOKEN. If your org has Saml SSO in front you will need to give your token those permissions as well, see this doc.
  2. For cloning GitHub Enterprise (self hosted github instances) repos you must set --base-url e.g. ghorg clone <github_org> --base-url=https://internal.github.com
  3. See examples/github.md on how to run

GitHub App Authentication (Advanced)

  1. Create a GitHub App in your Organization. You only need to fill out the required fields. Make sure to give Repository Permissions -> contents -> read only permissions
  2. Install the GitHub App into your Organization
  3. Generate a a private key from the GitHub App, set the location of the key to GHORG_GITHUB_APP_PEM_PATH
  4. Locate the GitHub App ID from the GitHub App, set the value to GHORG_GITHUB_APP_ID
  5. Locate the GitHub Installation ID from the URL of the GitHub app, set the value to GHORG_GITHUB_APP_INSTALLATION_ID. NOTE: you will need to use the actual GitHub url to get this ID, go to your GitHub Organization Settings Page -> Third Party Access -> GitHub Apps -> Configure -> Get ID from URL

GitLab Setup

  1. Create Personal Access Token with the read_api scope (or api for self-managed GitLab older than 12.10). This token can be added to your ghorg/conf.yaml or as a cli flag.
  2. Update the GitLab Specific config in your ghorg/conf.yaml or via cli flags or place it in a file and add the path to GHORG_GITLAB_TOKEN
  3. Update GHORG_SCM_TYPE to gitlab in your ghorg/conf.yaml or via cli flags
  4. See examples/gitlab.md on how to run

Gitea Setup

  1. Create Access Token (Settings -> Applications -> Generate Token)
  2. Update GHORG_GITEA_TOKEN in your ghorg/conf.yaml or use the (--token, -t) flag or place it in a file and add the path to GHORG_GITEA_TOKEN.
  3. Update GHORG_SCM_TYPE to gitea in your ghorg/conf.yaml or via cli flags
  4. See examples/gitea.md on how to run

Bitbucket Setup

Note: ghorg only supports bitbucket cloud, it does not support self hosted instances at this time

App Passwords

  1. To configure with bitbucket you will need to create a new app password and update your $HOME/.config/ghorg/conf.yaml or use the (--token, -t) and (--bitbucket-username) flags.
  2. Update SCM type to bitbucket in your ghorg/conf.yaml or via cli flags
  3. See examples/bitbucket.md on how to run

PAT/OAuth token

  1. Create a PAT
  2. Set the token with GHORG_BITBUCKET_OAUTH_TOKEN in your $HOME/.config/ghorg/conf.yaml or using the --token flag. Make sure you do not have --bitbucket-username set.
  3. Update SCM TYPE to bitbucket in your ghorg/conf.yaml or via cli flags
  4. See examples/bitbucket.md on how to run

How to Use

See examples directory for more SCM specific docs or use the examples command e.g. ghorg examples gitlab

$ ghorg clone kubernetes --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
# Example how to use --token with a file path
$ ghorg clone kubernetes --token=~/.config/ghorg/gitlab-token.txt
$ ghorg clone davecheney --clone-type=user --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples --scm=gitlab --preserve-dir --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples/wayne-enterprises --scm=gitlab --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone all-groups --scm=gitlab --base-url=https://gitlab.internal.yourcompany.com --preserve-dir
$ ghorg clone --help
# view cloned resources
$ ghorg ls
$ ghorg ls someorg
$ ghorg ls someorg | xargs -I %s mv %s bar/

Changing Clone Directories

  1. By default ghorg will clone the org or user repos into a directory like $HOME/ghorg/org. If you want to clone the org to a different directory use the --path flag or set GHORG_ABSOLUTE_PATH_TO_CLONE_TO in your ghorg conf. This value must be an absolute path. For example if you wanted to clone the kubernetes org to /tmp/ghorg you would run the following command.

    $ ghorg clone kubernetes --path=/tmp/ghorg

    which would create...

    /tmp/ghorg
└── kubernetes
    ├── apimachinery
    ├── gengo
    ├── git-sync
    ├── kubeadm
    ├── kubernetes-template-project
    ├── ...

  2. If you want to change the name of the directory the repos get cloned into, set the GHORG_OUTPUT_DIR in your ghorg conf or set the --output-dir flag. For example to clone only the repos starting with sig- from the kubernetes org into a direcotry called kubernetes-sig-only. You would run the following command.

    $ ghorg clone kubernetes --match-regex=^sig- --output-dir=kubernetes-sig-only

    which would create...

    $HOME/ghorg
└── kubernetes-sig-only
    ├── sig-release
    ├── sig-security
    └── sig-testing

Selective Repository Cloning

Creating Backups

When taking backups the notable flags are --backup, --clone-wiki, and --include-submodules. The --backup flag will clone the repo with git clone --mirror. The --clone-wiki flag will include any wiki pages the repo has. If you want to include any submodules you will need --include-submodules. Lastly, if you want to exclude any binary files use the the flag --git-filter=blob:none to prevent them from being cloned.

ghorg clone kubernetes --backup --clone-wiki --include-submodules

This will create a kubernetes_backup directory for the org. Each folder inside will contain the .git contents for the source repo. To restore the code from the .git contents you would move all contents into a .git dir, then run git init inside the dir, then checkout branch e.g.

# inside kubernetes_backup dir, to restore kubelet source code
cd kubelet
mkdir .git
mv -f * .git # moves all contents into .git directory
git init
git checkout master

Reclone Command

The ghorg reclone command is a way to store all your ghorg clone commands in one configuration file and makes calling long or multiple ghorg clone commands easier.

Once your reclone.yaml configuration is set you can call ghorg reclone to clone each entry individually or clone all at once, see examples below.

# To clone all the entries in your reclone.yaml omit any arguments
ghorg reclone

# To run one or more entries you can pass arguments
ghorg reclone kubernetes-sig-staging kubernetes-sig

# To view all your reclone commands
# NOTE: This command prints tokens to stdout
ghorg reclone --list
<p align="center"> <img width="648" alt="ghorg reclone example" src="https://user-images.githubusercontent.com/1512282/183263986-50e56b86-12b9-479b-9c52-b1c74129228c.png"> </p>

Setup

Add a reclone.yaml to your $HOME/.config/ghorg directory. You can use the following command to set it for you with examples to use as a template

curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-reclone.yaml > $HOME/.config/ghorg/reclone.yaml

Update file with the commands you wish to run.

Reclone Server Command

The reclone-server command starts a server that allows you to trigger ad hoc reclone commands via HTTP requests.

Usage

ghorg reclone-server [flags]

Flags

Endpoints

Examples

Starting the server. The default port is 8080 but you can optionally start the server on different port using the --port flag:

ghorg reclone-server

Trigger reclone command, this will run all cmds defined in your reclone.yaml:

curl "http://localhost:8080/trigger/reclone"

Trigger a specific reclone command:

curl "http://localhost:8080/trigger/reclone?cmd=your-reclone-command"

Get the statistics:

curl "http://localhost:8080/stats"

Check the server health:

curl "http://localhost:8080/health"

Reclone Cron Command

The reclone-cron command sets up a simple cron job that triggers the reclone command at specified minute intervals indefinitely.

Usage

ghorg reclone-cron [flags]

Flags

Example

Set up a cron job to trigger the reclone command every day:

ghorg reclone-cron --minutes 1440

Environment Variables

Using Docker

The provided images are built for both amd64 and arm64 architectures and are available solely on Github Container Registry ghcr.io.

# Should print help message
# You can also specify a version as the tag, such as ghcr.io/gabrie30/ghorg:v1.9.9
docker run --rm ghcr.io/gabrie30/ghorg:latest

Note: There are also tags available for the latest on trunk, such as master or master-<commit SHA 7 chars>, but these are not recommended.

The commands for ghorg are parsed as docker commands. The entrypoint is the ghorg binary, hence you only need to enter remaining arguments as follows:

docker run --rm ghcr.io/gabrie30/ghorg \
    clone kubernetes --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2

The image ships with the following environment variables set:

GHORG_CONFIG=/config/conf.yaml
GHORG_RECLONE_PATH=/config/reclone.yaml
GHORG_ABSOLUTE_PATH_TO_CLONE_TO=/data

These can be overriden, if necessary, by including the -e flag to the docker run comand, e.g. -e GHORG_GITHUB_TOKEN=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2.

Persisting Data on the Host

In order to store data on the host, it is required to bind mount a volume:

docker run --rm \
        -e GHORG_GITHUB_TOKEN=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2 \
        -v $HOME/.config/ghorg:/config `# optional` \
        -v $HOME/repositories:/data \
        ghcr.io/gabrie30/ghorg:latest \
        clone kubernetes --match-regex=^sig

Note: Altering GHORG_ABSOLUTE_PATH_TO_CLONE_TO will require changing the mount location from /data to the new location inside the container.

A shell alias might make this more practical:

alias ghorg="docker run --rm -v $HOME/.config/ghorg:/config -v $HOME/repositories:/data ghcr.io/gabrie30/ghorg:latest"

# Using the alias: creates and cleans up the container
ghorg clone kubernetes --match-regex=^sig

Tracking Clone Data Over Time

To track data on your clones over time, you can use the ghorg stats feature. It is recommended to enable ghorg stats in your configuration file by setting GHORG_STATS_ENABLED=true. This ensures that each clone operation is logged automatically without needing to set the command line flag --stats-enabled every time. The ghorg stats feature is disabled by default and needs to be enabled.

When ghorg stats is enabled, the CSV file _ghorg_stats.csv is created in the directory specified by GHORG_ABSOLUTE_PATH_TO_CLONE_TO. This file contains detailed information about each clone operation, which is useful for auditing and tracking purposes such as the size of the clone and the number of new commits over time.

Below are the headers and their descriptions. Note that these headers may change over time. If there are any changes in the headers, a new file named _ghorg_stats_new_header_${sha256HashOfHeader}.csv will be created to prevent incorrect data from being added to your CSV.

Converting CSV to JSON

go install github.com/gabrie30/csvToJson@latest && \
csvToJson _ghorg_stats.csv

Windows support

Windows is supported when built with golang or as a prebuilt binary however, the readme and other documentation is not geared towards Windows users.

Alternatively, Windows users can also install ghorg using scoop

scoop bucket add main
scoop install ghorg

Troubleshooting