Home

Awesome

UPM

GoDoc Run on Repl.it

UPM is the Universal Package Manager. It allows you to manage packages for any (supported) programming language through the same interface following the principle of least astonishment. At Repl.it, we use UPM to provide deep package manager integration for many different programming languages using the same infrastructure.

UPM does not implement package management itself. Instead, it runs a package manager for you. The value added is:

In other words, UPM eliminates the need to remember a huge collection of language-specific package manager quirks and weirdness, and adds a few nifty extra features like dependency guessing and machine-parseable specfile and lockfile listing.

Supported languages

coreindexguess
python-python3-uvyesyesyes
python-python3-pipyesyesyes
python-python3-poetryyesyesyes
nodejs-yarnyesyesyes
nodejs-pnpmyesyesyes
nodejs-npmyesyesyes
ruby-bundleryesyes
elisp-caskyesyesyes
dart-pub.devyesyes
rlangyesyes
javayesyes
rustyesyes
dotnetyesyes
phpyesyes

Installation

You have many options. UPM is a single binary with no dependencies, so you can install it anywhere. Tarballs are available on the releases page. Read on for instructions on installing via a package manager.

macOS

Available on Homebrew in a custom tap.

$ brew install replit/tap/upm

Debian-based Linux

.deb packages are available on the releases page.

RPM-based Linux

.rpm packages are available on the releases page.

Arch Linux

Soon to be available on the Arch User Repository. Right now, you can clone this repository and install with makepkg using the PKGBUILD in packaging/aur.

Windows

Available on Scoop in the main bucket.

$ scoop install upm

Snappy

Soon to be available on the Snap Store. Right now, .snap packages are available on the releases page.

Docker

You can try out UPM right away in a Docker image based on Ubuntu that has all the supported languages and package managers already installed.

$ docker run -it --rm replco/upm

Additional tags are also available. replco/upm:full is the same as the above, while replco/upm:light just has the UPM binary installed to /usr/local/bin and none of the languages or package managers installed. If you want to run a specific tagged release, rather than the latest development snapshot, use e.g. replco/upm:1.0, replco/upm:1.0-full, or replco/upm:1.0-light.

Quick start

Let's create a new Python project:

$ mkdir ~/python
$ cd ~/python

We'll start by adding Flask as a dependency. UPM will handle setting up the project for us:

$ upm -l python add flask
--> python3 -m poetry init --no-interaction

This command will guide you through creating your pyproject.toml config.


--> python3 -m poetry add flask
Creating virtualenv python-py3.7 in /root/.cache/pypoetry/virtualenvs
Using version ^1.1 for flask

Updating dependencies
Resolving dependencies... (0.6s)

Writing lock file


Package operations: 6 installs, 0 updates, 0 removals

  - Installing markupsafe (1.1.1)
  - Installing click (7.0)
  - Installing itsdangerous (1.1.0)
  - Installing jinja2 (2.10.1)
  - Installing werkzeug (0.15.4)
  - Installing flask (1.1.1)

UPM operates on a specfile and lockfile for each project. The specfile says what your project's dependencies are in a human-readable format, while the lockfile specifies exact versions for everything, including transitive dependencies. For Python, the specfile is pyproject.toml and the lockfile is poetry.lock:

$ ls
poetry.lock  pyproject.toml

We don't have to read them ourselves, because UPM can handle that. Notice that UPM is now aware that our project uses Python, because of the files that were created:

$ upm list
name    spec
-----   ----
flask   ^1.1

$ upm list -a
name           version
------------   -------
click          7.0
flask          1.1.1
itsdangerous   1.1.0
jinja2         2.10.1
markupsafe     1.1.1
werkzeug       0.15.4

Let's search for another dependency to add:

$ upm search nose
--> python3 -c '<secret sauce>' nose
Name                Description                                                              Version
-----------------   ----------------------------------------------------------------------   -------
nose                nose extends unittest to make testing easier                             1.3.7
nose-detecthttp     A nose plugin to detect tests making http calls.                         1.1.0
nose-picker         nose plugin that picks a subset of your unit tests                       0.5.5
nose-progressive    A testrunner with a progress bar and smarter tracebacks                  1.5.2
nose-unittest       UNKNOWN                                                                  0.1.1
nose-blockage       Raise errors when communicating outside of tests                         0.1.2
nose-watcher        A nose plugin to watch for changes within the local directory.           0.1.3
nose-bisect         A Nose plugin which allows bisection of test failures.                   0.1.0
nose-printlog       Print log to console in nose tests                                       0.2.0
nose-json           A JSON report plugin for Nose.                                           0.2.4
nose-faulthandler   Nose plugin. Activates faulthandler module for test runs.                0.1
nose-knows                                                                                   0.2
nose-pagerduty      PagerDuty alert plugin for nose                                          0.2.0
nose-logpertest     Logging nose plugin to create log per test                               0.0.1
nose-bleed          A progressive coverage plugin for Nose.                                  0.5.1
nose-numpyseterr    Nose plugin to set how floating-point errors are handled by numpy        0.1
nose-skipreq        nose plugin that will skip Google API RequestError exceptions.           2.0
nose-selecttests    Specify whitelist of keywords for tests to be run by nose                0.5
nose-pacman         A testrunner with a pacman progress bar                                  0.1.0
nose-switch         Add special switches in code, based on options set when running tests.   0.1.5

We can get more information about a package like this:

$ upm info nose
--> python3 -c '<secret sauce>' nose
Name:          nose
Description:   nose extends unittest to make testing easier
Version:       1.3.7
Homepage:      http://readthedocs.org/docs/nose/
Author:        Jason Pellerin <jpellerin+nose@gmail.com>
License:       GNU LGPL

For piping into other programs, the search and info commands can also output JSON:

$ upm info nose --format=json | jq
--> python3 -c '<secret sauce>' nose
{
  "name": "nose",
  "description": "nose extends unittest to make testing easier",
  "version": "1.3.7",
  "homepageURL": "http://readthedocs.org/docs/nose/",
  "author": "Jason Pellerin <jpellerin+nose@gmail.com>",
  "license": "GNU LGPL"
}

UPM can also look at your project's source code and guess what packages need to be installed. We use this on Repl.it to help developers get started faster. To see it in action, we'll need some source code:

$ git clone https://github.com/replit/play.git ~/play
$ cd ~/play
$ upm add --guess
--> python3 -c '<secret sauce>' '<secret sauce>'
--> python3 -m poetry init --no-interaction

This command will guide you through creating your pyproject.toml config.


--> python3 -m poetry add pygame pymunk setuptools
Creating virtualenv play-py3.7 in /root/.cache/pypoetry/virtualenvs
Using version ^1.9 for pygame
Using version ^5.5 for pymunk
Using version ^41.0 for setuptools

Updating dependencies
Resolving dependencies... (1.4s)

Writing lock file


Package operations: 4 installs, 0 updates, 0 removals

  - Installing pycparser (2.19)
  - Installing cffi (1.12.3)
  - Installing pygame (1.9.6)
  - Installing pymunk (5.5.0)

You can also just get the list of guessed dependencies, if you want. The -a flag lists all guessed dependencies, even the ones already added to the specfile:

$ upm guess -a
pygame
pymunk
setuptools

All of this might seem a bit too simple to justify a new tool, but the real power of UPM is that it works exactly the same for every programming language:

$ upm -l nodejs info express
Name:          express
Description:   Fast, unopinionated, minimalist web framework
Version:       4.17.1
Homepage:      http://expressjs.com/
Source code:   git+https://github.com/expressjs/express.git
Bug tracker:   https://github.com/expressjs/express/issues
Author:        TJ Holowaychuk <tj@vision-media.ca>
License:       MIT

$ upm -l ruby info jekyll
--> ruby -e '<secret sauce>' jekyll
Name:            jekyll
Description:     Jekyll is a simple, blog aware, static site generator.
Version:         3.8.6
Homepage:        https://github.com/jekyll/jekyll
Documentation:   http://jekyllrb.com
Source code:     https://github.com/jekyll/jekyll
Bug tracker:     https://github.com/jekyll/jekyll/issues
Author:          Tom Preston-Werner
License:         MIT
Dependencies:    addressable, colorator, em-websocket, i18n, jekyll-sass-converter, jekyll-watch, kramdown, liquid, mercenary, pathutil, rouge, safe_yaml

$ upm -l elisp info elnode
--> emacs -Q --batch --eval '<secret sauce>' /tmp/elpa552971126 info elnode
Name:           elnode
Description:    The Emacs webserver.
Version:        20190702.1509
Dependencies:   web, dash, noflet, s, creole, fakir, db, kv

That includes adding and removing packages, listing the specfile and lockfile, searching package indices, and guessing project dependencies. UPM knows all the best practices for each language so that you don't have to!

Usage

Explore the command-line interface at your leisure:

$ upm --help
Usage:
  upm [command]

Available Commands:
  which-language   Query language autodetection
  list-languages   List supported languages
  search           Search for packages online
  info             Show package information from online registry
  add              Add packages to the specfile
  remove           Remove packages from the specfile
  lock             Generate the lockfile from the specfile
  install          Install packages from the lockfile
  list             List packages from the specfile (or lockfile)
  guess            Guess what packages are needed by your project
  show-specfile    Print the filename of the specfile
  show-lockfile    Print the filename of the lockfile
  show-package-dir Print the directory where packages are installed
  help             Help about any command

Flags:
  -h, --help                       display command-line usage
      --ignored-packages strings   packages to ignore when guessing (comma-separated)
  -l, --lang string                specify project language(s) manually
  -q, --quiet                      don't show what commands are being run
  -v, --version                    display command version

Use "upm [command] --help" for more information about a command.

Here are useful things to know that aren't obvious:

Environment variables respected

Dependencies

UPM itself has no dependencies. It is a single statically-linked binary. However, if you wish to actually use it to manage packages for a language, then the relevant language package manager needs to be installed, as follows:

All of these dependencies are already installed in the replco/upm:full Docker image.

Contributing

$ make help
usage:
  make upm       Build the UPM binary
  make dev       Run a shell with UPM source code and all package managers inside Docker
  make light     Build a Docker image with just the UPM binary
  make full      Build a Docker image with the UPM binary and all package managers
  make doc       Open Godoc in web browser
  make deploy    Publish UPM snapshot Docker images to Docker Hub
  make pkgbuild  Update and test PKGBUILD
  make clean     Remove build artifacts
  make help      Show this message

To build UPM, run make upm (or just make), the built binary can be found in ./cmd/upm/upm. To remove build artifacts, run make clean.

Using Replit (simplest)

You can develop UPM on Replit. Simply click on this link and the repo will start cloning into a Repl.

Once you're in your new Repl, you can hit run to build a new binary (or make in shell). You can create a test folder and use the binary and test your changes. For example say you made a change to the nodejs-npm language and want to test it, you would hit run and do the following:

$ mkdir testnpm
$ cd testnpm
$ npm init -y
$ ../cmd/upm/upm add left-pad -l nodejs-npm

Using Docker

You can use Docker to avoid needing to install the package managers that UPM drives. To do this, run make dev. This will build an image and launch a shell inside the container with the UPM source directory on your computer synced with the filesystem inside the container. The same Makefile targets are available, and UPM is added to the $PATH automatically. You only need to restart the shell if you edit the Dockerfile or the scripts used by the Dockerfile. Aliases available inside the shell:

To build a Docker image which has only the UPM binary, for embedding in other images, run make light. The image will be tagged as upm:light. Alternatively, to build a Docker image which has the binary and all the package managers, but not the UPM source code, run make full. The image will be tagged as upm:full. These two images are automatically built and deployed to Docker Hub when a commit is merged to master.

UPM does not currently have any tests; however, we plan to fix this.

Deployment

Whenever a commit is merged to master, snapshot Docker images are built and pushed to Docker Hub by CircleCI. Whenever a tagged release (e.g. v1.0) is pushed to GitHub, the following happens:

Once you push a release and it passes CI, the following must be done manually:

<!-- Local Variables: --> <!-- truncate-lines: t --> <!-- End: -->