Setting up a private, team-wide PyPI repository

by Raphaël Barrois
in Python

tagged System, Python

When developing Python applications, it may be useful to store some applications in a private repository.

This provides several benefits:

  • Dependencies available even if PyPI and its mirrors are down
  • Storing custom forks of upstream packages
  • Providing private packages in a standardized manner.

For this purpose, three components are required:

  • A private index. This should be served over HTTP, with the index.html file containing links to available packages (see http://pypi.python.org/simple/ for a typical simple index. It can be protected by basic auth (e.g for private packages)
  • Properly configured local installation tools. The target is to have those tools looking at the private repository alongside PyPI.
  • Proper setup.py scripts for private packages, to prevent upload to the public PyPI repository.

Setting up the private index

Many tools exist for this, with two major families:

If your repository contains private packages, you should serve it only over HTTPS, since most Python packaging tools only use basic authentication.

Setting up the local tools

The ~/.pip/pip.conf file holds all pip-related configurations. It mostly supports a extra-index-url option for providing extra repository URLs:

[global]
; Low timeout
timeout = 20

; Custom index
extra-index-url = https://<user>:<pass>@pypi.private.example.org/

It is also possible to completely replace the PyPI index by setting the index-url option instead.

Note

The --extra-index-url can also be provided as a command line option. An alternate PIP configuration file can be used by setting the PIP_CONFIG_FILE env variable.

For distribute/distutils, use the .pypirc configuration file. This is mostly useful to register username/password combinations for private repositories:

[distutils]
index-servers =
    pypi
    my-private-repo

[pypi]
username=<your_pypi_username>
password=<your_pypi_password>

[my-private-repo]
repository=https://pypi.private.example.org/
username=<your_private_username>
password=<your_private_password>

Setting this will configure the login/pass to use for python setup.py register, upload, ... commands.

Protecting the private packages

I couldn't find any existing package for restricting upload/register/... of a package to a private repository, so I wrote the restricted_pkg package for that.

This package provides a custom setup() function, which performs the following actions:

  • Restrict upload, register and upload_docs command to the private repository, using the stored credentials or prompting if needed
  • Use the private repository along with PyPI for the install and easy_install commands. The private repository may replace PyPI if the --disable-pypi option is provided.

This means that the setup.py script for such packages should look like:

from restricted_pkg import setup

setup(
    private_repository="https://@pypi.private.example.org/",
    install_requires=[
        'restricted_pkg',
    ],
)

The @ in the repository URL indicates that this repository requires auth, but avoids committing a username:password string in the setup.py script.