When developing Python applications, it may be useful to store some applications in a private repository.
This provides several benefits:
- Dependencies available even if PyPI and its mirrors are down
- Storing custom forks of upstream packages
- Providing private packages in a standardized manner.
For this purpose, three components are required:
- A private index. This should be served over HTTP, with the index.html file containing links to available packages (see http://pypi.python.org/simple/ for a typical simple index. It can be protected by basic auth (e.g for private packages)
- Properly configured local installation tools. The target is to have those tools looking at the private repository alongside PyPI.
- Proper setup.py scripts for private packages, to prevent upload to the public PyPI repository.
Setting up the private index
Many tools exist for this, with two major families:
- Some provide only the 'mirror/index' feature (e.g https://github.com/brutasse/autoindex/)
- Others also support the upload and register commands.
If your repository contains private packages, you should serve it only over HTTPS, since most Python packaging tools only use basic authentication.
Setting up the local tools
The ~/.pip/pip.conf file holds all pip-related configurations. It mostly supports a extra-index-url option for providing extra repository URLs:
[global] ; Low timeout timeout = 20 ; Custom index extra-index-url = https://<user>:<pass>@pypi.private.example.org/
It is also possible to completely replace the PyPI index by setting the index-url option instead.
The --extra-index-url can also be provided as a command line option. An alternate PIP configuration file can be used by setting the PIP_CONFIG_FILE env variable.
For distribute/distutils, use the .pypirc configuration file. This is mostly useful to register username/password combinations for private repositories:
[distutils] index-servers = pypi my-private-repo [pypi] username=<your_pypi_username> password=<your_pypi_password> [my-private-repo] repository=https://pypi.private.example.org/ username=<your_private_username> password=<your_private_password>
Setting this will configure the login/pass to use for python setup.py register, upload, ... commands.
Protecting the private packages
I couldn't find any existing package for restricting upload/register/... of a package to a private repository, so I wrote the restricted_pkg package for that.
This package provides a custom setup() function, which performs the following actions:
- Restrict upload, register and upload_docs command to the private repository, using the stored credentials or prompting if needed
- Use the private repository along with PyPI for the install and easy_install commands. The private repository may replace PyPI if the --disable-pypi option is provided.
This means that the setup.py script for such packages should look like:
from restricted_pkg import setup setup( private_repository="https://@pypi.private.example.org/", install_requires=[ 'restricted_pkg', ], )
The @ in the repository URL indicates that this repository requires auth, but avoids committing a username:password string in the setup.py script.