6

Currently, we build our application on a build server where we create a virtual environment using virtualenv command, install all the Python dependencies into it, then "patch" it using the following bash commands to make it relocatable:

# Adopt virtualenv to the target system
echo "* Patching virtual envirinment for target system"
for f in "${VENV_PATH}/bin"/* ; do
    if [ -n "$(file "${f}" | grep "text")" ] ; then
    echo "  + ${f}"
        sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
    fi
done
rm -rf "${VENV_PATH}/lib64"
ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"

Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via yum.

The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to avoid installing the same dependencies on all the target servers.

The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).

Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate pip installation commands with, for example, pip-accel or caching) What are the general ways to tackle this problem?

  • 1
    unfortunately the --relocatable flag for virtualenv never did work well, as documented here virtualenv.pypa.io/en/stable/userguide/… If the project found it fragile I bet it will be for you too. deb's have dh-virtualenv but I don't know of one for rpm, but how about pip freeze and wheel? pip wheel --wheel-dir=./artifacts -r requirements.txt Package ./artifacts into your RPM, create and activate the venv then run pip install ./artifacts/ We shove our wheels into a gcp bucket and just add the urls for builds via jenkins. – gdahlm Sep 10 '17 at 0:21
4
+50

What are the general ways to tackle this problem?

The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.

Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:

python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.