Unable to run `conda activate` from Jenkins pipeline

Question

I'm running Conda 4.6.14 on Ubuntu 16.04.

I thought about using virtual environments with Conda in my Jenkins pipelines that needed Ansible, by having all nodes use the same environment, ansible-env to run playbooks. While the process seems pretty straight forward, I'm unable to call conda activate ansible-env, because it results in the following error:

+ conda activate ansible-env

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Here is the code I'm using to activate...pretty basic

sh("conda activate ansible-env")
sh("ansible --version")
sh("conda deactivate")

I ran conda init --system on the build node as the Jenkins user and then rebooted. No change.

I've spent a great deal of time trying to google a solution, but everything I've tried hasn't worked.

What I've attempted:

• Tried running sh("conda init bash") in the pipeline.
• Executing . /home/jenkins/miniconda3/profile.d/conda.sh from within the pipeline.
• Made sure the bin directory for miniconda3 is set in $PATH.
• Ran dpkg-configure dash to make sure /bin/bash is the default shell on the node.
• Tried sourcing the /home/jenkins/.bashrc file
• Calling the conda executable via the absolute path.
• Tried using source ansible-env or activate ansible-env without conda.

I'm pretty sure the issue has to do with the following lines added to ~/.bashrc of the jenkins user after running conda init --system, but still not being accessible from that same user when a job is running. Unfortunately, I've hit a wall and not sure where to go from here.

NOTE: Please don't suggest switching to VirtuanEnv unless there's a specific reason Conda won't work with Jenkins. I'm aware of that route, but went with conda for...reasons.

Answer 1

What we do is download and install miniconda in the first step in our (declarative) pipeline and create the environment(s) from a yaml file.

sh '''#!/usr/bin/env bash
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -nv -O miniconda.sh
bash miniconda.sh -b -p $WORKSPACE/miniconda
conda config --set always_yes yes --set changeps1 no
conda update -q conda

conda env create -f envs/ansible-env.yaml
'''

In each step we start the conda environment like so:

sh '''#!/usr/bin/env bash
source $WORKSPACE/miniconda/etc/profile.d/conda.sh
conda activate miniconda/envs/ansible-env/

# do stuff
'''

Answer 2

I've encountered the same issue when trying to use conda environments in CI builds and Docker images. The conda documentation does provide an example when using Travis CI, but it does not work for me.

However, via this blog I found a working solution! Instead of activating your environment, you can use it by calling conda run -n my_env python my_file.py. So, keep everything like it was until you activate your environment.

These build specs worked for me when using AWS CodeBuild:

{
  "version": "0.2",
  "phases": {
    "pre_build": {
      "commands": [
        "wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh -q",
        "bash miniconda.sh -b -p $HOME/miniconda",
        ". \"$HOME/miniconda/etc/profile.d/conda.sh\"",
        "hash -r",
        "conda config --set always_yes yes --set changeps1 no",
        "conda update -q conda",
        "conda info -a",
        "conda env create -q -p ./py_env -f environment.yml"
      ]
    },
    "build": {
      "commands": [
        "conda run -p ./py_env python app.py"
      ]
    }
  }
}

Answer 3

As a suggestion... Can you try calling the absolute path of the python executable in the environment? I mean: when you change conda environments, what happens is that the python executable you call changes path; you can find which python executable would be executed by

which python

Inside the environment. Then just forget about calling conda in the Jenkins pipeline. That should work if that’s what you’re changing environments for. Or are you changing the conda environment for some other reason?

EDIT: the same would happen when calling ansible as a library installed in the environment. It will change path to something like:

~/conda/envs/your_env/lib/ansible