4

I have quite some difficulties installing aws cli on a self-hosted Azure DevOps Windows agent as part of a pipeline. Does anyone have a working solution?

What I have tried so far:

a) install aws cli via pip

Firstly, I install Python (tried both; Python2 and Python3) within the pipeline (Powershell) by downloading and extracting a tar file. Example:

(New-Object System.Net.WebClient).DownloadFile("$(python3_download_url)", ".\Python.tar")
Expand-7Zip .\Python.tar .

Secondly, I use the freshly installed Python for installing aws cli. Example:

$(PYTHON364EXE) -m pip install awscli

Aws cli is then installed in the SCRIPTS directory, as expected.

When I try to execute aws.cmd, I receive an error message:

File association not found for extension .py
'""' is not recognized as an internal or external command,
operable program or batch file

Similar issues are described here: 1, 2, 3. The suggested solution is, to associate python files with the freshly installed python executable:

assoc .py=pyautofile
ftype pyautofile="$(PYTHON364EXE)" "%1" %*

I do not feel this is appropriate for an azure agent:

  1. It does not feel right to use Admin permissions for a simple build
  2. I am worried that the proposed change would impact all agents and the entire host server, while I need it for only one specific agent.

Another option would be, to install the required tools system-wide (python, aws). That would, however, make it difficult to try out a specific (new?) version of python or aws cli.

b) install aws cli via extracted msi package

... extracted the contents of the msi package via lessmsi, then run the aws cli:

C:\AWS> aws
failed to create process.

This obviously does not work either. - It actually does work properly with aws cli version 2, but not with aws cli version 1. Version 2 is not recommended for use in production yet.

c) Use AWS Tools for Microsoft Visual Studio Team Services

This is an extension for Azure DevOps, maintained by AWS. It works, but a) we have legacy scripts of a certain complexity using aws cli we cannot migrate, and b) there might be features available in aws cli which are not available in the extension.

d) call aws as python module

$(PYTHON364EXE) -m awscli --version

This works, but it means I would have to adapt and modify the legacy script.

1
  • Were you able to come up with a solution? I am trying to run the AWS CLI command aws s3 sync <s3 source> <azure data lake dest>. From the research, I've done, it appears that boto3 doesn't support the sync command. I have thought about a few different design patterns (including spinning up an Azure Data Science VM), but haven't come up with an elegant, simple, and cost efficient solution. – datalifenyc Mar 1 at 16:54
3

The best way to have AWS CLI tooling installed to your custom agent is by installing from Visual Studio Marketplace. AWS CLI is part of AWS Tools for Azure DevOps Service and Azure DevOps Server (the on-premise Azure DevOps) under the name of "AWS Tools for Microsoft Visual Studio Team Services".

Installing this AWS Tools will enable AWS tooling on CI (build pipeline) and CD (release pipeline).

This is the official landing page of AWS Tools in Visual Studio Marketplace: https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools

Official documentation is available at: https://docs.aws.amazon.com/vsts/index.html

2
  • Thanks - that would be the solution "c" as suggested above. Unfortunately, I need the CLI, because our existing scripts use them. The aws vsts tools do not contain the aws cli. This would therefore mean that we have to migrate everything. Any other suggestion? – hey Nov 25 '19 at 22:11
  • 1
    The AWS Toos for Azure DevOps does have the AWS CLI as long as your build agent runs on Windows. Please read the description of AWS Tools on the Visual Studio Marketplace link. – Eriawan Kusumawardhono Nov 26 '19 at 4:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.