I have quite some difficulties installing aws cli on a self-hosted Azure DevOps Windows agent as part of a pipeline. Does anyone have a working solution?
What I have tried so far:
a) install aws cli via pip
Firstly, I install Python (tried both; Python2 and Python3) within the pipeline (Powershell) by downloading and extracting a tar file. Example:
(New-Object System.Net.WebClient).DownloadFile("$(python3_download_url)", ".\Python.tar")
Expand-7Zip .\Python.tar .
Secondly, I use the freshly installed Python for installing aws cli. Example:
$(PYTHON364EXE) -m pip install awscli
Aws cli is then installed in the SCRIPTS directory, as expected.
When I try to execute aws.cmd, I receive an error message:
File association not found for extension .py
'""' is not recognized as an internal or external command,
operable program or batch file
Similar issues are described here: 1, 2, 3. The suggested solution is, to associate python files with the freshly installed python executable:
assoc .py=pyautofile
ftype pyautofile="$(PYTHON364EXE)" "%1" %*
I do not feel this is appropriate for an azure agent:
- It does not feel right to use Admin permissions for a simple build
- I am worried that the proposed change would impact all agents and the entire host server, while I need it for only one specific agent.
Another option would be, to install the required tools system-wide (python, aws). That would, however, make it difficult to try out a specific (new?) version of python or aws cli.
b) install aws cli via extracted msi package
... extracted the contents of the msi package via lessmsi, then run the aws cli:
C:\AWS> aws
failed to create process.
This obviously does not work either. - It actually does work properly with aws cli version 2, but not with aws cli version 1. Version 2 is not recommended for use in production yet.
c) Use AWS Tools for Microsoft Visual Studio Team Services
This is an extension for Azure DevOps, maintained by AWS. It works, but a) we have legacy scripts of a certain complexity using aws cli we cannot migrate, and b) there might be features available in aws cli which are not available in the extension.
d) call aws as python module
$(PYTHON364EXE) -m awscli --version
This works, but it means I would have to adapt and modify the legacy script.
aws s3 sync <s3 source> <azure data lake dest>. From the research, I've done, it appears that boto3 doesn't support thesynccommand. I have thought about a few different design patterns (including spinning up an Azure Data Science VM), but haven't come up with an elegant, simple, and cost efficient solution.