Questions tagged [python]
Python is a dynamic and strongly typed programming language designed to emphasize usability.
86
questions
0
votes
0
answers
17
views
Python scripts executor
I have multiple python scripts processing events from kafka sequentially. Each script has to be executed on remote machine (not all users have access to). I am looking for a way to automate script ...
- 11
1
vote
0
answers
53
views
Using a different Python version in AWX
I updated a recent playbook module that says I need to use at least Python version 3.8. I run this playbook on our AWX controller. When I run the cmd module and do a "which python3" I get ...
1
vote
0
answers
74
views
Root cause for my problem when deploying Spacy on AWS lambda (Klayers)?
I have been following this guide https://towardsdatascience.com/quickstart-spacy-on-aws-lambda-up-and-running-in-10-minutes-e347037a0ef8 to get Spacy up and running on AWS lambda. I have followed the ...
- 111
0
votes
0
answers
9
views
How to access docker container application from Google Compute Engine?
I am trying to deploy a Django application in Google Compute Engine using Docker and docker-compose. I installed docker-compose using this guide. The application I am trying to deploy is in ...
- 101
0
votes
0
answers
24
views
s3: reading parquet files with a subset of columns as argument; does this reduce data transfer?
I have set of large tables stored as parquets on s3.
In python, I'm using:
pd.read_parquet(...,columns=columns)
I'm reading the files directly from s3, without any database engine whatsoever for ...
0
votes
1
answer
22
views
Options to build a production server as a solo python developer?
I'm new to DevOps and overwhelmed with all the options.
I write python web applications as a solo developer, on my local machine. I have a "staging" server and I have multiple websites under ...
0
votes
0
answers
25
views
How can I force an immediate run of a job trigger
How can I activate a job trigger right after I create it. I am using this resource to create a job trigger - https://cloud.google.com/dlp/docs/creating-job-triggers and trying to activate the job ...
- 101
0
votes
1
answer
473
views
Multistage docker build for Python distroless image
This is my Dockerfile for distroless image similar to this example
FROM python:3.9-slim AS build-venv
RUN python3 -m venv /venv
# other installation steps go here
RUN /venv/bin/pip install --upgrade ...
- 103
0
votes
1
answer
758
views
How to get gcp project name by project id
Using cloud function - trying to make get call to get the project name and with the id - https://cloud.google.com/resource-manager/docs/creating-managing-projects#listing_projects
from ...
- 101
0
votes
1
answer
90
views
Publish python package into private repository behind VPN
I'm using Github Actions to publish my python packages automatically. When working with PyPI everything works fine and providing different config to point on different repository is not an issue.
But ...
- 103
0
votes
1
answer
193
views
Vscode/pytest gives me an error when importing
I asked this question on the regular stackoverflow but i think you guys will be much more competent about this topic. My guess is that is some sort of python setup issue.
When I write an import line ...
0
votes
0
answers
42
views
Should I build an API for my data ingestion/processing pipeline? (previously only backend, now building frontend)
I've built a fairly comprehensive data pipeline project. What it does:
ingests data from different sources
processes data, e.g. to calculate individual-level scores regarding a specific topic
sends ...
- 101
0
votes
0
answers
799
views
WORKDIR not working for docker in windows
I have a main.py file and a Dockerfile. The main.py file downloads and saves some .txt files. So, it is very necessary for me to specify the working directory.
I am working in a Windows machine.
The ...
1
vote
0
answers
647
views
python `subprocess.run()` returning unexpected empty stdout in gitlab-ci
I'm attempting to have my python blackbox test suite run through gitlab-ci. When running the test through pipenv run pytest on my personal machine everything executes correctly, but when running on ...
- 11
2
votes
1
answer
58
views
CI pipeline including upload to pypi repo
How should a ci pipeline be set up that publishes a python wheel to a private pypi server?
Say I have a publish job like this one:
publish:
image: $PYTHON_IMAGE
stage: publish
before_script:
...
- 121
0
votes
1
answer
905
views
My Python application doesn't communicate with MySQL inside docker. Can someone help me?
The connection is not succeeding in docker. Look at my settings:
Dockerfile
FROM python:3.8.5-alpine
WORKDIR /app
COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN pip install -...
0
votes
0
answers
955
views
Cant access gitlab ci variables from inside python script building with maven
I have a python project which I build using maven. In my .gitlab-ci.yaml file I set some global variables.
variables:
MAVEN_BASE_CLI_OPTS: ""
MAVEN_CLI_OPTS: ${MAVEN_BASE_CLI_OPTS}
...
0
votes
1
answer
3k
views
Python not connecting to mongodb in docker container
I have a simple python script which fetches certain values from the mongodb database. The script works fine on my local machine(host machine) but when i containarized my aplication it doesnt work. ...
- 101
2
votes
0
answers
406
views
How to speed up environment loading on network drive?
Background:
I have about 15 AWS EC2 instances running Windows. They each have local drives and share a single network drive. Each needs to be able to run certain in-development python tools from ...
- 21
1
vote
1
answer
3k
views
how to create dynamic stages in jenkins pipeline
I want to use a python script as input to create stages in jenkins pipeline
test.py
{'job-name': 'Test1', 'action': 'a', 'targetname': 'x'}
{'job-name': 'Test2', 'action': 'b', 'targetname': 'y'}
{'...
- 11
0
votes
0
answers
3k
views
how to execute a python script in jenkins pipeline as a stage and use the output of python script to declare further stages in pipeline as next stage
I have written a python script which provides me a output like this as a dictionary,
test.py:
{'job-name': 'Test1', 'action': 'a', 'targetname': 'x'}
{'job-name': 'Test2', 'action': 'b', 'targetname': ...
- 11
0
votes
0
answers
913
views
Ansible in venv: how to pip install globally on localhost?
Ansible is installed and is running in a virtual environment:
(env) ansible-playbook -vvvv -i localhost playbook.yml
I'm trying to install a Python package globally, but it either installs in venv ...
- 109
1
vote
1
answer
36
views
How to ensure data protection on remote server?
We have sensitive data from a customer and need to train a Neural Network for our customers.
However we have no severs to train the Neural Networks, so we want to rent GPU servers. There are many ...
- 11
1
vote
0
answers
67
views
Ansible : 'NetworkManagementClient' object has no attribute 'private_endpoints'
I try to deploy a private endpoint for a storage account with Ansible. I am using azure.azcollection.azure_rm_privateendpoint. My playbook code is as below:
- name: Create PE
hosts: localhost
...
- 111
3
votes
2
answers
191
views
Parametrize gitlab ci pipeline
There is the following simple pipeline:
build → test → deploy
that uses specified python version for all stages, for example 3.8.
How can I scale (parametrize) it for python with versions: 3.7, 3.8 ...
- 31
0
votes
1
answer
430
views
AWS FARGATE ECS - Exact steps to deploy new Docker images with aws CLI
We run services in AWS FARGATE... our Docker builds are tagged before we push them to AWS with a Makefile.
I committed changes to a Docker service, ran make, and I see the new Docker image tag when I ...
- 103
1
vote
1
answer
1k
views
How to add a string to a line when it doesn't exist?
I'm trying to solve a problem where I want to add a setting 'pci=noaer' to all kernel definitions in /boot/grub/grub.conf, but only if the setting doesn't exist. I've managed to get the basic ...
- 13
3
votes
3
answers
376
views
pypa/gh-action-pypi-publish token value 'does not start with "pypi-"'
I am trying to use the pypa/gh-action-pypi-publish action in GitHub to push a Python package to PyPI. I'm in the team that owns microbiomedata/nmdc-schema and successfully uses a GitHub action to ...
- 141
2
votes
1
answer
396
views
GitLab Maintain versions of the code in GitLab CI/CD pipeline for the develop branch
I would like to ensure that code merged via merge request in a GitLab from the feature branch to our develop branch has always a new unique version (gitflow strategy).
Version currently stored as the ...
- 121
1
vote
1
answer
169
views
Deploy to Ubuntu server with python script and awsebcli fails
Just yesterday everything worked fine on our AWS ElasticBeanstalk continuous deployment, but today suddenly fails with the following errors:
$ pip install awsebcli -q --upgrade
DEPRECATION: Python 2.7 ...
- 121
1
vote
1
answer
42
views
Testing docker image builds when low level libraries change
Background
In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:...
- 121
1
vote
1
answer
2k
views
Is there any Jenkins plugin for pylint results?
I have Pipeline which run Jenkinsfile. one of the stage is pylint:
python -m pylint --rcfile=.pylintrc --exit-zero ${MODULE} > ${WORKSPACE}/testreport/pylint.log
I know there was used to be ...
- 329
0
votes
1
answer
409
views
how can I import python module tkinter in gitpod
I can't import pyton module tkinter in my gitpod workspace.
My .gitpod.Dockerfile contains:
FROM gitpod/workspace-full-vnc
USER root
RUN apt update && apt install -y python-tk python3-tk tk-...
1
vote
0
answers
50
views
How to execute a "hello redis" sample python script on a container from the host?
How do I execute this "Hello Redis!!!" sample from the host?
root@mordor:~#
root@mordor:~# docker container ls
CONTAINER ID IMAGE COMMAND CREATED ...
1
vote
1
answer
3k
views
403 forbidden error on calling jenkins post api from python script
I am trying to build a python script that will build jobs externally but when I am executing it, throws 403 forbidden error for build jobs call other get APIs are working fine. Below is the python ...
- 11
0
votes
1
answer
32
views
Strange behaviour with docker-py
Consider the following setup.
Precondition: have a healthy image i.e. service starts up.
E.g.
docker run -d IMAGE
Then, with the Python plugin docker-py, consider the following code which should ...
- 6,782
1
vote
1
answer
26
views
Tool or way to support handling dependencies of modules in various platforms (for machine learning in Python)
I am Data Scientist and I have a problem with setting of versions modules like Keras, Tensorflow PyTorch, etc.
Although I am using tools like virtuaenv or conda, my code isn't cross-platform, version ...
- 111
2
votes
1
answer
75
views
Continuous deployment of package to virtualenv
My colleagues don't want to create their own virtualenv and deploy my tools from git on their own as part of their automation. Instead they want the tools pre installed on a shared server.
So I was ...
- 137
1
vote
2
answers
347
views
How to check minimum code coverage in pull request changes?
I want to force the developers to write unit tests for all new code they wrote. The code is written in python. I want to do it in the Jenkins pipeline. And here is a question. Is there any available ...
- 21
2
votes
0
answers
414
views
Connection refused when CURLing to docker container from within Jenkins pipeline
I have a situation with a CI Pipeline I'm trying to implement using Jenkins and Docker .
Inside my pipeline I want to: download a python uwsgi web app, build it in a docker container, run it, then ...
- 29
2
votes
1
answer
367
views
Is it a bad idea to use a web app including nginx, gunicorn and flask in a single docker container? [duplicate]
I want to deploy a web app as a docker container. Currently I need to set up nginx, gunicorn and flask to realize this. There are many guides around to do it in docker but always divide it in two ...
- 23
0
votes
1
answer
2k
views
How to inject credentials to python virtual env in jenkins?
I am trying to inject my credentials
to a Jenkins Freestyle project.
The project is building a python project in virtual environment
with "behave" created with ShiningPanda plugin
can someone help ...
0
votes
1
answer
5k
views
How to save groovy output to pipeline variable
I have the below code in the jenkisn pipeline:
stage ("amd_distribution_input_transformation"){
steps{
script{
amd_distribution_input_transformation url: ...
- 329
4
votes
1
answer
2k
views
How to install aws cli on a self-hosted azure DevOps Windows agent?
I have quite some difficulties installing aws cli on a self-hosted Azure DevOps Windows agent as part of a pipeline. Does anyone have a working solution?
What I have tried so far:
a) install aws cli ...
- 141
6
votes
1
answer
732
views
What's the benefit of using Nginx to serve a Flask API on AWS?
I want to deploy a Flask API on AWS using Docker. A common way seems to be using a combination of Nginx + app server (e.g Gunicorn) + Flask app.
The reason to use app server seems obvious to me. ...
- 61
2
votes
1
answer
710
views
Share variables across instances
I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the ...
- 121
0
votes
1
answer
469
views
How to run WSGI Python server without Apache or NGINX [closed]
Is there a way to run a Python server that uses WSGI bindings directly? I am trying to go with the one process per container philosophy which will naturally get rid of Apache/NGINX from the container ...
- 1
1
vote
2
answers
4k
views
Unit Test AWS Glue job in CodePipeline
I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, ...
- 3,123
1
vote
1
answer
2k
views
How to downgrade Jinja for Ansible?
I'm using Ansible 2.8.0 (ansible --version installed by brew on macOS) which is using Jinja 2.10.
I've tried to downgrade Jinja to 2.8 via pip3 install jinja2==2.8, and I can confirm the right ...
- 7,621
0
votes
1
answer
6k
views
Python Script in Jenkins
I am trying to run a python script in Jenkins. My Jenkins job is integrated with Jira & on build its gives JIRA_ISSUE_KEY. I want to save the value of JIRA_ISSUE_KEY in a variable.
import ...
- 77