Skip to main content
The 2024 Developer Survey results are live! See the results

Questions tagged [testing]

Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Use this tag on questions that talk about the testing of systems or software that would be on-topic for this site.

Filter by
Sorted by
Tagged with
0 votes
2 answers
137 views

Docker gets stuck when running 16 tests

I'm running my dotnet 7.0 tests using "testcontainers" on windows machine. The tests are parallelized integration tests which use SQL Server inside a Linux docker container. Each test ...
8 votes
3 answers
845 views

How to test a cluster of microservices?

I took a look into Server Spec and it says it clearly: Remark: serverspec test suites are meant to be run against a single machine (or docker container). In other words, you should not try to ...
6 votes
1 answer
255 views

What are the metrics to consider when doing API performance testing?

I want to establish the benchmark behavior of my API through performance testing. The purpose is to ensure that the product meet or exceed requirements such as: Throughput Response time Stability ...
0 votes
1 answer
314 views

Deploy A War/Ear To Container Marked build As failure When Deploying To Tomcat 9 Server

When I was Deploying A Sample Application On Tomcat9 I Faced This Issue. This is the Tomact9 Users File where I added credentials do I need to add anything more, Is There Any Configuration Which I ...
2 votes
1 answer
100 views

How to test production Ansible execution without having to provision target

We use ansible for our day to day operation and so far so good. We have been looking for a way to run local test etc just like chef guys and found molecule that we use with testinfra/vagrant. While I ...
0 votes
1 answer
42 views

QA instability due to lack of stable UAT/staging environment

Our testing teams are facing extensive downtime due to testing in a shared QA environment. When our front end teams perform end to end testing, their test (QA) environment points to the back end ...
4 votes
1 answer
1k views

How to share variables across instances?

I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the ...
0 votes
2 answers
2k views

TeamCity run step in docker

I'm trying to set up TeamCity to run tests on a .NET project. I have installed TeamCity and its agent (with access to docker) using docker compose : teamcity: image: jetbrains/teamcity-server ...
1 vote
1 answer
118 views

User-level secrets in GitHub Actions

I would like to create a GitHub workflow for running tests against a REST API. The API needs a key that is always tied to a user, there is no global "testing" key. Is there a way for each ...
1 vote
1 answer
282 views

How does one perform systems testing against multiple interdependent machines?

I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc....
5 votes
2 answers
940 views

How to test the CI/CD pipeline itself?

I have gitlab CI/CD defined through .gitlab-ci.yml. It runs different jobs if the pipeline is triggered manually, from a merge request or from a scheduled pipeline. It accepts a bunch of environment ...
0 votes
1 answer
275 views

Snapshot or Saving Prometheus Data for Performance testing and comparison

I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, ...
1 vote
1 answer
838 views

How can you do pre-merge testing in trunk based development?

Currently, we use a simple branching strategy, with a single main branch and one layer of feature branches; no branches on branches on branches. We also have many pipelines that run against each PR to ...
3 votes
1 answer
642 views

What is a good storage for storing performance results and visualizing in grafana

Intro I am a tester and need to analyze the results of automated tests (response times and etc.) Current situation 1) I run performance tests using locust.io 2) During test execution, response ...
37 votes
6 answers
6k views

How to test provisioning and configuration in Ansible setup?

Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration. I understand a few methods of testing on the configuration side of things but I'm ...
3 votes
3 answers
224 views

Should I shut down the staging environment when the deployment test ends?

I have a small application deployed on an EC2 instance. I need a staging environment so I've thought of two choices: Deploy the same app again on the same instance, and that would be a staging ...
1 vote
1 answer
57 views

Sorting out automated testing in CI pipeline and getting conflicting information related to building, unit testing and Docker images

I seem to get a unique response for every person that I ask, or every article that I read, which is just confusing me even more. Basically, I'm trying to setup automated testing in the CI pipeline ...
2 votes
1 answer
109 views

What is a good strategy to reliably and automatically perform remote system tests of a web service?

In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual ...
1 vote
1 answer
41 views

What is it called to deploy major changes for limited number of customers first?

Is there a common name for this deployment strategy which is to avoid deploying major updates across all the customers straight away but rather limit the initial number to see their feedback, find any ...
3 votes
0 answers
47 views

Is there an all-in-one control center software that can be used to manage build and test execution and to visualize test results?

If there are no constraints with regards to other components used in the system at all, do you know a software that can be used as a sort of "cockpit" / control center to visualize test ...
1 vote
1 answer
49 views

Testing docker image builds when low level libraries change

Background In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:...
25 votes
4 answers
47k views

Ansible: Other option available for telnet check of open ports?

I'm new to Ansible. Here's my task ... I have 400+ hosts, and I need to verify if 5 different ports are open from their end to our web server. Individually, I could log in and run: telnet ...
2 votes
2 answers
176 views

How can I test native macOS UI software in a container before deployment?

As part of the CI setup, I'd like to run (automated) tests on native macOS GUI software in a 'safe environment' (i.e. without messing with the host software installation too much). Virtual machines (...
8 votes
3 answers
1k views

How to implement a frozen test environment?

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?": This environment usually freezes during the tests. My ...
6 votes
1 answer
1k views

How to avoid a regular user can only test in production?

When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here: How to avoid a regular user can only test in production? There ...
8 votes
2 answers
340 views

What is "black box" testing? [closed]

What is "black box" testing and how it is different from ordinary testing? For example black box tests prior to deploying to production. Note: "black box" testing is a common requirement when ...
2 votes
0 answers
92 views

How to deploy large web app in container minimal data transfer? [closed]

In current deployment of a web app we are using svn to deploy a specific revision from a repository. The amount of data transferred is just the diff between the currently deployed revision and the ...
14 votes
1 answer
3k views

Testing Jenkins jobs and pipelines

Currently, we have a fair number of Jenkins jobs and pipelines for builds, testing, deployments and other automated activities. Every time we change or add a new job, we only test it manually - e. g. ...
0 votes
1 answer
460 views

Safari test automation using Robot Framework via Jenkins

I spent about one week trying to make my Robot Framework tests work for Safari browser in MacOS, when launched by Jenkins via ssh. The main issue was that Robot couldn't open a browser when executed ...
3 votes
1 answer
2k views

Is it possible to run every testing.xml file with different jenkins job?

I've around 100 test method in my testing.xml file. and that testing.xml configured in pom.xml file so I'm able to run my all those testcase as maven test and I'm getting the proper reports and all. ...
3 votes
1 answer
452 views

Trunk Based Development - release branch testing

We are currently using Git flow branching strategy and looking to go to a trunk based strategy - although we are quite a ways away from getting there I am trying to get all my ducks in a row, ...
51 votes
6 answers
23k views

How to test a Terraform Configuration?

If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / ...
2 votes
1 answer
49 views

A name for testing software migrated to a new environment against original application

This is more of an english language question - we have various types of tests - unit tests, A/B tests, etc... Suppose I need to migrate code to a different language, machine, or any other different ...
8 votes
3 answers
3k views

How to create staging servers for multiple git branches?

I have to create a new staging process for our development and testing. At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may ...
6 votes
0 answers
90 views

Blackbox testing vs openbox testing applications in a containerized environment [closed]

We use containers to deploy our Node.js applications. But currently, the development and a good share of testing is done by the developers on their own host system and thus using different versions of ...
1 vote
1 answer
102 views

Is Cypress ok for acceptance tests on APIs in a CI/CD pipeline?

I have a small team but a gigantic application. We don't have PHP unit testing, so we will be using Cypress as a baseline for quality. Everything will go through a CI/CD pipeline and use regression ...
1 vote
1 answer
280 views

Azure DevOps Server: Why does a Visual Studio Test step take so long to run unit tests?

In an Azure DevOps Server build system installed on premises, I have a Build Pipeline with a Visual Studio Test step. The step is configured to discover and run XUnit unit tests in a DLL. The step is ...
0 votes
1 answer
670 views

Running Protractor tests in docker through teamcity

I have a project for which I am using Teamcity to automate deployment, and I would also like to automate testing. It is an angular frontend, and we have some protractor tests. I would like to run ...
1 vote
2 answers
80 views

Code coverage versus functional coverage

We are working on a big Java project where we use JUnit tests. Our Jenkins helps us to monitor the code coverage and shows us the progress we make with this. But there seems to be something lacking ...
4 votes
2 answers
11k views

How to export Cucumber Reports via Jenkins Pipeline

I'm using the Cucumber Reports Plugin for Jenkins and I want to be able to export the HTML reports so they can be posted on an external portal. I can see that there is a folder named cucumber-html-...
5 votes
2 answers
1k views

Best practices for CI for embedded platforms (ESP32/Arduino)

Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools. ...
2 votes
0 answers
575 views

How to ensure testing execution order with GitLab CI? [closed]

I have about half a dozen of integration tests (handled with Robot Framework) to run in a GitLab CI pipeline. Once upon a time, we did not care about the execution order of these tests. The ...
17 votes
1 answer
6k views

A systematic way to test RESTful APIs with curl?

I have noticed during integration testing that I actually work on recurring use cases with restful APIs (or in general HTTP interfaces) I check here and there with bash+cURL. It starts looking quite ...
1 vote
1 answer
1k views

Jmeter simulating traffic and avoiding 429

I am using Jmeter to investigate the performance and capacity needs of a few web services. Jmeter sends traffic to the web-facing url (served by Nginx), with a rising rate, to see where the service ...
12 votes
4 answers
948 views

Infrastructure as code and TDD

Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while ...
4 votes
1 answer
3k views

How do I make read only mount in Docker container writable?

When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I ...
3 votes
1 answer
64 views

Tool preferences for rapidly spinning up and spinning down k8s clusters

My team hosts several completely different projects on a production k8s cluster with a given configuration. We use AWS (and are starting to use Google Cloud). We have a QA cluster as well, that "sort ...
2 votes
0 answers
79 views

What integration test framework has the best docker support? [closed]

We need integration testing framework - pass a url pass requests to that url and get responses and evaluate repsonses. Requirements: 1. We want a tool that can easily implement into a CICD pipeline ...
0 votes
1 answer
881 views

Automated Integration Testing of an API in Azure Functions using VSTS

I am building a HTTP API as an Azure Function App in C# and I would like to run automated integration tests (calls to the API) as part of the CI/CD. Is this possible to do using VSTS? I have ...
23 votes
5 answers
1k views

How to avoid continuous integration-caused instabilities in test environments?

Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of ...