Skip to main content

Questions tagged [testing]

Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Use this tag on questions that talk about the testing of systems or software that would be on-topic for this site.

Filter by
Sorted by
Tagged with
51 votes
6 answers
23k views

How to test a Terraform Configuration?

If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / ...
Richard Slater's user avatar
37 votes
6 answers
6k views

How to test provisioning and configuration in Ansible setup?

Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration. I understand a few methods of testing on the configuration side of things but I'm ...
Naphta's user avatar
  • 627
25 votes
4 answers
46k views

Ansible: Other option available for telnet check of open ports?

I'm new to Ansible. Here's my task ... I have 400+ hosts, and I need to verify if 5 different ports are open from their end to our web server. Individually, I could log in and run: telnet ...
AWhitaker's user avatar
  • 361
23 votes
5 answers
1k views

How to avoid continuous integration-caused instabilities in test environments?

Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of ...
Pierre.Vriens's user avatar
  • 7,205
17 votes
1 answer
6k views

A systematic way to test RESTful APIs with curl?

I have noticed during integration testing that I actually work on recurring use cases with restful APIs (or in general HTTP interfaces) I check here and there with bash+cURL. It starts looking quite ...
Ta Mu's user avatar
  • 6,782
14 votes
1 answer
3k views

Testing Jenkins jobs and pipelines

Currently, we have a fair number of Jenkins jobs and pipelines for builds, testing, deployments and other automated activities. Every time we change or add a new job, we only test it manually - e. g. ...
alecxe's user avatar
  • 849
11 votes
4 answers
942 views

Infrastructure as code and TDD

Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while ...
JackLeo's user avatar
  • 213
10 votes
3 answers
1k views

How to test VM provisioning script without provisioning

Currently I'm on a state that testing costs me money and a lot of time... Background: I'm deploying VMs at softlayer and using a post-deployment script(bash) that will install every software that I ...
Lucas Mattos's user avatar
8 votes
3 answers
848 views

What is the future role of tester?

If the new paradigm is "You build it, you run it" (Werner Voegels, Amazon CTO) which obviously puts much more responsibility - and pressure - on software engineers, what does this change introduce to ...
Ta Mu's user avatar
  • 6,782
8 votes
3 answers
3k views

How to create staging servers for multiple git branches?

I have to create a new staging process for our development and testing. At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may ...
max1221's user avatar
  • 81
8 votes
2 answers
340 views

What is "black box" testing? [closed]

What is "black box" testing and how it is different from ordinary testing? For example black box tests prior to deploying to production. Note: "black box" testing is a common requirement when ...
kenorb's user avatar
  • 7,901
8 votes
3 answers
1k views

How to implement a frozen test environment?

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?": This environment usually freezes during the tests. My ...
Pierre.Vriens's user avatar
  • 7,205
8 votes
3 answers
827 views

How to test a cluster of microservices?

I took a look into Server Spec and it says it clearly: Remark: serverspec test suites are meant to be run against a single machine (or docker container). In other words, you should not try to ...
nicocesar's user avatar
  • 197
7 votes
2 answers
821 views

What are the pros and cons of running application testing during vs. after a docker build?

Our dockerfile creates the application environment (e.g. env variables, install libraries with apt-get etc.). It also pulls and copies the python application code from its git repo. However, we are ...
cammil's user avatar
  • 171
6 votes
1 answer
1k views

How to improve DRP-testing of SE sites?

Have a look at what's described in the question about "Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)", which is about testing of the existing "...
Pierre.Vriens's user avatar
  • 7,205
6 votes
1 answer
1k views

How to avoid a regular user can only test in production?

When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here: How to avoid a regular user can only test in production? There ...
Pierre.Vriens's user avatar
  • 7,205
6 votes
1 answer
143 views

How to test database applications in QA testing environments?

Process Our current process is: Build process produces a windows setup (msi). This setup consists of a web site deployment and also database scripts - which can create a new database or upgrade ...
user5078's user avatar
6 votes
1 answer
236 views

What are the metrics to consider when doing API performance testing?

I want to establish the benchmark behavior of my API through performance testing. The purpose is to ensure that the product meet or exceed requirements such as: Throughput Response time Stability ...
storm's user avatar
  • 1,759
6 votes
0 answers
89 views

Blackbox testing vs openbox testing applications in a containerized environment [closed]

We use containers to deploy our Node.js applications. But currently, the development and a good share of testing is done by the developers on their own host system and thus using different versions of ...
Sylvain Leroux's user avatar
5 votes
2 answers
1k views

Best practices for CI for embedded platforms (ESP32/Arduino)

Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools. ...
Moritz's user avatar
  • 1,217
5 votes
2 answers
912 views

How to test the CI/CD pipeline itself?

I have gitlab CI/CD defined through .gitlab-ci.yml. It runs different jobs if the pipeline is triggered manually, from a merge request or from a scheduled pipeline. It accepts a bunch of environment ...
John Doe's user avatar
4 votes
2 answers
288 views

Continuous Integration and Isolated Feature Testing

I've been given the task at work to move everyone from using TFS Source Control to Git as the Development Team has been lead to believe that Git may well be the silver bullet that will fix alot of the ...
Stu1986C's user avatar
  • 141
4 votes
2 answers
3k views

macOS testing on Desktop: Docker containers vs. Homebrew installation

Often I test some stuff on my MacBook. Mostly database and data store related stuff (MongoDB, MariaDB, Postgres). I install the software with Homebrew and test. I wonder if I should switch to docker? ...
Sybil's user avatar
  • 452
4 votes
2 answers
10k views

How to export Cucumber Reports via Jenkins Pipeline

I'm using the Cucumber Reports Plugin for Jenkins and I want to be able to export the HTML reports so they can be posted on an external portal. I can see that there is a folder named cucumber-html-...
lawnmowerlatte's user avatar
4 votes
1 answer
1k views

How to share variables across instances?

I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the ...
Abel's user avatar
  • 141
4 votes
1 answer
48 views

Automated Testing Jobs

We have multiple environments close to 7 for an application that has UI and also integration server that accepts SOAP xmls. Its hard to maintain the environment integrity when many people working, ...
Tech Dude's user avatar
4 votes
1 answer
3k views

How do I make read only mount in Docker container writable?

When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I ...
Jiri Klouda's user avatar
  • 5,807
3 votes
3 answers
217 views

Should I shut down the staging environment when the deployment test ends?

I have a small application deployed on an EC2 instance. I need a staging environment so I've thought of two choices: Deploy the same app again on the same instance, and that would be a staging ...
mr L's user avatar
  • 131
3 votes
1 answer
438 views

Trunk Based Development - release branch testing

We are currently using Git flow branching strategy and looking to go to a trunk based strategy - although we are quite a ways away from getting there I am trying to get all my ducks in a row, ...
VKR's user avatar
  • 71
3 votes
2 answers
738 views

Best tool for Docker Container test automation?

In my project i am building about 200 containers I want to check in a smoke test if the containers are still running after a rebuild in pipeline. I want to bring up the container with certain ...
Mandragor's user avatar
  • 133
3 votes
1 answer
63 views

Tool preferences for rapidly spinning up and spinning down k8s clusters

My team hosts several completely different projects on a production k8s cluster with a given configuration. We use AWS (and are starting to use Google Cloud). We have a QA cluster as well, that "sort ...
swagrov's user avatar
  • 131
3 votes
1 answer
2k views

Is it possible to run every testing.xml file with different jenkins job?

I've around 100 test method in my testing.xml file. and that testing.xml configured in pom.xml file so I'm able to run my all those testcase as maven test and I'm getting the proper reports and all. ...
NarendraR's user avatar
  • 233
3 votes
1 answer
637 views

What is a good storage for storing performance results and visualizing in grafana

Intro I am a tester and need to analyze the results of automated tests (response times and etc.) Current situation 1) I run performance tests using locust.io 2) During test execution, response ...
GrimSmiler's user avatar
3 votes
0 answers
46 views

Is there an all-in-one control center software that can be used to manage build and test execution and to visualize test results?

If there are no constraints with regards to other components used in the system at all, do you know a software that can be used as a sort of "cockpit" / control center to visualize test ...
Hunter2's user avatar
  • 31
2 votes
1 answer
108 views

What is a good strategy to reliably and automatically perform remote system tests of a web service?

In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual ...
d33tah's user avatar
  • 71
2 votes
1 answer
49 views

A name for testing software migrated to a new environment against original application

This is more of an english language question - we have various types of tests - unit tests, A/B tests, etc... Suppose I need to migrate code to a different language, machine, or any other different ...
kabanus's user avatar
  • 123
2 votes
1 answer
2k views

Validating custom Jenkinsfile

Currently we're using Jenkins shared libraries which we're planning to use with custom Jenkinsfiles. Idea here is that developers should be able to manage those custom Jenkinsfiles without any ...
zrks's user avatar
  • 21
2 votes
1 answer
82 views

What are best practices for running Pipeline tests into a CI framework?

According to this github repo it is possible to test the Jenkinsfile, but this is run by Jenkins and if there are issues it will not run. Should one run such unit test in another CI? What are best ...
030's user avatar
  • 13.2k
2 votes
2 answers
163 views

How can I test native macOS UI software in a container before deployment?

As part of the CI setup, I'd like to run (automated) tests on native macOS GUI software in a 'safe environment' (i.e. without messing with the host software installation too much). Virtual machines (...
Frerich Raabe's user avatar
2 votes
1 answer
98 views

How to test production Ansible execution without having to provision target

We use ansible for our day to day operation and so far so good. We have been looking for a way to run local test etc just like chef guys and found molecule that we use with testinfra/vagrant. While I ...
black sensei's user avatar
2 votes
0 answers
570 views

How to ensure testing execution order with GitLab CI? [closed]

I have about half a dozen of integration tests (handled with Robot Framework) to run in a GitLab CI pipeline. Once upon a time, we did not care about the execution order of these tests. The ...
avazula's user avatar
  • 949
2 votes
0 answers
91 views

How to deploy large web app in container minimal data transfer? [closed]

In current deployment of a web app we are using svn to deploy a specific revision from a repository. The amount of data transferred is just the diff between the currently deployed revision and the ...
Jiri Klouda's user avatar
  • 5,807
2 votes
0 answers
79 views

What integration test framework has the best docker support? [closed]

We need integration testing framework - pass a url pass requests to that url and get responses and evaluate repsonses. Requirements: 1. We want a tool that can easily implement into a CICD pipeline ...
DemiSheep's user avatar
  • 163
1 vote
1 answer
799 views

How can you do pre-merge testing in trunk based development?

Currently, we use a simple branching strategy, with a single main branch and one layer of feature branches; no branches on branches on branches. We also have many pipelines that run against each PR to ...
James Geddes's user avatar
1 vote
1 answer
101 views

Is Cypress ok for acceptance tests on APIs in a CI/CD pipeline?

I have a small team but a gigantic application. We don't have PHP unit testing, so we will be using Cypress as a baseline for quality. Everything will go through a CI/CD pipeline and use regression ...
Daniel Vianna's user avatar
1 vote
1 answer
1k views

Jmeter simulating traffic and avoiding 429

I am using Jmeter to investigate the performance and capacity needs of a few web services. Jmeter sends traffic to the web-facing url (served by Nginx), with a rising rate, to see where the service ...
Bruce Becker's user avatar
  • 3,603
1 vote
2 answers
78 views

Code coverage versus functional coverage

We are working on a big Java project where we use JUnit tests. Our Jenkins helps us to monitor the code coverage and shows us the progress we make with this. But there seems to be something lacking ...
Frank's user avatar
  • 111
1 vote
1 answer
96 views

User-level secrets in GitHub Actions

I would like to create a GitHub workflow for running tests against a REST API. The API needs a key that is always tied to a user, there is no global "testing" key. Is there a way for each ...
Parzival's user avatar
1 vote
1 answer
265 views

How does one perform systems testing against multiple interdependent machines?

I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc....
vpseg's user avatar
  • 11
1 vote
1 answer
56 views

Sorting out automated testing in CI pipeline and getting conflicting information related to building, unit testing and Docker images

I seem to get a unique response for every person that I ask, or every article that I read, which is just confusing me even more. Basically, I'm trying to setup automated testing in the CI pipeline ...
cheslijones's user avatar