Questions tagged [testing]

Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Use this tag on questions that talk about the testing of systems or software that would be on-topic for this site.

Filter by
Sorted by
Tagged with
2
votes
0answers
51 views

How to test production Ansible execution without having to provision target

We use ansible for our day to day operation and so far so good. We have been looking for a way to run local test etc just like chef guys and found molecule that we use with testinfra/vagrant. While I ...
2
votes
1answer
105 views

Trunk Based Development - release branch testing

We are currently using Git flow branching strategy and looking to go to a trunk based strategy - although we are quite a ways away from getting there I am trying to get all my ducks in a row, ...
2
votes
1answer
42 views

A name for testing software migrated to a new environment against original application

This is more of an english language question - we have various types of tests - unit tests, A/B tests, etc... Suppose I need to migrate code to a different language, machine, or any other different ...
0
votes
1answer
139 views

Safari test automation using Robot Framework via Jenkins

I spent about one week trying to make my Robot Framework tests work for Safari browser in MacOS, when launched by Jenkins via ssh. The main issue was that Robot couldn't open a browser when executed ...
1
vote
1answer
108 views

Azure DevOps Server: Why does a Visual Studio Test step take so long to run unit tests?

In an Azure DevOps Server build system installed on premises, I have a Build Pipeline with a Visual Studio Test step. The step is configured to discover and run XUnit unit tests in a DLL. The step is ...
1
vote
1answer
44 views

Is Cypress ok for acceptance tests on APIs in a CI/CD pipeline?

I have a small team but a gigantic application. We don't have PHP unit testing, so we will be using Cypress as a baseline for quality. Everything will go through a CI/CD pipeline and use regression ...
2
votes
1answer
159 views

Share variables across instances

I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the ...
0
votes
1answer
446 views

Running Protractor tests in docker through teamcity

I have a project for which I am using Teamcity to automate deployment, and I would also like to automate testing. It is an angular frontend, and we have some protractor tests. I would like to run ...
2
votes
2answers
31 views

How can I test native macOS UI software in a container before deployment?

As part of the CI setup, I'd like to run (automated) tests on native macOS GUI software in a 'safe environment' (i.e. without messing with the host software installation too much). Virtual machines (...
6
votes
0answers
49 views

Blackbox testing vs openbox testing applications in a containerized environment [closed]

We use containers to deploy our Node.js applications. But currently, the development and a good share of testing is done by the developers on their own host system and thus using different versions of ...
2
votes
0answers
401 views

How to ensure testing execution order with GitLab CI? [closed]

I have about half a dozen of integration tests (handled with Robot Framework) to run in a GitLab CI pipeline. Once upon a time, we did not care about the execution order of these tests. The ...
2
votes
0answers
49 views

How to deploy large web app in container minimal data transfer? [closed]

In current deployment of a web app we are using svn to deploy a specific revision from a repository. The amount of data transferred is just the diff between the currently deployed revision and the ...
1
vote
0answers
35 views

How to manage Pull request or branch environment databases? [closed]

As an organisation we are (finally) moving to a good CI/CD setup. We use Github for development and now have a nice CI setup where we auto deploy specific tags to different environments. Now I want ...
1
vote
1answer
318 views

Jmeter simulating traffic and avoiding 429

I am using Jmeter to investigate the performance and capacity needs of a few web services. Jmeter sends traffic to the web-facing url (served by Nginx), with a rising rate, to see where the service ...
3
votes
1answer
468 views

What is a good storage for storing performance results and visualizing in grafana

Intro I am a tester and need to analyze the results of automated tests (response times and etc.) Current situation 1) I run performance tests using locust.io 2) During test execution, response ...
1
vote
0answers
113 views

Tools for provisioning windows testing environment on Hyper-V [closed]

I'm looking for tools that would help our team create and manage environments for our internal testing and debugging. Each environment consists of single windows server and multiple windows clients. ...
1
vote
2answers
52 views

Code coverage versus functional coverage

We are working on a big Java project where we use JUnit tests. Our Jenkins helps us to monitor the code coverage and shows us the progress we make with this. But there seems to be something lacking ...
3
votes
1answer
48 views

Tool preferences for rapidly spinning up and spinning down k8s clusters

My team hosts several completely different projects on a production k8s cluster with a given configuration. We use AWS (and are starting to use Google Cloud). We have a QA cluster as well, that "sort ...
2
votes
0answers
73 views

What integration test framework has the best docker support? [closed]

We need integration testing framework - pass a url pass requests to that url and get responses and evaluate repsonses. Requirements: 1. We want a tool that can easily implement into a CICD pipeline ...
3
votes
1answer
1k views

How do I make read only mount in Docker container writable?

When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I ...
0
votes
1answer
782 views

Automated Integration Testing of an API in Azure Functions using VSTS

I am building a HTTP API as an Azure Function App in C# and I would like to run automated integration tests (calls to the API) as part of the CI/CD. Is this possible to do using VSTS? I have ...
1
vote
1answer
2k views

Validating custom Jenkinsfile

Currently we're using Jenkins shared libraries which we're planning to use with custom Jenkinsfiles. Idea here is that developers should be able to manage those custom Jenkinsfiles without any ...
5
votes
2answers
654 views

Best practices for CI for embedded platforms (ESP32/Arduino)

Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools. ...
1
vote
1answer
151 views

Chef (well, Test Kitchen) hanging when using local cookbook

I am new to Chef, trying to put together a POC for my company. So I've got a project that has two cookbooks under the cookbook folder. One cookbook is a "base" cookbook called 'hydra-base', the other (...
4
votes
2answers
150 views

Continuous Integration and Isolated Feature Testing

I've been given the task at work to move everyone from using TFS Source Control to Git as the Development Team has been lead to believe that Git may well be the silver bullet that will fix alot of the ...
3
votes
1answer
1k views

Is it possible to run every testing.xml file with different jenkins job?

I've around 100 test method in my testing.xml file. and that testing.xml configured in pom.xml file so I'm able to run my all those testcase as maven test and I'm getting the proper reports and all. ...
10
votes
3answers
952 views

How to test VM provisioning script without provisioning

Currently I'm on a state that testing costs me money and a lot of time... Background: I'm deploying VMs at softlayer and using a post-deployment script(bash) that will install every software that I ...
11
votes
4answers
752 views

Infrastructure as code and TDD

Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while ...
4
votes
1answer
42 views

Automated Testing Jobs

We have multiple environments close to 7 for an application that has UI and also integration server that accepts SOAP xmls. Its hard to maintain the environment integrity when many people working, ...
1
vote
1answer
111 views

How to setup testkitchen with ansible-provisioning from localhost for different vagrant-vms?

As pointed out here, testkitchen is in principable usable with ansible if the right plugins are installed. Now I've done all the installing and am at the point, where I wonder, what I missunderstood: ...
14
votes
1answer
4k views

A systematic way to test RESTful APIs with curl?

I have noticed during integration testing that I actually work on recurring use cases with restful APIs (or in general HTTP interfaces) I check here and there with bash+cURL. It starts looking quite ...
6
votes
1answer
104 views

How to test database applications in QA testing environments?

Process Our current process is: Build process produces a windows setup (msi). This setup consists of a web site deployment and also database scripts - which can create a new database or upgrade ...
-4
votes
1answer
49 views

How to implement Devops environment test section? [closed]

In our university, our instructor gave Devops environment improvement as a project. I am in the test section of this project and I have a few questions that I can not understand. Our instructor said ...
3
votes
2answers
521 views

Best tool for Docker Container test automation?

In my project i am building about 200 containers I want to check in a smoke test if the containers are still running after a rebuild in pipeline. I want to bring up the container with certain ...
2
votes
1answer
74 views

What are best practices for running Pipeline tests into a CI framework?

According to this github repo it is possible to test the Jenkinsfile, but this is run by Jenkins and if there are issues it will not run. Should one run such unit test in another CI? What are best ...
11
votes
1answer
1k views

Testing Jenkins jobs and pipelines

Currently, we have a fair number of Jenkins jobs and pipelines for builds, testing, deployments and other automated activities. Every time we change or add a new job, we only test it manually - e. g. ...
16
votes
4answers
24k views

Ansible: Other option available for telnet check of open ports?

I'm new to Ansible. Here's my task ... I have 400+ hosts, and I need to verify if 5 different ports are open from their end to our web server. Individually, I could log in and run: telnet ...
8
votes
3answers
2k views

How to create staging servers for multiple git branches?

I have to create a new staging process for our development and testing. At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may ...
4
votes
2answers
2k views

macOS testing on Desktop: Docker containers vs. Homebrew installation

Often I test some stuff on my MacBook. Mostly database and data store related stuff (MongoDB, MariaDB, Postgres). I install the software with Homebrew and test. I wonder if I should switch to docker? ...
7
votes
2answers
742 views

What are the pros and cons of running application testing during vs. after a docker build?

Our dockerfile creates the application environment (e.g. env variables, install libraries with apt-get etc.). It also pulls and copies the python application code from its git repo. However, we are ...
1
vote
1answer
87 views

Installing phantomjs karma launcher in Ubuntu-Docker context

Karma is a nodejs tool to automate browser testing. It seems that it depends on nodejs plugin fsevents. But, fsevents is for processing Mac OS X events, therefore I get the error "not supported on ...
7
votes
3answers
789 views

What is the future role of tester?

If the new paradigm is "You build it, you run it" (Werner Voegels, Amazon CTO) which obviously puts much more responsibility - and pressure - on software engineers, what does this change introduce to ...
4
votes
2answers
7k views

How to export Cucumber Reports via Jenkins Pipeline

I'm using the Cucumber Reports Plugin for Jenkins and I want to be able to export the HTML reports so they can be posted on an external portal. I can see that there is a folder named cucumber-html-...
0
votes
2answers
38 views

System test frameworks for scale testing on physical and virtual systems

I'm searching for a framework for running system tests. The requirement are: Able to manage provisioning to multiple physical machine Able to manage provisioning to multiple virtual environments (AWS,...
6
votes
1answer
1k views

How to improve DRP-testing of SE sites?

Have a look at what's described in the question about "Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)", which is about testing of the existing "...
45
votes
6answers
18k views

How to test a Terraform Configuration?

If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / ...
7
votes
2answers
601 views

How to test a cluster of microservices?

I took a look into Server Spec and it says it clearly: Remark: serverspec test suites are meant to be run against a single machine (or docker container). In other words, you should not try to ...
8
votes
3answers
1k views

How to implement a frozen test environment?

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?": This environment usually freezes during the tests. My ...
20
votes
5answers
1k views

How to avoid continuous integration-caused instabilities in test environments?

Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of ...
6
votes
1answer
1k views

How to avoid a regular user can only test in production?

When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here: How to avoid a regular user can only test in production? There ...