Software Security Testing in a DevSecOps Pipeline

Question

This is a somewhat general question by design, intended for discussion of process. At my current (and all previous) companies we had issues with releasing insecure software into production. This cut across the divide from mobile, web, API's, internal, external, critical and non-critical applications and systems. If we built it, it had/has security issues... just like your company. In addition, we lack support stack (qualified application security engineers who "knew code" and not just glorified pen testers who didn't know JavaScript from Java) so we could not scale services as needed. Hence automated software security scanning in the SDLC and now in the DevSecOps CI/CD pipelines. We face the issues of:

1. Service Level Agreements - our dev teams want/need results in minimal time, we all know that's not happening with either DAST or SAST tools without heavy configuration.

2. Allowing teams to choose their own language and not even having static code support... leaving us with DAST if applicable or, ugh, pen tests, which in my view are essentially useless.

   We need to shift software security "left" - from point of inception, not wait until code is in production and hope we detect the vulnerabilities - I am sure someone will find them, most likely not the pen test team we are paying huge money for, however.

   Automation of static and dynamic tools, properly configured, driven off a threat model seem to be our only options to get ahead of the game. We all know "developer education" is a lost cause.

   How is your company/team solving this, assuming they even are or recognize it as a problem to address?

Answer 1

I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, DAST and SAST can be covered with:

1. Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is integrated with your development process (continuous integration). Implement processes that automatically block changes to your code when they fail.

2. Testing against known vulnerabilities: These are automated DAST tests that test your application from outside. Programs that handle this task include BDD and mittn which is inspired by GAUNTLT.

Integrate everything properly with a CI server (Jenkins, Gitlab, Travis) and stop having to rely on your programmers' "best effort security"

Note: my opinion of proper CI integration, from a language/platform agnostic view, covers the following

Tests/examples: - unit tests that cover at minimum 80%, better 90% - integration tests that ensure the modules work with one another - example programs that show how a user should use the API

Build automation: - Automatically build every pushed commit - Build library, test and example code - Build for each platform/configuration - Run the tests

Merge/pull requests: - Block merges into master branch if any thing fails - Require code review of at least one other colleague

Code analysis: - Run nightly static code analysis (style, anti-conventions, dangerous constructs) - Run nightly dynamic code analysis (memory leaks, race conditions, known security flaws)

Publish results: - Publish graphs/trends of test/analysis history - Talk about changes/targets in weekly meetings - Act on problem areas with sprints/work packages

Answer 2

This is a little bit of a general question as you said, so my answer is a little general too. We have always made security a part of the development process and we enforce policy to make sure our code is secure. This would include forcing developers to only use languages that can work with your security tools. This needs to be part of the review process. Part of this could be in your automated testing process too.

In my experience many DevOps companies view security as an afterthought because of the fear of slowing down development. This runs counter to the reason companies adopt DevOps in the first place. As you said, security must shift left in the development process. I believe this is in line with DevOps / agile principles. Rather than waiting until a product is complete to think about security. Trying to secure something once it is built or already breached will be far more costly in the long run.

Answer 3

While I completely agree with

security must shift left in the development process

though I can't completely agree with

We all know "developer education" is a lost cause.

Because I think of the answer to a Quora question I've seen recently, something like "What is the hardest requirement to a software engineer?".

The answer there was like "*a feature which has highest priority unless it requires too much work".

You probably don't want to do same to your security requirements!

Because many software engineers are very eager to experiment with and learn new things if there is an exactly defined desired outcome.

By this I mean to say that the following steps should be undertaken:

• security design: what are exactly the security requirements by DevOps lifecycle phases coming from your customer/business/regulatory bodies?
• security implementation: for every requirement, you will either find an applicable tool you could plugin to the CI/CD or you will need to build a custom solution (which is part of work requirement and task to your developers).

A great pitfall is from my experience to ask for "state of the art security" either "what developers consider best to realized" because this produces a lot of opinionated discussion including in some cases questioning tool reliability either metrics applicability as such.

To have a good implementation strategy, some research will be required. I will try what comes to my mind mostly from the open source world. In a large scale business critical industrial environment, investing many thousands to licensing of specialized tools is not much compared to financial and image damage numbers we know about from media.

• Requirements: If written requirements exits, do people responsible for that aka business analysts iterate with stakeholders on specific security requirements if such exist? Left shift means also customer should be also aware of the information security and privacy protection as business topic. Which assets are to be protected, which risks are acceptable?
• Static code analysis: Sonarqube and underlying tools provide metrics on potential vulnerabilities
• Packaging: modern artifact repositories provide dependency management tools to highlight problems with package vulnerabilities and licenses (which is though not for free). Also information that some very old package version, either discontinued open source project could be already a source risk. In Java/Maven, you've got the dependency-check-maven based on the US NIST national vulnerability database. For Docker images, you have got security scanners as well.
• Deployment You can grade TLS implementation with the open source Qualys SSL Labs' scanner, there are penetration automation tools from OWASP.

Side note. An additional assessment is also a potential conflict between advanced CI automation and existing organizational processes which have implemented secure processes bound to human interaction. Large organizations where software is business critical therefore should connect the lines between their digital transformation programs and the role of CI/CD systems in the value delivery chain.