Automating security for applications with DevSecOps

Bob, after conversing with Jenny, learned about GitOps and is using it effectively. Previously what took him 8-12 weeks to develop due to operations issues can now be resolved in days. Every other week Bob builds a new version of his application.

At some point, Bob's user base reached 1 million users. The insane popularity of his application got the attention of black hat hackers. The very next day, his application was down. Since Bob didn't know how to build secure applications, he hired a team for red-blue testing.

Bob kept developing his applications with the push-pull model, whereas the cybersecurity team kept testing each version. When the crew reported a bug, the application was three versions ahead. It became a nightmare for Bob to keep using the DevOps cycle and ship secure products. He called up Jenny again.

Why do we need DevSecOps?

Jenny: First of all, let's recall the DevOps lifecycle and specifically the stages of it. In DevOps flow, everything is automated and fast. Developers push the code, and an application delivery pipeline starts. The tests are checked with automated tests, and a new version is built and deployed in the test environment. After being deployed to the test environment, more automated tests are run, and finally, when all checks are green, a new version is deployed to the production.

The application delivery pipeline makes everything fast without manual intervention.

Now, since do all the checks make this ready for deployment in the production environment? Absolutely not.

Suppose you are building banking software. Without proper security testing, deploying it would be detrimental to the business. The need for a security team arrives.

It takes days to scan the application for the security team. The version has already changed when the security team finds vulnerability issues and sends them to the developers. Now even though a new version is ready, the security team blocks the process due to security concerns, and hence the process is delayed by weeks each time.

Why does Security Audit take so long?

Jenny: Mostly due to the change in the application development framework from monolith architecture to microservices. With that, the points for security concerns have increased. Microservices use APIs to communicate, implying more surfaces for attacks. Microservices use many third-party applications such as databases, message brokers, service mesh applications, and more. These run in containers which creates more attack surfaces again.

The primary reason for the delay is because:

1. The security team sometimes needs to learn about modern development architecture;
2. Security paradigms and methodologies were developed before the new model;
3. The tools used by security personnel are generally outdated.

While the security team might need better tools, its engineers are also humans. There is a chance of human error in the process, which can take longer than expected.

All of these reasons create a bottleneck in the application development process.

How does DevSecOps fix the problem?

Jenny: Basically, by integrating the security component in DevOps. For more effectivity, it shifts "Sec" to the left. Shifting left saves time as well as money by making processes more efficient.

What does shifting left mean?

Jenny: Instead of thinking of security after the development and deployment pipeline processes, it integrates it after every stage. Now, the developers take responsibility for developing secure applications. The security team becomes an advisor for the developers and ops team instead of acting like external police who only blocks the process. The security team creates security policies, selects automation tools for detecting security issues, and trains developers and the operations team on the fundamentals.

After every process, a security check is followed. If a check fails, the DevOps workflow stops, and the developers fix the issue. Building on this pipeline, developers are becoming autonomous, making the entire process faster, more reliable, and quicker.

Bob: I understand DevSecOps makes it easier to manage the rapid pace of development and large-scale secure deployments. But I have another question that is bothering me.

How do we implement DevSecOps?

Jenny: DevSecOps includes a lot of tooling, but moving forward without accepting it as a culture change is impossible. First, you need to integrate its principles at every stage.

Bob: The above flow diagram includes a lot of methods and tools. What do these things mean, and how do they help?

Jenny: Don't worry, I will break down everything for you. Let's compare security to an onion. 🧅 Every layer has its very own responsibility.

Let's start peeling off the layers to understand them.

Layer 1: Pre-commit Hooks

Those hooks are run in the developer workstation. Often, sensitive information, such as access keys, access tokens, or SSH keys, is erroneously leaked due to accidental git commits. A pure regex-based approach can be used for filtering sensitive data. Be aware that developers could circumvent this step, hence use it as a defense, but don't entirely rely on it.

Layer 2: IDE Security Plugin

Plugins are pre-installed in the IDE. You won't spot level 4 injection, but you can find basic mistakes. A plugin will spot simple mistakes like printing a get parameter in PHP before they get into the final code. IDE plugins provide quick, actionable intelligence pointers to the developers.

Layer 3: Secrets Management

In any application development, we deal with multiple access tokens and credentials. Since we can never allow developers to push them into repositories, there must be a uniform digital locker for storing the secrets. Keeping credentials in configuration files is not safe enough. Secrets management allows you to tokenize the information and store it safely.

Layer 4: Software Composition Analysis

The modern software development process includes many third-party modules and libraries. On average, 15% of the total run code is written by the developers building the software, while the rest of the code is fetched from other resources. Software Composition Analysis performs checks to identify vulnerable or outdated third-party libraries.

Layer 5: Static Analysis Security Testing

SAST is about running automated tools capable of picking up easy wins, such as SQL injections, Cross-Site Scripting (XSS), insecure libraries, and more. SAST tools are generic and need manual training for managing false positives. Using SAST tools is tricky on the first go as it can return a report ranging from 500 to 900 pages. 😅 It requires around 4-5 iterations to find what we are looking for. On a bird sight view, it is a part of white box testing.

Layer 6: Dynamic Analysis Security Testing

Even after training and configuring SAST tools, there are still points that are neglected and blindspots. The environmental setup and workload might influence the security results, and they may be different than expected, looking from the source code. DAST tackles that, and it is part of black box security testing.

Results of DAST and SAST can be compared to weed out false positives.

Layer 7: Security in Infrastructure as Code

Infrastructure as code allows you to document and version controls the infrastructure. It enables you to perform audits on the infrastructure with ease. As Docker and Kubernetes rely on base images, the environment is only as secure as the base images. The base images need to be minimal and need to be assessed to identify inherited vulnerabilities.

Layer 8: Compliance as Code

Compliance could be industry standards such as PCI DSS, HIPAA, or SOC. Compliance is essentially a set of rules and hence can be converted into written test cases. Having written code format, this can again be version controlled.

Layer 9: Vulnerability Management

All the following eight layers would generate reports, ultimately creating a massive document to read! Thus we make a central store to manage all the documents. This software subsystem can be placed between a bug tracking system — which developers are familiar with — and security tools. A central dashboard to normalize the data, which is initially created by the security team.

Layer 10: Alerting and Monitoring

When everything is deployed, that's when we would start looking for feedback. We need to make sure everything worked as it was designed to. To test security control effectiveness, we check when an attack occurred, whether it was blocked, what level of access was achieved, and what data was brought in and brought out.

How to decide which tools to choose from?

Jenny: DevSecOps pipelines consist of a lot of tools. Below I have attached a set of tools that you can choose from in the pipeline. Many of the tools present are open source, and apart from the open-source alternative, there are a couple of commercial tools that can be used.

Every environment is different. There is no one-size-fits-all solution. We cannot take someone's DevSecOps practice and copy it into our own lifecycle. Choosing the perfect one needs brainstorming.

Bob: Then how am I supposed to choose one?

Jenny: Let me tell you the parameters I look into before choosing a tool.

1. The tool should have API as well as command line access;
2. Execution start to final output should not be more than 15 minutes;
3. Tools should be containerized or scriptable;
4. Minimal licensing limitations for parallel scans;
5. Output format should be parsable and machine-readable;
6. Configurable to counter false negatives or false positives.

Bob: But how many tools should I use? Because using too many tools will increase the pipeline time, right?

How to optimize the pipeline?

Jenny: Breaking down the pipeline is the best option in this case. Make one pipeline that runs on every commit and another every other day or when a significant change happens. The pipeline optimization depends on how you want your application delivery pipeline to look.

For example, whenever a CSS file is changed, there is no need for Software Composition Analysis, SAST, or DAST tools to run. Look at what has changed and build the pipeline accordingly to this.

Bob: If we configure a DevSecOps cycle, do we still need a dedicated security team?

Is DevSecOps enough for security?

Jenny: As weird as it sounds, the security team should try eliminating the security team. Building a secure DevSecOps cycle should be the first aim for any security team in an organization opting for the DevSecOps approach. However, while companies like Netflix have claimed to have a fantastic and safe DevSecOps cycle, they still have periodic penetration testing and run continuous bug bounty programs for feedback.

Remember that we are using a piece of code to perform security checks. Something still needs to check on the security of the security code from the security team. 😉