Best Practices for CI/CD

June 21, 2022
8 min read time
Best Practices for CI/CD


If there is one thing the software development world loves, it’s following the latest development trends. Call it the “cool factor” or simply a desire to be at the cutting edge of technology, but there is a big rush for many companies to adopt certain development and software processes. One such trend—which admittedly has been around for many years now—is that of continuous integration or continuous development (CI/CD), commonly associated with the DevOps world.

The principles behind CI/CD encourage development teams to deploy code to production more frequently than is typical with monolithic software and traditional release cycles. The core principle is that big releases are fraught with risk, so focusing on making smaller releases more often actually poses less risk and makes it easier to maintain the code in production.

It’s a great idea that makes a lot of sense for teams to adopt. However, if companies are not prepared to make changes to how they operate, then their journey to adopting CI/CD is going to be a difficult one. While there are certainly technical changes that need to be made to the architecture and tooling to facilitate a move to CI/CD, it is as much of a cultural change as it is a technical one.

This guide will help you to follow best practices – both cultural and technical – when implementing CI/CD in your organization.

Why the Move to CI/CD and DevOps?

Before we discuss these best practices, though, it’s perhaps best to look at the evolution of CI/CD and what brought the software development world to adopt this approach so widely.

A lot of this has to do with the move to agile development, which was very popular in the early 2000s. The agile mindset lowered the amount of time required to develop new application functionalities, encouraging development teams to release early and often (or at least that was its intention).

Certain catastrophic outcomes prevailed. Deploying to production would still take weeks or months because teams did not emphasize their requirements and software design or did not understand the complexities of software testing and automation. This led to excessive scope creep, rework, and the hard toil of regression testing, which only became more necessary and time-consuming.

Thus, the principles of CI/CD and DevOps emerged to help realign teams from working on big releases and adopt shorter release schedules instead.

The new development principles behind CI/CD came around the same time as the growing popularity of cloud computing in the early 2010s, which allowed teams to deploy directly into a production environment.

Even if businesses weren’t interested in using the cloud, tools such as Docker and Kubernetes emerged allowing teams to deploy virtualized infrastructure in any environment and to explore ways of adopting CI/CD into their deployment processes.

Now that we understand why teams want to use DevOps and CI/CD principles, it’s time to look at the best practices that will help them do this successfully.

1. Shift Left

As mentioned above, one of the core issues with agile development is the continued need for regression testing, something that only increases if you’re looking to deploy daily or multiple times a day. Therefore, teams need to revisit the way they test.

“Shifting left” is a concept that means moving tests as close to the code development as possible, with a greater emphasis on unit tests with a high code coverage percentage and automation at every stage of the testing cycle.

Testing close to the code allows errors to be caught earlier and prevents the pipeline from building further with faulty code. This makes it a lot cheaper to fix errors.

Testing works well with automation. Tests are easier to write, more reliable, and faster to execute than many other automation tasks.

However, shifting left is more than just writing unit tests and automation scripts. You also need to focus on test-driven development (TDD), where these tests are written before the code is written. Otherwise, unit tests can easily be retrofitted to pass the existing code.

TDD is best done in conjunction with professional testers, who can help analyze the software requirements and identify all the test scenarios needed to increase coverage, along with code coverage tools to make it easier to measure the effectiveness of the testing.

It’s worth noting, though, that automated unit tests alone are not enough. You also need other end-to-end test cycles for full coverage. Automated unit tests simply reduce how many test cycles you need to run.

2. Commit Daily

CI/CD works best when every small code change is put through a testing cycle. This prevents untested code from becoming too big and from requiring significant fixing to get through its testing phase. It also reduces the risk at the point of deployment.

Committing daily to either the main branch or the significant development branch also reduces the risk of complex branching and code-merging strategies in the development phase.

3. Fix the Broken Builds

Having fewer branches reduces maintenance in the development cycle. However, it’s not just about making regular deployments but also about ensuring that the pipeline stays green. When there are failures in the build process, you should fix these immediately.

The idea behind CI/CD is that teams should be able to deploy into production frequently. By always keeping their code error-free, teams can deploy their main branch into production at any time, knowing that it is in good working order, stable, and well-tested. This is especially useful when you need to release an urgent change or code fix.

4. Adapt and Reduce Technical Debt

Every organization has a process for creating and delivering code. This process may change over time due to new technologies, teams, or processes. However, if you want to make a success out of CI/CD, you need to be prepared to constantly re-evaluate and improve these processes.

Our CI/CD pipelines reflect where teams are in terms of DevOps maturity. Typical delivery responsibilities include provisioning infrastructure, deploying applications, approving deployment changes, quality assurance/testing, and monitoring. The first iteration of your CI/CD pipeline might not include all these responsibilities and capabilities, but you can add more when your teams are ready.

There is no one pipeline template that suits everyone, so teams need to be able to adapt to find the approach that works best for them and constantly re-evaluate how they can improve their CI/CD process along the way.

Another important point here is to reduce the amount of technical debt incurred by any team. The health of your deployment pipelines is threatened when you do not upgrade software and processes to the latest versions or when you do not address known technical or testing problems. Pipelines that are badly maintained can easily become out-of-date, unreliable, and expensive to fix.

Your CI/CD pipeline should be ready to deploy into production at any time. Reliability issues in your pipeline undermine your entire CI/CD strategy.

5. Define a Release and Rollback Strategy

No matter how well you test or plan your deployment, the risk of things going wrong will remain. You need to be prepared with a process for both releasing software and rolling it back accordingly.

Any time we release software, we introduce the risk of vulnerabilities, issues, bugs, and non-performant software. There could be any number of reasons for rolling back a deployment or producing a hotfix. For this reason, it’s important that you define a release strategy that guides teams on how to release in any given situation. The different types of testing and rules that need to be met in the pipelines ensure that teams can fix and support production while keeping risk at a minimum.

Release strategies should also include how to roll back software. Rollback strategies tend to mirror release strategies and are needed to recall earlier versions of the software when different issues occur, thus making it easy for teams to test and deploy when required. Whether you decide to keep the current version of your application running while deploying a new instance is up to your team to decide, although the decision should be guided by some sort of advice from your release strategy. Just ensure that you have a workflow that addresses any downtime and data loss.

The beauty of your release and rollback strategies is that once they become more mature and refined, they can be automated and built into your pipeline tooling, minimizing the need for any form of overhead and allowing teams to make quick and clear decisions based on what the tooling is telling them.

It’s also worth integrating your metrics, monitoring, and alerting solutions into your CI/CD pipeline to operationalize your deployments. Much like how you should leverage unit tests in the earlier stages of a CI/CD pipeline, it’s important to ensure quality after deployment. Data-driven continuous delivery allows teams to improve their delivery capabilities and amplify feedback to encourage continuous improvement in DevOps teams.

Learn more about how to implement modern deployment strategies such as blue-green and canary deployment.

6. Prioritize Security

While your CI/CD pipelines should alleviate a lot of the functional risk by ensuring high test coverage, this is not the only risk that teams need to be aware of. Security is equally critical.

Your CI/CD approach must include security scanning and testing to ensure that deployed software doesn’t introduce a security vulnerability.

With CI/CD pipelines often having a route directly to the production environment, it’s important that you manage credentials carefully and apply rules for who can authorize certain deployments. Use “secrets management” to protect sensitive deployment or environment information from being compromised.

7. Amplify Feedback

Reliability is everyone’s responsibility, so it’s important that your whole organization adopts a culture of feedback and improvement.

Problems in software development and delivery often occur as symptoms of anti-patterns and worst practices. Too often, organizations take their issues too lightly and fail to learn from them, repeating critical mistakes that hinder their ability to deliver reliable software.

Use retrospectives to understand pain points related to incidents, bugs, and defects and target specific outcomes for improvement. To make this work effectively, ensure you do a proper root cause analysis so that all parties understand the issues properly and that everyone follows the correct mitigating actions.

Some incidents are caused by known bugs or defects, so even setting errors or incident errors can be helpful for better software delivery. It’s common to freeze or lock production deployments when there is a history of poor application performance or frequent incidents.

Technical Difficulties in Building CI/CD Pipelines

Even following best practices, you still need to consider technical factors. Not every architecture is compatible with the quick deployment and scalability of CI/CD. Some common difficulties in building CI/CD pipelines include:

  • Creation and management of deployment scripts
  • Lack of self-service delivery capabilities
  • The upgrade, configuration, and management of plugins
  • Platform stability and scalability.
Ensure that your software architecture and engineers can deliver on these elements so that your organization can deliver on its CI/CD strategy and software ambitions.

One critical component of platform stability and scalability is your load balancer or application delivery controller (ADC). A static, monolithic load balancer or ADC can struggle to keep up with fast-moving CI/CD pipelines and multiple dev, test, and production environments. On the other hand, cloud-native load balancers like Amazon ALB can be limiting in hybrid environments and hard to manage at scale.

Base your CI/CD strategy on an app services platform that is dynamic, multi-cloud, and container-friendly, with scalable ephemeral services and centralized control, security, and observability. Ensure that you can automate your app services with your existing tooling, so you can deploy and scale app services automatically as you push new code through dev, test, and into production.

Is CI/CD Right For You?

While CI/CD is clearly a great strategy for teams and companies that want short and frequent releases, it’s not without its challenges.

The best practices defined above are crucial in helping make CI/CD a success. If you’re looking to adopt CI/CD in your development approach or have been struggling to make a success out of your existing CI/CD strategy, hopefully, the tips in this article can help set you in the right direction.

If your organization is unable to adopt these practices, or if your software architecture is not compatible with this approach, DevOps and CI/CD might not be right for you today.

Get load balancing and security optimized for CI/CD

Snapt Nova provides load balancing and WAAP security on-demand for modern distributed apps, with a full REST API for integration into your tooling and workflow.


Subscribe via Email

Get daily blog updates straight to your email inbox.

You have successfully been subscribed!