DevOps is not an event: it’s a culture. As such, it demands continual improvement in order to stay relevant and competitive. Improvement, however, requires measurement. KPIs are measurements that allow DevOps leaders to see where their teams are, and map out where they are going.
KPIs are metrics that help everyone in an organization from the engineer to the CTO, CIO or CISO. When people at all levels are on the same page, using the same data to build upon, progress happens more easily and quickly. The engineering team uses the information to improve, while the executive team uses it to make decisions that reshape the DevOps in the organization.
KPIs also align everyone with the few things that matter the most. They clarify and help align with company goals, increase collaboration and transparency.
Think DevSecOps metrics, not just DevOps
Recently, and due to the digital transformation and quicker time to market DevOps has produced, new needs have cropped up in DevOps. KPIs have always measured key components of DevOps success such as velocity, quality, and productivity as referenced in DORA metrics. Increasingly, the need to measure and improve security has become critical. Read on to learn more.
How often you deploy, time to market and change volume, etc. all play a role in your DevOps velocity. There’s a distinct advantage for organizations that can move quickly, and a few KPIs can help effectively measure speed.
High deployment frequency and change volume keep one another in check: moving fast means nothing, after all, if no real change is happening. What is the deployment’s actual impact on customers, whether it’s a new software release or a response to a production outage? These KPIs provide a feedback loop that informs the production deployment process for future features.
Analyzing deployments by build, developer, and other factors also provides useful end-to-end insight to the process. This measures developer efficiency and productivity, revealing how many code commits, merge requests, builds were made to each environment before the code was moved to production.
These value-add KPIs allow everyone - Developers, DevOps engineers, Security and Quality teams, and Executives to understand deployment patterns and identify gaps around quality and security.
This refers to the frequency with which new features, updates or capabilities are launched for public use. Generally, frequency is measured daily, though some organizations do tend to measure weekly metrics. However, daily deployments do provide a more accurate indication of team efficiency.
Deployment frequency should remain reasonably consistent or increase at a moderate pace. Sudden decreases in the number usually denotes some kind of obstacle in the pipeline that affects the workflow at large. On the other hand, if deployment frequency sees a sharp hike, it may point to possible high failure rates once the update/feature is actually in production.
This refers to the lines of new code pushed to production in each deployment. Even if deployment frequency is high, it wouldn't mean anything if teams are just releasing a large number of insubstantial, minor changes. If 50 deployments take place, but with only a few changes or 2-3 lines of new code, it does not mean much for team efficiency. In that case, these deployments are of little consequence.
Track the amount of changed vs. static code in each deployment. If changed code is minimal, then an issue with progress exists in the pipeline. Updates can't just be numerous, they must also be impactful.
Refine metrics further and track deployment frequency and change volume by each developer and project. Without this overview, managers can overlook poorly performing individuals or projects. Additionally, this may also cause them to overlook issues or bottlenecks that may prevent those particular devs/testers from offering their best potential.
On noticing lower deployment numbers, start with assessing why that particular or individual may be exhibition sub-par performance. Examine any gaps in access to resources, as that tends to be the most common reason for flabby numbers. In larger, generally well-performing teams, constrictions faced by specific operators are easy to miss.
Lead Time measures the time required to implement any change in software, while cycle time measures how much time it takes to get from the brainstorming phase to getting user feedback.
Measuring Lead Time and Cycle Time gives insight into the efficiency of the development workflow - from ideation to deployment. It must be judged in relation to the user base's current expectations and preferences. Generally longer lead and cycle times point to efficiency gaps, while shorter ones indicate that user feedback is being addressed swiftly.
However, note that shorter lead and cycle times should not be pursued at the cost of higher volume of defects. Don't rush through user feedback just to reduce time, as that would lead to long-term loss of credibility and revenue.
Testing is key to quality assurance. KPIs focused on quality offer a feedback loop that allows teams to move towards test-driven development.
A robust quality process paired with automation gives the whole team a clear picture of an application’s predictability, availability, and performance. They also provide information to teams on single points of failure and time to recovery as part of testing.
Quick detection, response, and recovery time in the case of an outage is critical to keeping availability high. Availability, of course, is a major factor in perceived software quality—it’s hard to be high-quality when an application is down, after all. The KPIs focused on Quality include:
Change Failure Rate denotes how often software releases lead to unexpected failures, outages or major anomalies. In other words, how often is released software failing to serve its purpose?
Low change failure rate indicates smooth and positively impactful deployment. The converse denotes inadequate application performance stability, which is a massive red flag and requires immediate investigation and resolution. A high change failure rate signals significantly poor end-user experience which, once again, would be extremely negative for brand image, credibility, and revenue.
Build Failure Rate tracks the number of builds that have failed (for any reason) within a certain period of time. It seeks to be a barometer for product health while it is being developed. In this case, the extent of build failure or its reason is not monitored. Rather, it just tracks the number of ultimate successes and failures.
High build failure rates signal an unstable product, possibly suffering from hard-to-find bugs, design flaws or even anomalies in the development pipeline. Ensure that failure rates are being measured within a specific window of time so as to paint an honest picture of software performance.
Keep an eye on how automated tests are performed in relation to manual tests. If failure rates are suspiciously high, consider that the fault may lie in the framework’s setup mechanism, the usage of the automation tools, or the test scripts.
Automated tests are an integral part of an effective CI/CD pipeline, and consequently, a DevOps-based development model. Therefore, this metric must be closely monitored, and anomalies must be immediately addressed.
As technology advances, security has become increasingly important to organizations, businesses, and customers. Decreasing security risks is critical to providing a secure and competitive application.
Embedding security as a DevOps stage shifts the process to DevSecOps and allows every organizational team to identify risks and SLA compliance in pre-production. This helps to prevent outages and compliance or audit issues.
Read more: How to integrate security to the DevOps toolchain
Having visibility into security KPIs also helps the team decrease security-related tickets, and allows the technology team to run a secure application in production. As businesses grow increasingly mindful of security risks in technology, these KPIs can demonstrate an organization’s competitive edge. Measure your security with:
This metric tracks how frequently defects are identified in the pre-production vs in production. In other words, how many defects are escaping into the hands of the end-users? It is one of the most valuable measures of application quality, especially in post-production.
Bear in mind that some defects will almost invariably escape into production. If you're lucky, some of them will be detected via acceptance testing, but some might only show up to end users.
However, the latter number must be as minimal as humanly possible. Defect escape rate reflects the quality of the software itself as well as the development process that created it.
This metric actually includes a number of other metrics, all meant to check the health, efficacy and security of operations and business results. A few common vulnerability metrics that can be effective across DevOps engines would be: time to detect vulnerability, time to contain/mitigate a vulnerability, patch management efficiency, and system hardening timelines.
In DevOps-based structures, vulnerabilities refer to gaps in efficiency or security, anything that might adversely affect the quality of the developed product or the organizational stability at large.
Code smells refer to situations in which code has been written in violation of fundamental principles, which lower its quality and efficacy. To optimize code so that it performs better and more sustainably, it has to adhere to a set of ideal practices.
Now, smelly code won't necessarily fail, but it will be of poor quality with adverse consequences. It may reduce processing speed, pose increased risk of failure and error. It may even render the software more vulnerable to bugs. Essentially, it increases technical debt and makes life difficult for everyone involved.
The good thing about code smells is that they are easy to "sniff" or identify via the right tools.
Container scans are required to verify images that are pushed to the production environment. It ensures that container images do not have critical, high, medium vulnerabilities (CVE’s). Container scans are critical for developing any cloud based applications to avoid security attacks. Therefore, it should ideally be a mandated step for any cloud based deployments.
The Operations team is responsible for driving the operational efficiency of the entire process. They care about metrics across the entire life cycle.
True DevOps success means increasing the speed and agility of your teams. Operations usually carries the onus of sharing key performance metrics with all stakeholders to improve success at every stage.
Below are two key KPIs that matter the most.
MTTR is the average time it take for the system to correct and recover from a failure. It includes the time from when the system fails to when it return to full functional capacity. To get the MTTR, calculate the average of all the times it took for all failures to be resolved for a particular system.
As is obvious, MTTR is integral to facilitating effective incident management by offering necessary insights into the speed with which downtimes issues are solved to reboot the system into full functionality.
MTTR = sum of the time to recovery duration / number of incidents to be solved
Read more: Our blog on reducing MTTR
An SLA is a contract (written) between a service provider and a company. It details the kind and level of service to be provided, the metrics that will measure said service and how long it will take to restore the service to optimal levels in the event of a problem.
The metric in question here is the SLA compliance ratio, which determines the impact of its IT service on end-user experience. SLA compliance ratio is the % of total service problems fixed in the purview of the previously agreed-upon SLA criteria as well as parameters like issue category, cost, time, priority and the like.
SLA compliance ratio = No. of IT incidents resolved according to SLA compliance / Total no. of IT incidents
Calculating and analyzing KPIs manually is time-consuming and resource-draining. With more than 10-25 tools in your CI/CD environment, it is often difficult to piece together intelligence from individual tools. You need unified analytics with searchable logs to troubleshoot issues or identify redundancies or efficiencies. This is best accomplished with a platform that integrates data across tools to provide holistic reporting and dashboards, including everything from planning to production deployment and the embedded quality and security gates.
Read more:
Our in-depth blog on DevOps observability
An analysis on DevOps monitoring
An ounce of prevention is worth a pound of cure, as the saying goes. Using KPIs to track and measure DevSecOps trends helps to stop issues before they arise, conserving resources’ time and energy spend fixing issues.
When release risks are predictable, the need to be reactionary dissipates, easing the pressure on teams to constantly put out unexpected fires. It also allows leaders to identify which team, developer, or build present issues with security quality and where bottlenecks may happen. In a nutshell, KPIs allow you to plan ahead instead of reacting after it’s too late.
Get unified insights from across all your DevOps tools from planning to production with Opsera. Troubleshoot faster with contextualized and searchable logs, and build manifests. Improve security and quality posture with end to end visibility. Identify gaps and efficiencies and scale up what works.
To see your DevOps KPIs in action, contact us for a personalized demo.
