I implemented DORA in an organization and this was the result
The value of any metric lies not in its collection, but in its use." — W. Edwards Deming
When I was leading a cloud digital transformation practice at a consulting company, I implemented DORA metrics to boost customer satisfaction within 12 months.
We had an issue with quality of deliverables. Customer surveys highlighted it had to do with delays, feedback not being incorporated in time, leading to frustration and loss of trust with delivery teams.
As this was a consultancy, different development teams would put on to various projects. So the experience of the team played a role in the delivery.
My success was predicated on outcomes for clients – whether they wanted to migrate in to the cloud, or produce a PoC to integrate emerging technologies I needed to make sure they met the underlying goals. The initial sales, architecture, and planning phases were easier to manage as I was deeply involved, but once handing off development to teams was finding myself being parachuted into projects at the 11th hour to “fix” and get back on track. This was not sustainable for me or the company.
Compounding to the problems, I had challenges with the teams including:
We had local and offshore development teams with differing backgrounds of experience
Offshore development teams were in multiple regions globally
These teams also supported multiple clients on different projects based on client needs and their availability
What I could measure would help me improve the delivery of the teams. I decided to adopt DORA metrics (I talked about the various metric in a previous article located here) as SPACE wasn't available at the time. I knew it would be a cultural shift for the organization, and would take time. Prior to jumping in I asked the question:
What are we trying to improve?
All the work that I put in would need to align to a common goal for a successful rollout. Should teams not buy-in to the goal, they will ignore and morale will drop or they look to gamify any metrics (the pitfalls of measuring developer productivity is covered here).
Working in conjunction with leadership and the development management teams we concluded that we need to improve our customer satisfaction. But given that this was not a SMART goal, we couldn't know if we achieved it. We then refined it to:
“Increase customer satisfaction scores by 20% in 12 months”
The company had already been collecting results from their clients via satisfaction survey allowing it to have an established baseline from which to begin.
Cultural Factors
Knowing what the company’s goal was allowed for looking at the DORA metrics with an objective purpose. For example, “frequency of deployment” for the sake of just increasing that number is easy enough to do with no appreciable benefit. But looking at “frequency of deployment with the goal of increasing customer feedback” allows teams to examine their workflows, processes etc.
Additionally, I had to align the rollout of metrics based on the team culture and company dynamics. In this case, the development teams were:
Working primarily in silos with little to no communication or collaboration between other teams
Highlighting key risks and dependencies to projects wasn't the norm, it meant issues were raised late
DevOps processes were early stage with a lot of manual steps for deployment, testing and minimal standardization of systems
Payment of services depended upon successful completion and delivery of the work
Communicating risks and dependencies between teams was imperative, so I began by picking metrics that would reward team members for exactly that. Since the company operated in constant delivery mode, I knew from experience, an all-in approach was not going to be successful and as a result I created the following 5-step rollout:
Redefine failure - not that something was implemented poorly but if we hadn't learnt from it. It meant teams needed to do in depth retrospectives
Changes would be treated as a series of experiments allowing to see results and walk back on what wasn’t working
A subset of teams would be targeted for selection of implementing new metrics based on where they were in delivery of projects
A champion model would be used to rollout beyond initial teams to other teams in the organization, they would share their lessons learnt in this journey
Vendors of dependent products and clients would be included as part of the feedback loop early in the process
A key insight during this discovery was that vendors (in our case we were directly working with Microsoft's Azure Technical Team given that the services we adopted were in pilot/new) were key to our success with delivery. If some resource such as Kubernetes (AKS) was having issues upgrading to the latest release, it would impact not only our client delivery but also their satisfaction of choosing that platform. So I decided to start working with them closely to open communication both ways.
Rollout Plan and Timelines
With the overall goal set and a plan to tackle the intricacies of the culture, the next step was to determine which teams and how they would work together, including:
What are a team’s specific actions that can help increase customer satisfaction by 20%?
How were we measuring those metrics?
What did those metrics mean and how to interpret?
Who and how is it being rolled out to?
What was the timeline to onboard each team?
The following chart shows a snapshot of each teams specific goal as it related to the overall objective. The was key to the planning prior to rolling out, aligning the impacts each team would be making to the primary objectives.
If we look at the Web Development Team’s impacts from the chart, delivering HQ (high quality) code was a prime example of an outcome that would align with customer satisfaction. The team worked to:
Increase frequency of deployments to reduce the number of changes resulting in less risk and better quality
Reset expectations based on team capability of what lead times for changes would be. Smaller changes could be delivered faster, allowing for feedback earlier.
Tackling issues earlier in the development cycle, meant faster feedback and eventually reducing error rates prior by the time demo features to clients.
Some benefits from reducing time to restore also meant deployment times between environments were streamlined, developers changed habits to take ownership of deployments which led to addressing customer issues earlier.
Knowing that these changes were taking place while teams were still working on active projects the changes were small, especially within the first quarter where primarily teams worked to gather additional baseline data to work from.
Was it successful?
The short answer is Yes (and then No).
All planning was also done while continuing to delivery on existing work and responsibilities. It did take additional resourcing time and overhead up front (10-20%) from meetings, reviews, etc.
Rollout to the teams in the chart above was successful, I, along with the teams, were able to get them communicating on key risks and issues early. Some DORA metrics improved quickly while others took longer. Client satisfaction increased in the form of recommendations to other projects, and improved satisfaction scores.
Was 20% increase achieved in the 12 months? No.
The company had a change in CEO who re-prioritized the focus to billable hours with a mentality of "if the client doesn't pay for it we're not doing it". As a result, I didn't get a chance to rollout to data engineering, science, mobile development, and a few other teams (even though I had done much of the planning for them). But the impacts to the teams that transitioned were significant, it became a part of the culture they would actively talk about metrics. It also meant the teams that had adopted those metrics, the morale was much higher.
Eventually the impasse meant that the improvements stalled out with no active push, and I decided to stop working with this company.
I learnt from that in order to rollout successfully, buy-in needs to be across the organization.
That said, even if you don't have executive support for a broad rollout, it's definitely worth pursuing at the team level. Once other teams see the benefits they'll start asking for it themselves. Additionally, always ensure you can relay the story behind the metric - a short term loss may be necessary for long term gain.
I write about this experience to show that not always will the best intentions and planning succeed. We may have missed the target, but we did still provide value and some change was achieved. Do you think it was enough? Write your thoughts below.