Introducing the SCORP Review Cycle – part 1
Security, Cost, OpEx, Reliability & Performance (SCORP)
All three leading Cloud Providers (AWS, Azure, GCP) in recent times have produced well-architected frameworks and guidance for teams and organisations to guide engineering and architecture approaches within their platforms.
What is the well-architected framework?
AWS Well-Architected helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for their applications and workloads. Based on five pillars — operational excellence, security, reliability, performance efficiency, and cost optimisation — AWS Well-Architected provides a consistent approach for customers and partners to evaluate architectures and implement designs that can scale over time.
The AWS Well-Architected Framework started as a single white paper but has expanded to include domain-specific lenses, hands-on labs, and the AWS Well-Architected Tool. The AWS WA Tool, available at no cost in the AWS Management Console, provides a mechanism for regularly evaluating your workloads, identifying high-risk issues, and recording your improvements.
Image from https://www.awsgeek.com
How organisations embrace this guidance has been of particular interest to me in my day job as a Technical Architect. Encouraging teams to adopt the Well-Architected Framework in their architecture and engineering practices truly is a significant step forward. When the organisation applies it at scale, the organisation can get much more economies of scale and create healthier development environments for their teams.
My organisation has started with promoting the use of Well-Architected and we have indeed begun to experience the benefits it brings. We are in the early stages of adoption, and we are currently driving it at an individual team and workload level. More recently, however, I have been thinking about getting more productive with it through supporting dedicated facilitation and using it as a basis for achieving much broader economies of scale when it comes to driving continuous improvement for our teams. Around six months ago, I started with the thought/hypothesis that:
‘If the organisation invested in facilitating the integration of the Well-Architected-Framework into to their core ways of working then we would quantifiably see organisational improvements in the quality of our team execution and the quality of our outcomes.’
What is behind my hypothesis on organisational Well Architected Facilitation?
Being a Technical-Architect working for a relatively large Engineering organisation with multiple development teams working together and not necessarily always on the same engagements, there are many aspects of our operation that I am responsible for and need to consider, such as:
- Reducing the distance between our most effective teams and the least effective (emerging) — More commonly measured in the DORA Metrics.
- Defining and supporting the architectural enabling constraints to guide all teams? Consistency and benchmarking of engineering standards.
- Creating Efficacy (Efficiency and Effectiveness) through prevention of duplication in terms of problem solving and creation of situational awareness between teams. Reuse.
- Highlighting and Celebrating good engineering rigor and exposing shortcutting and technical debt.
- General engineering proficiencies and development of those proficiencies.
Inspired from recently reading the book Atomic Habits, I realised good psychology in slow, deliberate and sustainable incremental improvement. In meeting the responsibilities that I called out above, slow continuous and gradual improvement would be the approach I would take. I just needed a framework to base my plan for supporting this incremental improvement around. A direction to steer individuals and teams towards AWS’s Well-Architected Framework gave me the solid fundamentals to work from.
Why does the well-architected framework tick the boxes of organisational scale?
The Well-Architected-Framework being a commoditised set of opinionated standards means that I, or other architects for that matter, don’t have to invest a lot of time crafting and agreeing standards, as the subject matter experts from the cloud providers have already done that and will continue to evolve them with their platforms. Yes, you are correct in thinking that not all these standards apply all of the time, but as an architect, I focus on the principles and guidance that apply, and I get right into the ways we can implement or meet the said standard. The Well-Architected Lenses are also an essential subset of the Well-Architected-Framework tailored for specialised domains. (Serverless, FinTech, Machine Learning, IoT etc.).
The framework also gives us the power or portability, i.e., if a developer moves from one team to another, or indeed from one supporting org to another separate org, their experience and expectations remain consistent, their skills and experience, you could say, have become very much transferable. My challenge was to create a process that brought the Well-Architected Reviews, the Collaboration, the enterprise direction and integration with team backlog. The process would need to allow me to meet the teams where they were at (experience and capability wise) and bring them together in the spirit of learning and collaboration, which allowed us to make and celebrate small wins through deliberate, incremental improvement. A process that would ultimately be sustainable and not overbearing on the development process.
Working with ten squads, I attempted to make my hypothesis a reality and integrate Well-Architected into our ways of working through a Process that supported regular facilitation and collaborative coaching techniques. Here is my story so far.
Before introducing the SCORP process cycle, I had facilitated roughly around 15 Well-Architected Reviews with different teams. The process itself, when performed correctly, was quite enjoyable for me as the Architect and the engineers participating (even if I do say so). The review provided me an entry point to meet the team and allowed me to understand the nuts and bolts of the architectural implementation of their workload. As well as being able to advise and share thoughts and direction to the team, the team would escalate and discuss challenges, successes and experiences of their own, which for me is invaluable in terms of facilitating the reviews with other teams; I was effectively a conduit for well-architected practice across teams. The reviews themselves would typically end up with me updating the AWS Well-Architected Tool, creating a milestone and following up a few days later to see what the team had added to their backlogs.
The subsequent follow-up, well-architected reviews with the teams, typically around three months later, my observations on progress were reasonably similar. Teams were progressing, but was it progress enough? It’s not to say I was disappointed with the lack of progress, but I knew that we could be better, we should be covering more ground, and we were missing tons of low-hanging fruit type improvements. Your typical review can last anywhere between 2–10 hours depending on the lens being used, so this in itself can be a tiring exercise when the team hasn’t made much progress and can maybe create a sense of frustration if continued.
What I was observing was that the teams were falling into a sort of build trap. With being able to move fast, teams typically were more focused on shipping to production and maybe not as invested in engineering excellence around the Five Pillars as they could be. The focus was not on continuous improvement (proactivity); it was more on reactive corrections and rapid delivery. In some cases where teams start with small workloads, and the dev-ops overhead is negligible. However, as time moves on, the team has scaled out the workload and the dev-ops overhead increases significantly. In these cases, the teams always ended up with epics to resolve issues with tech debt or reliability.
These sorts of trends are not, in my experience, trends that can be corrected by directing technical leads at books or meeting them once a quarter for a technical review. The teams are typically succumbing to the everyday pressures of delivery. These trends in my experience require leadership to engage in the delivery process itself; it would require clarity of direction and a way for the teams to introspect themselves continuously and their performance and capacity to act. It would also need to be consistent for all teams regardless of how they were engaged.
I set about this with the introduction of the SCORP Process Cycle, which we will explore in part 2 of this blog series.