At Ventera Corporation, we service both the federal and commercial sectors, focusing on management consulting, software engineering, and data solutions. In 2008, we turned to Scrum to improve our development life cycle, including quicker feedback cycles, improved vision translation, and shorter development iterations. Implementing Agile brought many positive benefits to our organization. At the team level, Scrum proved excellent for managing projects with numerous risks or changing requirements. It also allowed for efficient development teams, which consisted of developers, testers, and a Scrum Master. With positive long-term client relationships and longstanding development teams, adopting Scrum was highly successful for Ventera.
There were, however, challenges along the way. We struggled with Agile teams operating in silos, reactive/ad hoc change control that led to untimely value delivery, and limited release coordination among teams. While the development team had successfully implemented a formal way to coordinate different priorities and activities, it became apparent that this communication was also necessary at the client level. We found that many of the Scaled Agile Framework (SAFe) principles helped us address these problems with scaling the Scrum process from one team to multiple teams.
What is SAFe?
The Scaled Agile Framework (SAFe) was developed to deal with the obstacles presented by large scale adoption of Agile. SAFe addresses these challenges by utilizing a three layer model:
- The Team Layer, which is very similar to traditional Agile, but with broader communication channels
- The Program Layer, which coordinates the teams, their dependencies, and most importantly, the releases (Release Train)
- The Portfolio Layer, which keeps the development aligned business strategy and vision on par with enterprise architecture and standards (Business Epics and Architectural Epics)
SAFe is currently on its fourth iteration and has garnered a lot of excitement and attention in the market as an important framework to coordinate multiple Agile software development teams. A number of large organizations, including federal agencies, have also become interested in SAFe because of its scalability. Ventera has successfully implemented several aspects of SAFe, customizing the framework to work in each customer’s environment, whether that entails solely Agile or traditional waterfall-like components, or even a hybrid. With the application and customization of SAFe, we have seen substantial benefits for our customers.
While SAFe is tremendously valuable, it is not a universal remedy. Our experience demonstrates that it requires tailoring to work well in large environments, especially ones that have not fully adopted Agile practices or ones that have hybrid approaches within the organization. In this paper, we will discuss our challenges with scaling Agile, how we used key SAFe ideas to address them, and how you can apply similar concepts to scale Agile on your own projects.
Success Brings Challenges
As a result of our successful implementation of Scrum on several projects, we began to increase the number of Scrum teams working in parallel. Through several projects, we observed three major challenges that emerged as Agile was scaled across multiple releases:
- Agile teams operating in silos
- Reactive / ad hoc change control and untimely value delivery
- Limited release coordination
Challenge #1: Agile Teams Operating in Silos
Increasing our number of Scrum teams meant that at any given time, a project team could be working on multiple releases and tracks. Clients had competing priorities, varying from new feature enhancements, technical debt, and upgrades, to production-support related tasks and activities. Without a formal communications structure, teams were operating in silos and critical dependencies between teams were not acknowledged or properly addressed. It became challenging for anyone tasked with development to see the bigger picture.
By design, Scrum teams are smaller groups that are meant to overcome communication hurdles; however, issues such as communication, integration amongst cross-functional aspects of the business requirements and code, and attempts to reintegrate into the larger group still arose. Teams operating in silos gave rise to a much higher risk of potential defects and a missed opportunity to optimize and streamline code.
Challenge #2: Reactive / Ad hoc Change Control and Untimely Value Delivery
In the older process, there was a defined Change Control practice that ensured proper delivery of the desired outcome; however, the value of the fix was diminished as it arrived after the fact. This approach ensured the change was implemented quickly, but at the risk of additional ripple effects that could cascade into further defects, date slippage, or other program issues.
In the new Agile process, change was well accommodated from a release perspective, but often addressed in an ad hoc and reactive manner. Scenarios arose that required applying reactive changes that did not take into account the larger strategic perspective of the program. These changes were often introduced into a sprint along with existing high-priority requirements to be completed. There was a need to implement a solution that would support changes as a normal part of the business process for the entire program, not only for a sprint.
Challenge #3: Limited Release Coordination
Release Planning was often done at the team level. The coordination of the various releases was based on a queue that was developed in advance and not dynamically updated as teams would update their priorities and backlogs. Communication of these release-related modifications to the original release plan tended to be broken out functionally; for example, the technical teams were concerned with technical debt while the business side was more focused on new features.
In addition, there were broader client challenges. For some clients, the knowledge base for Agile and Lean thinking was not as advanced as the development teams implementing Agile. By nature of the clients’ work, it was also not possible for them to participate in the selected Scrum methodology on a daily basis. As a temporary solution, we instated a product owner proxy who would serve on the various teams to answer business questions.
At Ventera, our goal is to provide the best value quickly to our clients with our products and services. To do this successfully, we found it beneficial to implement only aspects of the SAFe framework that would work for the client at that time (based on items such as project complexity, team sizes, and Agile savviness).
In this section, we will discuss how we were able to utilize an adapted version of SAFe at the various SAFe levels. This solution enabled the identification of cross-functional client needs via a roadmap and product backlog, all while facilitating the execution of the overall vision through our tightly controlled release coordination. The solution included:
- A progressive planning approach
- A single vision and roadmap
- A dedicated strategic product owner team
- A master product backlog
- Program release coordination
- Scheduled release trains
- Architecture railroads
Progressive Planning Approach
As stated before, one of the challenges we faced was the reactive approach to change requests. Most of the time, the requested changes were valid and necessary; however, the approach to planning, requirements gathering, development, and deployment were not built around accommodating changes quickly. While the existing change control was organized and effective, it was also built with reactive decisions and wait times that ultimately slowed down delivery.
By focusing on value, it was only natural to take on a more progressive approach to planning; therefore, we decided to implement Rolling Wave Planning (RWP). This planning method encourages iterative planning and adaptability, and specifically addresses projects with changing scope and emerging requirements. RWP allowed us to set near-term work based on high level assumptions and milestones for immediate business needs. By adopting it, we provided greater value to our clients by switching our focus to near-term tasks, those which required the most immediate attention, instead of spreading out the focus on both near and long-term items.
A Single Vision and Roadmap
Within a release, the first few iterations were more precisely defined than later iterations. To increase visibility between product teams and allow for early identification of any potential dependencies, we laid known information out on a roadmap. In addition to planning iteratively, the roadmap was clearly communicated across teams for the upcoming iterations and releases. By keeping a constant view on the entire picture, the objectives of the product and client needs were better understood.
Strategic Product Owner Team
While the RWP approach helped us better manage requirements and the roadmap helped us steer the teams in a common direction, we still needed a way to provide sufficient level of detail for the requirements feeding into each iteration. To facilitate this, we created a Strategic Product Owner (SPO) team on the program level. This team consisted of a cross functional group of resources, including representatives from different disciplines, such as business analysis, application development, database, and testing, as well as the client.
Through the incremental planning and organizing of all requirements and process improvements by the SPO team, we were able to effectively respond to teams operating in silos that were facing challenges such as competing priorities or missing critical dependencies. With the SPO team in place as a ‘one-stop shop’, the program benefited from better foresight and communication at the portfolio, program, and team levels.
Master Product Backlog
The SPO team owned the Master Product Backlog and ensured quality across the program. Once a week or more, the SPO team conducted a product backlog meeting with the client in order to funnel all high level epics into the Master Product Backlog and prioritize them accordingly in a single list. We used these meetings to involve the client into our Agile processes and ensure that expectations were aligned.
Prior to the creation of the SPO team, the same group of clients had to manage several disparate communication streams as multiple releases were going on in parallel. We streamlined this process by creating a predictable communication channel – the SPO team – and having a single location – the Master Product Backlog – for all requirements. This has consequently eliminated messaging discrepancies or gaps in requirements.
Program Release Coordination
Another direct benefit of the SPO team was the provision for program release coordination. While being highly adaptive is an effective measure for governance and release coordination, we sought and succeeded in evolving the culture of the entire program, not just the individual teams. The establishment of quarterly Release Trains (discussed in more detail below) enabled us to elaborate features and operate in a regular pattern – or cadence – for the entire program. Once cadence was established, a regular rhythmic flow of operations formed and perpetuated, enabling us to fully deploy product increments that could be placed on any number of Release Trains.
Scheduled Release Trains
The ability to manage the product backlog, prioritize the details, and arrange cross-functional collaboration amongst the groups also enabled our teams to utilize Release Trains in full capacity. Release Trains ensured that there was always the ability to deploy a release as needed, with each “train” loaded with the latest agreed upon developed requirements/code. Having a solid relationship with the operation team was key for coordination of various releases. There were several development integration environments and product increment integration environments that could easily support several parallel Release Trains. This enabled the program to have frequent automated builds in the lower environments for continual testing and verification.
In addition, having Release Trains meant we could also create an entirely new Release Train as needed for a specific request. We found that the concept of “Special Events” trains was especially useful in situations of emergency or unplanned releases due to existing system defects or last minute client enhancement requests that have become an urgent need. The Release Trains structure gave us the flexibility to deploy a release as needed since we were always prepared and could fully absorb the unexpected changes in a new release (without affecting our existing “trains”). Ultimately, through the adoption of the Release Train concept, we were able to effectively respond to the ever-growing needs of the clients. We evolved to a point of high adaptability whereby we can coordinate planned or unplanned development as needed.
Under the SAFe framework, architecture is equally important. The architecture lays the “railroads” for the next key features or product increment and guides the larger technology decisions. Envisioning and planning for the architectural railroads significantly reduced the amount of risk by saving time and allowing enhancements to be done in critical releases. Using architecture railways made both functional and non-functional upgrades clearer and easier to manage. Since the roadmap and future railroad needs were transparent under the entire portfolio, the program could now add features as appropriate within each release, instead of doing special releases to address technical debt as was often done in the past.
Client Transformation Story
Below is a simple transformation view of what we witnessed from one of our larger clients. Based on previous experience, the client was already aware of Ventera’s record of good products, and was not concerned with our development framework as long as work was completed. The primary driver for the program was to evolve the product or system as systematically as possible. As we scaled Agile and adopted more SAFe practices into the program, our clients saw the value of the development framework and have not looked back since.
The current modified SAFe framework for this client takes into account the three recommended layers – Portfolio, Program, and Project. The Portfolio level is primarily managed by the client at a business epic level, while we provide input on architectural epics. The dynamics of this relationship creates a strategic partnership between Ventera and the client and enables the identification of cross-functional needs. The client has the autonomy to develop their own portfolio while having the transparency necessary to help facilitate what is being implemented on the various Release Trains.
At the Program level, Rolling Wave Planning helps manage our incremental delivery process and schedule. It allows us to keep the overall vision intact by determining the incremental delivery process and creating the value streams necessary to define the correct number of releases, technology required, and release timeline.
At the Team level, we operate with a core team, where resources are organized into Scrum teams based on their expertise level with each value stream and knowledge base of the requirements. The team remains unchanged for the duration of the release to create and maintain the team’s momentum. Scrum of Scrums, meetings where the ScrumMasters from various teams attend, help resolve impediments in a timely manner and identify key dependencies that impact multiple teams.
Overall, the simplified process at the portfolio level with a receptive environment for planning both architectural and business epics, the highly coordinated program level releases and processes, and the self-organizing teams all contribute to a highly successful framework for our client and Ventera. On a more important note, the client has seen that the program can [easily] accommodate scope changes based on changing business needs, while also increasing quality and throughput overall. In summary, two main concepts from SAFe worked well for this client:
- Maintain a single core dynamic backlog that is shared across the portfolio, program, and team
- Have one or more consistently scheduled release trains based on value streams of the client
Based upon our success to date, we are continually adapting additional SAFe principles along the way as part of our larger framework of project development.
In the client story above, we deployed aspects of SAFe over the course of a year to address program-specific challenges, customizing and tailoring the framework to work in the client’s environment. Throughout the process, we learned the importance of focusing on value and building on success.
Focus on Value. Ventera’s driver is to evolve the business and our clients by delivering value sooner than later. We focus on very specific activities that help answer the question, “Does this activity provide value to the client?” As demonstrated in our client story, by adopting practices incrementally and over time, we were able to deliver value to the customer sooner, versus waiting to see how an entire framework adoption would pan out.
Build on Success. We found that an important part of adopting SAFe was to quickly identify what works and what does not, and build upon each success. To do so, it was important to consider several aspects of the organization and adapt with proven practices. Adapting to one’s environment is key to achieving success, and makes it possible to offer rapid, accurate development and deliver products successfully.
How can you scale Agile for your organization?
Take a modular and incremental approach to scaling. Try concepts that work for your organization and create an environment that fosters continuous improvement with activities you choose to adopt. There is a whole art to changing organizational processes. This art is better understood by people who have had years of Agile experience under their belt. Needless to say, having the right staff or coaches is crucial. As a recommendation, rather than trying to tackle an entire framework rollout at once, try to roll out a few of our key solution ideas, share successes, and then build upon those practices.
As the saying goes, “expect the unexpected.” Ventera’s ability to say “Yes” when the clients’ priorities change is a normal part of our business. Projects, by nature, are rapidly developing on their own and represent a living system, even more so today with the quick advancement of technology. Our scaled Agile implementation based on SAFe enables us to respond to living project systems by creating an environment where multiple teams are adaptive, fluid, highly organized, and successful.
Try this approach and success should shortly follow!