Welcome!

XML Authors: Carmen Gonzalez, Nikita Ivanov, Mike Kavis, Tom Leyden, Elizabeth White

Related Topics: SOA & WOA, Java, XML, .NET, AJAX & REA, Web 2.0

SOA & WOA: Article

The Odd Couple: Marrying Agile and Waterfall

Interfacing between Linear Waterfall and Agile Approaches

This article depicts the best practice approach for integrating Agile approaches and specifically Scrum development with traditional overarching linear approaches, specifically waterfall methodology. The agile PMO, properly defined, can be positioned to secure Agile-Scrum benefits while maintaining the necessary overarching control.

The challenge
Over the last two decades, various Agile approaches have been introduced and practiced. Of these, in last 5 to 7 years, Scrum has gained the most popularity resulting from a combination of simplicity, ease of use, and effective public relations. Scrum success in software development organizations has been a powerful driver for roll outs across products, industries and businesses. As described, this was exacerbated by a focused marketing effort from Scrum evangelists. Unfortunately, most of these organizations were not structured in a way that supports the Agile-Scrum approach and methods. Even more so, scrum in its raw and pure form is not suitable for the majority of organizations.

The first wave of failed Agile-Scrum implementations brought about an even stricter admonition, based on an unwavering belief from Scrum zealots. The main assertion has been that failed implementations are a result of partial embracing of the true scrum spirit and the full benefits of the approach can only be attained if the entire organization is reengineered. This fanatical attitude left many project teams across organizations big and small, struggling with their already idealistic implementations. Some have been figuring out on their own, how to combine the contemporary and traditional worlds. Other teams have completely abandoned the Agile-Scrum concepts reverting back to the traditional linear waterfall approach and method. Yet other teams, ridden with guilt, manage Scrum by name only, and hiss vehemently at any project management proponent who is unfortunate enough to advise on re-embracing Agile in a more cognizant approach.

The concepts which are presented and embodied in Agile-Scrum are too good to be ignored; however implementing it outside pure software development requires adaptation.

Complex scenarios for Agile
The main hurdle in achieving the benefits of Agile- Scrum outside software development is integrating it with existing control mechanisms. These control mechanisms are stipulated due to various organizational prerequisites and are normally actualized by using the Linear Waterfall approach and methodology. Four of these typical organizational prerequisites are depicted below:

  • Big global corporates: in these hierarchical matrix organizations, top down portfolio control is the rule of the day. The free spirited agile approach has a tough time adjusting to the rigorous controls. Specifically the inherent Agile plan-free concepts, make Agile-Scrum difficult for the organization to swallow.
  • Highly regulated industries: some industries are required by compliance and governance bodies to have a strict binding control mechanism. These can be for example medical equipment, aircraft, and pharmaceuticals research and product development business units. While individual teams might operate Agile-Scrum, the development process must follow a rigid Linear Waterfall approach method for traceability and governance.
  • Complex predefined products: some integrated products which include both hardware, software, imbedded and more are developed as a contract with an end customer under a predefined set of requirements. In these cases the degree of requirement flexibility is small, though larger than what is anticipated initially. The concept of a fully flexible backlog of Agile-Scrum suffers considerably in these cases.
  • Generic IT departments: much of the daily and weekly activities in maintenance driven IT departments is ad hoc. Changes to the daily schedules are numerous and immediate. Constant interferences in the teams work are the norm. The concept of time boxing and no interference is difficult to maintain in these situations.

Naturally - many times the four discrete categories detailed above, actually mix; so it is common to find a complex product in a global big corporate which is required to comply with firm regulation.

Based on practical experience, the recommended method to tackle these scenarios and others is by empowering the Agile PMO to act as an enabler, driver and translator between the emerging Agile-Scrum teams and the Linear Waterfall elements.

Refer to the table below for specific guidelines

The Agile PMO - leading the hybrid organization - guidelines

Scenario

Challenge

Possible solution

Comments

Insights

Big global corporates

Strict controls manifested in Linear Waterfall

The Agile PMO is the buffer between Agile-Scrum teams and the Linear view

Burn down charts are translated to phases for control;

Requirement traceability done by PMO architect;

Agile PMO maintains the dictionary between sprint planning, execution and the phase gating mechanism

Product owners can be part of the Agile PMO;

Project initiating and closing managed by the PMO

Scenario

Challenge

Possible solution

Comments

Insights

Highly regulated industries

Strict compliance and paper trail requirement including product risk analysis

The Agile PMO is also resourced by administrative staff to ensure compliance with regulations

Product risk is managed on a lifecycle view with members of the Scrum-Agile team;

Backlog populated by Non-functional yet critical requirement and owned by the Agile PMO

Agile PMO staff maintains traceability of these requirements.

Necessary documentation is part of the backlog

The added administrative effort handled by the PMO is compensated by the increased velocity of the Agile teams.

Administrative PMO staff can also be non-functional product owners to ensure compliance aspects

Scenario

Challenge

Possible solution

Comments

Insights

Complex predefined products

Limited flexibility in product scope tends to deteriorate  Agile implementations to Agile by name only; Also, hardware elements of product can't be performed in an Agile approach

The Agile PMO owns the backlog interfacing with the various components of product development - managing a hybrid Agile-Linear project

This is probably the most difficult and tricky scenario to tackle;

It requires technical as well as leadership propensity and know-how.

Experience shows that by investigating creatively - Agile concepts can be implemented in rigid hardware development environments

Also - rigid product requirements still allow usually 20% flexibility

the most value added can be reaped in this scenario by developing a customized mixed approach;

Agile stage deliveries can be used to increase flexibility.

Concepts of incremental deliveries may sometimes not be achievable in all product aspects

Scenario

Challenge

Possible solution

Comments

Insights

Generic IT departments

Constant changes to team's work, inability to see the big picture due to ad-hoc work interfering;

missing a true product owner

The Agile PMO substitutes the product owner role in acting as a buffer to oncoming requests also protecting effort to reasonable levels

Many disheartened IT departments have become bitter when trying to use Agile to their development and ongoing work; the result has been fatigue laden team, viewing Agile as a vicious manipulation to increase output without genuine management support; more than a single project management approach can be practiced

Noticeably, Kanban works better for these environments;

Time boxing still makes sense, however a certain predefined buffer for ad-hoc work should be built into each sprint; Sprint durations should be flexible

 

 

Important best practices to remember that go hand in hand with the Agile PMO:

  • Implementing Agile-Scrum as a restricting admonition is exploiting the adaptive nature of Agile
  • There is no one right way - no one size fits them all;
  • There is no silver bullet - you can create what works for you;
  • Being agile and adaptive also allows being flexible with how one uses the methods, process and Methodology;
  • Time boxing is great as long as you are flexible to change the durations of the time box if necessary;
  • Sometimes the client isn't directly available, in these cases marketing and product management are a proper alternate;
  • Arbitrary rules don't complete projects, people do! Empower your team and yourself to choose the appropriate mix of approaches that enable product delivery.

Summary
In my book about the Agile PMO I describe how PMOs can succeed only if they create and enhance value through smart portfolio selection (more about that in a future white paper). With the emerging of Agile approaches and specifically Scrum methods new opportunities have become apparent. Integrating them into an existing control structure - typically presented by a waterfall lifecycle - can be frustrating. We have defined a new key player - the Agile PMO which can be positioned to create a transformation / translation layer between the approached and methods, contributing to higher success levels of these integrations.

 

The Agile PMO Michael Nir

More Stories By Michael Nir

Michael Nir - President of Sapir Consulting - (M.Sc. Engineering) has been providing operational, organizational and management consulting and training for over 15 years. He is passionate about Gestalt theory and practice, which complements his engineering background and contributes to his understanding of individual and team dynamics in business. Michael authored 8 Bestsellers in the fields of Influencing, Agile, Teams, Leadership and others. Michael's experience includes significant expertise in the telecoms, hi-tech, software development, R&D environments and petrochemical & infrastructure industries. He develops creative and innovative solutions in project and product management, process improvement, leadership, and team building programs. Michael's professional background is analytical and technical; however, he has a keen interest in human interactions and behaviors. He holds two engineering degrees from the prestigious Technion Institute of Technology: a Bachelor of civil engineering and Masters of Industrial engineering. He has balanced his technical side with the extensive study and practice of Gestalt Therapy and "Instrumental Enrichment," a philosophy of mediated learning. In his consulting and training engagements, Michael combines both the analytical and technical world with his focus on people, delivering unique and meaningful solutions, and addressing whole systems.

@ThingsExpo Stories
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it's a mix of architectural styles ...
The definition of IoT is not new, in fact it’s been around for over a decade. What has changed is the public's awareness that the technology we use on a daily basis has caught up on the vision of an always on, always connected world. If you look into the details of what comprises the IoT, you’ll see that it includes everything from cloud computing, Big Data analytics, “Things,” Web communication, applications, network, storage, etc. It is essentially including everything connected online from hardware to software, or as we like to say, it’s an Internet of many different things. The difference ...
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, and physical persons. In the IoT vision, every new "thing" - sensor, actuator, data source, data con...
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, demonstrated how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
SYS-CON Events announced today that Gridstore™, the leader in hyper-converged infrastructure purpose-built to optimize Microsoft workloads, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Gridstore™ is the leader in hyper-converged infrastructure purpose-built for Microsoft workloads and designed to accelerate applications in virtualized environments. Gridstore’s hyper-converged infrastructure is the industry’s first all flash version of HyperConverged Appliances that include both compute and storag...
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect at Hookflash, will walk through the shifting landscape of traditional telephone and voice services ...
"Matrix is an ambitious open standard and implementation that's set up to break down the fragmentation problems that exist in IP messaging and VoIP communication," explained John Woolf, Technical Evangelist at Matrix, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and assessments, including a decade of leading incident response and digital forensics. He is co-author of t...
We are reaching the end of the beginning with WebRTC, and real systems using this technology have begun to appear. One challenge that faces every WebRTC deployment (in some form or another) is identity management. For example, if you have an existing service – possibly built on a variety of different PaaS/SaaS offerings – and you want to add real-time communications you are faced with a challenge relating to user management, authentication, authorization, and validation. Service providers will want to use their existing identities, but these will have credentials already that are (hopefully) i...
The 3rd International @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
WebRTC defines no default signaling protocol, causing fragmentation between WebRTC silos. SIP and XMPP provide possibilities, but come with considerable complexity and are not designed for use in a web environment. In his session at @ThingsExpo, Matthew Hodgson, technical co-founder of the Matrix.org, discussed how Matrix is a new non-profit Open Source Project that defines both a new HTTP-based standard for VoIP & IM signaling and provides reference implementations.
Scott Jenson leads a project called The Physical Web within the Chrome team at Google. Project members are working to take the scalability and openness of the web and use it to talk to the exponentially exploding range of smart devices. Nearly every company today working on the IoT comes up with the same basic solution: use my server and you'll be fine. But if we really believe there will be trillions of these devices, that just can't scale. We need a system that is open a scalable and by using the URL as a basic building block, we open this up and get the same resilience that the web enjoys.
Connected devices and the Internet of Things are getting significant momentum in 2014. In his session at Internet of @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, examined three key elements that together will drive mass adoption of the IoT before the end of 2015. The first element is the recent advent of robust open source protocols (like AllJoyn and WebRTC) that facilitate M2M communication. The second is broad availability of flexible, cost-effective storage designed to handle the massive surge in back-end data in a world where timely analytics is e...
There's Big Data, then there's really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at Big Data Expo®, Hannah Smalltree, Director at Treasure Data, discussed how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other machines...
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...