When can Scrum take credit for Success

I have seen many contexts in software development that are not congenial for IID and that higher value can be generated faster, cheaper and better, by doing it right the first time, in such contexts. Retrofitting Scrum in such contexts, with BUFD, in an incremental but non iterative mode, sometimes did improve the way teams did things right the first

time, though they implemented Scrum upside down.

I am not trying to be antithetical to Scrum, but am trying to identify and discover new options to generate value that are clearly outside the contexts of Scrum, based on patterns in value generated by Teams implementing Scrum upside down in traditional waterfall environments.

I have no where said that it is a pattern of Scrum, but have clearly debated that this is a useful pattern of anti-Scrum. FScrum, is and will be an anti-Scrum pattern, so everyone knows that this is NOT Scrum, but still has a potential to enhance Value in contexts outside of Scrum identified contexts.

How do we know whether Scrum is working or an upside down implementation of Scrum is working ?

We could easily verify and validate if improvements in an organization can be credited to Scrum.

Here is a simple way of doing it. In any organization that claims improvement with using Scrum, we can verify and validate if the following had occurred in the way they did their work:

1. There were small teams ( 5-9 members) who were fully empowered, self-organzing and cross-functional (SO-CF). The Teams were supported by the Management through Servant Leadership.

2. The SO-CF Teams demonstrated that through iterative "inspect and adapt" practices, they accomplished empirical learning with feedback and demonstrated improvements in their iteration deliverables, successively improving with each iteration.

3. Customers provided regular feedback on the deliverables made by the team at the end of each iteration and found that with successive iterations, the teams had developed a capability to anticipate, understand, meet and exceed their expectations, and improved their capability do so, with each successive iteration.

4. The Product Owner and the Team together, co-created mutual Value and harnessed change to enahance Value. There was a high degree of collaboration, communications, interactions and a high degree of mutual trust between the Product Owner and the Team.

5. That there was complete mutual trust and transparancey between the vendor organization and the customer organization, between the vendor management and the SO-CF Teams, and within the team members. All activities, practices and policies which the three stakeholders adopted in their mutual relationships and transactions, through the duration of the project till the end, confirms this.

5. Teams ability to deliver Value to Customers improved successively with each iteration due to the ability of the teams to continuously learn and improve through "inspecting and adapting" their way of doing things based on empirical feedback and learning

6. That the work started when requirements was not fully clear and without a BDUF, the cutomers and developers worked together evolving and changing scope over successive iterations, based on their learning and feedback, in a manner that both the teams and customers were learning new things, adding and modifying scope all the way that reflected their learnings and got a product in the end that fully met their expectations.

7. At the end of the Scrum project, The Customers were delighted, The Team was delighted and the Mangement was delighted since they made good profits too.

8. And that in this whole project start to finish, Scrum Roles, Artefacts and Ceremonies were practiced exactly as in the book, without any ScrumButs, and in the right spirit of Scrum Principles and Practices as published.

If this evidence can be found, I will surely agree that the credit of its success should go to Scrum. If even one of these were not true in the context, I would surely not pass the credit to Scrum and will investigate to find out what else contributed to their success.

SCRUM should be GOAL driven

We have a great saying "Systems should be Customer driven, Customers should not be System driven" . Similarly I feel that any approach or strategy we embrace should be appropriate and effective to accomplish a clear set of "GOALS" (or objective) for a given endeavour in a given context.

Our "GOALS" in different project situations can be different and we need to ADAPT our approaches to accomplish the GOALS most efficiently and effectively in that context.

Imperative therefore that before we choose to adopt SCRUM, we should know why we are doing so, what we are trying to accomplish, and how this will help us accomplish our GOALS.

If my GOALS in one project context is to increase efficiency only (less cost, less time), I should be able to ADAPT my Scrum Practices to that context to enable me accomplish my GOALS in that context.

If my GOALS in another project situation is to deliver VALUE to my customers by harnessing the "Change" and "Uncertainty" to the customer's competitive advantage, I should be able to ADAPT my Scrum Practices to that context to enable me accomplish my those GOALS in that context.

Simply adopting a strategy or an approach like SCRUM, without understanding why we should do that, may be meaningless and counterproductive. I now am more confident, and more convinced that we should be able to ADAPT Scrum Practices to different situations differently.

Therefore it becomes relevant to explore how SCRUM can benefit those who have only "EFFICIENCY" (reducing cost and time without compromising quality and scope) as their GOALS in their "waterfall" projects and "waterfall" organizations. With a few modifications in SCRUM Principles, it can work for them to accomplish their GOALS without changing the 3 Roles, 3 Artifacts, 3 Ceremonies and the core practices prescribed in SCRUM.

I am at it again, the FScrum will be a critical need for such teams, working in such contexts.

Agile, Shuhari and Commandoes

Guys operating in a Ri stage are like trained and experienced commandos. Commando training in the martial arts, plus orientation on a different Value System and Collaborative Team Work. In a given endeavor and its context, they need to manage both the "outputs" and the "outcomes" appropriately.

Scrum only talks about managing the outcomes. Without understanding how to manage the outputs, achieving outcomes would remain a mere passion. I was greatly inspired by a statement of Tom Peters in his book "Thriving on Chaos". He said "Most Quality Programs fail for two reasons : they have Passion without Systems or Systems without Passion". Now we can see this in real within the practice of Agile Software Development or in the rote implementations of heavy processes in so called L5 organizations.

I have been delivering free short-duration (2hrs) Tech Talks on Agile to larger software organizations in the recent past and have covered about 20 organizations so far and am to cover another 20 in the next couple of months. One participant asked me : if this is true, how do we do Agile ? I said, I would not know your organization context but you could do the following:

1. Identify guys in your organization who are potential commando material. Create an inventory list of such people. All in your organization may not fit to be commandoes. It requires an EQ, Team Play and being a good learner with a willingness to be honest, mature, committed and transparent.

2. Once you have identified such people, begin to train them in the software engineering skills needed to build software incrementally and iteratively. The ones I referred in my earlier post. Give them an orientation on the agile value system of collaboration, team work and customer value focus. This may be equivalent to the Shu stage. At the end of this training and orientation, identify now who qualify and who do not qualify to go further.

3. Now only those who qualified and passed the Shu stage, can be put on Agile Projects. This engagement will be for them to experience the Ha stage. Coach them as they do their job if required, to ensure that they now know how to apply their learning into real situations. You could disqualify members at this stage too if you find that they may not be suitable for Agile for one reason or the other.

4. When they successfully deliver value consistently in different project situations, they can be selected, groomed and used as Coaches, where they get a chance to work at the Ri stage with other Agile Teams either in their Ha stage or use them as trainers to train fresh intakes for the Shu stage.

This is how we train Pilots in the Indian Air Force (I am an ex-IAF officer from the Air Defence Stream). And this is how Army trains and maintains a group of Commandoes who during a war can work as soldiers but during a situation like 26/11, can work as commandoes, working towards a given goal, empirically and iteratively in an empowered self-organizing cross-functional team mode, strictly and clearly within a set of value system meant for them, which is different from the army way of process orientation.

Here too please note that out of 100 soldiers in the army, only a few qualify for getting trained as commandos and fewer still qualify to get posted to a commando unit for one term.

This is what I am trying to incorporate into my FScrum hypothesis.

Shuhari and Scrum

A discussion at agileindia forum on this topic was so enlightening, I felt it should be recorded here too for the benefit of all others not on this forum.

In the context of our discussions on this "Scrum for Waterfall" I had said :

......I am only pained when I see people embrace Agile without knowing what they are trying to achieve by doing so and practice it more as a "ceremony" . In such cases they are not able to understand the imperatives, implications and consequences of Agile......

To which one of the learned members in the forum responded:

http://en.wikipedia.org/wiki/Shuhari

..... I believe some of it is caused by a Shu level person heading a 'Ri' level operation. I hope an organization that is about to embrace agile employs a Ri level person or at least learn from and have one to guide at all times. It isn't bad in the initial stages of adoption to practice it as a ceremony as long as it is guided by someone who understands the implications.

Here is what I responded to his post which I wish to preserve here for others to read.

.......Your post on "Shuhari" has been a real eye opener for me in understanding why people doing Agile upside down, do it that way. I surely would like to pass this credit to you for sharing your views on it that has truly enhanced my wisdom... if I have any :)

I have seen many Western Agile Evangelists comparing Martial Arts to Agile and try to cut-copy-paste their approaches into the way we learn to do Agile.

Here is my 2 cents on it.

When you become a student of any martial-art, you first undergo a highly rigorous training on the basic Values, Principles and Practices of the martial art under a teacher, who is more seen as a tyrant. In such schools, students learn under the direct supervision of a teacher, for several hours a day, for several years, to master their learning of it. These schools more look like concentration camps. (Shu )

Once the teacher feels that you have learnt the art correctly, you are allowed now to practice the martial art, you graduate and you now leave the teacher and go on your own to apply your learning in real-time scenarios. You use your learning in a manner that suits the context of its use, strictly within its value system, and you start empirically evolving your own strategies and approaches to overcome your enemy in different situations and contexts. (Ha)

After years of such practice and success, your learning and approaches become your reflexes and now these competencies can be used naturally. This highest stage of mastery is also called "Unconscious Competence" and your skills are naturally ingrained in your action, reactions and deeds. (Ri).

Now my question to you is:

Where have the agile developers gone through the Shu, and Ha stages before they think they are operating in a Ri stage of maturity. ( My answer to this would be: The two day CSM course under a CST is the Shu stage, After becoming a CSM, you first failed agile project is your Ha stage, therefore now in your next Agile Project you think you are operating in a Ri stage, so you need a Ri level coach. I agree. But I cannot run my company on this basis and still satisfy my customers and make profits consistently.

Similarly guys who pass a Black-Belt certification examination in Six Sigma feel that they are now in the Ri stage.(Sorry..they are no where near it)

A lot of guys have become "Agile Evangelists" and preach/teach Agile and Scrum without ever having written one line of code. Ask them to teach you "Incremental Architecture", "Evolving Design", "Test Driven Development", "Refactoring" (refactoring of architecture, design, code and tests) and "Continuous Integration" principles first before they talk Agile or Scrum to you. Most of them will run away or disappear instantly.

How do you expect to learn to do Agile anyway at the Shu and Ha stages from them and expect to operate at a "Ri" stage with your eyes, ears closed and standing on one leg on the tip of a pole.

Spend some time and introspect on what I am trying to say here, and we may have some ray of hope for Agile to survive in the near term at least.

What is Scrum ?

Thanks to our discussions in the agileprojectmanagement@yahoogroups discussion forum, many of my assumptions and understanding of Scrum has now been clarified formally.

For those of you who are not very clear of what Scrum is and what it is not, here is what I was given to understand of what Scrum is, by a CST on this forum. I am taking this at face value pending its endorsement by the official big wigs in the Scrum Alliance.

--------------------

I do not believe that the Scrum Alliance is under the umbrella of the
Agile Alliance. Many members of the Scrum Alliance are members of the
Agile Alliance, of course, but it would be incorrect to say that scrum,
itself, as a process for Agile Software Development. As Ken has pointed
out, one of the major scrum teams mentioned in the "Enterprise and
Scrum" book (the transition team) does not have software as a product.
Of course, most of us see scrum as a good process for Agile Software
Development, but it is not only for that. The Scrum Alliance defines
"Scrum: A team-based framework to develop complex systems and products."
right on the front page.

Dan Rawsthorne, PhD, CST
Senior Coach, Danube Technologies
dan@danube.com, 425-269-8628

------------------------------------------------------------------------------------

Based on this, I am now very clear on many of the aspects of Scrum implementation as an Agile Software Development Methodology, in software development environments.

So far everyone in the Agile Community have been adopting Scrum as an Agile Software Development methodology within the ambit of Agile Values and Principles. What we are now learning implies that "Scrum can also be used in Agile Software Development while it was never designed exclusively for Agile Software Development"

I recall a blog of mine titled "Scrum in not Agile" more thant 2 years ago that was frowned upon by the Agile guys ( http://scrumtales.blogspot.com/ :19th March 2007 ) Now it looks like afterall it was not so much out of the context.

With this, as a veteran member of the Agile Community I feel that the Scrum Alliance, for the sake of their own credibility, should clearly communicate this to their potential audience and may need to clearly spell out different approaches for implementing Scrum in the following varied contexts.

This also reinfornces my belief that we need to have separate versions of Scrum for the following contexts. Using a single generic approach in the practice of Scrum in all contexts may spell disaster.

1. Scrum for Agile Software Development (as in xp@Scrum)
2. Scrum for Agile Product Development (as suggessted in Ken's book on it)
3. Scrum for the Product Production Industry (sequential process or waterfall - as in manufacturing and construction industry) in line with what we are planning for FScrum
4. Scrum for Research and Development (as in Jim's Adaptive Systems Development)
5. Scrum for the Creative Industry (Advertising, Hollywood and other industry involved in Commercial Arts) that engages a true iterative empirical approach.
6. Scrum for PM BOK ( I know of this initiative by the Scrum Alliance to co-opt Scrum with PMI, IPMA and other project management associations worldwide )

This further reinforces my belief that working to develop an FScrum - a version of Scrum for the waterfall types, may not be invain.

I am aware that Scrum was created to be truly "ADAPTIVE" and it reflects in the evolving "Values and Principles" of the Scrum Community, who have been "EVOLVING" Scrum in line with changing market and economic considerations, over the years, for the benefit of all.

Clarifying the need for FScrum

Thanks to some great discussions on other sites, I realize that I have to clarify why I thought there was a need for something like FScrum.

Scrum is currently under the Agile Umbrella and is labelled very much an Agile Methodology. Hence in its implementation, we cannot voilate or be in conflict with the Agile Values and the Agile Principles clearly mandated by the Agile Alliance.

Therefore implemting Scrum as an Agile Methodology in a predictive (fixed scope, fixed time, fixed cost and fixed quality) plan-driven, defined process enviroment with traditional management structure (supervision and control) is a definite no-no, will be very hypocritical and treated as anti-Agile.

If we can use Scrum outside the Agile Values and Principles, in such contexts as illustrated above, and still be acceptable as in order by the Scrum Alliance, then we do not need FScrum.

However in such a scenario we may need to clearly distinguish between Agile Scrum and non-Agile Scrum. This is what I was trying to formalize and call FScrum (Formal Scrum) or Scrum for Waterfall.

Further to make this work in formal (non-agile) environments, we may need to modify Scrum Principles slightly so that it is in sync with "Waterfall" culture. Scrum Practices however (Roles, Ceremonies, Artefacts) can remain the same but "who does what and how" may need to change a little so suit a "Waterfall" environment.

VISION for FScrum - Brainstorming

Let's first try to build a VISION for our "FScrum", its potential Features, Advantages and Benefits, so we are able to clarify our own expectations from it. I am using this space to brainstorm, speculate, collaborate and learn to create a right VISION for FScrum, so we can architect its design correctly to enable it accomplish its goals and expectations correctly.

Needs Analysis - Why do we need FScrum

Organizations are looking for something like Scrum to help them improve in a Traditional Management Environment. When I say Traditional Management environment, it is where we have clear management structure and hierarcies, defined roles, responsibilities in the organization and is functioning in an environment of formal supervision and control systems.

Now these organizations cannot suddenly invert their pyramid structure, change their management style into an environment of empowerment and servant leadership, change the way they work with customers with contracts, and create new recognition and reward systems to promote and nurture group accountability for those working in self-organizing cross-functional teams delivering value to customers, in an empirical process environment driven by feedback.

This is an Agile Culture IMPERATIVE to be able to harness the Agile Values and Principles and implement Agile Practices like as in Scrum. Without this Scrum cannot work.

So, we need to have something like Scrum, which is not necessarily Agile Scrum but a Scrum that can work in formal environments....so again the name FScrum looks good.

Here is our first list of expectations from FScrum :

1. Something that does not mandate empowerment and self-organizing teams becuase we work in a traditional hierarchial structure with supervision and control, and we cannot invert our organization structure.

2. Something that does not manadate empirical process environment as we have a defined process environment and cannot invert it.

3. Something that allows us to work with Fixed Scope Contracts with our Customers because that's they we work and that's the way our Customers want it. We cannot change the way we do our business with our customers.

4. Something that does not mandate active Customer Collaboration and Co-creation, because our customers are not inclined to give us iterative feedback during development. They wish to have a clearly defined and approved scope upfront, before the project is contracted. We cannot change this way of working, and if we do this, we will loose all our current customers.

5. We will be happy to build software incrementally and iteratively in time-boxed iterations, as long as the iterative nature of adaptive planning is targetted at internal process improvements for the outputs, but is not targetted to iterate changes in Scope by customers for improving outcomes. Scope is frozen in a contract. (note the two words used here...outputs and outcomes). Outputs are for internal benefit (improving efficiency, quality, cost, time) while outcomes are for the benefit of Customers (customer satisfaction, Value, ROI, effectiveness). We cannot do this in our current environment even if we wished to do so.

6. We need something like Scrum that can address all our issues accross our entire Product Design, Development, Integration and Maintenance life-cycles. We are engaged in building large Mission Critical products for a wide range of our Customers that needs to be maintained and serviced for a long Product Life period. They are complex products with multiple technologies and interfaces operating in a complex IT environment of legacy and new applications. Hence they have be meticulously engineering for scalability, compatability, performance and compliance to industry and domain strandards. Therefore the scope, architecture and design of our products are centrally controlled and managed by core Product Engineering Teams and Product Managers very formally to ensure the overall integrity of its sub-systems. They are large and hence need many large teams working to develop it in a distributed and off-shored environments spread out worldwide. Agile Scrum cannot work as it does not scale up to large non-collocated teams in an off-shore product development environment nor was it designed to do so.

7. We still want to improve the way we do what we do. We also feel that there could be many ways to do it and find that some of the Principles and Practices in Scrum, Lean, TOC, CCPM, SixSigma can still be harnessed to our advantage, to help us improve within our own organizational contraints and limitations.

Can we think out of the box and build a process-innovation framework that can help us do better in our situation.

YES, WE CAN.

Scrum for Waterfall

Many Software Product Development Organizations working in a traditional plan-driven and process-driven environment are not able to apply Scrum in its right spirit.

Scrum is an adaptive management framework that is used only in iterative development, in the face of volatile and evolving requirements accompanied by high degree of unpredictabilities in the development scenario.

It is based on an empirical process approach that helps empowered self-organizing cross-functional teams "inspect and adapt" to fast changing situations in the development environments, achieved through a high degree of collaboration with their customers. Trust and Tansparency is an imperative in Scrum environments.

If Scrum is applied in a predictive, traditional management structure, it becomes counter-productive. Many times, when Scrum is implemented upside down in such contexts it causes such projects to move from "frying pan to fire". This is referred to as "Flaccid Scrum" or in Ken Schwaber's own term "ScrumBut".

On this blog-site we are trying to evolve a set of practices that can help deliver "Waterfall Organizations" similar kind of benefits that Agile organizations reap from methodologies like Scrum.

We wish to evolve a new variant of Scrum for the "waterfall" organization by drawing upon from the best of the paradigms in Agile, Lean, Six-sigma, TOC/CCPM and the likes, as appropriate, practical and feasible for such organizations using watefall as their main software product development approach.

We like to call this FScrum to mean, "Scrum for Waterfall"