The Fallacy of Converting Requirements into User Stories

Borrowed from www.itworld.com

The Fallacy of Converting Requirements into User Stories

I don't know how many of my Product Management peers (or those others of my friends who work with scrum teams) have experienced a transmogrification/migration of a waterfall/RUP team to agile, but one common element seems to be the initial attempt of moving existing requirements into User Story format. Invariably a list of requirements is simply moved into an agile project with little or no editing - and it's amazing how many shops think that this completes the transfer. I also had a very interesting experience while working with a fledgling team in a new jira implementation which I'd like to relate.

The story of Excess

I had come into a new team that was trying to adopt to a scrum framework. While the product manager had some agile experience, it turned out to be rather sub-optimal to what would be considered a good, best-practices based agile approach (that PdM wasn't me, by the way). The requirements that he had inherited when hired defined a fairly large online application with multiple components include most of those you usually find: user creation, onboarding and maintenance; order lifecycle management; business process management; reporting and analytics, etc. Those requirements had been collected via existing business analysts and put into a traditional requirements document by an outsourced company via a product management requirements application (I don't recall which one, but basically each topic, section, sub-section, use casee, etc cascaded in an orderly fashion using numbering: 1.1.1.1 - you've probably encountered this before).

In a rush to get these into user story format, this PdM had all the data exported from this tool into a flat file (spreadsheet) and then imported into the adopted agile management software (in this case, jira). Since very little manipulation of the data was done before the import, this resulted in line items defining topic headers turning into User Stories with little to no content. I'm talking thousands of user stories with hundreds of summaries and no descriptions. Trying to sort this out, this PdM then had the company license a plug-in called "structure" (or something similar) that would then straighten out all the stories into some type of readable hierarchy. The whole process ended up taking several months. At some point I was asked my opinion about the whole thing (or perhaps I saw the result in conversation and just opened my big mouth, I don't recall which) I asked "Why don't you just start with the first component and decompose it into useful stories, describing an MVP to get the project going?" - actually my question wasn't that elegantly phrased, but I want to get this story going). The argument against doing this method was that so much time had already been spent with the current export/import mess. Of course this went on for several months - and the development team wasn't very happy with these stories or the way things were being prioritized. I eventually convinced the PdM to try my way and junked all those existing stories into an unused project.

So what went wrong, exactly? 

1. The product manager was pushed into doing something with the existing requirements so the project could get moving. For the sake of speed and being able to claim "There's 2000 user stories in the backlog" the export/import wasn't thought out very well. At minimum, when you use this type of approach, you should consider how the data is formatted (perhaps by doing a few sample records and seeing how things go?) and making adjustments on either side of the ETL or via a manual clean-up of the data file. To me pulling in requirements like this is probably a good way to have starting, stubbed-in topics but lack just about everything else necessary for a good agile backlog of user stories.
2. Trying to fix this kind of mess causes it's own time-suck - months were used in trying to first interpret the lack of canonical structure, then by selecting software that made an agile tool more like a traditional waterfall requirements tool, the stories themselves gained very little (it ended up just making things easier to read for the PdM, otherwise there was little to gain). Months ended up being used to fit all the user stories into the hierarchy that the structured tool provided. That time would have been better spent establishing a method of establishing MVP for each project phase and working with the scrum team to determine the best approach (this last bit was made difficult as the team was still being hired - the product manager did the best he could with what he had).
3. The third factor had more to do with time pressure and the need to derive some value from the work that had already been done. I'll focus on that a bit in the next section as a potential process solution.

So what is a better way?

It's always easier to look back and figure out what went wrong and fortunately we all survive to learn from mistakes. I'm going to cheat a bit and limit the solution to converting existing documentation rather than go all high-and-mighty about how the requirements were derived. We who have been practicing agile and scrum for a long time realize that there are many approaches to translating requirements into user stories and much depends on the expectations of the team, company and established principles. Some methods are good, some not so good, and sometimes we have little choice in the matter. For me, I like to have conversations with the people using the software (the business, consumer or end-user) and work backwards towards the solution, focusing on the desired outcome rather than on the tactical implementation - that latter part I think ideally should be determined by the entire team. Unfortunately in shops that are adopting agile, there's a lot of baggage that we all have to contend with and sometimes it's just not worth the effort to overhaul the entire process - better to make small improvements that lead to big changes over time. So how would I approach a similar scenario as my story above?

1. First I would carefully read and try to understand the existing documentation. Presumably it's been done well and is worth circulating to engineering and executive leadership - after all someone had to go to the effort of putting it together. I would also do some spot checking against any critical requirements to ensure that enough due diligence was done while putting the document together, by validating those assumptions with end-users.
2. Next, I would circulate the overall effort with engineering leadership to help define the near and future goals of the software. This is to establish some design principles and at least create a contextual framework focused on data, objects, interfaces and general principles, especially around the moving, calling and updating of data. Your engineers usually already have some coding standards - now is the time to get buy-in from the group so there's a bit of consistency in implementation against the overall project scope. The outcome of this should be some good architectural diagrams of the various components necessary to get to the endgame. This can then suggest the best method of execution and the start of user story creation.
3. Using the existing document, I would then see if the topic headers fit the suggested method of execution - the higher-level topics become epics that are further decomposed into user stories. Sometimes the sub-topics fit well, and other times they don't so they may not be very helpful (generally they are).
4. The guts of the requirements are then used to write the user stories, which should be abstracted. Typical requirements are too specific - they tell the user what to do. Instead the approach should be one that tells the developer what the end-user want's to accomplish as a foundation of a collaborative user story. The baseline requirements could still be useful in determining the number of field values that may be required due to some external dependencies, etc but in general the perspective of the story itself should focus on who has a need rather than what needs to be done.
5. At this point the high-level topics for the near-term should be fairly well defined, with less definition for the other parts of the project. As each component part is elevated in priority in the backlog, each story is evaluated by the entire scrum team with a focus on context and what the feature is trying to solve. The story is edited to make sure that we're all on the same page and there's a clearly defined acceptance criteria so we all know whether the story has been successfully completed.

So easy, right? Like I wrote earlier, its much easier to determine what went wrong and suggest something better in retrospect, rather than while things are happening. The fallacy that I refer to in the title is the assumption that a Requirement from traditional waterfall or RUP development means the same thing as a User Story in Agile/scrum. They simply don't, and because the intent of each is different, relabeling doesn't work.

Borrowed from http://agile4freshmen.blogspot.com/

I remember my first experience with agile - we did exactly the wrong thing. I basically took a traditional requirements document and did a search-and-replace of "Requirement" to "User Story" - see, since we were also doing scrums, we're doing agile! It actually took quite a bit of time and experience to realize that what we were doing was adopting some of the ceremonies and artifacts without really understanding how things fit together. The problem is that often even adopting small parts of the agile framework can produce a positive result where a development team sees improvement. It's my opinion that there are as many teams that think they have successfully adopted agile as there are those where agile adoption has failed, each for the same lack of understanding.

-- John

(also published on LinkedIn)

The Argument for Social Business

There's currently quite a bit of buzz around the term Social Business - especially around HCM (Human Capital Management) so I thought I would address the term. I think what makes it a bit confusing is that the term is already used as defined by Muhammad Yunis in 2009 in his books "Creating a world without poverty - Social Business and the future of capitalism" and "Building Social Business -  The new kind of capitalism that serves humanity's most pressing needs." The books in general are about changing business prime-directives from maximizing profits to financial self-sustainment and re-investment. This is much different from the term Social Business used to describe a socialization trend happening in businesses that refers to the use of social tools in everyday business practices.

A better term might be "Collaborative Business" and in general refers to the use of some of those abstract constructs used in social networks but applied at the business level. Most of these constructs involve the "group think" concepts of shared posts, transparent threading of messages/emails/posts, the aggregation of interactivity across multiple tools and the community aspect of collaboration as embraced by businesses, at minimum internally and ultimately with external clients and experts. Ideally Social Business incorporates the various communities involved with the business and introduces a higher degree of interaction between the communities. The stress is on engagement and the result can be measurable depending on how things are defined (aligning the goals of the business with the efforts involved in sharing information). HCM has begun adopting some of these constructs, especially for internal communication but I believe to be effective these tools should be implemented beyond the enterprise - however it's still a good first step at implementation.

IBM has done a rather remarkable job introducing the concept into it's core communication - much of this came about through a corporate initiative to allow its employees to blog articles to help disseminate information which then lead to their hosting both internal and external blogs - this was followed up by the extension of the old Lotus Notes into the current Domino product. I recently attended a TAG Social Society event (and if you haven't joined TAG - the Technology Association of Georgia, then this single group may make it worth while) that had the topic What Will My Email Look Like “Tomorrow”? (it was actually a combined event of separate TAG groups but the presentations were mostly around Social Business). Louis Richardson from IBM was the keynote - he started out with the the quote "Email is where messages end up to die" and spoke about the relative waste of using email to communicate, especially between a group trying to get consensus on a decision - his example had 6 people in it who all had to copy the entire group every time a comment was made. By the time several people commented (and some copied as yet other interested parties) there were quite a few emails in total (I believe he said the average is 150 emails to make one decision). He then referenced the way Domino addresses the problem - I won't go into detail but if you're really interested you may want to look up the feature description.

I'm using the IBM and the TAG Social Society reference above to illustrate my point - there are many thinking about the future of communication and collaboration and how it will be addressed by business, especially enterprise business. Why all the interest? If you look at the demographics, particularly of the Millennials and post Millennials that are making their way into business, you'll understand that this latest generation interacts differently with information than most of the "gray hairs" currently controlling business. This new group is used to multitasking using a variety of electronic devices, combining messaging, video, text-in-email, social media - you name it and they use it, often all at once. It's been observed that the group in general has become "wired" differently in the way they think - with information coming into the funnel (so to speak) in multiple work streams. The biggest change is the way information is collated by multiple participants into threads (the advantage is that information is shared to a larger group without multiple posts like in the email example provided by L. Richardson above).

Think about being a Millennial and being used to this work stream way of thinking, then hitting the plodding methods traditionally (well, at least the last 20 years) used by business where email is king - I can imagine the frustration and lack of enablement - and this may explain why companies who have adopted a more collaborative way of communicating using social tools have become so successful - Google is a good example. It might also indicate why so much of the talent pool is moving towards these "hot" companies (yeah you get free lunches and you always hear about the company culture - but is that enough?). My theory is that if we don't provide a communicative atmosphere of collaborative tools we aren't just missing the boat, we're basically given a paddle and told to make the best of it.

That that end, I think that the beginnings of the solution has already been adopted by technology teams (they tend to be the first to adopt good ideas) in most software development shops through the use of collaboration tools like Campfire and github. Many of the tools and ceremonies have also been adopted by the Agile and scrum shops to improve team communication - and it works! so I've pitched the idea to those I know who facilitate the agile/scrum model. To further examine how Social Business might be implemented, I've created a new MeetUp group called Atlanta Social Business Product Management (yeah it's obnoxious but I couldn't think of anything that had more zing). Please join if you're interested.

The Three Laws of Software Development - Conway's Law

During a recent class, Peter Saddington referenced three laws which collectively are knows as the Three Laws of Software Development:

1. Humphrey's Law
2. Ziv's Law
3. Conway's Law

I wrote about Humphrey's and Ziv's Laws in my previous posts. Conway's Law is perhaps the one that is most well known of the three.
Looking up "Conway's Law" you'll find it in Jeff Sutherland's book "Exploring Scrum: The Fundamentals." Conway's Law is an adage named after Melvin Conway who first mentioned the idea in 1968. It states "Organizations which design systems...are constrained to produce designs which are copies of the communication structures of these organizations" or it's also written as "If you have four groups working on a compiler, you'll get a 4-pass compiler" (sic). That second version is a bit dated by today's standards. I believe Peter stated it a bit more colloquially as "Software always mirrors the group that built and designed it - if the company is dysfunctional due to poor communication then the software will have the same dysfunction" or simply "everyone needs to communicate better."
This law is as proverbial as the previous two and like them, is more of a valid sociological observation. I found this diagram by Manu Cornet on Bonkers World that I thought was pretty funny and actually a fair representation of Conway's Law...


Manu's diagram isn't specific to Conway but you can infer what you like. The Microsoft and Oracle diagrams are especially telling. Peter tied this law into the need for excellent communication and collaboration as a necessity in producing great software. I've personally seen this law represented myself in the many start-up companies that I've had the honor of working with and the opposite end of the spectrum, large corporate software endeavors (at some point I'll relate a project that had over 100 sign-offs).
Early stage Start-ups are typically small and have a tiny organizational chart - the software produced is usually fairly simple structurally and the communication is more one-to-one due to the tight org structure. This ability to be nimble due to size produces many advantages, especially the ability to go to market quickly. Larger organizations begin struggling to produce software with any speed - there are so many check-points and stakeholders involved in the production that there are often small changes that impact both the software in complexity and the company's ability to market quickly. Large corporate entities are often bound to many revisions, checks-and-balances, approvals - you get the idea. Not only does the software fill with complexity but releases become a nightmare.
Tying back to the class, the emphasis is to make small incremental changes and expose those changes for approval back to the stakeholders early and often so the team can adjust and do a better job meeting expectations. Not only do you produce better and more efficient software, but the release becomes less problematic - there are fewer surprises. I don't know how many times I've worked on a project where the smaller changes truly delight the customer while the larger changes produce confusion and revolt - this is usually ascribed to poor training but even taken that as a viable excuse, doesn't it amount to a lack of communication?

The Three Laws of Software Development - Humphrey's Law

(previously published on LinkedIn)
During a recent class, Peter Saddington referenced three laws which collectively are knows as the Three Laws of Software Development:

1. Humphrey's Law
2. Ziv's Law
3. Conway's Law

I had never heard of these observations referenced as "laws" and wanted to explore them further with my own research. Apparently Conway's Law is used in some scholastic software development classes but the other two are a bit obscure to me. We'll start with Humphrey's Law.




Looking up "Humphrey's Law" you first encounter a Wikipedia entry regarding something called "The Centipede's Dilemma" - which is something psychologists call the centipede effect or the centipede syndrome. The reference has to do with what happens when a normal action is interrupted by the person's own awareness of the action. An example might be a baseball player who normally uses his training, muscle memory and a bit of instinct to swing and hit a home run, but because he's consciously aware of what he's doing, he basically over-thinks the action, causing him to strike. The effect is also called hyper-reaction or Humphrey's Law, named for English psychologist George Humphrey, who wrote about the effect in 1923 using a poem by Katherine Craster, usually titled "The Centipede's Dilemma." - yeah I know I'm getting fairly esoteric in this but I think it's good to understand where this stuff comes from.


The Centipede's Dilemma
A centipede was happy – quite!
Until a toad in fun
Said, "Pray, which leg moves after which?"
This raised her doubts to such a pitch,
She fell exhausted in the ditch
Not knowing how to run.


While the poem is interesting it does illustrate the main point, which is "Thus, the eponymous "Humphrey's law" states that once performance of a task has become automatized, conscious thought about the task, while performing it, impairs performance. Whereas habit diminishes and then eliminates the attention required for routine tasks, this automaticity is disrupted by attention to a normally unconscious competence." - rather obscure, right? So the question remains, how does this fit into the Humphrey's Law as referenced by Jeff Sutherland, co-creator of Scrum and where the Three Laws of Software Development idea was presented?


Jeff Sutherland's version of Humphrey's Law states that "users don't know what they want until they see working software" or to paraphrase Peter Saddington, "People don't know what thy want until they see what they don't want." This seems to be the opposite of what "Humphrey's Law" as referenced by psychologists means, although they do share some common threads:

1. Both ideas are about human response to exterior stimuli
2. Both ideas are observations which seem true.
3. Both contain an element of subconscious objectivity
4. Both share the same name

I had original concluded that they are the same law, only cross-referenced from the perspective of the observer for the Software Development reference, and from the subject (actor(s) aka centipede) in the psychologists' version. Of course this was purely conjecture - and someone commented on my thread on LinkedIn to say that it's actually a different Humphrey - a software developer.


So what about application? These types of laws represent an observable problem, so using the adage that one must first recognize the problem before it can be addressed, the next step is to identify how Humphrey's Law is impacting the development of good software and using various methods to resolve the issue. Peter did a great job illustrating the problem using a "I want a car" technique that I hope to demonstrate to the product guys and my teams at Altisource Labs. I'm working through this now in the form of an interactive class and demonstration which I'll share with you in a subsequent post. In the meanwhile, I'll continue exploring the other two Laws in my next posts.


-- John

Building Personas

(also published on LinkedIn)
From my last post you may have garnered my desire to begin using some of the Pragmatic Marketing training and incorporating the best ideas into my current projects. Specifically, I began creating personas to use within the user stories and to support each development effort by providing additional context for the team members. When I broached the idea with the development team I initially was met with silence. On further discussion with some of the team members I received a rather interesting response "Well, yeah those are fine for UI but they really aren't useful to a developer." I met this with a bit of dismay - I really thought they would buy into the idea of providing additional context, as the current development groups (we went from a highly efficient team to several inexperienced teams) need all the context they can get. The counter-argument I received had to do with security settings and how typical UI-type personas did not provide enough information to determine access controls. It's interesting how some of those same members who desire efficient agile teams argue against using tools to make it so.




To reinforce the idea, a recent CSPO class emphasized the same thing - provide personas that can be used to help get the team into the mind of the user. At this point I could use some hard data to support the idea (mostly to get buy-in from the team) so anyone out there reading, if you have some fact-based analysis I that I can use it would be helpful in supporting my ideas. In the meanwhile, I'm continuing to build my personas to support epics/stories going forward, hopefully this will nudge the team into being more accepting. As to the question of security settings and access controls, it seems that if the personal has a well defined description and these are of importance, the information can be added into the personal description, no?




In any case, I'm currently defining personas for about a dozen different user-interacting roles into the current application. More updates to follow.




-- John

The Accidental Product Manager

I've been doing Product Management for over 13 years. When I reflect back upon my entry into that field, I find that there isn't any single event that led me there. Even my entry into software development was roundabout - having come from a graphics background. I've also found that this is a common trait among other product managers I've met or worked with - it seems we all started elsewhere then gravitated towards what we like best about software development, the actual design and creation of something new. Continuing on this thread, I haven't seen any specific courses for product management (sure there are certification courses by companies such as Pragmatic Marketing, but I've never seen any specific course work offered in universities to prepare a student for software product management). I have encountered some common characteristics of the best PdMs that I've met worth sharing:

1. Most of the professional product people I've met enjoy a high degree of creativity. The best excel at designing new products that not only provide a great deal of interactivity with the user, but also exceeds the perceived need - it takes genuine creativity to accomplish both goals.
2. Most product managers are super-detail oriented, with the ability to talk high-level, but then dive deep into the minutiae. There's no room for sloppiness or a lack of details and you'll rarely find a PdM that exhibit those traits.
3. The best product managers make the user the primary driver of the application and strive to enhance the user-experience. If you haven't heard of heuristics you probably shouldn't be a product manager.
4. Contrary to the belief of many, a good PdM also talks frequently with his/her technology teams to make sure what's being asked for doesn't have an extreme cost (not just money, but also time, resources and a consideration of the overall technical design). If you're not talking to your teams you should consider doing something else.
5. Product managers should strive to stay abreast of the latest technology - at minimum to keep things fresh and ideally to take advantage of new opportunities. This also includes training in disciplines outside of his/her comfort zone. If you haven't done Agile you should look into it and learn about being a product owner in a scrum team.
6. I don't think I could ever take myself seriously, or call myself successful without having actually taken a product from design all the way to market. I both count my wins and learn from my mistakes.

Just a few thoughts - thanks for reading!

-- John Eaton

An Introduction to Product Management and Agile Product Ownership

I've been writing about the various topics that interest me for many years - I was an early adopter of Blogger and continue to use the platform - I like the flexibility and ease-of-use. Also, at Blogger I'm able to have as many blogs that I could ever want, and explore the many things I may want to highlight through writing and opinion. I started thinking about Product Management more-and-more recently as I'll explain shortly and decided it was time to start opining about it and how it integrates into Agile Product Ownership as part of an agile scrum team.

But first, I'd also like to define a few terms so those of you unfamiliar with them can follow along.

Product Management - actually this should be Software Product Management. Wikipedia defines it as: "...an organizational lifecycle function within a company dealing with the planning, forecasting, or marketing of a product or products at all stages of the product lifecycle."

Product Manager - kind of obvious based on the definition above - this is someone who manages products. In my case since I do software development, the role has more to do with taking the various inputs from the business, users and others, and distilling them into information that can be used by software developers to make changes or create new software products.

Agile Software Development - this is a software development methodology. Once again, Wikipedia defines it as "...a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change. It is a conceptual framework that promotes foreseen interactions throughout the development cycle. The Agile Manifesto^[1] introduced the term in 2001.

scrum - (just assume that all of these definitions are from Wikipedia going forward, with some notes from me) - "...is an iterative and incremental agile software development framework for managing software projects and product or application development. Its focus is on "a flexible, holistic product development strategy where a development team works as a unit to reach a common goal" as opposed to a "traditional, sequential approach". Scrum enables the creation of self-organizing teams by encouraging co-location of all team members, and verbal communication between all team members and disciplines in the project.
A key principle of Scrum is its recognition that during a project the customers can change their minds about what they want and need (often called requirements churn), and that unpredicted challenges cannot be easily addressed in a traditional predictive or planned manner. As such, Scrum adopts an empirical approach—accepting that the problem cannot be fully understood or defined, focusing instead on maximizing the team's ability to deliver quickly and respond to emerging requirements."

kanban - "...is a method for managing knowledge work with an emphasis on just-in-time delivery while not overloading the team members. In this approach, the process, from definition of a task to its delivery to the customer, is displayed for participants to see and developers pull work from a queue."

scrum master - "Scrum is facilitated by a Scrum Master, who is accountable for removing impediments to the ability of the team to deliver the sprint goal/deliverables. The Scrum Master is not the team leader, but acts as a buffer between the team and any distracting influences. The Scrum Master ensures that the Scrum process is used as intended. The Scrum Master is the enforcer of the rules of Scrum, often chairs key meetings, and challenges the team to improve. The role has also been referred to as a servant-leader to reinforce these dual perspectives. The Scrum Master differs from a Project Manager in that the latter may have people management responsibilities unrelated to the role of Scrum Master. The Scrum Master role excludes any such additional people responsibilities."

Product Owner - "...represents the stakeholders and is the voice of the customer. He or she is accountable for ensuring that the team delivers value to the business. The Product Owner writes (or has the team write) customer-centric items (typically user stories), ranks and prioritizes them, and adds them to the product backlog. Scrum teams should have one Product Owner, and while they may also be a member of the development team, this role should not be combined with that of the Scrum Master. In an enterprise environment, though, the Product Owner is often combined with the role of Project Manager as they have the best visibility regarding the scope of work (products)."

I think that covers most of what I'll write about and I apologize if all of the above is already familiar to you. More in the next post...

-- John