Wednesday, November 4, 2015

Re-inventing usable requirements: Value-driven projects & the failure of the Agile user story

Hubble Helps Find Smallest Known Galaxy Containing a Supermassive Black Hole
I'm fascinated by the attraction of the Agile user story with requirements writers and the effort to make it be a final usable requirement. Stakeholders format their needs from their view using the simplistic Agile user story. That list of stories then becomes the requirements document or backlog with little analysis. It can reel in any amount of confusing scope easily much like a black hole will attract anything it gets close to.

I wrote about rewriting user stories into business' goal targeted requirements in my last blog and my best words to describe it is as a value-driven approach. After writing that I found that Elizabeth Keogh (@lunivore) suggested that business value is more important than the user role back in 2008. But I want to go deeper. I find that the detail level of the user story is unusable as a coder except to start making informed guesses.

Project value

Needs elicitation is the common starting point of a project. This is normal. Needs are selfish. But needs can't be met until everyone decides what the business goal is that the needs address. The goal should be to solve a business problem. The agreement on the problem statement becomes the starting point for gathering focused needs.

Once there is a problem statement, the needs can be framed in that context instead of the individual project stakeholder or customer. I don't think the stakeholder is going to give you any amount of money to write your code so why tailor the project to each one of them? It's the business that gains the value; it's the business that pays the bills; it's the business that should be the context of the requirements.

User story versions

Scott Ambler, a prolific Agile writer and process thinker, outlines the user story on his Agile Modeling site:
  • pre-2008: user story begins as an informal simple "I want a ..." kind of statement
  • 2008+: Mike Cohn adds the party benefiting from the value In User Stories Applied with the format of "As a (role) I want (something) so that (benefit)."
Scott has a list of user stories on his site in the informal style that include:
  • Students can purchase monthly parking passes online.
  • Parking passes can be paid via credit cards.
  • Parking passes can be paid via PayPal.
The Mike Cohn user story of the first bullet point above according to Scott would be:
As a Student I want to purchase a parking pass so that I can drive to school.
The Elizabeth Keogh user story of the same story would probably look like
In order to drive to school, as a Student, I want to purchase a parking pass.
Both of those examples have the Student getting the value. If I'm the starving student, I want to park free. Did you people ask me? None of those are real benefits. After I get done changing this need into a value-driven requirement it will look completely different:
The school sells permits to park vehicles on school property.
You'll see the benefit as a large scope requirement that traces down (is a parent item) to this requirement. The user story is a needs elicitation tool but with a process flaw. The user stories are passed on the the programmer who think they have structure. The reductionist user story rearranges the words of the "requirements" you find in an Excel spreadsheets and passes them on to development.

Needs vs requirements

A need is a statement of any scope granularity, any priority, with or without a responsible party, and either functional or not. It could even be a design as envisioned by the stakeholder trying to grasp how to convey what they think will get the job done. Needs have so many shapes and forms that they can't be processed easily. They have no structure which is why "requirements" meetings last so long. They are really needs elicitation brainstorming sessions.

Needs have to be rewritten into organized requirements. A good requirement has structure. That structure is testable. A good set of requirements is complete. That completeness is validated by a model and the stakeholders. Structure and models take training and time to do well. Stakeholders don't have that time. Analysts do.

The biggest problem for my students in my Business Analysis class seems to be stopping at a good enough stage where the needs are all lined up neatly with enough words so it looks passable. The critical thinking skills should kick in to test the requirements but the tests are not well understood about what the goal is for a good requirement. Programmers don't communicate and use their tool-driven BDD/TDD/XDD style processes to wrangle the statements into some code.

Value-driven requirements

My value-driven approach for the first parking pass requirement would be much more detailed and focused on the school that is paying for the project. A goal accomplishes business value in a functional requirement statement and can be rolled up to a goal group. A goal group expresses larger scoped business values. For the goal, rules express how the functional requirement is to be executed.
  • Goal group: The school offsets direct costs for parking.
    • Goal: The school sells permission to park vehicles on school property.
      • RULE: parking permissions are valid in one month increments
      • RULE: permissions are only offered online
      • RULE: permissions can be paid for with credit cards (Visa, MasterCard, AmEx) or PayPal.
  • Goal group: The school provides a secure environment.
    • Goal: The school controls vehicular access to school property.
      • RULE: Drivers access school parking lots through entrances requiring permission.
  • Design recommendation: parking passes should be used as permission evidence.
The Student of the user story is a valuable piece of requirements information but not for the requirements statements. It should become a persona as long as descriptive details are added to make them more real. Personas should be used as a design step in the project and for testing but not for requirements. You'll also need an Instructor or Employee persona.

A parking pass is a permission mechanism that can be replaced with another solution and not affect the requirements. That makes it a design choice and unless there is a constraint on the project for doing it, it is not worded in the requirements statements. It should be captured as a stakeholder idea for a solution in a category of design recommendations.

My goal follows a similar structure as Alistair Cockburn's goal driven use case except for adding the rules directly into that scenario if the rules are not reusable in other use cases. Many goals should be able to be found that roll up to a goal group. And goal groups can be drilled down into to find other goals. Business value becomes more strategic as you roll up your goals.

It's the focus on solving our own needs as we see them instead of creating value out of solving a business problem that has us creating confusion in requirements. Encourage the stakeholders to start thinking as a team from the business viewpoint and find the value that is in your project.

Thursday, October 22, 2015

As an Agile user story writer, I want a kitten. - Rewriting user stories for business value.

source: Flickr
Teaching yet another iteration of my Business Analysis class here in Kansas City, I was struck by the two versions of Agile user story techniques I had somehow collected over the years. They were different and would yield two different results. And I didn't like that.

Project phases

There's a basic flow to managing a project:
  • Figure out the problem (strategy)
  • Break the problem down into chunks (analysis)
  • Plan what we have to do (design)
  • Do/build each of the chunks (development)
  • Make sure we did it right (testing)
  • Use the solution (production)
But I think we should add a step that will influence how we write our documentation. It's the step where someone pays for the project. That someone is the project sponsor and they want to see some sort of value returned for the investment of time and money on the project.

The User's Story

The common form of a user story in the Agile world goes something like this:
As a <type of user>, I want <some goal> so that <some reason>
The <type of user> is important so that we can wrap our security access controls around them in the design stage. And the type is also useful when we are eliciting other types of user stories during a brainstorming session of some sort so that we can come up with related user stories from the same sort of user. A use case would combine similar types of users into one actor so that the interface is simplified when moving into design.

The <some goal> is important because it provides a similar sized chunk of scope that allows a project manager to create a work breakdown structure with work packages. I also find it useful to provide an architectural sized component based on the goal in software.

But there is trouble here. We can ask for the goal from the perspective of the user or from the business. It's been my inflexible rule that you never ask a stakeholder the priority of a requirement. The expected answer from any manager worth their salt is that they gotta have it. Ask a room full of managers what the priority is and you'll make your meeting much longer than you wanted to.

One of the lessons I learned from Incident Management in ITIL is that you measure priority of incidents as a combination of impact and urgency giving you some hard numbers to work with. Impact is the number of people affected or marketplace and urgency focuses on the perceived value often backed up by a level of management to manage that process or measured by how much you want that new 4k monitor.

But this priority should be from the view of the business and not from the user of the product or service. If we asked the user what they wanted (akin to asking a 7-year old what college they want to attend) they will tell us all sorts of useless goals. Maybe it really is an important goal of that stakeholder to own a kitten or draw seven red perpendicular lines but the business is not going to find value in it.

The <some reason> then expands on the trouble that started with the user's goal. They'll tell you a wonderful reason why it's important to them. But we're looking for business justification here and in my mind, I'd want that justification to be in the form of a higher level business goal. When things go right, the business goal is in mind, but it's not a constraint of the format.

I know that in Extreme Programming, where the user story came from, user stories were seen as an estimation tool. They also were written from the perspective of the system. That makes sense to me. I need a metric to measure my project where all the units are of the same size. That way I can make that PERT chart make sense in terms of scope.

The System's Story

And why is a basic functional requirements statement written from the perspective of the system? Because the requirements have to be executed by the system and tests confirm that. So we write it that way. But why not write from the perspective of the user? Let's try a test. A user story to run a report for monthly sales goes like this:
As a sales manager, I want to see a sales report, so I can identify the sales reps who are under-performing.
What's the value there? Aren't the under-performing sales reps making money for the business? Is this a personal agenda or a goal that isn't well thought out? How about we try the story from a business perspective?
As a sales manager, the system shall create a sales report, so that sales trends can predicted.
Or another version from a different user might be:
As a marketing manager, the system shall create a sales report, so that marketing can budget for advertising. 
I can see the business value in sales forecasting and marketing budgeting. What I see in the other story is a sales manager who creates churn in their sales division which is not value to the business. The requirement is the same.

User story to requirements

So I'm not going to try and change the user story because it's a good elicitation tool. It does not have enough detail to be a good way to deliver requirements to anyone downstream in design or programming. But I did want to raise the issue of the two-faced nature of the statement. In the statement, we can direct the value to the stakeholder or better, we should direct the value to the business.

My recommendation is when using user stories, that we take those stories as a user's statement of what they want and then rewrite them as a good requirements statement. In a use case format we would have
Stakeholders (interests): sales manager (wants forecasting), marketing manager (wants budgeting)
Actor: report reader
Use case: create sales report
As it was, I figured out that all the business value reinforcement I had been getting from other sources over the years caused me to rewrite the user story in a different format when adding it to my lecture. So in the end, I made a mistake about the real format of the user story and learned from my mistakes. No, I didn't get a kitten.

PS. The rant continues on my next blog, Re-inventing usable requirements: Value-driven projects & the failure of the Agile user story.

Friday, May 15, 2015

Blame chain or value chain? Value drives a project from strategy to test.

This series of blogs is based on a talk entitled "Is Your Programmer a Better Business Analyst Than You Are" given by the author to the Kansas City IIBA user group meeting in April of 2015.

It's always been a problem for me that programming students have no clue about working with existing project requirements. I've never seen any teaching nor any methodology come from the Agile movement let alone any other technical cultural phenomenon. If no one is concerned that the project moves from the business analyst role to the programming role, that must mean that the project really starts with the programmer because that's where the project deliverables really begin. And so, programmers have reverse-engineered analysis and design skill sets to cope by themselves,

Of course, real world projects are the practical training ground for understanding how to work with project teams, but couldn't we make this a part of the skill set for a programmer before they get thrown into the war room?

Web programmers are used to this. They are a department of one. They gather information on understanding customers' needs, write copy, design layouts, put the code together, and deploy and maintain a site. As the web application world expands, this puts immense pressure on the technical web employee to know about any new development and keep up their own skills tied to the legacy apps they maintain.

Programmers I talk to have an implicit distrust of the project. They find a purpose but no real detail in it and muddle through the development as best as they can. In fact, most businesses without a solid project management process don't have much to work with. The management will throw stuff out into the market and claim that the market is crazy. The analysts are documenting whatever anyone says and then complaining that the users don't know what they want. Coders code whatever they think is right and try to make the project manager's deadlines.

And testers. Standing at the end of the line with a deadline rapidly approaching doesn't make anyone try to establish any standards knowing that they are holding up the deployment. They just try to find as many bugs as possible in the time they have.

The programming students I see have not been trained in business so it's reasonable they don't know how good goals can drive their behavior. A couple of questions have always intrigued me when working with a software project:

  • What if all the people on the project have the same goal? 
  • What if they all had documents that built on the previous documents and were traceable back to the original value-driven goals? 
  • What if the project was measured on how it realized that goal?
A project proposal should be estimated on value. Not only just a return on investment but understanding what the priority is for the enterprise. Value for internal projects can be seen as basically two parts but more can be added if justified:
  • The number of people in the company that will use the project in their processes
  • The visibility of that project in the organizational chart of the business so that if that project fails, the level of management involved determines the severity ticket level.
Value for external projects is estimated the same way. There are the same two metrics involved:
  • The number of people in the market that will find value in using the product/service.
  • The amount of money that the market is willing to spend to get that product/service
The two measures are both based on quantity of value. The project then can look back to this value of opportunity and see how to organize the project pieces. For instance, business goals, the basic business driven goals that begin the requirements process, are based on what value the business gets. Requirements meetings should never ask the question "What do you want the project to do for you?" because it's not about the person. The better question is "What should the project do to produce the best value for the business?"

Then, the business can see that the customers have needs where they should adapt their product/service to achieve the best solution to a large market segment. The question for the strategists and enterprise analysts would be "What can I do to help our customers/employees solve their problems?"

Business analysts would then understand that the value perceived at the strategy level should be communicated downstream to the programmers clearly. That communication should be able to move directly into testing of that code because the target of the project development effort is to communicate business value to the technical staff. The analysts' question then is "How does what the customer/staff do right now help them get tasks done?" 

The designers have the task of taking the current processes and possible new processes and shoehorning them into a fit with the business assets so that it improves the value. They have to think about improving life in general. The question for them is "How can we translate the requirements into code/real life? The trick is to make it look like it's simple.

The final stage in the value transition from idea to final service/product is that someone double-checks that the value has actually arrived. That's the job of the tester who assures that the quality will be there though verifying minor things throughout the project and validating that the expectations were met at the end. Their questions are "How can I guarantee quality?"and "How can I let everyone know how well they did?

In making the move to measuring by value, the chain of FUD will change to the chain of quality. But there are lots more questions that I've asked and I'll continue this series by taking on those questions. Here's a sample of some:
  • How should scope be measured by value?
  • How should scope be organized by value?
  • What is the role of analysis?
  • What does a programmer need requirements to look like?
  • What are good business quality metrics for code?
  • Is TDD dead or alive?

Tuesday, July 8, 2014

New ThoughtWorks Technology Radar out today

Four trends are identified in the Thoughtworks strategic IT report called Technology Radar published twice a year. The most important one for me is the one concerning JavaScript technologies and the challenge it brings for understanding. That is the reason that I'm developing my new course, JavaScript Powered Web Apps focusing on the language's application to building client-side logic for web pages and developing more with the browser. And it's been challenging without the guidance of book authors and coordinating corporations.
Here's the full excerpt on JavaScript:
Churn in the JavaScript World — We thought the rate of change in the Ruby open source space was rapid until the full rush of JavaScript frameworks arrived. JavaScript used to be a condiment technology, always used to augment other technologies. It has kept that role but expanded into its own platform with a staggering rate of change. Trying to understand the breadth of this space is daunting, and innovation is rampant. Like the Java and Ruby open source spaces, we hope it will eventually calm to at least a deluge.
The other three trends were 
  • Microservices and the Rise of the API (also somewhat aligned with the JavaScript trends), 
  • Conway's Law ("organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations") and
  • Re-decentralization 

Monday, April 21, 2014

JavaScript Powered Web Apps - new programming course

The world of web programming is moving ahead into what should be called Web 3.0 (the semantic web is just a pipe dream). It's using JavaScript as a unifying layer and re-imagining how the web can do what it does without the benefit of large back end frameworks. Web 1 was delivering files to a client. Web 2.0 was letting clients think they were in control by faking a desktop application over HTTP. And now we can have a true application that has been enhanced with networking to services in 3.0.

For the next several months I will be developing a new course to complement the classes that I wrote a few years back on HTML5, CSS, jQuery, jQuery Mobile, and JavaScript that bring these components together. I see tooling on web applications starting to mature and it is the right time to start to promote a new style of web application development. Except that there's no one way to do anything yet. Adobe is getting close to providing another great IDE with Edge Code but I think we're a ways off yet. Even Google might become a player here with their IDE code named Spark built with Dart and Polymer.

The new course, JavaScript Powered Web Apps, will walk students through building a "site" using combinations of node.js, nginx, SASS, Mongo, Bootstrap, Github, jQuery, jQuery Mobile, Grunt, AngularJS, Knockout, Express, etc. I'll probably do four days of sample sites and then show a web work flow and let students choose their own tools.

The course will assume some programming or design experience with a web site but most of the exercises will be scripted so that anyone can follow them. The students that likely will be most interested are those that don't have the back end skill set and see a tremendous advantage in learning only one language to do both front and back end coding. That means that anyone from high school that has learned the basics of HTML and CSS can enroll.

A lot of the training is on the administration and workflow of the tools which is harder to learn from books. I'll try to capture what I can for the exercises but if anyone has suggestions, I'm willing to listen. And as always, the class will continually be updated as the tools rev and newer tools emerge.

For instance, I'm still waiting to see how is going to impact the animation tools. It looks incredible but won't be completely public until May 19th when HTML5 Dev Conference in San Francisco starts. And I want to spin up a partial.js site as well I think. But I'll never know until it's finished. After all, a project's requirements are never finished until the project is over.

Thursday, January 23, 2014

Documentation as a control mechanism - not! Think communication.

People associate documentation in Agile with waste. In fact, the Agile Manifesto prefers "working software over comprehensive documentation" but does that mean that the purpose of documentation is secondary to the output of the project? Agile set us up with a poor dichotomy. I mean I prefer getting a paycheck over driving to work. Maybe we're asking the wrong question.

I don't see documentation as an optional part of the project. It's definitely an output and can be measured and that is the allure of the artifact. The traditional usage of documentation in a project is as a control mechanism when I look at the process outputs. It's often managers and specifically project managers who finalize a phase with the documentation. It's what they do. They design their work package by understanding the scope of the effort and if there is something measurable at the end it becomes a good work package.

Back in the darker ages of procedural coding, there was a movement to measure code by what it did. That entailed putting an estimate on the smallest granular operations of the computer in the code itself. That worked for awhile when the code was consistent in its granularity. But code has changed and what we can hide in a line of code has become enormous. The function point estimation methods died.

Now we estimate, if at all, with a measurement of work package time completion and complexity (from the programmer's perspective) or confidence (from the project manager's perspective). The dependency is on the work package design. If we don't have a good description of what it is we're going to do, we can't plan it. Many projects flounder from a lack of analytic description.

So where does this good description come from? Well, your requirements are the place to find these descriptions. Of course, the best requirements are ideally the needs and wants of the stakeholders massaged in to testable work packages detailed down to repeatable tasks so that no business questions have to be asked constrained by project limitations. But in reality, they are more of a garbage dump of what people said in excruciating long meetings.

The requirement document gets written and then we're on to the next phase. Follow along on your Gantt chart please. As a programmer, my input should be the output of the analyst. But with all the hubbub in programming circles about how to organize and manage testability, it looks like the analyst isn't doing a very good job. I don't see much evidence of usable requirements from my point of view either.

So, the Agile people are right. Processes without consumers are pure waste. Let's right-size this documentation by eliminating it. No one used it anyway. But what are we losing? We're losing the ability to record a decision and the to think about the design of the business. Of course, if it wasn't good for the programmer, then it was useless.

But let's consider that the analyst can benefit the programmer. Then the documentation becomes a stepping stone to better code. Then there is communication of the needs of the business. Then there is less thinking on the role of the programmer who gets to focus on writing well-structured and reusable code.

The role of the documentation is really that of communication when the project scales. If you are the sole stakeholder and programmer, you probably have all the requirements circling in your head at any one time. No documentation is necessary. If two people know exactly what has to be done after a good agreeable meeting, no documentation has to be created. But if there is a memory loss, a sick day, a new member to the team, you will need some documentation. The need for documentation increases as the need to communicate increases.

The management of documentation needs a metric instead of lazily setting the goal to that of creating a greater return on investment (ROI). Just how do you measure that? My standards for measuring are in the more subjective realm whereby you produce the documentation, ask the user of that documentation if they understand, and then see if they are able to do their tasks without any further questions. The quality score descends as the need for answers or the amount of perceivable confusion increases. Get feedback.

Documentation is not a gate to the next phase and to be signed off. I'll take the stance a little further than the traditional "living document" style of writing. Since it is to be a communication mechanism, it has to always communicate the current understanding of what the project is about. Anyone and everyone can be a contributor but the use cases / user stories / work packages should be maintained by the analyst / technical writer role so that they achieve the best level of testability and detail. Wikis are good.

The trend of "barely good enough" documentation, I think, is allowing the programmer to use their analytic skills in place of poor business analysis skills in the workplace, which is sad but the best workable solution to getting the job done. Stop producing unusable business documents and let the programmer get on with the code. Why are programmers strongly in favor of commenting their code and following Test Driven Development? Because those are the tools that get the documentation done a better way.

So, lets eschew the notion of controlling the project by requiring the project members to produce a result that isn't used in the next phase. Control the project by understanding the work package completions. The artifact that completes the work package is the code or the pseudo-code (the use case) in some form or another, not a project document.

So, is documentation is secondary to the output of the project? Getting a paycheck over driving to work is comparing the result with a task for getting that result. It depends whether you have to go to the office or not. Documentation is not secondary. It's just a question of whether you need to communicate more or not.

Wednesday, May 22, 2013

Thoughtworks Technology Radar

Free software, free books and Technology Radar. If you've sat any of my classes, you've heard me talk about those three things every time.

I have been reading Martin Fowler's writing for most of my IT training career and have found him practical, in-depth, and current. He also satisfies that craving for a little higher level analysis thinking that developer blogs don't always cover well enough exposing their lack of experience with other technologies and limiting their authority on changing my opinion.

The new edition of TR is out today. It's taken me years to work through some of the recommendations that they've put together in their think tank, ThoughtWorks, that are considered to be the cutting edge worth keeping abreast on. They also don't mind telling you when a technology is not worth your time. Both are worth my time to read and understand.

The trends highlighted in this issue are:

  • Falling boundaries - cloud development, co-location, perimeterless enterprise
  • Proven practices to areas that missed them - CSS frameworks, database migrations for NoSQL, etc.
  • Lightweight analytics
  • Infrastructure as code
The four major areas that are reviewed are
  • Techniques
  • Platforms
  • Tools
  • Languages & Frameworks


For me, being mostly a web developer, I was happy to see that the Adopt recommendations in the Techniques section were for mobile testing on mobile networks moving away from the fake simulators as well as using promises for asynchronous programming, giving assurance of feedback for us JavaScript/AJAX coders.

The next level of recommendation down from the Adopt level is the Trial level which should be approached with a little more thought. You see HTML5 storage replacing cookies and Mobile First here as well as responsive web design. I agree with all those because they're not total solutions to a problem. What's interesting is their lack of concern for exhaustive browser based testing in the Hold level which means don't worry about it.

For Windows and Powershell people, you'll be glad to know that Chef, Puppet and Octopus (automated deployment of ASP.NET apps without PowerShell)  to support infrastructure tasks has made Windows automation a much better choice.


Martin is following the NoSQL movement very closely and has put MongoDB as the choice for his Adopt level. CouchBase, Hadoop and BigQuery are down one level in Trial. Node.js is down there too probably as a technology too green to make it worth our while yet. But I'm waiting for their take on Polymer and Meteor. Also interesting in the next level up from the basement, Assess, is PhoneGap (Apache Cordova) and Zepto.js, the smaller relative to jQuery.

I wasn't surprised to see WS-* holding the bottom place in the platforms with REST taking over web services slowly but surely.


I'm using NuGet for .NET development and was happy to see it in the top level. Check out Chocolatey NuGet as well if you do Windows administration. Maven is on Hold.

I've been searching for that right observer component that Google's Angular.js, Knockout.js  or Ember.js cover but also includes the whole MV* framework thing which I have covered with either a Java or .NET web framework. Reactive extensions for .NET (RxJS) didn't fare too well in the Assess level. My choice here for a solution to the observer pattern could be ReactJS or RxJS. 

The one tool that surprised me was D3. I have been recommending Raphael for JavaScript charting and watching D3 some but it shows up now in the Adopt level due to better complementary libraries such as Rickshaw and Crossfilter.

Languages and Frameworks

CSS frameworks like SASS/SCSS and Compass were staying in the Adopt level. The web apps that are moving away from traditional client/server architecture have much to learn yet but many frameworks are beginning to have business value so that they show up on the Trial level. These are HTML5 for offline applications, JavaScript as a platform and JavaScript MV* frameworks. Twitter Bootstrap also shows up as an Assess. But Backbone.js  and handwritten CSS are as good as last year in the Hold level.

A surprising observation was that Team Foundation Server caused productivity problems as a version control system. ThoughtWorks recommends Git, Perforce, or Subversion instead. It's a good thing that Visual Studio works with Git.

And just when you thought analytics couldn't get any better than Google Analytics, they see great promise in the data set aggregation and AWS/Hadoop management of your billions of web hits with Snowplow Analytics.

I'm sure I've missed some recommendations and packages people are using such as that new fangled language Mel Tillis and Kenny Rogers started about a paralyzed vet's wife going in to town for the evening without him. But read through the assessment and mine the results for some great improvements to your technology stack.