The Path to a Better Way

In the last 2 posts we considered how some prominent Agile concepts and ideas may not be delivering their full potential. First, we looked at how work items tend to be treated as to-do lists, often arranged in a hierarchy. Then we examined the literary story/epic metaphor and the role it plays. In both cases, there were benefits at a small scale but they were lost as projects scale up in size.

In this post, we will explore an alternative approach, tackling the structure and the metaphor together. At the start, I suggested reconsidering long held assumptions. Now I am asking you to rotate your thinking 90 degrees, from a vertical hierarchy with a literary metaphor, to a horizontal flow and a journey metaphor.

Consider how you plan a journey on a GPS. First, you enter your final destination. If it is a long journey, you may have other interim destinations you need to pass through so you add them in too. Once that information is provided the system will suggest all the individual steps you need to take to get to each waypoint and ultimately to your destination.

It’s the same way products evolve. They start with a lofty goal; an objective or desired outcome. For any decent sized project these goals will be too big to achieve all at once. There will be checkpoints along the way to confirm it is on track to deliver the intended value and if it’s not, then adjust as necessary. Once the checkpoints have been identified, the individual work items to achieve each of them can be fleshed out.

Hence, the foundation of the journey metaphor for Agile development are: Destination, Waypoint and Step.

Destination

A significant goal, achievement, outcome or objective.

What constitutes as significant may vary from project to project. This is acceptable since it does not have to be “final” in the sense that there will not be anything after it. In the same way, reaching the destination you put in your GPS doesn’t mean you won’t later proceed to another place.

A Destination would represent a state where a value proposition can be realised. They would be set by the product manager.

Waypoint

An interim milestone or checkpoint.

A Waypoint is a logical point to evaluate how the work is progressing and if any changes need to be made based on what has been completed and what has been learned.

Waypoints should be agreed between the team and the product manager. The team will have insights for any risks that need to be explored and mitigated, and a Waypoint is the time to review the risk. So it makes sense that the team should have input to that. A Waypoint is also a checkpoint to ensure assumptions about the value proposition still hold and the product manager must be included as well.

Step

A small, incremental development or other activity that progresses towards a Waypoint and ultimately, a destination.

The team should decide what the necessary steps are to reach a Waypoint.

It may be tempting at this juncture to look at these and think that a Step corresponds to a story, a Waypoint corresponds to an epic and a Destination corresponds to a feature. But there is more to it than that. We now look at how to apply these structurally and metaphorically.

Step, Waypoint and Destination are not hierarchical; instead, they represent a progressive flow. The relationship between items is not parent/child. In some cases, it could represent a hard dependency. It can also just be a preferred sequencing. This means they can be fluid and change according to the circumstances.

Also, note that a Destination may have links to what will come after it - and that could be the next step. Metaphorically, when you make it to one Destination, you know the journey will continue, even if you don’t know where to at the time you set out.

Not only are these items not hierarchical, they are also not mutually exclusive. Completing the last leg on a journey can be seen as arriving at the final checkpoint;  that checkpoint can also be the final destination. It follows that a step can also be a waypoint and a destination.

With the foundational concepts in place, there are many other journey related terms that can be applied that work with the metaphor.

Path: A linear progression from one item to another

Detour: An alternate path to follow when the intended path is blocked

Dead end: A path that does not end at a desired location

Pothole: Some repair work is needed here

Scout ahead/Exploration: The path forward is not clear, send a party ahead to find a way

Shortcut: An alternative path that is shorter or quicker than the original

The visualisation of this metaphor naturally lends itself to be a directed graph and opens up a variety of new user experiences and ways to explore an agile plan. The example below shows some of the immediate benefits and possibilities:

• Immediately you can see and understand the relationship between things (It is the same dataset that was shown as a hierarchy in the first post.)

• It looks like an actual roadmap

• The sequencing of work is clear

• The size of an item can be visualised for a quick relative comparison

• Colour can be used to show status and progression

• Work that does not align to goals is obvious

Simple interactions quickly enable these actions directly:

• Adding and removing items

• Reordering items

• Expanding to see what is next

• Collapsing to hide what is not relevant

A key element is the concept of a path defined earlier as a linear progression from one item to another. The image above is really a collection of paths that together present a plan. Paths can be useful in several ways:

• Stakeholders may want to view the progress towards a destination

• A product owner may want to view the progress towards a waypoint

• A team may want to view the steps required to make their next milestone

All of the above can be derived from the same plan. They are all paths at different levels.

It is interesting to consider the path of a team towards a Waypoint. The path from one Waypoint to the next could be considered as the next “unit of work” for a team. So then, instead of having a time-based sprint, they could have an objective-based sprint which is naturally defined by the path between two consecutive Waypoints. Also, notice how a “sprint” works nicely within a journey metaphor.

If a sprint can be defined by the paths to be completed, is a traditional sprint board required? The path shows all the steps that need to be completed as well as their current progress. It is clear when something has not started yet. Even more than that, it is possible that the current sprint could be a filtered version of a larger plan.

That all seems pretty cool, but is it better? Does it address the shortcomings of other approaches?

Completion Focus: Work items are not presented as to-do lists. They show a progression to meet an objective. Completing work is still important and it is represented in a consumable manner.

Value Delivery: This is closely related to completion focus above. This approach naturally shows the progression to achieve the next value goal, and the next one and the one after that. The value proposition is always accessible by following the path.

Improve Collaboration: This is achieved in three ways. Firstly, the representation is easy to consume and understand quickly. Secondly, it is clear what value the team is striving for and what it will take to get there. Lastly, there is a consistent vocabulary and metaphor that can be used at all levels of the organisation.

Scalability: There can be any number of paths to reach a destination and each path can be an arbitrary length. The same concepts can be applied consistently regardless of the size of the project.

Flexibility: Just as in real life when travel plans change, the route may be altered. Paths can be added, removed or re-routed as necessary .These changes are easy to make and are easy to assess with this representation.

Query Support: A rethink of what a query should be for a journey is necessary and deserves its own post. For now, it is sufficient to say that it is possible, and it is really useful.

Have you ever found yourself referring to product development as a journey or a progression? Have you ever been frustrated by how much time you spend trawling through work items? If that is the case, then hopefully some of these ideas resonate with you. I’ve only just scratched the surface of where this is going. A teaser if you will. There is more to follow.

If this post has excited you, or reinvigorated a passion for truly Agile development, feel free to reach out to me. We are on the lookout for great people to evaluate, improve and refine these ideas. If that is something you would like to be a part of, let me know.

Do epics tell the story?

In the previous post, I started exploring the way many Agile methodologies and tools structure work items as hierarchical to-do lists and the consequences of it. I looked at the way the structure encouraged a short term completion mindset and made it harder for teams to keep aligned to the product vision. I said we would be looking at some fundamental concepts to see what opportunities for improvement might exist.

The second opportunity we are going to look at considers the role of language and vocabulary. It goes without saying that words are important and our choice of words can affect how well our message is received. So let’s take a look at some language constructs common in Agile today.

Metaphors are a literary tool that are used to help understand a complex topic by relating it to something that is already familiar. In technology, metaphors often become abstractions and show up in code, in patterns and, as it turns out, in the tools we use. In all cases, they are supposed to help understand a specific topic.

Epics and stories are the go-to metaphor for planning software development activities today. It is a concept that has been part of the agile vocabulary since the early days and is familiar to all developers, project managers and product managers.

The purpose of a story, or user story, is to describe a small aspect of a system being built in the style of a narrative about the user’s interaction. Storytelling is a great way to communicate ideas, and it’s something that people have done for millennia. It’s a very relatable way to focus on something from the perspective of the customer.

Where a story is intended to be small - something that can be completed in a sprint - the original intent of an epic was that it was a big story. Something that was too big to fit into a single sprint. Put another way, an epic is the same as a story, only bigger and longer.

If we only consider the application of stories and epics to software development, the metaphor works. There are small work items and big work items, and they are expressed in a meaningful, story-like way. Of course, stories and epics are just one part. There are many other aspects of Agile development. So let’s consider how well it expands if we consider the broader ecosystem.

Recall that whether something is a story or an epic depends only on if it fits in a sprint. A sprint is another commonly used Agile term, but outside of Agile, a sprint is unrelated to storytelling. Sprinting is more readily associated with athletics, the Olympics or sport. It does not fit with the literary story metaphor.

If the intent of an epic is that it is a big story that cannot be completed in a single sprint, the implication must be that an epic is indivisible. If that wasn’t the case, then it would have been broken down into stories and the epic wouldn’t be necessary.

In practice, what most projects do is introduce a hierarchy into the relationship between story and epic. We have already discussed the limitations of hierarchies, and I’m not going to repeat that here. In the context of a literary metaphor, introducing a hierarchy here breaks the original intent of an epic as a large, indivisible piece of work. In an Agile project today, an epic represents a grouping of stories. In a literary metaphor, it is more like a series.

Where the literary metaphor really starts showing its limitations is when attempts are made to scale it. Take features for example. All products have features. So the concept of a feature as it applies to a product is understood. But as part of a literary metaphor, it doesn’t fit. Outside of software development, there is nothing that combines epics, features and stories. Even within software development, they are not defined consistently - sometimes feature is above epic, sometimes it is the other way around.

A metaphor is supposed to help understand a complex topic by relating it to real world experiences. The original story/epic metaphor could do this. Over time it has expanded and evolved into what we see today. Most Agile teams have been conditioned to use stories, epics and features. It no longer functions as a metaphor; it is now a hierarchy to scope groups of items in a to-do list.

Some may say “It’s only a metaphor, and no metaphor is perfect”. Does it really matter?

Here’s the thing: building software is not just about writing code and saying it is done. A huge part of a successful project comes down to communication, collaboration and alignment of everyone involved. There is a good reason why the Agile Manifesto values “interaction and collaboration over processes and tools”. It is within the interactions that the value will be discovered and for them to be effective there needs to be a common vocabulary to describe the product and the vision.

As much as this post has talked about tools and processes, the intention is for them to facilitate and support interaction and collaboration. These are not independent things. The output of any collaboration activity should be represented in the tools, and the tools should help inform discussions and decisions about the product. If there is any participant at any level, be it a developer, tester, business analyst, product manager or stakeholder who cannot easily use a common tool to find something that matches their understanding of the product, it increases communication friction and makes collaboration harder.

Stories and epics are simple for developers, but any hierarchy above that starts to lose relevance. Stakeholders will understand features and objectives, but the progression of epics and stories to deliver them is too complicated.

Does it matter? Yes, it does. Especially when you consider how much more cohesive collaboration can be under a suitably structured and unified metaphor.

Next: The Path to a Better Way

Previous: How Hierarchy Hurts

How Hierarchy Hurts

I’ve been in the Software Development industry for well over 20 years now. Starting out as a developer, I have filled many roles over the years with both a technical and product focus at small startups, medium businesses and large corporates. All of them have sincerely tried to be Agile in one flavour or another yet somehow the promised benefits of Agile have not fully materialised. I know I am not alone in that experience.

There are lots of factors that can contribute to a successful implementation of Agile and I am not going to try to cover all of them. If you want to read about stand-ups or story points or retrospectives, this is not the place. They are covered in plenty of other places and I’m not going to repeat them here.

In this series we will look at some of the fundamental concepts present in many Agile methodologies and tools, and consider how they contribute to the success of a project. When I say fundamental, I mean things that you may take for granted without a second thought. Things that are so established that it is assumed there is no other way. Some things may even be considered controversial. Even if you do not agree with me, I encourage you to take a few minutes, step outside the box and reconsider your assumptions. If the discourse piques your interest, then I invite you to journey with me and explore the opportunities ahead.

There is an opportunity to rethink the way work is structured. When you think about it, regardless of what labels are used - whether they are stories, epics or anything else, work items are fundamentally a to-do list. A very sophisticated to-do list with lots of details and fields and conversations and definitions of ready and states and workflows and, and, and….  And all those things are useful but ultimately it is still a to-do list.

The thing with to-dos is the main focus is to get it done. Whilst it is very important to complete the work that is set, it is even more important that the work is relevant, adds value and is aligned with the product vision. To-do items have a high risk of falling out of alignment with the product direction. It is easy to become too focused on just completing the work so that the burn-down chart looks good or so that management can see that metrics are being achieved. But what should be be front-and-center for everyone is the value it will bring to the customer. Too often constant external reminders are necessary to keep work focused and progressing in the direction of customer value.

I have simplified it by calling it a to-do list; in reality it is more complex than that. There are two common ways work items are presented: a flat list or a hierarchy. However, even when arranged as a hierarchy, it is still fundamentally a list; or more precisely, a list of lists. Each level of the hierarchy is still treated like a to-do item with the potential to obscure the product vision.

It is not sufficient that the product vision be encapsulated at the higher levels of hierarchy since the people building the product tend to work at the bottom of the hierarchy and rarely look above that. Often, when it is there, it is so dispersed across items that it is hard to grasp how everything comes together and where it is heading.

Hypothetical work Items for a web store

Take the very simple example of some epics from a hypothetical online webshop shown above. How long does it take you to work out what is planned, what order they should be done in, what is done and what doesn’t belong? If you have to read every row, that’s not going to scale very well when you have hundreds of rows.

Furthermore, a strict hierarchy has these other deficiencies:

1. Lack of Flexibility: Agile projects require flexibility to respond to changing priorities and customer feedback. If new requirements or insights emerge, restructuring the hierarchy is cumbersome and time-consuming. Inserting new items, moving items from one branch to another, making sure everything is where it belongs all takes time.

2. Dependency Management: In a hierarchical structure, dependencies between different levels or across hierarchies can lead to confusion. For example, if a story depends on an epic which is not its parent. Or it depends on a story that is not a sibling. Since Agile projects often involve concurrent development over many iterations, it is important to handle dependencies efficiently.

3. Difficulty in Scaling: As Agile teams grow or projects become increasingly complex, hierarchical models struggle to scale effectively. Large hierarchies lead to increased overhead and difficulties maintaining a clear overview as it gets spread across all levels. It is typical for the number of levels in a hierarchy to increase as the size of a project increases.
   It is not uncommon for higher levels in the hierarchy to be focused on product management and the lower levels on execution. Unfortunately, if the process between these is not unified putting them in the same hierarchy will not magically unify them. It may just further separate the builders from the vision.

4. Completion Focus: Agile methodologies prioritise delivering value to the customer. As mentioned, a rigid hierarchy leads to a focus on completing tasks in the hierarchy rather than delivering incremental value to the customer.

Related to hierarchies is the limitations of the queries to search them. The filtering of items is not sufficiently sophisticated for the complexity of projects and creates inefficiencies. The inefficiencies can manifest in a number of ways:

1. Duplication of Queries: If a query exists but does not quite meet the need of another user, then the query will be duplicated and adjusted to suit. For example, when the definition includes the fields to be returned in the results, if the query provides the right items that someone is looking for, but excludes a field they are interested in, they will create a duplicate query just to have the desired field. Now when the conditions of the original query need to be changed, all copies that were based on it also need to be changed.

2. Unable to retrieve data: This usually only applies to hierarchical data where the query mechanism does not support querying across relationships. For example, “Get all open stories for epics in the 3.5 release.” This requires filtering on the status field on the story end of the relationship and the release field on the epic. Most tools have little or no support for queries like this.

3. Duplication of Fields: This is usually a consequence of the query limitations noted above. A common solution is to duplicate fields at all levels of the hierarchy. This allows the data to be retrieved but comes at the cost of additional maintenance overhead.

Of course, most Agile tools also have some sort of a kanban or sprint board view. It is a different view than a list of to-do items. The focus of a board is even more on the completion of work as each list item moves its way across the status columns. Boards will also have swim lanes to group related work together which provides some immediate context; however, the big picture is still hidden. The appropriate use of boards is a whole other topic to be explored another time.

It’s easy to imagine how work tracking tools evolved from simple to-do lists to the complex hierarchies we see today. For me, I could see incremental changes to the tools that appeared on the surface to be making things better. But over time, I started to notice small limitations and inconveniences. Maybe you did too? Eventually, I got to the point where I thought there must be a better way. To discover what it was, I had to reconsider some other long held ideas.

Continued in part 2: Do Epics tell the Story