Do epics tell the story?

In the previous post, I started exploring the way many Agile methodologies and tools structure work items as hierarchical to-do lists and the consequences of it. I looked at the way the structure encouraged a short term completion mindset and made it harder for teams to keep aligned to the product vision. I said we would be looking at some fundamental concepts to see what opportunities for improvement might exist.

The second opportunity we are going to look at considers the role of language and vocabulary. It goes without saying that words are important and our choice of words can affect how well our message is received. So let’s take a look at some language constructs common in Agile today.

Metaphors are a literary tool that are used to help understand a complex topic by relating it to something that is already familiar. In technology, metaphors often become abstractions and show up in code, in patterns and, as it turns out, in the tools we use. In all cases, they are supposed to help understand a specific topic.

Epics and stories are the go-to metaphor for planning software development activities today. It is a concept that has been part of the agile vocabulary since the early days and is familiar to all developers, project managers and product managers.

The purpose of a story, or user story, is to describe a small aspect of a system being built in the style of a narrative about the user’s interaction. Storytelling is a great way to communicate ideas, and it’s something that people have done for millennia. It’s a very relatable way to focus on something from the perspective of the customer.

Where a story is intended to be small - something that can be completed in a sprint - the original intent of an epic was that it was a big story. Something that was too big to fit into a single sprint. Put another way, an epic is the same as a story, only bigger and longer.

If we only consider the application of stories and epics to software development, the metaphor works. There are small work items and big work items, and they are expressed in a meaningful, story-like way. Of course, stories and epics are just one part. There are many other aspects of Agile development. So let’s consider how well it expands if we consider the broader ecosystem.

Recall that whether something is a story or an epic depends only on if it fits in a sprint. A sprint is another commonly used Agile term, but outside of Agile, a sprint is unrelated to storytelling. Sprinting is more readily associated with athletics, the Olympics or sport. It does not fit with the literary story metaphor.

If the intent of an epic is that it is a big story that cannot be completed in a single sprint, the implication must be that an epic is indivisible. If that wasn’t the case, then it would have been broken down into stories and the epic wouldn’t be necessary.

In practice, what most projects do is introduce a hierarchy into the relationship between story and epic. We have already discussed the limitations of hierarchies, and I’m not going to repeat that here. In the context of a literary metaphor, introducing a hierarchy here breaks the original intent of an epic as a large, indivisible piece of work. In an Agile project today, an epic represents a grouping of stories. In a literary metaphor, it is more like a series.

Where the literary metaphor really starts showing its limitations is when attempts are made to scale it. Take features for example. All products have features. So the concept of a feature as it applies to a product is understood. But as part of a literary metaphor, it doesn’t fit. Outside of software development, there is nothing that combines epics, features and stories. Even within software development, they are not defined consistently - sometimes feature is above epic, sometimes it is the other way around.

A metaphor is supposed to help understand a complex topic by relating it to real world experiences. The original story/epic metaphor could do this. Over time it has expanded and evolved into what we see today. Most Agile teams have been conditioned to use stories, epics and features. It no longer functions as a metaphor; it is now a hierarchy to scope groups of items in a to-do list.

Some may say “It’s only a metaphor, and no metaphor is perfect”. Does it really matter?

Here’s the thing: building software is not just about writing code and saying it is done. A huge part of a successful project comes down to communication, collaboration and alignment of everyone involved. There is a good reason why the Agile Manifesto values “interaction and collaboration over processes and tools”. It is within the interactions that the value will be discovered and for them to be effective there needs to be a common vocabulary to describe the product and the vision.

As much as this post has talked about tools and processes, the intention is for them to facilitate and support interaction and collaboration. These are not independent things. The output of any collaboration activity should be represented in the tools, and the tools should help inform discussions and decisions about the product. If there is any participant at any level, be it a developer, tester, business analyst, product manager or stakeholder who cannot easily use a common tool to find something that matches their understanding of the product, it increases communication friction and makes collaboration harder.

Stories and epics are simple for developers, but any hierarchy above that starts to lose relevance. Stakeholders will understand features and objectives, but the progression of epics and stories to deliver them is too complicated.

Does it matter? Yes, it does. Especially when you consider how much more cohesive collaboration can be under a suitably structured and unified metaphor.

Next: The Path to a Better Way

Previous: How Hierarchy Hurts

How Hierarchy Hurts

I’ve been in the Software Development industry for well over 20 years now. Starting out as a developer, I have filled many roles over the years with both a technical and product focus at small startups, medium businesses and large corporates. All of them have sincerely tried to be Agile in one flavour or another yet somehow the promised benefits of Agile have not fully materialised. I know I am not alone in that experience.

There are lots of factors that can contribute to a successful implementation of Agile and I am not going to try to cover all of them. If you want to read about stand-ups or story points or retrospectives, this is not the place. They are covered in plenty of other places and I’m not going to repeat them here.

In this series we will look at some of the fundamental concepts present in many Agile methodologies and tools, and consider how they contribute to the success of a project. When I say fundamental, I mean things that you may take for granted without a second thought. Things that are so established that it is assumed there is no other way. Some things may even be considered controversial. Even if you do not agree with me, I encourage you to take a few minutes, step outside the box and reconsider your assumptions. If the discourse piques your interest, then I invite you to journey with me and explore the opportunities ahead.

There is an opportunity to rethink the way work is structured. When you think about it, regardless of what labels are used - whether they are stories, epics or anything else, work items are fundamentally a to-do list. A very sophisticated to-do list with lots of details and fields and conversations and definitions of ready and states and workflows and, and, and….  And all those things are useful but ultimately it is still a to-do list.

The thing with to-dos is the main focus is to get it done. Whilst it is very important to complete the work that is set, it is even more important that the work is relevant, adds value and is aligned with the product vision. To-do items have a high risk of falling out of alignment with the product direction. It is easy to become too focused on just completing the work so that the burn-down chart looks good or so that management can see that metrics are being achieved. But what should be be front-and-center for everyone is the value it will bring to the customer. Too often constant external reminders are necessary to keep work focused and progressing in the direction of customer value.

I have simplified it by calling it a to-do list; in reality it is more complex than that. There are two common ways work items are presented: a flat list or a hierarchy. However, even when arranged as a hierarchy, it is still fundamentally a list; or more precisely, a list of lists. Each level of the hierarchy is still treated like a to-do item with the potential to obscure the product vision.

It is not sufficient that the product vision be encapsulated at the higher levels of hierarchy since the people building the product tend to work at the bottom of the hierarchy and rarely look above that. Often, when it is there, it is so dispersed across items that it is hard to grasp how everything comes together and where it is heading.

Hypothetical work Items for a web store

Take the very simple example of some epics from a hypothetical online webshop shown above. How long does it take you to work out what is planned, what order they should be done in, what is done and what doesn’t belong? If you have to read every row, that’s not going to scale very well when you have hundreds of rows.

Furthermore, a strict hierarchy has these other deficiencies:

1. Lack of Flexibility: Agile projects require flexibility to respond to changing priorities and customer feedback. If new requirements or insights emerge, restructuring the hierarchy is cumbersome and time-consuming. Inserting new items, moving items from one branch to another, making sure everything is where it belongs all takes time.

2. Dependency Management: In a hierarchical structure, dependencies between different levels or across hierarchies can lead to confusion. For example, if a story depends on an epic which is not its parent. Or it depends on a story that is not a sibling. Since Agile projects often involve concurrent development over many iterations, it is important to handle dependencies efficiently.

3. Difficulty in Scaling: As Agile teams grow or projects become increasingly complex, hierarchical models struggle to scale effectively. Large hierarchies lead to increased overhead and difficulties maintaining a clear overview as it gets spread across all levels. It is typical for the number of levels in a hierarchy to increase as the size of a project increases.
   It is not uncommon for higher levels in the hierarchy to be focused on product management and the lower levels on execution. Unfortunately, if the process between these is not unified putting them in the same hierarchy will not magically unify them. It may just further separate the builders from the vision.

4. Completion Focus: Agile methodologies prioritise delivering value to the customer. As mentioned, a rigid hierarchy leads to a focus on completing tasks in the hierarchy rather than delivering incremental value to the customer.

Related to hierarchies is the limitations of the queries to search them. The filtering of items is not sufficiently sophisticated for the complexity of projects and creates inefficiencies. The inefficiencies can manifest in a number of ways:

1. Duplication of Queries: If a query exists but does not quite meet the need of another user, then the query will be duplicated and adjusted to suit. For example, when the definition includes the fields to be returned in the results, if the query provides the right items that someone is looking for, but excludes a field they are interested in, they will create a duplicate query just to have the desired field. Now when the conditions of the original query need to be changed, all copies that were based on it also need to be changed.

2. Unable to retrieve data: This usually only applies to hierarchical data where the query mechanism does not support querying across relationships. For example, “Get all open stories for epics in the 3.5 release.” This requires filtering on the status field on the story end of the relationship and the release field on the epic. Most tools have little or no support for queries like this.

3. Duplication of Fields: This is usually a consequence of the query limitations noted above. A common solution is to duplicate fields at all levels of the hierarchy. This allows the data to be retrieved but comes at the cost of additional maintenance overhead.

Of course, most Agile tools also have some sort of a kanban or sprint board view. It is a different view than a list of to-do items. The focus of a board is even more on the completion of work as each list item moves its way across the status columns. Boards will also have swim lanes to group related work together which provides some immediate context; however, the big picture is still hidden. The appropriate use of boards is a whole other topic to be explored another time.

It’s easy to imagine how work tracking tools evolved from simple to-do lists to the complex hierarchies we see today. For me, I could see incremental changes to the tools that appeared on the surface to be making things better. But over time, I started to notice small limitations and inconveniences. Maybe you did too? Eventually, I got to the point where I thought there must be a better way. To discover what it was, I had to reconsider some other long held ideas.

Continued in part 2: Do Epics tell the Story