How Hierarchy Hurts
I’ve been in the Software Development industry for well over 20 years now. Starting out as a developer, I have filled many roles over the years with both a technical and product focus at small startups, medium businesses and large corporates. All of them have sincerely tried to be Agile in one flavour or another yet somehow the promised benefits of Agile have not fully materialised. I know I am not alone in that experience.
There are lots of factors that can contribute to a successful implementation of Agile and I am not going to try to cover all of them. If you want to read about stand-ups or story points or retrospectives, this is not the place. They are covered in plenty of other places and I’m not going to repeat them here.
In this series we will look at some of the fundamental concepts present in many Agile methodologies and tools, and consider how they contribute to the success of a project. When I say fundamental, I mean things that you may take for granted without a second thought. Things that are so established that it is assumed there is no other way. Some things may even be considered controversial. Even if you do not agree with me, I encourage you to take a few minutes, step outside the box and reconsider your assumptions. If the discourse piques your interest, then I invite you to journey with me and explore the opportunities ahead.
There is an opportunity to rethink the way work is structured. When you think about it, regardless of what labels are used - whether they are stories, epics or anything else, work items are fundamentally a to-do list. A very sophisticated to-do list with lots of details and fields and conversations and definitions of ready and states and workflows and, and, and…. And all those things are useful but ultimately it is still a to-do list.
The thing with to-dos is the main focus is to get it done. Whilst it is very important to complete the work that is set, it is even more important that the work is relevant, adds value and is aligned with the product vision. To-do items have a high risk of falling out of alignment with the product direction. It is easy to become too focused on just completing the work so that the burn-down chart looks good or so that management can see that metrics are being achieved. But what should be be front-and-center for everyone is the value it will bring to the customer. Too often constant external reminders are necessary to keep work focused and progressing in the direction of customer value.
I have simplified it by calling it a to-do list; in reality it is more complex than that. There are two common ways work items are presented: a flat list or a hierarchy. However, even when arranged as a hierarchy, it is still fundamentally a list; or more precisely, a list of lists. Each level of the hierarchy is still treated like a to-do item with the potential to obscure the product vision.
It is not sufficient that the product vision be encapsulated at the higher levels of hierarchy since the people building the product tend to work at the bottom of the hierarchy and rarely look above that. Often, when it is there, it is so dispersed across items that it is hard to grasp how everything comes together and where it is heading.
Hypothetical work Items for a web store
Take the very simple example of some epics from a hypothetical online webshop shown above. How long does it take you to work out what is planned, what order they should be done in, what is done and what doesn’t belong? If you have to read every row, that’s not going to scale very well when you have hundreds of rows.
Furthermore, a strict hierarchy has these other deficiencies:
Lack of Flexibility: Agile projects require flexibility to respond to changing priorities and customer feedback. If new requirements or insights emerge, restructuring the hierarchy is cumbersome and time-consuming. Inserting new items, moving items from one branch to another, making sure everything is where it belongs all takes time.
Dependency Management: In a hierarchical structure, dependencies between different levels or across hierarchies can lead to confusion. For example, if a story depends on an epic which is not its parent. Or it depends on a story that is not a sibling. Since Agile projects often involve concurrent development over many iterations, it is important to handle dependencies efficiently.
Difficulty in Scaling: As Agile teams grow or projects become increasingly complex, hierarchical models struggle to scale effectively. Large hierarchies lead to increased overhead and difficulties maintaining a clear overview as it gets spread across all levels. It is typical for the number of levels in a hierarchy to increase as the size of a project increases.
It is not uncommon for higher levels in the hierarchy to be focused on product management and the lower levels on execution. Unfortunately, if the process between these is not unified putting them in the same hierarchy will not magically unify them. It may just further separate the builders from the vision.Completion Focus: Agile methodologies prioritise delivering value to the customer. As mentioned, a rigid hierarchy leads to a focus on completing tasks in the hierarchy rather than delivering incremental value to the customer.
Related to hierarchies is the limitations of the queries to search them. The filtering of items is not sufficiently sophisticated for the complexity of projects and creates inefficiencies. The inefficiencies can manifest in a number of ways:
Duplication of Queries: If a query exists but does not quite meet the need of another user, then the query will be duplicated and adjusted to suit. For example, when the definition includes the fields to be returned in the results, if the query provides the right items that someone is looking for, but excludes a field they are interested in, they will create a duplicate query just to have the desired field. Now when the conditions of the original query need to be changed, all copies that were based on it also need to be changed.
Unable to retrieve data: This usually only applies to hierarchical data where the query mechanism does not support querying across relationships. For example, “Get all open stories for epics in the 3.5 release.” This requires filtering on the status field on the story end of the relationship and the release field on the epic. Most tools have little or no support for queries like this.
Duplication of Fields: This is usually a consequence of the query limitations noted above. A common solution is to duplicate fields at all levels of the hierarchy. This allows the data to be retrieved but comes at the cost of additional maintenance overhead.
Of course, most Agile tools also have some sort of a kanban or sprint board view. It is a different view than a list of to-do items. The focus of a board is even more on the completion of work as each list item moves its way across the status columns. Boards will also have swim lanes to group related work together which provides some immediate context; however, the big picture is still hidden. The appropriate use of boards is a whole other topic to be explored another time.
It’s easy to imagine how work tracking tools evolved from simple to-do lists to the complex hierarchies we see today. For me, I could see incremental changes to the tools that appeared on the surface to be making things better. But over time, I started to notice small limitations and inconveniences. Maybe you did too? Eventually, I got to the point where I thought there must be a better way. To discover what it was, I had to reconsider some other long held ideas.
Continued in part 2: Do Epics tell the Story