Is Agile still appropriate for modern digital data-driven businesses?

7 min readOct 13, 2020

Most businesses adopt some version of iterative product development within their delivery lifecycles. Agile is by far the most common approach to iterative development in any industry and has been a mainstay as such almost 20 years.

But it’s a changing world, one enriched with more data and analytics than we know what do with and where collaboration tools make sure we are connected and accountable at any given moment. Digital product innovation has never been more competitive, more optimized and even more commoditized in terms of its delivery and approach to team utilization.

Agile frameworks are not formally adapting to these changing times and there is no formal versioning process that can centrally influence how organizations can best apply or optimize over time.

Within the last 5 years, Machine Learning has become a core capability of any new digital product but there is no formal language for data science in Agile*.

Likewise, management approaches like Servant Leadership and Lean Startup Methodology (not to be confused with Agile-Lean), are gaining significant traction within the product and technology leadership circles and already influence how a team prioritizes and allocates tasks across a project team.

So given these new ways of working, should we still use Agile?

excluding formal methodologies for iterative Data Science (i.e. CRISP-DM)

What does Agile mean again?

The term “Agile” is perhaps the single most widely over-used term in product development. So much so that as companies describe their inner processes as “Agile”, it’s easy to mis-understand what that actually means…so what does it mean to be Agile?

This term that started as a software development methodology has been adopted by organizations as a broader movement for digital innovation at times encompassing everything from people management, project delivery and corporate marketing initiatives.

The term “Agile” is perhaps the single most widely over-used term in product development.

Agile in Software Development advocates for fast delivery cycles, direct stakeholder involvement and reactive re-prioritization over multiple iterations. It is by far the most popular approach to building modern digital platforms and frameworks such as Scrum, Kanban, Extreme Programing as well as the wider governance framework SAFe make up the majority of the industry-adopted approaches.

Agile has plenty of vulnerabilities…and a dark side.

What follows here is a short list of vulnerabilities within Agile frameworks that have not adapted with the interaction of newer capabilities or have become unforeseen frictions over time.

In each case I’ve also added suggestions on how businesses might mitigate and address these issues.

The quickest way to kill the morale of a team member is to take all decision-making responsibilities away from them.

1. Micro-management

The quickest way to kill the morale of a team member is to take all decision-making responsibilities away from them.

Agile development has been the focus for better tooling for many years, each new platform coming with the promise of squeezing productivity and showing greater metrics and transparency.

Greater scrutiny and less control will always work inversely with team member happiness. This is a great article I recently stumbled upon concerting the issues of over-measurement which does a great job or walking through how greater degrees of controls within common tools like Atlassians Jira (the worlds most popular Agile Project Management Tool) can impact team morale.

Suggested Approach: Incorporate thoughtful task allocation and team management into the process

I have found that these principles are effective in both delivering work and ensuring a happier workplace:-

Organizing as a mostly flat hierarchy with each squad-sized of 4–6 members including a lead (depending on team size)
Driving for an inclusive culture and removing ego where appropriate by thoughtful work allocation and share-outs
Allocating tasks to team members to take advantage of develop growth opportunities
Driving a culture that self-expression in work (as well as more broadly) is welcomed, not managed out.

2. Lack of formal ROI (Return of Investment) controls

Building new software products relies on the involvement of customer stakeholders, which, for the most part is represented by a Product Manager, Sales or customer support team member.

It’s absolutely fundamental that work is prioritized through a customer-centric lens but Agile doesn’t go further to provide the constructs for businesses to manage the ROI of decisioning and prioritization.

Given how Agile has evolved from a purely software development process to a digital delivery team process, I think ROI should now be a formal input to the process.

Most companies don’t yet have a formal approach on how to incorporate data insights back into product development and they’re falling behind.

Suggested Approach: Introduce formal controls to quantify the ROI for each new story or work item

Introduce quantitative scoring for both analytics ( how we measure the success of the change) and the cost of the change in $ as well as the anticipated value to the company that this change means. These signals will help customer stakeholders think much more carefully about how work is prioritized and provide a longer term measure of build efficiency.

3. Data Science need a bigger seat at the table

The major role that analytics and machine learning now plays in building modern digital platforms doesn’t have enough formal representation in Agile. With no specific constructs for task prioritization based on informed-data decisions, the responsibility of prioritizing tasks remains a purely subjective strategy from the customer stakeholder.

Recent history has shown that the adoption of products can be significantly improved with the introduction of data-inferred changes that might have seemed unintuitive at first pass.

A recent McKinsey report describes where businesses have experienced considerable improvements on feature and design changes through the incorporation of data insights: https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/fusing-data-and-design-to-supercharge-innovation-in-products-and-processes

Most companies don’t yet have a formal approach on how to incorporate data insights back into product development and they’re falling behind, failing to keep up with the speed of innovation as a result.

An issue not addressed here is the organizational integration of Data Science and software development teams — this topic will be covered at a later date.

Suggested Approach: Treat Data Science as an integral input and stakeholder for acceptance criteria

Task prioritization and technical design should widely incorporates information from any insights garnered from behavioral analytics or elsewhere. Without that consideration, businesses simply aren’t getting a fair return from the investment in machine learning.

4. Lean Startup Development and Agile

Agile-Lean development (Toyota Product System) within the context of Agile has been written about for a while but there are principles within a completely different methodology commonly followed by new product businesses that also have merit.

The Lean Startup Methodology LSM (http://theleanstartup.com/principles) advocates for an approach that involves continuous validation, learning and the removal of uncertainty within a product build process. Perhaps the best anecdote that best describes LSM is the phrase “Love the problem, not the Solution”.

Within a delivery team setting, the solution strategy tends to be baked into a wider technology strategy and with little or no language in Scrum, XP, Kanban or SAFe on the maturity of a solution delivered within each iteration, the phased approach of first delivering something that works (MVP) before refactoring and further iterations is left down to technology strategy.

Without reducing quality, there is an opportunity to help delivery teams more formally think about the level of maturity required in regards to the phase of an overall solution which would impact task estimation and mitigate the risk of over-engineering.

Suggested Approach: Only build what is truly desirable and can be easily measured.

Taking a cue from the innovation playbook that decodes meaningful definitions on “Desirable”, “Feasible” and “Viable”, incorporate lean processes into task prioritization that ensures you’re building the right functionality at the right time. Complimentary to that effort is being able to explain to a wider audience why we’re building what we’re building and how we are thinking about each feature’s evolution.

it’s important for any company to remain receptive to what makes sense for them.

Conclusion

It’s fair to say that iterative product development is a core capability here to stay…but in changing times with new capabilities and management philosophies propagated so quickly these days, it’s important for any company to remain receptive to what makes sense for them and to be empowered to change accordingly.

Additional to the above suggestions, I would also advocate for Companies hiring and up-skilling teams to look more formally at certification processes for specific versions of Agile. I.e the SAFe 5.0 framework which is a specific version and certification courses have been available for a while: https://www.scaledagile.com/certification-and-exam-information-safe5-cmu/

Final Thoughts: Development of digital platforms is a fast-moving space and as such, modern tooling and methodologies should have the ability to be proactively updated. More formal methodology versioning from industry leaders like Atlasssian would help organizations tackle these challenges.

===

Thanks for taking the time to read this — as always I welcome all comments and feedback. I also included some useful links worth reviewing for this context: