In this blog post, we share some thoughts and experiences on myths, mistakes or misinterpretations seen in the delivery of Agile-AI projects. The critical success factors structure & discipline are widely misunderstood and have to be well balanced to drive successful impact.
SwissCognitive Guest Blogger: Michi Wegmüller, Co-Founder Artifact SA – Empowering Agile-AI at Scale
When I had my first training on agile delivery, I recall very well the agile coach discussing the Agile Manifesto with the four principles and the twelve practices we know by now all inside out. On this day, more than ten years ago, we had a long discussion regarding the 2nd principle, “Working software over comprehensive documentation” – and whether agile delivery means no documentation or how to interpret this principle. It was a discussion that ended as many discussions do in the consulting space – with an answer “it depends”. It pointed out the ambiguity that is embedded in agile. And it changed the way how I today practice agile, especially in AI and Data Science project work.
Over time we’ve seen many agile projects, some of them more successful, some of them less. Some of them claimed to follow agile, some of them actually did agile delivery. What makes a difference? Where are the small nuances that make a project successful, and what makes them fail? In this blog article, I would like to share my views on common mistakes, clean out some misbeliefs regarding agile, and provide a simple formula to keep in mind when you set up and run your next agile project.
How to understand the Agile Manifesto
Back to the Agile Manifesto and its four principles:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
While the first part of the principles (before the word “over”) seem to make common sense – I was very much interested in the second parts. We had many discussions in the past on how to interpret that part of the Agile Manifesto. A bit provocative, one could state that agile projects have no processes, tools, documentation, negotiations or plans at all.
After many years of experience in agile delivery, I have seen many cases that all show that such a ruthless interpretation is far from what the principles want to say. The Agile Manifesto wants to get the priorities right – but the principles are by no means exclusive nor radical statements.
On the other hand, a very naïve view would be to assume that agile and its manifesto is simple and straightforward. What initially seems simple and straight (at least looking at the principles) is after a closer look, far away from being that. In my experience, I have hardly seen any straightforward and easy agile project – every project has its complexity, its specialty, its character. And so does the Agile Manifesto – hence, the answer on how to interpret the manifesto truly depends on the actual situation a project is in and the ambition, what the project wants to achieve.
Let’s have a look at some examples around the four principles.
Agile-AI and processes
First, while it’s essential to be human-centered, agile is far more than anything else, structured and radically steered through a strong governance process. That seems counter-intuitive – but taking a closer look, it makes a lot of sense. This gets clearer if you look at it from various angles. E.g. agile has a single dedicated role in facilitating the method, process, and remove impediments – the scrum master. Where have you seen this in other delivery methods? It seems that agile is far more structured than initially it appears to be. Also, have a thought about the ceremonies such as the daily stand-ups, the scrum of scrums, and the retrospectives or playbacks – the iteration planning. Agile intentionally sets a frame, a structure around the delivery.
Agile-AI and plans
Second, agile delivery follows an incremental plan that allows for change – but it is far from operating without a plan. Here again, agile puts structure as an answer to keep the flexibility for change – agile’s answer to planning are the sprint & increment grid in addition to the product owner, who sets the right priorities. Have you ever been in an agile project where the product owner was not able to make decisions on the backlog priorities? Relatively quick, the project turns from a green status to dark red as the team needs continuous validation and discussion with the best person to set the priorities. Consequently, the burndown chart or velocity is hit, and you’re getting behind the plan. Rethinking KPIs like the ones mentioned above, in fact, try to give visibility on how much ahead or behind an expectation (aka a plan) a team/project is. The structures support the measurement of agile projects.
Agile-AI and documentation
Third, I recall an intense discussion regarding documentation requirements in agile projects. The client believed this is not required at all, and hence, we should increase our velocity expectations for the initial iteration of a project. I had to object to this view. To me, there is one thing obvious – there is no agile project, especially not in the AI and Data Science space, that will be delivered and accepted without documentation. Traceability, transparency, and explanations are essential to foster trust and proper interpretation of the results such projects give. Without adequate documentation, service receivers tend to criticize, question and challenge results. Clients will only apply to the recommendations from AI & Data Science algorithms if they understand what the development team has done. Additionally, teams work in iterations and push boundaries and support the cross-functionality in such scrum teams, it is vital to have job rotations. To support such motivating and experience building activities, documentation will be essential.
Agile-AI and contracting
Fourth and finally, agile sets with the structure it has the right guidelines to steer projects towards success – this is done through strong integration of the end customers (often the business) and additional stakeholders from the customer’s organization such as IT, Change & Communication, and senior leadership executives. Hence, a good contract has to set the framework of delivery right from the beginning. It is a substantial success factor that the team can perform efficiently. As such, the contract must agree on the ways of working, the delivery model & methodology, the escalation paths, the success criteria, and the way the teams measure themselves. The contract has also to define the client-owned roles and expected dedication (i.e. percentage of time spent) to the project. At least the roles of the product owner, the product management, the business owners. the epic owners, the shared services teams, and the system teams have to be defined as key counterparts in the client’s organization. There is a lot of collaboration, discussion and alignment required in agile delivery to get things right. These counterparts must be available. E.g., one of the most complex points in AI and Data Science projects is gaining access to the data. This is a task that generally must be supported and led by the internal IT department. Good contracts outline such responsibilities and, to some extent, also Service / Support Level Agreements (SLAs) with such stakeholder groups. Good contracts give the right framework to deliver in an efficient and targeted way. Hence, it’s vital to agree on the ways of working. Based on my learning, the most crucial part in contract negotiations is to convince and motivate the procurement department, which naturally prefers to have fix price, fix outcome projects (best to control), to support an agile delivery contract. Discussions like clearly defined deliverables and implemented functionalities are harder to embed into agile contracts. All stakeholders must deal with ambiguity, which normally requires a good understanding of how agile delivery works, a strong relationship based on trust (that has normally to be earned), and support from the business to convince procurement for the agile contract form.
By now, we have seen that the Agile Manifesto is more a framework of principles that allows for flexibility & judgment, and it does not rule-based nor black and white by any matter. Agile is built on communication, exchange, and alignment. To make this efficient, the schedule and the roles are very clearly defined upfront – that reduces at least on the governance side the ambiguity – it is clear who is responsible for what.
Maximizing the delivered value with Agile-AI – a heuristic formula
Now taking a step back – agile tries to maximize the delivered value by focusing on the right things while allowing for quick change adoption in an environment that supports creative working. It is based on motivated & self-directed teams that “buy-in” on the sprint backlog. This is supported & guided by a structural framework that defines the roles, responsibilities, escalations, and ways of working (processes). While this is all good, there needs to be strong discipline and experience from each stakeholder in an agile project. The expertise to deliver in a disciplined and structured way is one of the key success factors. It demands a mature organization with experienced scrum masters (at least) to be successful. Hence, I came up with the following heuristic & illustrative formula, which positions discipline & structure as important factors for the maximization of delivered value in agile projects.
This formula, of course, indicative and illustrative, is based on the logic of two concepts, i.e. f(x) and g(y), which have a strong relationship to each other. This follows the mathematical formula outlined hereunder.
What can we learn from the formula?
What does it mean? As outlined in the graphs below, the combination of capability, flexibility and creativity will increase the value of agile delivery in a linear way. This means – very simplified – the more, the better. It is more complex for the combination of discipline and structure. Here it is crucial to find a good balance. If, e.g. you have not enough discipline and structure, you will miss the opportunity to maximize the delivered value. On the other hand, if you have too much of it, you will again destroy value. There needs to be a good level of discipline and structure to unlock the value of the available capability, flexibility, and creativity.
Fun fact – the maximum on these graphs is always a multiple of e, Euler’s number. Maybe you could say that Euler already figured out back in 1731 the golden formula for value in agile delivery. This has yet to be proven. Let’s see.
Key takeaways and how to start right
Consequently, we strongly suggest ensuring your agile squad has a foundational discussion early in the project (best before its start) regarding how to implement the Agile Manifesto and its principles best. Teams must find the optimal balance of discipline and structure to maximize the value of the delivery.
Net-net – the Agile Manifest intends to set up good practices – and practices must not be mistaken with black-and-white rules. It’s not binary – it depends on the situation. And it also requires expertise, know-how, continuous re-validation and steering and above all, applied common sense. Hence, agile delivery is powerful in mature and well-trained organizations and can have many pitfalls for projects that only claim to follow “agile delivery” without proper setup. We generally suggest starting with smaller & simpler projects to create an environment to learn and grow. Making sure that the team gets the right support from expert scrum masters and have access to training. Agile must be lived – hence, people delivering following this method must truly buy-in. Agile eventually is also change of an enterprise’s culture, which normally requires time and continuous efforts and role modelling from the very top.
About the Author:
Michi Wegmüller is co-founder of Artifact SA and responsible for Artifact’s Analytics Garage offering. He has more than 15 years of experience in Data and Analytics consulting and has supported a diverse set of Swiss and international clients across industries. He has helped realize analytics initiatives that are sustainably growing and continuously delivering value to the business and functional units. He is passionate about agile analytics at scale.