By Francis Miers, Director of Automation Consultants

By Francis Miers, Director of Automation Consultants

Lean methodologies and Agile methodologies are often compared, but how much do they really have in common? Lean methodologies originated at Toyota in the late 1940s and early 1950s. The emphasis was on eliminating waste and continuous improvement in the manufacture of cars. Toyota’s methods contrasted with the received wisdom of the day, which held that economies of scale were the key to manufacturing efficiency. Large inventories of parts would be built up because it was cheaper to make them in bulk, and the production line would be kept going at all costs, because every hour of lost production was expensive. Any defects which crept into the cars in production would be rectified in post production processes outside the main production area. Toyota’s way was very different. If any defects were spotted in the components entering the production line or the methods used in the line, production would be stopped immediately until the problems were resolved. Inventory was kept to an absolute minimum. One of the ways this was done was through the use of Kanban, meaning board or sign in Japanese. When a certain process was running low on components, it would send an empty trolley with a Kanban (a board) to the part of the factory which made the components (or via goods inward to an external supplier), and the trolley would then be filled with the component and sent back. Components would only be supplied on receipt of the Kanban, which ensured that they were only produced to fulfil real demand, greatly reducing waste. Toyota placed waste reduction and continuous improvement at the centre of everything it did, and strove to train all its employees in these principles and empower them to apply them. These methods led to Toyota’s great success and have been widely admired and adapted, as “Lean” management methods, to processes beyond car manufacturing, such as project management and software development.

How did Agile get started? Agile software development was born of the failure of many software projects which had used traditional waterfall methods. In traditional waterfall methods the steps of a project are done in strict sequence, with each step carefully documented and approved. No phase is started without documented completion and approval of the previous one. The main weaknesses of waterfall methods are that they are poorly adapted to changes in customer requirements, and any defects in the software tended to be discovered only towards the end of the project. Changes in customer requirements cause problems because the customer requirements are extensively documented, and a whole hierarchy of other documents is based on the requirements documents, such as functional and non-functional specifications, specifications, design documents, development plans, budgets and test plans. A change to a customer requirement may flow through to dozens of other documents which must in theory all be revised and re-approved. In large projects, these documents may run to thousands of pages, and have dozens of approvers. Obtaining sign-off can take months. If changes to the requirements occur (which they do in most projects) the project either takes on major delay and cost overrun, or the strict adherence to the documentation and sign-off requirements is abandoned in order to make progress. To make matters worse, all too often customers only notice that changes are needed to the requirements at the end of the development phase when they get their first glimpse of the software in action. The redesign, rework and loss of time are thus much greater than they would have been if the changes to the requirements had been made in the early stages of the development phase. Many important projects failed or had huge delays and cost overruns as a result of these issues. Examples in the UK include the Taurus trading system of the London Stock Exchange, which was cancelled after ten years and £75 million spent; and NERC, the National Air Traffic Services’ air traffic control system, which did go live, but only after six years’ delay and £500 million of overspend on an original budget of about £200 million. In both cases, shifting customer requirements were a major factor in the delays. The experience of Taurus and NERC were widely repeated on a smaller scale across the IT industry. This was the context which led to Agile, and the term Agile was coined to contrast with the supertanker-like unsteerability of a large waterfall project.

Agile methods originated as a number of different software project management disciplines which had in common an emphasis on developing working code early on in the project, obtaining early and frequent feedback from the customer and less emphasis on formal documentation as a prerequisite for progress. These methods included Scrum, Extreme Programming (XP), Feature-driven development (FDD) and the Unified Process (best known as the Rational Unified Process). In 2001, representatives of these methodologies came together and published the Agile Manifesto, which sets out twelve principles which capture the essence of Agile. At the time of the publication of the Agile manifesto, Agile was regarded with suspicion by many organisations, especially large ones with well established rules. There was suspicion that Agile did not produce enough documentation for complex IT systems, and there was a worry that Agile did not lend itself to planning to meet deadlines. Since then, however, Agile has shown itself to deliver better results than waterfall for most types of software development project, and it has gradually become the preferred method of software development across the industry.

Agile and Lean do not have the same origins. One came out of a need to improve on software development method; the other from vehicle manufacturing. They do, however, share a remarkable amount in common. Common traits include the following.

  • Emphasis on short work cycles: in Agile, the customer is consulted frequently – in Scrum after every sprint. This reduces the risk of code being developed which the customer does not want. In Lean, small batches of components reduce the risk of components being produced and not used.
  • Emphasis on quality: in Agile, testing is done often and early, with every sprint or even as code is written. Defects are fixed early, at the latest in the next sprint. In Lean, the production line is stopped if defects are detected. The idea is to get it right first time and avoid the waste that comes with rework.
  • Emphasis on training and empowerment. In Agile, teams are encouraged to “self-organise” and find their own solutions to problems they encounter. In Lean, there is an emphasis on training and on decisions being taken at the lowest practical level. The production line can be stopped on quality grounds by very junior people.
  • The idea of the Kanban board has made it directly into Agile. One of the leading Agile methods is Kanban.

Once an organisation decides to adopt Agile, what is the best way to do it? Unfortunately, Agile cannot simply be mandated by top management and adopted overnight. A major cultural change is necessary at all levels of the organisation concerned with software development. The twelve principles of the Agile Manifesto give an idea of the mindset required: frequent customer consultation, short delivery cycles of working software, placing trust in the members of the development team. Another important element of the Agile Manifesto is “we value individuals and interactions over processes and tools”. An important starting point, therefore, is to bring in expertise, whether by using consultants, training existing staff in Agile methods or hiring new staff with Agile expertise. From here, pilot projects can be set up in preparation for a wider roll-out. In any case, it will take time for the culture of Agile to bed down the organisation, and the primary focus should be on the people, and imbuing them with the Agile ethos. If this is done successfully, the people will decide on which flavour of Agile and which tools are best for them.

The team is likely to choose a tool from one of the leading ones summarised in the table below. There are, however, hundreds of Agile project management tools available and those listed below is just a small selection.

ToolProsCons
JIRA SoftwareIntegrates with rest of Atlassian suite.Some features depend on add-ons which are not available in the cloud.
VersionOneGood for larger organisations. Has integral portfolio management.More complex product, harder to learn.
LeanKitGood for Kanban.Does not support Scrum.
CA (formerly Rally) Agile Project ManagementHas integral portfolio management.Complex user interface, lack configurable reporting.
Microsoft Visual Studio Team ServiceIntegrates with rest of Microsoft technologiesWeak for non-Microsoft projects.
Pivotal TrackerSlick visual interfaceWeak at scale, less well adapted to large organisations.

 

Even before choosing a tool, the Agile leaders in the organisation will most likely choose an Agile methodology. The leading methodologies are listed below.

MethodRemarksProsCons
ScrumBy far the most popular type of AgileGood for software projectsDoes not normally put specific deadlines on tasks
Extreme Programming (XP)Often combined with Scrum. Closely related to test-driven development (TDD).Engineering-style disciplines are applied to different parts of software development, which can yield improvements in quality.Requirements are expressed as tests. Limited scalability.
KanbanInspired by Lean management and just-in-time manufacturing as at Toyota.Allows specific deadlines. Well suited to IT projects outside software development.Limits capacity to perform retrospectives.
Iterative DevelopmentA forerunner of Agile.Evolved from waterfall, easy to combine with aspects of waterfall.There is no set way to do it. Not ideal for inexperienced teams.
Lean DevelopmentA direct application of Lean principles to software development.Easy to adopt in workplaces which are already using Lean management techniques.Relatively small community in the world of software.

 

Scrum is the most popular method. In a recent survey, 58% of the teams surveyed used it, compared to 10% of the teams using the next most popular methodology, a hybrid of Scrum and XP. Scrum is designed for software development projects. Activity is organised into “sprints” of 2-4 weeks, during which the aim is to produce working software. The customer is consulted at the end of each sprint and shown the progress of the sprint. Scrum specifies a number of other procedures, such as a short daily meeting of the team, a “stand-up”, and a retrospective at the end of each sprint.

XP is similar to Scrum, but has shorter iterations and is more prescriptive about how the software is developed. For example, it advocates performing a lot of testing, and this has evolved into what most now be seen as a separate methodology, Test Driven Development (TDD). In TDD, tests are written for each piece of code before the code itself is written.

Kanban takes a slightly different approach to Scrum. Unlike Scrum it does not mandate sprints, retrospectives or daily stand-ups, although they can be added to a Kanban team. Kanban does, however, limit the amount of work in progress by inhibiting the starting of new tasks before those in progress are finished, i.e. it limits work in progress. This is a direct use of the methods pioneered by Toyota in vehicle manufacturing. Kanban lends itself well to work which is not easily divided into sprints, such as IT helpdesk work or major change projects with hard deadlines.

Iterative development is one of the contributory methodologies to Agile and influenced the Agile Manifesto. It stemmed from the fact that many software engineers realised that several cycles of development would be needed to achieve a project’s goals. They adapted the traditional waterfall techniques to include several pre-planned iterations of the development phase. Several NASA projects used iterative development well before the Agile Manifesto including the Mercury programme of the 1960s and the Space Shuttle programme of the 1970s and 1980s. Unlike Scrum, there is no set way to do iterative development. Any project management method in which there are several cycles of development and testing can be termed iterative. Iterative development is perhaps best suited to teams which have evolved an effective project management methodology over time. That evolution can be steered in a more iterative (and thus Agile) direction without losing the successful aspects of the existing methodology. For new teams seeking to adopt Agile, it is perhaps less well suited, because there is no set way to do it, and there is much scope for different team members to adopt different interpretations on the details of the new practices to be adopted.

Lean software development is a direct application of Lean management practices to software development. It originated in a book, Lean Software Development by Mary and Tom Poppendieck, published in 2003. As mentioned above, many of the concepts of Lean, when applied to software development, correspond closely with those of Agile. Lean software development is perhaps best suited to organisations which already use Lean methods in other types of management. It will be easier for staff who already know Lean, simply to apply the principles to software development, rather than having to learn a new vocabulary and set of habits that would come with, say Scrum or XP. For organisations in which Lean methods are not already well-known, Lean is perhaps not the best method, because the user community in software is smaller, and there are greater resources available for individuals and teams to learn Scrum: more books, more outside experts, more tools which support it etc.

In summary, Lean and Agile are an example of convergent evolution. They have different origins, but many goals in common, and when their key concepts are analysed, they turn out to be remarkably similar. An organisation which has adopted Agile successfully will most likely fulfil many of the criteria needed to be seen as Lean. In the adoption of Agile, the most important place to start is with the people. Once they have grasped the key concepts and are motivated, they will make it happen. As regards methods, Scrum is by far the most popular for software development, and is the best method for teams adopting Agile for the first time, because it is a good method and it has a wealth of resources to help the team learn. Kanban is great for work other than software development such as helpdesks and transformation programmes. Other methods such as XP, Iterative and Lean are best where there is pre-existing knowledge or pre-existing success which could be utilised and developed.