The Suitability of Agile Methods in Large Systems

1.0 Introduction
Designing and developing information systems involves carefully managing people, processes, tools and technology. It necessitates having a clear understanding of the role each entity plays, and overseeing the relationships between them. The intricacies of designing and developing systems vary, depending on the size of the systems in question. Over the years, software and technology experts have adopted various methodologies that assist system architects and solution providers in the design and implementation process. These methodologies are continually evolving, seeking better ways of addressing common problems faced during the development life cycle of a system. One of these emerging methodologies is the use and adoption of agile methods in the design and development of systems.

Agile methods propose a different approach to developing software systems, by paying less attention to conventional methods of excessive planning, heavy documentation, and legal contracts, and focusing more on the light-weight methods such as quick response to changes, teamwork and effective communication, amongst others. Many arguments have been put forward to suggest that agile methods are applicable only in the development of small to medium sized systems, questioning their suitability in large systems. This research aims to determine the suitability of agile methods in large systems. The strategic objectives of this research are:

• To investigate and analyse the characteristics of large systems
• To research and understand the agile approach to software development and examine the use of agile methods in software development processes
• To examine the suitability of agile methods for large systems

2.0 Characteristics of large systems and inherent challenges of developing large software systems
Large software systems are naturally complex. Heun (2007) agrees with this, as he takes the position that the fundamental challenges faced in designing large systems is as a result of the complexities inherent in them. Large software systems consist of varying components, which are often intertwined and closely coupled. Jennings (2001, p. 35) further buttresses that “developing such software in domains like telecommunications, industrial control and business process management represents of the most complex construction tasks human undertake”.
Understanding why developing large systems is difficult will give us insight as to how best prepare to palliate the challenges. Wen and Dromey (2009, p. 292) argue that “there are three main reasons” for the difficulties of developing large systems. These are:

• Complexity of large systems
• The continuous changes to the system requirements
• The shortcomings of system requirements

The above concerns are argued to be peculiar to large systems, however there are generic challenges that are shared by all software systems. These challenges are clearly pointed out by Gennings (2001, p. 30); he argues that the software industry as a whole faces the challenges of “poor quality, delivery delays, long delivery cycles and burned-out engineers”. Although these challenges are generic, it is worthy to note them while the consideration for an appropriate methodology for large systems is being undertaken.

3.0 Overview of Agile Methods
There is no definitive agreement on what Agile is (Abrahamsson et al., 2002); hence the term has different meanings, wholly dependent on the context in use. Larman (2004, p.25) also argues in this regard, stating, “It is not possible to exactly define agile methods, as specific practices vary”. However, some authors make attempts to define agile methods. Williams (2009) argues that agile methods are evolutionary methods and are based on iterative development processes. This viewpoint stems from a software development perspective and suggests that the development of software systems should be broken down into mini-projects and handled independently. Aoyama (1998, p.4) also supports this proposition by arguing that agile methods have “light-weight processes” and are “leaner than conventional process models”. Utilising agile methods in design and development, invariably involve breaking down a project into smaller bits and independently taking them through the processes of analysis, requirements definition, implementation and testing (Larman, 2004).

Generally, agile methods are seen as a breakaway from the conventional approaches to software development. As Salo and Abrahamsson (2008) point out, agile methods change the paradigm adopted by organisations, which is the plan-driven approach to software development. Szalvary (2004) agrees to this, and remonstrates that in adopting agile methods, a larger priority has to be placed on productivity and values, than on traditional, colossal processes and bulky artefacts development. Larman (2004, p.26) also argues that agile methods are unique and different from traditional systems development methodologies, stressing that agile methods embrace principles of “simplicity, lightness, communication, self-directed teams”.

These arguments provide insights on the concept of agile methods, however, to get a more comprehensive and holistic understanding of the agile methodology, it is necessary to take a look at the value and principles at its core.

3.1 Agile Methods and the Manifesto for Agile Software
The agile manifesto seeks to define a different ideology as to how software systems should be developed. It proposes that people think differently of the software development life cycle, and its associated processes. In 2001, a group of “people who referred to themselves as software anarchists” (Glass, 2001, p. 12) shared their expectations on software design and development, and put together their thinking on how software development should be handled. Their interest was in the use of iterative and agile methods in the development of systems (Larman, 2004). After their sessions, they came up with what is called the “Manifesto for Agile Software Development”. The Manifesto for Agile Software Development explicitly states that “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan”

Highsmith and Cockburn (2001) argue that agile methods are not new, but rather the focus on people as the drivers for success. The agile manifesto thus points out that fundamentally, people and the communication between them are the most important aspect of the development life cycle of systems. Cockburn (2001) argues this further, by stressing that managing communication is key to mastering agile software development. Abrahamsson et al. (2002, p.100) summarises the agile methodology by pointing out clearly that “agile methods claim to place more emphasis on people, interaction, working software, customer collaboration, and change, rather than on processes tools, contracts and plans”. This presentation is simply a reflection of the agile manifesto.

4.0 Analysis on the suitability of agile methods in large systems
In order to analyse the suitability of agile methods in large systems, we will look at the applicability of agile methods in different design and development dimensions, and draw relevant examples where necessary. We will consider agile methods under the following categories:

• Adaptability to changes and flexibility: The traditional approach to developing large systems adopts a completely sequential, phase-driven methodology where the requirements of the system to be built are defined at the onset of the project development, followed closely by the design of the system, then implementation, and finally testing (Szalvary, 2004). The immediate challenge with this approach is that during the development life cycle of a project, the requirements of the system always change, thus leaving teams developing systems that do not address the needs of the users (Tudor and Walter, 2006). The inflexibility and inability of traditional development methodologies to quickly adapt to the changes being proposed by users leads to the development of systems that might not necessarily meet the requirements of users. On the other hand, agile methods are inherently flexible and were conceived with adaptability in mind. Aoyama (1998, p.4) stresses that agile software processes “emphasise rapid and flexible adaption to changes of the process, product and environment”. An example of an adaptation of agile methods for the purposes of flexibility and urgency to address changing needs is seen by the Highways Agency, United Kingdom (Grice, 2010). The Highways Agency, which is charged with the responsibility of operating, maintaining and improving the road network in England, had to adopt the use of agile methods in its project implementations in order to successfully respond to changes in a timely manner.
• Guarantee of end product quality: The Manifesto for Agile Software Development has one of its core principles as: “Working Software is the Primary measure of Progress”. The manifesto makes it explicitly clear that agile methods are deeply rooted in the quality of the output of the system development life cycle. Glass (2001) argues that the traditional approach to software development at times focuses on the wrong deliverables, for instance, extensive documentation over working software. Agile methods focus on working software, which thus amounts to quality being a priority in the execution of projects. Lindwall et al. (2004) examined the use of agile methods in DaimlerChrysler and found that using agile methods was responsible for the production of high-quality software across various projects. Highsmith and Cockburn (2001, p.120) also argue in this regard, and are clear that “agile software development stresses quality in design”. From these arguments, one can summarise that adopting agile methodologies in the development of large systems can lead to the development of quality software which meet user-defined requirements.
• Reduced time-to-market: Another fundamental area of concern is the time needed for products/services to reach the market. In designing large systems, designers are usually faced with this challenge. Therefore, adopting development methodologies that mitigate this will go a long way in determining the success or otherwise of a project. Agile methods bring about a level of productivity that reduces the time-to-market. Lindwall et al. (2004) evaluated large firms adopting agile methods, and attest that while agile methods were in use, there was an increase in the pace of development efforts and change requests. There are real-life examples of large companies that have used agile methods to reduce their time-to-market. GAP Incorporated, one of the world’s largest retailers, was facing the problem of time-to-market while adopting traditional methods of systems development. The company was continually missing delivery dates on its systems development efforts and decided to opt for agile methods (Goodman and Elbaz, 2008). Gap Incorporated took a bold step by developing one of its largest projects – Piperline.com, by using agile methods. Another large company that embraced agile methods with success is Ericsson (Telecommunications Technology Provider). Goos and Melisse (2008) attest that Ericsson was able to reduce its time-to-market by over 50% by adopting agile methods in its processes.
• Increased Productivity and Cost Management: One of the key drivers for organisations remains the need for them to increase productivity in their processes (Lindwall et al., 2004) Productivity and cost savings have a direct relationship in that increasing the productivity of the processes in the development life cycle invariably leads to a reduction in the costs of systems development. In traditional systems development methodologies, challenges with requirements definition, project planning and implementation, are some of the core reasons why the development of large systems fails (Hatton, 2009). These failures come about as a result of lack of effectiveness and efficiency in the organisation’s systems development processes. One of the largest electronic retailers in the United States experienced this, and had to adopt agile methods in order to preverse its fortunes. Illmensee and Muff (2009, p.405) point out that the electronics retailer was adopting traditional methods in developing its systems but was having challenges of “the enormity of the requirements sessions, overwhelming documents and painstaking reviews, change requests and elaborate development processes”. This invariably consumed the company’s resources and time. In addition to this, DaimlerChrysler’s use of agile methods was able to reduce its development costs significantly, while achieving higher standards, increasing flexibility and satisfying the requirements of its customers (Lindwall et al., 2004).
• Ability to Scale: One of the key arguments against the use of agile methods is the inability of agile methods to be able to handle the actual design and development process of large systems. Rodriguez et al. (2009) argue that even if there are obvious benefits that have been seen in the use of agile methods in small systems, it remains a challenge to scale large systems using these methods. A counter argument is sighted with an example from Fecarotta (2008), which explores the adoption of agile methods across all Boeing IT processes. The adoption of agile methods at Boeing started with a single project called “MyBoeingFleet”. MyBoeingFleet.com sought to provide an extranet that would be used by Boeing to provide resources to its customer base. Like other organisations, the management of Boeing was skeptical that agile methods could be used for their processes. However, MyBoeingFleet was hugely successful and as a result Boeing further adopted agile methods in use in other areas of their business. Similarly, Aoyama (1998) attests that for many years Fujitsu has been successfully using agile methods in the development of large-scale telecommunications software. Also, Lindwall et al. (2004) research findings in the use of agile methods at Motorola convinces them that agile methods can be used to develop large, complex systems.

5. Conclusion
There are numerous propositions on the best approach to develop large systems. Some experts are accustomed to the traditional approaches to software development, while a few others are willing to explore the realms of agile methodologies. Based on my research findings, my position is that agile methods can be utilised effectively in the design and development of large systems. My view is that organisations that have large systems are accustomed to the traditional ways of doing things and hence are skeptical about delving into the world of agile methods. Agile methodologies if adopted rightly can increase productivity, reduce the costs of development and reduce the time-to-market for large systems, as has been seen with the numerous cases of large organisations.

References
Abrahamsson, P. Salo, O. and Ronakainen, J. (2002) Agile Software development methods – Review and analysis, Finland: VTT Publications.

Abrahamsson, P., Warsta, J., Siponen, M and Ronkainen, J. (2003) ‘New Directions on Agile Methods: A Comparative Analysis’, International Conference on Software Engineering, Washington DC, pp. 244-254.

Aoyama, M. (1998) ‘Agile Software process and its Experience’, International Conference on Software Engineering, April, pp. 3-12.

Cockburn, A. (2001) Agile Software Development, Addison-Wesley Professional

Cohen, D., Lindvall, M. and Costa, P. (2004) ‘An Introduction to Agile Methods’s, Advances in Computers, vol. 62, January, pp.1-66.

Fecarotta, J. (2008) ‘My Boeing Fleet and Agile Software Development’, Agile Conference, Toronto, pp. 135-139.

Glass, R. (2001) ‘Agile Versus Traditional: Make Love, Not War!’, Cutter IT Journal, vol. 14, no. 12, December, pp. 12-18.

Goodman, D. and Elbaz, M. (2008) “It’s not the pants, it’s the people in the pants – Learnings from the Gap Agile Transformation – What Worked, Howe We Did it, and What Still Puzzles Us”, Agile Conference, Toronoto, pp. 112-115.

Goos, J. and Mellisse, A. (2008) ‘An Ericsson Example of Enterprise Class Agility’, Agile Conference, Toronto, pp. 154-159.

Grenning, J. (2001) ‘Launching extreme programming at the process-intensive company’, IEEE Computer Society, vol. 18, no. 16, November/December, pp. 27-33.

Grice, C. (2010) An Agile approach to Software System Development for the Highways Agency, [Online] Available at: http://www.dsdm.org/wp-content/uploads/2011/02/An-Agile-Approach-to-Software-Systems-Development-for-the-Highways-Agency-Sept-10.pdf [17th March, 2011]

Hatton, L. (2009) How to build successful complex systems: lessons from open source [Online], Available at: http://www.leshatton.org/wp-content/uploads/2012/01/how-to-build-successful-complex-software-systems.pdf [21st March, 2011]

Heun, W. (2007) ‘Systems Engineering of Complex Software Systems’, ASEE/IEEE Fronters in Education Conference, Milwaukee, pp. F1A-16.

Highsmith, J. and Cockburn, A. (2001) ‘Agile Software Development: The Business of Innovation’, IEEE Computer Society, vol. 34, no. 9, September, pp. 120-122.

Illmensee, T. and Muff, A. (2009) ‘5 users Every Friday: A Case Study in Applied Research’, Agile Conference, Chicago, pp. 404-409.

Jennings, N. (2001) ‘An Agent-based approach for Building Complex Software Systems’, Communications of the ACM, vol. 44, no. 4, April, pp. 35-41.

Larman, C. (2004) Agile and Iterative Development A Manager’s Guide, Addison-Wesley Professional.

Lindvall, M., Muthig, D., Dagnino, A., Wallin, C., Stupperich, M., Kiefer, D., May, J. and Kahkonen, T. (2004) ‘Agile Software Development in Large Organisations’, IEEE Computer Society, vol. 37, no. 12, December, pp. 26-34.

Prell, E. and Shen, A. (1984) ‘Building Quality and Productivity into a Large Software System’, IEEE Computer Society, vol. 1, no. 3, July, pp. 47-64.

Rodriguez, P., Yague, A., Alarcon, P. and Garbajosa, J. (2009) ‘Some Findings Concerning Requirements in Agile Methodologies’, Product-Focused Software Process Improvement, vol. 32, no. 4, June, pp. 171-184.

Tudor, D. and Walter, G. (2006) ‘Using an Agile Approach in a Large, Traditional Organization’, Agile Conference, Minneapolis, pp. 367-373.

Salo, O. and Abrahamsson, P. (2008) ‘Agile methods in European embedded software development organizations: a survey of the actual use and usefulness of Extreme Programming and Scrum’, IET Softw., vol. 2, no. 1, February, pp.58-64.

Szalvay, V. (2004) An Introduction to Agile Software Development, [Online], Available at: http://www.danube.com/docs/Intro_to_Agile.pdf [24th March, 2011]

Wen, L. and Dromey, L. (2009) ‘A Hierarchical Architecture for Modeling Complex Software Intensive Systems Using Behaviour Trees’, Software Quality Institute, Griffith University, pp. 292-299.

Williams, L. (2009) A Survey of Agile Development Methodologies, [Online], Available at: http://agile.csc.ncsu.edu/SEMaterials/AgileMethods.pdf [26th March, 2011]