The Role of Iterative Usability Evaluation in Agile Development: A Case Study

The Need for Usability Evaluation of Healthcare Systems

Many new healthcare applications—particularly personal health–management systems—require continual, iterative usability evaluations while they are under development. Our research identified and provided an example of the explicit processes of iterative usability evaluation and its implementation in an agile development environment. The findings from a previous study by Ammenwerth and others [10] stated that, considering the different stakeholders who are involved in the development of a healthcare system, it is necessary to plan for iterative usability evaluation over the long term, allowing a development team working in an agile environment to integrate the new findings and user feedback into their implementation.

The principal strategy for usability evaluation should be to take into consideration the complexity of healthcare systems, while the consequence of usability evaluation should be decreased complexity for users. [17] Our study provided evidence that the rapid development and iterative prototyping that characterize an agile approach [12] should rely on usability evaluation to provide designers and developers with the necessary inputs to enable them to improve the usability of complex healthcare software. Because agile development emphasizes collaboration and communication among developers, usability experts, and users, it can lead to a blending of these roles. [15] However, usability experts can serve as a bridge connecting users and system developers—enabling them to understand the needs of the users of a personal health–management system.

Our Recommendations to Usability Professionals

The integration of usability evaluation into an agile development approach for a healthcare system requires a high degree of adaptability to change, savvy collaboration between the system developers and the usability team, and timely planning for the involvement of users. Therefore, usability experts need to work with system developers on sprint planning and tracking. Long-term agile development requires continuous cycles of design, usability testing, and making design improvements throughout all development iterations.

Based on our study, here are a few tips for usability professionals conducting usability evaluations in an agile development environment:

• To ensure the satisfaction of multiple user personas working with health-information technology, conduct usability evaluations on a regular basis throughout the agile product development lifecycle.
• Identify the design changes that have been made to an application since the last development cycle, and ensure that the system developers understand the usability-evaluation goals by maintaining effective communication with them—especially when the design changes are subtle.
• Prior to doing usability testing, conduct a pilot study, then develop an appropriate interview protocol.
• In comparison with non-agile development processes, agile approaches require intensive collaboration between system developers and usability professionals and short cycle times, which makes the recruitment of usability test participants a highly demanding process.

The Methodology for Our Study

This study explored a specific case: the agile development of a personal health–management system. The development team worked in collaboration with a usability testing laboratory at a large Midwestern university in the United States. The usability testing laboratory conducted 14 cycles of usability evaluations over a period of 12 months, in 2011 and 2012. A total of 59 users participated in the usability evaluation sessions, during which the usability team asked each participant to perform a list of predefined tasks in a controlled lab setting. The usability team asked participants to verbalize their thoughts and actions, or think aloud, while performing the tasks. Each usability test session lasted approximately 45–60 minutes. The usability team recorded each session, using Morae 3.2.1.

Data Collection and Analysis

To answer our first research question—What constitutes an iterative usability-evaluation process in agile development?—we analyzed all 14 cycles of usability evaluation. The data that we analyzed included meeting minutes, documents on the development team’s expectations, explanations of the applications being tested, task lists, the usability team’s notes, and their final reports. Our analysis focused primarily on the activities that were part of each cycle, the people who were involved in product development and usability evaluation, and their responsibilities. We also analyzed data about each cycle of usability testing, including the duration of each cycle, the number of participants, the richness of the feedback that the participants provided, and the development team’s satisfaction with the results of the usability evaluations.

To answer our second research question—How do usability experts perceive their roles in an agile development process?—we examined the role perceptions of the usability professionals who participated in the usability evaluations. According to Biddle, [18] most versions of role theory assume that people’s expectations are the primary generators of their perceptions about roles that they learn through experience, and people are aware of the expectations that they hold. Therefore, we conducted semi-structured interviews with five of the usability professionals who were actively involved in multiple cycles of usability evaluation of the personal health–management system. We open coded the data that we collected from the interviews and identified common themes.

Future Research

Our future research might examine the role perceptions of other stakeholders who were involved in the usability-evaluation process—for example, the system developers—to allow us to build a better, more comprehensive understanding of iterative usability evaluation in the agile development of a healthcare system. 

Acknowledgments

We would like to express our deepest gratitude to the people who helped and supported us throughout our study, especially to our colleagues Josipa Basic and Ben Richardson for their insightful comments.

References

[1] Singh, Mona. “U-SCRUM: An Agile Methodology for Promoting Usability.” AGILE ’08 Conference. New York: IEEE, 2008.

[2] De Vreede, Gert-Jan, Ann Fruhling, and Anita Chakrapani. “A Repeatable Collaboration Process for Usability Testing.” HICSS ’05. Proceedings of the 38th Annual Hawaii International Conference on System Sciences. New York: IEEE, 2005.

[3] Sy, Desirée. “Adapting Usability Investigations for Agile User-Centered Design.” Journal of Usability Studies, May 2007.

[4] Ferreira, Jennifer, James Noble, and Robert Biddle. “Agile Development Iterations and UI Design.” Agile Conference (AGILE), 2007. New York: IEEE, 2007.

[5] Highsmith, Jim, and Alistair Cockburn. Agile Software Development: The Business of Innovation. New York: IEEE Computer Society, 2001.

[6] Paetsch, Frauke, Armin Eberlein, and Frank Maurer. “Requirements Engineering and Agile Software Development.” IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. New York: IEEE Computer Society, 2003.

[7] Nawaz, Ahsan, and Kashif Masood Malik. Software Testing Process in Agile Development. Computer Science Master Thesis, Blekinge Institute of Technology, 2008.

[8] Göransson, Bengt, Jan Gulliksen, and Inger Boivie. The Usability Design Process: Integrating User-Centered Systems Design in the Software Development Process. San Francisco: Wiley Online Library, 2003.

[9] Eklund, John, and Ciaran Levingston. “Usability in Agile development.” UX Research, 2008.

[10] Ammenwerth, Elske, Stefan Gräber, Gabriele Herrmann, Thomas Bürkle, and Jochem König. “Evaluation of Health Information Systems: Problems and Challenges.” International Journal of Medical Informatics, 2003.

[11] Kushniruk, Andre W., and Vimla L. Patel. “Cognitive Evaluation of Decision Making Processes and Assessment of Information Technology in Medicine.” International Journal of Medical Informatics, 1998.

[12] Kushniruk, Andre. “Evaluation in the Design of Health Information Systems: Application of Approaches Emerging from Usability Engineering.” Computers in Biology and Medicine, 2002.

[13] Cockburn, Alistair, and Jim Highsmith. “Agile Software Development: The People Factor.” Computer, November, 2001.

[14] Wusteman, Judith. OJAX: A Case Study in Agile Web 2.0 Open Source Development. Bingley, UK: Emerald Group Publishing Limited, 2009.

[15] Kollmann, Johanna, Helen Sharp, and Ann Blandford. “The Importance of Identity and Vision to User Experience Designers on Agile Projects.” AGILE ’09. New York: IEEE, 2009.

[16] Callero, Peter L. “Role Identity Salience.” Social Psychology Quarterly, September, 1985.

[17] Rouse, William B. Health Care as a Complex Adaptive System: Implications for Design and Management. Washington, DC: Bridge-Washington-National Academy of Engineering, 2008.

[18] Biddle, Bruce J. “Recent Development in Role Theory.” Annual Review of Sociology, 1986.