An investigation into restart errors and the effectiveness of cues presented under high memory working load

ABSTRACT
Human error is an unintentional activity caused by decision-making failure. In addition, the criticality of human error is not directly measurable but rather inferred from performance. This paper describes the experimental methodology used to investigate restart errors in procedural steps. The aim is to study the design issues with the occurrence of procedural errors and their effect on performance. Furthermore, this paper will investigate the causal factors of restart errors and the effectiveness of cues in the prevention of errors under high working memory load.

The methodology considers restart error in a set of systematic procedural tasks in a game-like environment (microworld). Participants are tested in groups for experimental methodologies focusing on variance in the design of observable cues and restart slips under high working memory load. Participants are interrupted at a certain point during the procedural task. The interruption lasts for 30 seconds to allow sufficient decay in memory. The point of interruption is selected to be when the participant is working under high memory load and is prone to error commission. After the interruption, the task is restarted and the errors committed in accordance with observable design are reported. Finally, the study produces the factors for a better interactive design which can help the user’s psychological decision making with the help of cues. Cues serve as a directly observable design paradigm to help users to observe the behaviour design.

1. INTRODUCTION
Errors play a role in upgrading performance. Often, an error leads us to backtrack one step or, in some cases, several steps. In a field requiring an efficient performance, errors play an important role in driving the procedure properly. e.g., in aviation, a minor error can lead to crashes and mishaps. In contrast, an error made while heating a meal using a programmed chip is inconsequential compared with an air mishap. Moreover, the errors in a set of procedural steps commonly have an influence on the performance of users; the result is low efficiency and a failure of the intended outcome.

The pervasiveness of computers which deliver high performance tasks in the everyday lives of ordinary people means that computer scientists need to be more responsible about system design and reliability. The computer scientists should understand human psychology; mental errors can be made by the users of a system, and turn them pale. Paul Curzon in ‘The Dog, Hen and Corn’ argues that the software developer should change from stereotypical development and take the extra step required to understand the psychological issues of design. Moreover, cognitive knowledge of a system is still is losing the edge over the slips made. [7] These are identified as the slip errors within the procedural set even when expert knowledge is involved behind the task. To support the slip-error commission, [2] demonstrated experimentally the post-completion error occurrence as a slip within the procedure. The work-blamed slip errors are the consequence of a high work memory load. It has been observed that it is often under conditions of a high work memory load that a person doesn’t remember the sequence and omits the final step.

The effect of the point at which the task was interrupted and the duration of the interruption was studied. [7] The study reported that the duration should be enough to incur a substantial decay in memory so that a participant would be prone to slip. The literature further investigated the when the interruption occurred while studying post-completion errors. The study reported that an interruption occurring before the task-completion step resulted in maximum errors within the procedure. Moreover, [7] studied the implications of cues for future actions. The implication of creating cues was reported. When the user can leave sensory notes for future action, a window of opportunity was likely to open as the commitment of errors would be less likely.

Procedural errors have gained much attention from researchers, and in depth insight has been gained through the research of post-completion errors, initialisation errors, interruptions and cues. Most of the research has overlooked the restart errors being committed. Restart error is the omission of a step while continuing the procedure after interruption of a certain length of time at a particular step. The interval turns the psychological state of mind to the other side and incurs a substantial decay of the memory corresponding to the procedural task. The decay disturbs the systematicity of the procedure and it is more likely that a user will commit an error at this stage. However, the design should allow users to create visual/sensory cues for future action and thus lessen error-rate and increase the safety and performance of the system and the user.

2. RELATED WORK
Interactive system design study is about the design of computer interaction systems developed by humans. Finding a design flaw in controlled conditions is a step towards improving the effectiveness and efficiency of a system. But commission of an error in an uncontrolled situation exposes design flaws to the users of a system. To increase the system’s robustness and reliability, it is the role of the designers to look for the needs of the users. Manufacturing a system to be design-proof needs deep insight into possible human errors. [4] identifies some of the design flaws where computers showed their dominance over the usability needs of a user. [10] Describes the same design flaws prone to human error, reporting these in terms of unsuitable behaviour that can affect system efficiency and safety. [4] identifies an insight in design procedures in terms of the user goals. In the literature it is argued that even if an application hubs thousands of features but does not satisfying the basic goal of a user, that application is void. Designing for features makes an application error-prone.

[12] Investigated the statistics of 34 incidents and reported that 92% of the deaths in those incidents were due to the human–computer interaction and that the other 3% were credited to software error in application. Such reports show the criticalities of design issues within a highly reliable system. Thus human error has been a concern of research over the last few decades [13], where models of cognition and experimentation of human error are being performed. [14 categorised error on the basis of the intention as: ‘If the intention is not appropriate, this is a mistake. If the action is not what was intended, this is a slip.’
[15] introduced a cognitive error classification framework as skill-based slips and lapses, rule-based mistakes and knowledge-based mistakes. Whilst studying information processing, [6] drafted an influential classification system of information processing in HCI (Human Computer Interaction). The author identified the system of information processing related to the degree of consciousness and hence derivables of an error. Based on his classification of degree of consciousness, processing systems were diversified into skill-, rule- and knowledge- based. Brief definitions of the above mentioned processing systems can be given as as:

Skill based processing category: the smooth execution of a highly complex task in response to an event.

Rule based processing category: a user performs a task with transitional conscious control and executes the rules learnt in training.

Knowledge based processing category: a user performs a task with high consciousness scaling as if he is new to activity.

[2] Introduced the term ‘Post-Completion Errors’ (PCE) in research methodologies of human error. The report argues that humans are capable of doing certain things in a right way and proper manner. However, the introduction of one extra step in a procedure, after the main goal has been achieved, makes it prone to errors. Researchers credit this kind of error to working memory load in this particular instance. The findings of their research reported the participants doing the same task in a procedural way without committing errors when working memory load is low. This created a relationship between PCE and working memory load. However to lessen the omission of the last step (slip) due to high working memory load, [3] studied the introduction of visual hints (cues). The study reported the elimincation of slip error when a specific visual cue was presented just in time. This illustrates the role visual cues play in controlling the procedural errors. Furthermore, to fully investigate the effect of interruptions, which are, more often than not, the logical cause for PCE, [11] analysed the effect of the point at which the interruption occurred and its duration on the rate of PCE. The results of the experiment indicated that most PCEs were committed when people were interrupted at the penultimate step of the task compared to when interrupted at any other step. [1] introduced an insight into the concept of systematicity with the hypothesis questioning ‘Does being motivated to avoid procedural errors influence their systematicity?’. The corresponding findings implied that users’ performance is prone to PCEs.

Nevertheless, it is evident from the literature that there has been a little lapse in concentrating on another kind of error namely: restart error. This dissertation is concerned with research methodology to study restart errors. The method requires the user to work in a game-like environment. While under high memory load, the user will be paused for an interval of some time and then asked to restart performing the same task. Control of the ‘in between’ group experimental research is noticeably in cue generation. Whilst users will be interacting with goal-oriented Microworld rather than feature hub applications, the environment will be studied to calculate the restart errors which occur within the Microworld variations of design.

Research observations will direct the error rate to be calculated. Project goals are set to reduce the errors by studying slips, cues and working memory load. Variations in design will provide comprehensive knowledge for the interactive system design issues and factors of human error.

3. PROBLEM
This paper is concerned with the research of restart procedural errors. The research is based on firm literature of procedural error research in the past. Thus the independent variables e.g. point at which the task is interrupted and the duration of the interruption are not part of the study. The methodology of previous research has provided the infrastructural basis for future research in these areas. Nonetheless the problem of incurring a high working memory load needed the project to study a game-like environment. Whilst working under high memory load, the user evaluation was not selected to be ‘think aloud’ because speaking while performing would affect the performance of users. Therefore, discarding the reason to increase the error rate, the application was designed to generate the reports of errors committed rather incurring extra memory load for the participants. Further, selecting the control between the two designs needed to be simple enough to notice the errors. To make errors noticeable, the control was chosen to be within the number panel as a cue. Moreover the Microworld state cannot be made goal motivated/driven for each participant because of the consideration of ethical issues.

4. METHODOLOGY
The project mainly considers the restart errors using the Sudoku gaming application as found on Microworld. The recruited participants will be trained to gain exposure towards this particular application. In the training session, no cues will be supplied so that users do not get used to cues. Moreover, participants will be asked to speed up from time to time. During the interruption, a similar secondary arithmetic task will be performed which will distract their attention from a high memory load game to another game. The participants will have to remember the sequence of numbers appearing on coloured of balls. Users will have to re-write numbered sequences of coloured balls. E.g. a when the blue ball appears, the user will have to select which number was etched on the ball; in this case, number one.

Figure 1: Sequence of coloured balls

Microworld Procedures:
Step 1: Click on 3*3 inner Matrix A of 9*9 matrixes.
Select a particular 3*3 matrix to fill up the numbers within it.

Step 2: Click on a box within 3*3 Matrix B
Choose one box in a 3*3 matrix.

On the same screen a number panel appears, providing the options of clicks that will fill up that particular box corresponding to the number clicked in the panel.

In matrix B, the user can cross out the options of those that are not valid to be filled in that particular box within matrix A; i.e. cross out the numbers in matrix B which are declined for choice to be filled in a particular box.

Step 3: Lock the choice of the number to be filled in the box within matrix A.
Crossing out numbers in matrix B leaves the user with one number to be filled within the box of matrix A. thus, finalise the content of a box within 3*3 matrixes (matrix A). The same pattern is to be repeated for each box within the 3*3 matrix

Step 4: Confirm the number to be filled
In order to confirm the number which emerges after crossing out numerals from matrix B, the user can now insert the number within the box by pressing the corresponding numeral (1-9) from the number panel. This confirms the box contents.

Step 5: Commit the step:
The user has to press the commit button to finalise the insert operation. This commitment can help the user to be sure about whether the number to be committed is in the right place. Whenever the user rolls the mouse over the uncommitted box, a cloudy message appears that tells the user whether the particular button is committed or not. In this way, the user can be sure about the choice of number and corresponding box, while he/she is committing every box after full consideration.

Step 6: Rollback
If the user makes a mistake and needs to un-commit a number which has been incorrectly filled in a box, she/he can press the rollback button over that box, which will result in an empty box. The same procedure can then be repeated. However, the rollback of a step can result in an increase in the number of steps taken to complete the game.

Control lies in the number panel. Users will be interrupted at the end of step three, when they would have decided which number to insert within the box. Group1 users will have to unlock the number panel when they return to play. Duration of the interruption will be 30 seconds so as to enable a substantial decay of working memory from step three; that is, after a certain time interval, the number panel will be locked to clicks. Moreover the research conducted by [5], in considering the interruption duration, reported that the duration of the interruption was independent of the global task performance of the user. Though the study was similar in terms of interruption, complexity and memory load, an interruption delivers determining factors of performance. [11] states that the interruption duration should last long enough to incur a decay of the goal to be resumed and should prevent it from being rehearsed. In this experiment, an interval of 30 seconds will be incurred. This will ensure enough decay in memory so that the effects of the interruption can be noted. The effects of interruption will be noted by means of the occurrence of slips when the game is resumed. The control group’s performance will give an indication of whether the slip occurred or not.

It was decided that the control would be the click on the number panel. For one group’s experiment, the number panel is locked and for the other it is not. If the users start trying to click on the matrix A, a pop-up message will appear, reporting an error. The visual cues, like a blinking number panel, are used to help the users of the locked number panel gaming interface, as a hint. The users who don’t get a locked number panel in the application will carry on without receiving a cue over the locked panel.

Two groups, with 30 participants each, will be studied for five gaming session. The errors reported in two different environments (with cues and without cues) will be statistically analysed to deliver results and test the hypothesis.

5. EVALUATION
An evaluation of the Microworld will be conducted with the groups, based on the procedure described above. A report of any errors will be generated for each participant and the data will be stored as per ethical norms. The experimental hypothesis, ‘Does Restart in a Procedural Step Imply more Issues in the Design of a System?’ will be tested according to the results of the experiment. The time and the length of the interruption are not studied as these aspects have already been considered in the literature by [11]. However, conducting an experiment with a large number of participants will provide qualitative methodology for future research. Moreover, to establish a solid qualitative basis for design implications, data collected will be statistically analysed. Analysis of variance (ANOVA) tests will be selected based on the common procedure for future verification of research results.

6. CONCLUSION
In previous literature, major conclusions have been reached with regard to just-in-time cues and the effects of low working memory load on participants. [8] established a goal model based on the implications of omission errors. The study notes omission errors are low at self activation influence and might fail due to environmental influences. This infers that design issues are very important for reducing omission errors. Stereotypical development focuses on increasing the complexity of the system, but this increases the rate of omission. However, the study will confirm the issues and implications of design. Further, the results will solidify the significance of high working memory load and assess personalised cue creation and its effectiveness. As well as the above implications, the study investigates their influential behaviour on restart errors and overall capability of lowering the error-prone behaviour of interactive system design.

7. ACKNOWLEDGEMENTS
This work is carried out as part of the module requirement towards the MSc degree in Software Engineering from Queen Mary, University of London. The work is based on the proposed topic of a dissertation supervised by Dr. Paul Curzon. The author would like to extend heartfelt thanks to Paul, who has been supportive and offered valuable guidance. His generosity in providing the reading material is appreciated. The author is grateful for the support and assistance received from the module coordinator, Mr. Nick Bryan-Kinns. Last but not the least, the author would like to thank the School of Electronic Engineering and Computer Science for providing the opportunity and facilitating the timely individual requirements.

8. BIBLIOGRAPHY
Back, J., Cheng, W.L., Dann, R., Curzon, P. & Blandford, A. (2006). Does being motivated to avoid procedural (DOI: 10.1007/978-1-84628-664-3_12 ). People and Computers XX – Engage Proceedings of HCI 2006 (Vol. 1). London.

Bovair, Micheal D. Byrne & Susan (1997). A Working Memory Model of a Common Procedural Error. Cognitive Science Session 21 (1), (pp. 31-69). Atlanta.

Byrne, Philip H Chung, & Micheal D. (2008). Cue effectiveness in mitigating postcompletion errors in a routine procedural task. International Journal of Human-Computer Studies , 217-232.

Cooper, A. (1999). Inmates are Running the Asylum. Indiana Polis: SAMS.

D.Broadbent & T.Gillie (1989). What makes interruptions disruptive? A study of lenght, similarity and performance. Psychological Research, (pp. 243-250). Oxford.

Embrey, David. (2007). Understanding Human Behaviour and Error. Lancashire: Human Reliability Associates Ltd.

J. Back, A. Blandford & P. Curzon (2007). Slip Erros and Cue Salience. ECCE, 2007 (pp. 221-224). London: ACM, NY, USA.

J. Gregory Trafton, Erik M. Altmann & Raj M. Ratwani (2002). A memory for goals model of sequence errors. Conference of Cognitive Modeling , (pp. 39-83). Manchester, UK.

Lamming, William M. Newman & Micheal G. (1995). Interactive System Design. Boston, MA, USA: Addison Wesley.

Lee, Carrie A. (n.d.). Human Error In Aviation. Retrieved February 19, 2010, from carrielee.net: http://www.carrielee.net/pdfs/HumanError.pdf

Li, S.Y.W., Cox, A.L., Blandford, A., Cairns, P., & Abeles, A., (2006). Further investigations into post-completion error: the effects of interruption. Cognitive Science Conference (2006). London.

Mackenzie, Donald, Knowing Machines: Essays on Technical Change. Cambridge, MA: MIT Press, 1996

Mortenson, Ian Christian (2004). An investigation of working memory areas. London.

Norman, D. A. (1983). Design Principles for Human Computer Interaction. DOI: 10.1145/800045.801571. Conference on Human Factors in Computing Systems (pp. 1-10). Boston, Massachusetts, US: ACM, NY, USA.

Reason, James. (1990). Review of Human Error. New York: Cambridge University Press