Computers can be valuable tools, and networked resources via the Internet can be beneficial to many different populations and communities. Unfortunately, when people are unable to reach their task goals due to frustrating experiences, this can hinder the effectiveness of technology. This research summary provides information about the user frustration research that has been performed at the University of Maryland and Towson University. Causes of user frustration are discussed in this research summary, along with the surprising finding that nearly one-third to one-half of the time spent in front of the computer is wasted due to frustrating experiences. Furthermore, when interfaces are planned to be deceptive and confusing, this can lead to increased frustration. Implications for designers and users are discussed.

Jonathan Lazar is a professor at the Department of Computer and Information Sciences & Center for Applied Information Technology, Towson University, Towson, Maryland 21252.

Katie Bessiere is a graduate student at the Human-Computer Interaction Institute, Carnegie-Mellon University, Pittsburgh, Pennsylvania 15232.

Irina Ceaparu is a graduate student at the Department of Computer Science, Human-Computer Interaction Laboratory, Institute for Advanced Computer Studies & Institute for Systems Research, University of Maryland, College Park, Maryland 20742.

John Robinson is a professor in the Department of Sociology at th e University of Maryland, College Park, Maryland 20742.

Ben Shneiderman is a professor in the Department of Computer Science, Human-Computer Interaction Laboratory, Institute for Advanced Computer Studies & Institute for Systems Research, University of Maryland, College Park, Maryland 20742.

© 2003 Stanford University

The Internet holds great promise for improving people’s lives, through increased access to medical and educational information. Unfortunately, users frequently report having problems finding what they want on the Web. Users can’t find the site that they want, and once on a Web site, they can’t find the specific content that interests them. What are the specific causes of user frustration related to Web navigation?

This article presents a research summary of recent work related to user frustration, with a focus on Web navigation. A specific concern is the possibility that poorly-educated and/or low-income users will have an especially difficult time in navigating the Web. Their limit ed experience with computers, the Internet and Web services means that extra attention may need to be paid to the design of Web sites for government- and public-assistance programs. The Digital Divide has been well-documented within the United States (National Telecommunications and Information Administration 2002), and it is a major problem throughout the world. Economic factors are a major influence, but training, education and good design can play a key role in bridging the divide by providing people, via the Web, with information that is empowering.


Users frequently report being frustrated with the Web because they cannot find what they are looking for. There are two separate issues with Web navigation: trying to find a Web site and then trying to find specific content on that Web site. Search engines can be confusing for users who are not familiar with how to form appropriate queries—a common problem for novice users (Marchionini 1995). Even if users are able to find a Web site that interests them, they frequently report having problems finding the content on a Web site that they are interested in. The focus on frustration in Web navigation is especially important because the majority of Web users tends to be newer users without a lot of computer experience, and therefore can easily get frustrated. In addition, the Web is frequently used in a home setting, without the technical support available in a workplace (Cummings and Kraut 2002).

Navigation on Web sites can be improved by providing multiple navigation techniques or paths to the same content. For instance, navigation can be provided by links organized both by topic and target audience (Fleming 1998). In addition, path navigation (also called “breadcrumbs navigation”) can help to show users where they currently are located within the information architecture (Nielsen 2000). Good Web navigation must be provided by the Web site itself, because the navigation cues provided by the Web browser are limited (Cockburn and Jones 1996). For instance, navigation aids in the browser (such as the Back button) generally allow users to trace only one path taken in an information hierarchy (Cockburn and Jones 1996).

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003

In addition, users frequently do not use the browser mechanisms to trace their paths, but rather use the navigation provided by a site to navigate forward (Spool, Scanlon, Schroeder, Snyder and DeAngelo 1999). Whatever navigation is provided by the Web site should be consistent throughout the site, so that users do not get confused and perceive that they have entered another Web site (Shubin and Meehan 1997).

Navigation should typically be text-based, for two main reasons. Text-based navigation loads faster, and in addition, graphical navigation can provide problems for users with disabilities (Paciello 2000). Sufficient testing should be done with users to ensure that the descriptions of navigational links are clear, and that the content is organized into topics that generally match the user’s model of how the content should be organized (Lazar 2001). Improved site navigation can help users find the content that interests them.


The authors have recently been examining what frustrates users when they interact with computers. One hundred eleven (111) subjects took part in a research study examining user frustration with computers. The data were collected using a modified time diary. Other research methods, such as surveys, often ask the respondent to estimate the answer to questions from memory, which can often lead to inflated or incorrect answers related to time. Time diaries minimize the reporting burden on the respondents by allowing them to record their time use immediately after it occurs, instead of attempting to remember an aggregate amount of information at a later date.

Users spent a minimum of an hour using the computer, and no specific tasks were assigned or expected. Rather, users were simply asked to carry on with their normal tasks, and then to report the experiences that were frustrating. This approach to collecting data was more likely to result in data that were representative of the actual tasks that users would perform.

Before beginning their sessions, users filled out a pre-session survey, noting their mood and computer experience. Whenever users encountered a frustrating experience, they filled out a “frustrating experience form,” detailing the cause of the frustration and the time lost. After completing a minimum of an hour (average time spent was 2.5 hours) at the computer, the subjects filled out a post-session form detailing their current mood.

In this study of 111 subjects, Web browsing was cited as the task that most frequently caused frustration. Subjects reported 122 frustrating experiences with Web browsing out of a total of 373 frustrating experiences— Web browsing was largest source of frustration. The second and third most frustrating experiences cited were e-mail (49) and word processing (45) (Ceaparu, Lazar, Bessiere, Robinson and Shneiderman 2002). Since subjects chose their own tasks, this is not representative of the fact that Web browsing in

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003

general is the most frustrating application; however, it is clear from these statistics that Web navigation is frustrating.

Some of the specific causes of the frustration relate directly to Web navigation. The five most-cited causes of frustration on the Internet were: 1) dropped connections, 2) long download times, 3) Web page not found (404-type of error), 4) e-mail not sent/received properly (many users reported using Web-based e-mail applications) and 5) pop-up advertisements. Long download times, pages not found, and pop-up advertisements are all related to Web navigation.In other words, when users cannot find what they want, cannot reach the Web pages that interest them without a long wait, or are hindered in accessing those pages by pop-up advertisements, this is clearly a Web-navigation problem.

One of the most surprising findings of this research project is the time lost due to frustrating situations. In terms of minutes lost, the study subjects estimated that one-third to one-half of the time spent in front of the computer was lost, due to frustrating experiences. The total time (in minutes) was defined as the total time in front of the computer (recorded by the subject in the modified time diary). Minutes lost were defined as the sum of the minutes spent to solve the problem with the minutes spent to recover from any work loss due to the problem. Much of this lost time was caused by problems with Web navigation. This is obviously a troubling finding, pointing to the problems that users face in their everyday interactions with their computers.

The social-psychological factors involved with computer frustration were also examined. Based on data collected from the subjects, the strongest predictors of frustration with a specific incident are the severity of interruption (i.e., the time lost) and the importance of the user’s goal (Bessiere et al. 2002). However, for the overall frustration at the end of a session, both computer experience and self-efficacy (the belief that one is capable of completing the tasks) were the strongest predictors of overall lasting frustration, with those most experienced with computers feeling the least frustrated (Bessiere et al. 2002). This points to the need for more user training. However, much of the frustration experienced was due to poor design or unpredictable interfaces. Designers need to make clearer, and more predictable interfaces for Web browsing, both in the Web browser and in Web site design.

A second study was conducted to analyze users' experiences with searching governmental statistical data on the Web (Ceaparu 2002). The study focused on the portal , which provides access to statistical data collected by over 70 United States federal government agencies.

In this usability study, 15 subjects were asked to look for answers to three queries (scenarios) using the index page of FedStats, an alphabetical listing with approximately 700 entries. Out of the total 45 queries, subjects were unable to answer 31, or almost 70%. Subjects found the correct answer for only 7 queries and partial answers for another 7—or less than a third. The questionnaires used in the study revealed that over 50% of the subjects reported high levels of frustration related to the usefulness of the results, the granularity

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003

of the results, the usefulness of the Web site, ease of use and the time spent on searching the for answer. Recommended improvements to lessen frustration when using the Web site include terminology changes, organization into thematic categories, and provision of metadata to support searches.


User interfaces should be easy to use and predictable (Shneiderman 1998). Many of the frustrations related to Web navigation occur when the interface is unpredictable. An increasing unfortunate trend is to design Web interfaces that are purposefully deceptive and unpredictable. These Web interfaces provide confusing navigation, forcing users to go to sites that they did not intend to visit. For instance, pop-up advertisements are simply annoying; however, when the pop-up advertisement mimics an error message, this is deceptive, and will confuse users as to the appropriate next step. Figure 1 displays an example of a banner advertisement that mimics an error message.

Another instance of anti-usability occurs when standard interface widgets, which have well-defined roles, are used deceptively. For instance, the “X” symbol in the upper-right hand corner of a window or dialog box has long represented the action required to “close” the window. When users clicks “X” in the upper-right hand corner of a window, they expect the window to close. Instead, some advertisers are using the first, almost automated user action of clicking on the “X” to take users to an advertising Web site.

Usually, clicking the “OK” button will also close a dialog box. However, Figure 2 displays an example of deception. If the user clicks either the “OK” button or the “X” widget in the upper-right hand corner, the user will instead be taken to an advertisement site. The users are trying to close this window to get back to their task, but instead they are taken to another site.

Figure 3 shows another example of deception leading to an advertisement—a window that mimics a MS-Windows tip. Users may perceive this as an actual tip from the operating system, but in reality, it is an advertisement that, if clicked on, will take the users to an outside Web site. These examples of deceptive navigation make it harder for users to find the information that they want, because users cannot predict how their interfaces will react. This only makes Web navigation harder and more unpredictable.


Designers have a key role in improving user navigation on the Web, thereby lessening frustration. There are many useful documents providing usability guidelines for web sites. documents. (See for links and sour ces of information about universal usability. See

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003



FIGURE 3: AN ADVERTISEMENT THAT MIMICS A TIP FROM THE WINDOWS OPERATING SYSTEM WHILE BROWSING THE WEB for a starting point and Shneiderman 2000). A few, important criteria to support navigation are as follows:

1. Fast downloads: Research studies have repeatedly shown that users want Web pages that download quickly (Ramsay, Barbesi and Preece 1998; Sears, Jacko and Dubach 2000). Although the actual download time is a result of a number of factors, Web designers can improve the

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003

download time by creating Web pages that include minimal graphics or plug-ins, and that do not have extra, unneeded HTML tags.

  1. Consistent and predictable interfaces: Novice users are disturbed by unexpected change s to terminology, layout, color and fonts. For instance, if users are familiar with the term “shopping cart” on an e-commerce site, there is no need to use another term that could be confusing. Deceptive interfaces that do the opposite of what users predict are especially problematic. Interfaces that are predictable and consistent help users.
  2. Text-based navigation: Simple navigation paths described by meaningful textual names not only help novices, but enable users with disabilities to appropriately use voice browsers, mobile users to get access on small displays and readers from other countries to apply translation tools.
  3. Broad, shallow tree structures for information architecture: With large Web sites, it is most effective to have fewer levels, with many links at each level. The links act as descriptive terms and fewer steps mean more successful navigation. Users should not have to click through more than 4-5 levels of a specific Web site.


Training and experience can greatly improve Web-navigation experiences. Professional instruction, tutorials and guidebooks can provide introductions to Web resources, but after that, personal experience makes for more successful users. Users who persevere in asking questions to friends or professionals in person, or through online discussion groups, are more likely to succeed. With a better understanding of the limitations and structure of the Web environment, users may also feel less frustration and find more success in their tasks. The first principle in Karat’s User’s Bill of Rights (1998) is: “The user is always right. If there is a problem with use of the system, the system is the problem, not the user.”


While the World Wide Web is heralded as a revolutionary development that will greatly improve the lives of users, the reality of usage is more sobering. The above research studies and other published studies of Web usage revealed high levels of frustration and low rates of success for many typical tasks. In addition, novice users with low education and little computer experience seemed most at risk, in terms of failing to benefit from Web services. However, careful design and effective training seem likely to improve the user’s Web-navigation experience.

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003


Bessiere, K., Ceaparu, I., Lazar, J, Robinson, J. and Shneiderman, B. 2002. Understanding Computer User Frustration: Models and Measures of Incident and Individual Factors. Paper under review. Tech report available at:

Ceaparu, I. 2002. Governmental Statistical Data on the Web: A Case Study of FedStats. Paper under review. Tech report available at:

Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J. and Shneiderman, B. 2002. Determining Causes and Severity of End-User Frustration. Paper under review. Tech report available at: reports.shtml

Cockburn, A. and Jones, S. 1996. Which Way Now? Analyzing and Easing Inadequacies in WWW navigation, International Journal of Human-Computer Studies, 45(1), p. 105–129.

Cummings, J. and Kraut, R. 2002. Domesticating Computers and the Internet, The Information Society, 18 (3), p. 221–231.

Fleming, J. 1998. Web Navigation: Designing the User Experience. Sebastopol, CA: O’Reilly and Associates.

Karat, C.-M. 1998. Guaranteeing Rights for the User, Communications of the ACM 41, (12), p.29–31.

Lazar, J. 2001. User-Centered Web Development. Sudbury, MA: Jones and Bartlett Publishers.

Marchionini, G. 1995. Information Seeking in Electronic Environments. Cambridge, England: Cambridge University Press.

National Telecommunications and Information Administration, US Dept. of Commerce 2002. A Nation Online: How Americans Are Expanding Their Use of the Internet, Washington, DC. Available at:

Nielsen, J. 2000. Designing Web Usability: The Practice of Simplicity. Indianapolis: New Riders.

Paciello, M. 2000. Web Accessibility for People with Disabilities. Lawrence, KS: CMP Books.

Ramsay, J., Barbesi, A. and Preece, J. 1998. A Psychological Investigation of Long Retrieval Times on the World Wide Web, Interacting with Computers 10(1), p. 77–86.

Sears, A., Jacko, J. and Dubach, E. 2000. International Aspects of WWW Usability and the Role of High-end Graphical Enhancements, International Journal of Human-Computer Interaction, 12(2), p. 243– 263.

Shneiderman, B. 1998. Designing the User Interface: Strategies for Effective Human/Computer Interaction. Reading, MA: Addison-Wesley.

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003

Shneiderman, B. 2000. Universal Usability: Pushing Human-Computer Interaction Research to Empower Every Citizen, Communications of the ACM 43, (5), p. 84–91. Available online:

Shubin, H., and Meehan, M. 1997. Navigation in Web applications, Interactions 4(6), p. 13–17.

Spool, J., Scanlon, T., Schroeder, W., Snyder, C. and DeAngelo, T. 1999. Web Site Usability: A Designer’s Guide. San Francisco: Morgan Kaufmann Publishers.

IT&SOCIETY, Vol. 1, Issue 3, Winter 2003