Supporting Creativity with

Advanced Information-Abundant User Interfaces

Ben Shneiderman ben@cs.umd.edu July 29, 1999

Department of Computer Science,

Human-Computer Interaction Laboratory, Institute for Advanced Computer Studies &

Institute for Systems Research

University of Maryland

College Park, MD 20742 USA

Position Paper for National Science Foundation & European Commission meeting on human-computer interaction research agenda, June 1-4, 1999, Toulouse, France. To be published in book form.
 
 
 
 

Abstract

A challenge for human-computer interaction researchers and user interface designers is to construct information technologies that support creativity. This ambitious goal can be attained if designers build on an adequate understanding of creative processes. This paper describes a model of creativity, the four-phase genex framework for generating excellence:

- Collect: learn from previous works stored in digital libraries, the web, etc.

- Relate: consult with peers and mentors at early, middle and late stages

- Create: explore, compose, discover, and evaluate possible solutions

- Donate: disseminate the results and contribute to the digital libraries, the web, etc.

Within this integrated framework, there are eight activities that require human-computer interaction research and advanced user interface design. This paper concentrates on techniques of information visualization that support creative work by enabling users to find relevant information resources, identify desired items in a set, or discover patterns in a collection. It describes information visualization methods and proposes five questions for the future: generality, integration, perceptual foundations, cognitive principles, and collaboration.

1. Introduction

Ambitious visions can be helpful in shaping more concrete research agendas. Vannevar Bush's vision of a system to support memory expansion (memex) inspired researchers for a half century in the development of what has become the World-Wide Web. Engelbart’s goal of augmenting human intellect (Engelbart and English, 1968) led to innovations such as the mouse and windows. Later, Brooks’s belief in the importance of toolmaking (Brooks, 1996) led to innovations such as haptic feedback in 3-d graphical environments. Hiltz's recognition of the power of online communities (Hiltz, 1984), inspired early software development (Hiltz and Turoff, 1993) and the emergence of ideas such as the collaboratory (National Research Council, 1993). These inspirational visions were important predecessors for genex (generator of excellence), an integrated framework for creativity support tools (Shneiderman, 1998b, 1999).

1.1 Creativity Support

Supporting creativity is a bold ambition, but it is becoming feasible because of refined understandings of the creative processes (Rosner and Abt, 1970; DeBono, 1973; Boden, 1990; Mayer, 1992; Czikszentmihalyi, 1996; Couger 1996) and the emergence of advanced user interfaces to support creativity (Massetti, 1996; Massetti, White, and Spitler, 1999). While theories of creativity vary widely, common features describe a preparatory phase that deals with the need to find information, understand the problem, and explore alternatives privately, followed by discussion with peers and mentors. These steps lay the basis for an incubation phase, and moments of inspiration to break a too rigid mind set. Then come the hours of perspiration to evaluate possibilities, refine potential solutions, implement the chosen solution, and disseminate it.

The diverse theories contributed to the four phases in an integrated user interface framework, called genex:

- Collect: learn from previous works stored in digital libraries, the web, etc.

- Relate: consult with peers and mentors at early, middle and late stages

- Create: explore, compose, discover, and evaluate possible solutions

- Donate: disseminate the results and contribute to the digital libraries, the web, etc.

Across these four phases, at least eight activities are opportunities for research in user interface design and human-computer interaction (Shneiderman, 1999) (Figure 1):

- Searching and browsing digital libraries

- Consulting with peers and mentors

- Visualizing data and processes

- Thinking by free associations

- Exploring solutions- What if tools

- Composing artifacts and performances

- Reviewing and replaying session histories

- Disseminating results

These activities are richly interwoven, for example, visualizing supports searching, exploring, reviewing, etc. Each activity deserves consideration as part of a research agenda, but this paper focuses on the strategies for visualizing. This paper focuses on the third item – visualizing data and processes – and discusses its place the broad array of genex activities.

1.2 Visualizing as a support for creativity

Visualizing is one of the eight activities that support the genex framework because visual information processing is central to many problem-solving tasks and creative explorations. Information visualization can contribute to early stages of collecting information from vast digital resources, to exploring alternative solutions, and to identifying appropriate people for consultations or dissemination.

Evolutionary needs have made the human visual system extremely well adapted to recognizing patterns, extracting features, and detecting unexpected items. Humans can rapidly process enormous amounts of visual information and take action rapidly. The human perceptual apparatus integrates interpretation so that people can rapidly identify familiar faces or recognize threats.

Linear or tabular presentations of text, numbers, and music extend human memory, enable transmission of information across time and space, and support creativity. The process of recording an idea facilitates innovation and discovery by compelling an author to produce a coherent presentation, develop a consistent notation, and present a logical argument. The linear or tabular formats enable the author and others to review, edit, refine, compare, and search presentations rapidly.

The goal for visual designers is to match the task to the presentation (Tufte, 1983, 1990, 1997). Bertin (1983) dramatically portrays the possibilities by showing one hundred different presentations of information about French provinces, each suited to a specific task.

Two-dimensional visual presentations such as drawings, tree structures, flowcharts, and maps have additional advantages. They can present affinities among multiple items, enabling users to see relationships that might be difficult to discover in linear textual presentations. Proximity or links in two-dimensional presentations can show rich structural relationships. Coding by size and color are easily perceived and further coding by shape, texture, rotation, or markings can highlight additional relationships.

World maps are especially rich information sources that enable designers to show complex relationships while allowing users to plan trips and make further discoveries. Mental maps of information spaces and special designs such as the periodic table of elements, monthly calendars, or the Linnaean taxonomy of animal phyla also support creativity. By placing known information in an orderly compact structure they support users in solving problems, planning activities, and making further discoveries.

1.3 Information Visualization

Human perceptual abilities are remarkable and still largely underutilized by the current graphical user interfaces. Computer based visual presentations bring additional opportunities and dangers. Adding animations such as blinking, color shifts, and movements enrich the possibilities for presentations but risk overwhelming readers. However, a great benefit of computing environments is the opportunity for users to rapidly revise the presentation to suit their tasks. Users can quickly change the rules governing proximity, linking, color, size, shape, texture, rotation, marking, blinking, color shifts, and movements. In addition, zooming in or clicking on specific items to get greater detail increases the possibilities for designers and users (Bederson and Hollan, 1994). The simple process of viewing a year-long display, clicking on the month, and then the day enables users to get an overview and quickly move to details. A picture is often said to be worth a thousand words. Similarly, an interface is worth a thousand pictures.

A reasonable expectation of the future of user interfaces is for growth of larger, information-abundant displays. While small portable personal devices will proliferate, the attraction of large desk-mounted or projected displays will be great. Human perceptual skills are remarkable and largely underutilized in current information and computing systems. Based on this insight, designers at several leading research centers developed advanced prototypes and a variety of widgets to present, search, browse, filter, and explore rich information spaces (Card, Mackinlay, and Shneiderman, 1999).

We can define information visualization as: A compact graphical presentation and user interface for rapidly manipulating large numbers of items (102 - 106), possibly extracted from far larger datasets. Effective information visualizations enable users to make discoveries, decisions, or explanations about patterns (correlations, clusters, gaps, outliers,...), groups of items, or individual items. This definition emphasizes the user interface for control of the presentation (location, color, size, rotation, shape, texture, blinking, movement, animation, and other variables).

For the purposes of this discussion, information visualization deals with abstract multi-dimensional and multi-variate data and is quite different from scientific visualization or 3-D modeling. Scientific visualization typically deals with three dimensional real-world phenomena such as storms, crystal growth, or human anatomy and 3-D modeling typically deals with mechanical parts, architectural walk-throughs, or aircraft design.
 

2. Examples of information visualization supporting creativity

One of our early visualizations was based on dynamic queries, which are animated user-controlled displays. These update immediately (no 'run' button) in response to movements of sliders, buttons, maps, or other widgets (Shneiderman, 1994). For example, in the HomeFinder the users see points of light on a map representing homes for sale (Ahlberg, Williamson, and Shneiderman, 1992; Williamson and Shneiderman, 1993). As they shift sliders for the price, number of bedrooms, etc. the points of light come and go within 100 milliseconds, offering a quick understanding of how many and where suitable homes are being sold. Clicking on a point of light produces a full description and, potentially, a picture of the house. A controlled experiment with 18 subjects confirmed the performance and preference advantages of dynamic queries over a natural language interface and a paper database.

A next step was the starfield display, which was created for the FilmFinder to provide visual access to a film database (Ahlberg and Shneiderman, 1994). The films were arranged as color coded rectangles along the x-axis by the production year and along the y-axis by popularity. Recent popular films were in the upper right hand corner. Zoombars (a variant of scroll bars) enabled users to zoom-in on a single axis in milliseconds to view the desired region. When less than 25 films were on the screen, the film titles appeared and when the users clicked on a film’s rectangle, a dialog box would appear giving full information and an image from the film. The commercial version of starfield displays became available late in 1996 (Figure 2) (www.spotfire.com).

Figure 2: Multi-dimensional film database viewed with a 2-dimensional starfield display in Spotfire. The x-axis is the years and the y-axis is the popularity of the film. Color coding is film type (action, drama, mystery, etc.) and larger size dots indicate longer movies.

In our LifeLines prototype, we applied multiple timeline representations to personal histories such as medical records (Plaisant, Rose, Milash, Widoff, and Shneiderman, 1996). Horizontal and vertical zooming, focusing, and filtering enabled us to represent complex histories and support exploration by clicking on timelines to get detailed information.

Information visualization supports creative work by enabling users to:

- find relevant information resources in digital libraries,

- identify desired items in a set, or

- discover patterns in a collection.

Figure 2 demonstrates how a digital library of films could be viewed in a way that presents large amounts of information in an orderly way. Users can understand the distribution of films in this library and find specific films to satisfy their needs. Similarly, in a legal information library, users may be seeking the relevant precedents to support their arguments. The West key number system organizes information in a hierarchy whose first three levels have 470 items, which then expand into a tree with 85,000 nodes. Figure 3 shows how the results of a search might be displayed to reveal where the cases fall within the key number hierarchy (Shneiderman, Feldman, and Rose, 1999). Additional information is supplied on the x-axis, currently organizing the cases by year, and by color coding, currently showing the region the case came from. This visualization makes it clear that the major topic is Criminal law. This visualization enables users to find the relevant resources that would be very difficult to discover in a typical paged list with 20 cases per page.
 
 

Figure 3: Results of a search in a legal information library. The x-axis is the topic, organized by the West key numbers and the y-axis is the year. Color coding is by region.

A second form of creativity support is to identify items in a set. The legal information library is an example for this as well, but the drug discovery task is more typical. In this example, 379 compounds are viewed at once, organized by the amount of carbon and oxygen, color coded by dipole moment and size coded by polarizability (Figure 4). The unusual compound (the selected square at the upper right) is clearly visible by its distinct color coding (may not be as easy to see in the black and white print).

Figure 4: Information visualization of 379 chemical compounds. The x-axis is the amount of carbon and the y-axis is the amount of oxygen. Color coding is by dipole moment and size coding is by polarizability.

Important patterns can also be seen in visual displays. The familiar chemical periodic table of elements becomes more informative when color and size coding are added, and when users can make selections by moving the double box sliders. Figure 5 shows the usual layout but the high electronegativity is immediately visible by the color coding. Figure 6 shows a strong correlation between electronegativity and ionization energy, with two dramatic outliers: helium and radon.
 
 
 
 

Figure 5: The periodic table of elements with color coding to show electronegativity and size coding by ionization energy.

The three ways that information visualization interfaces support creativity are not a complete set, but they give designers some specific goals to work towards.
 
 

3. Questions for the Future

The computing industry and the research community are moving ahead with a new generation of systems. In addition to our work, research on information visualization is emerging at key sites such as Georgia Tech's Graphics Visualization and Usability Center, Xerox's Palo Alto Research Center, and Lucent Technologies (formerly AT&T Bell Labs) in Naperville, IL. Commercial activity ranges from expansion of existing statistical, spreadsheet, or database packages to include visualization capabilities, for example SPSS and SAS. Specialized visualization tools have emerged from new companies such as Inxight, Visual Insight, Visible Decisions, and Spotfire. However, numerous questions remain that are the basis for this research agenda.

Figure 6: Chemicals organized by electronegativity and ionization energy, revealing a strong correlation and two outliers: helium and radon.
 
 

Generality: Many creativity support and information visualization tools are designed for a specific type of data and task, so generalization is an important issue. Since the range of information visualization situations includes at least 1-d, 2-d, 3-d, multi-dimensional, temporal, hierarchical, and network data types, it is not clear a single tool can be useful to a wide range of users (Robertson, Card, and Mackinlay, 1993; Becker, Eick and Wilks, 1995; Shneiderman, 1998a). Tasks include presenting meaningful overviews, zooming in on desired items, filtering out undesired items, obtaining details-on-demand, showing relationships among items, extracting information for use in other programs, and keeping a history of user actions to allow review and replay. Developing general purpose tools that support the range of data and tasks would be a major step forward.

Integration: Successful support for creativity and practical application of information visualization requires a smooth integration with other tools. The results of a web or database search should be easily imported (for example by cut and paste) into a visual presentation. Then users should be able to filter the data appropriately and adjust the visualization features, such as x,y-axes, color, size, or other codings. When an interesting group of items is found, users should be able to select them and paste them into a spreadsheet or statistics package for further processing. At the next stage, the visualization and processed items should be embeddable in a written report, slide presentation, or email note. Email recipients should be able to manipulate the visualization or report still further (Roth, Lucas, Senn, Gomberg, Burks, Stroffolino, Kolojejchick, and Dunmire, 1996).

Perceptual foundations: A necessary foundation is an understanding of the perceptual principles concerning location, color, size, shape, animation and other codings (Rohrer, Ebert, and Sibert, 1999). While much is known about static displays, the dynamic environment of user interfaces is in need of extensive human factors analysis. Pre-conscious recognition of small numbers of items, simple patterns, or outliers from a large group occurs very rapidly (less than 400 milliseconds), but recognition of more complex relationships involving multiple colors or shapes can take much longer. Strategies for rapid panning or zooming are beginning to be understood, but comparisons between these animated approaches, distortion oriented (fisheye), and dual views would be very helpful (Plaisant, Carr and Shneiderman, 1995; Schaffer et al., 1996). Another key comparison that deserves extensive study is between 2-d and 3-d visual presentations (Sutcliffe and Patel, 1996). Advocates of each style claim superiority, but the empirical evidence is still shallow.

Cognitive principles: A solid theory for creativity support by way of information visualization would rest on a thorough understanding of cognitive principles to guide design (Card, 1996). Existing design principles such as direct manipulation have been demonstrated to be successful by empirical studies, and they have been widely applied in word processors, spreadsheets, drawing tools, and many other environments:

- Visual representation of the "world of action"

- Objects & Actions are shown

- Tap analogical reasoning by appropriate metaphors and icons

- Rapid, incremental, and reversible actions

- Replace typing with pointing/selecting

- Continuous feedback

These principles are helpful to designers but need to be more rigorous if they are to provide predictive power. Another basic principle that has been applied for browsing and searching, might be summarized as the Visual Information Seeking Mantra:

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Overview first, zoom and filter, then details-on-demand

Each line represents a project in which we struggled with a design only to rediscover this principle, and therefore I wrote it down as a continuing reminder. If we can design systems with effective visual displays, direct manipulation interfaces, and dynamic queries then users may be able to responsibly and confidently take on more ambitious tasks. However, empirical studies would be helpful in assessing the benefits.

Collaboration: A key phase of genex, and many creativity models, is the process of consultation with peers and mentors (Olson and Olson, 1997). Such consultations may occur at early, middle, or late stages of a creative problem solving process. Users need to conference over some materials using an appropriate communications medium such as a face-to-face meeting, videoconference, telephone, email exchange, or printed documents. Therefore, creativity support and information visualization tools need to make it easy for users to save, extract, replay, and annotate their activities.
 
 

4. Conclusions

Creativity support is a risky term because it may sound vague and difficult to evaluate. However, it can become a vigorous research topic if work focuses on more identifiable user activities, such as the list of eight offered in the opening. This paper concentrates on the activity of visualizing and explores how information visualization techniques can support creativity. Researchers will have to deal with at least five key questions in order to develop useful software: generality, integration, perceptual foundations, cognitive principles, and collaboration.

However, implementation of novel tools is not a sufficient goal. New visualizations and their use must be subjected to rigorous empirical studies to get past the developer's bias and wishful thinking. Evaluations, ranging from controlled experiments to field trials with ethnographic observations, will validate or overturn hypothesis, refine theories, and sharpen our understanding of what to measure. Such studies are likely to be the rapid route to development of advanced information-abundant user interfaces.

Acknowledgements: I greatly appreciate my partners at IBM, NASA and WestGroup for partial support of this research. Thanks to Benjamin Bederson, David Ebert, and Jenny Preece for their thoughtful comments on earlier drafts.

References

Ahlberg, Christopher and Shneiderman, Ben, Visual information seeking: Tight coupling of dynamic query filters with starfield displays, Proc. ACM CHI'94 Conference: Human Factors in Computing Systems, ACM, New York (1994), 313-321 + color plates.

Ahlberg, C., Williamson, C., and Shneiderman, B., Dynamic queries for information exploration: An implementation and evaluation, Proc. ACM CHI’92: Human Factors in Computing Systems, ACM, New York (1992), 619-626.

Becker, Richard A., Eick, Stephen G., and Wilks, Allan R., Visualizing network data, IEEE Transactions on Visualization and Computer Graphics 1, 1 (March 1995), 16-28.

Bederson, Benjamin B. and Hollan, James D., PAD++: A zooming graphical user interface for exploring alternate interface physics, Proc. User Interfaces Software and Technology '94, ACM, New York (1994), 17-27.

Bertin, Jacques, Semiology of Graphics, University of Wisconsin Press, Madison, WI (1983).

Boden, Margaret, The Creative Mind: Myths & Mechanisms, Basic Books, New York (1990).

Brooks, Frederick, Jr., The computer scientist as toolsmith II, Communications of the ACM 39, 3 (March 1996), 61-68.

Bush, Vannevar, As we may think, Atlantic Monthly 76, 1 (July 1945), 101-108. Also at

http://www2.theAtlantic.com/atlantic/atlweb/flashbks/computer/tech.htm

Candy, Linda, Computers and creativity support: Knowledge, visualisation and collaboration, Knowledge-Based Systems 10 (1997), 3-13.

Card, Stuart C., Visualizing retrieved information: A survey, IEEE Computer Graphics and Applications 16, 2 (1996), 63-67.

Card, Stuart, Mackinlay, Jock, and Shneiderman, Ben, Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann Publishers, San Francisco, CA (1999).

Couger, D., Creativity & Innovation in Information Systems Organizations, Boyd & Fraser Publ. Co., Danvers, MA (1996).

Csikszentmihalyi, M., Creativity: Flow and the Psychology of Discovery and Invention, HarperCollins, New York (1996).

De Bono, E., Lateral Thinking: Creativity Step by Step, Harper Colophon Books, New York (1973).

Engelbart, Douglas C. and English, William K., A research center for augmenting human intellect, AFIPS Proc. Fall Joint Computer Conference 33 (1968), 395-410.

Hiltz, R. S., Online Communities: A Case Study of the Office of the Future, Ablex, Norwood, NJ (1984).

Hiltz, R. S. and Turoff, M., The Network Nation: Human Communication via Computer: Revised Edition, MIT Press, Cambridge, MA (1993).

Massetti, Brenda, An empirical examination of the value of creativity support systems on idea generation, MIS Quarterly 20, 1 (1996), 83-97.

Massetti, Brenda, White, Norman H., and Spitler, Valerie K., The impact of the World Wide Web on idea generation, Proc. 32nd Hawaii International Conference on System Sciences, IEEE Press (1999).

Mayer, Richard E., Thinking, Problem Solving, Cognition: 2nd Edition, W. H. Freeman, New York (1992).

National Research Council - Committee on a National Collaboratory, National collaboratories: Applying information technology for scientific research, National Academy Press, Washington, DC (1993).

Olson, Gary M. and Olson, Judith S., Research on computer supported cooperative work, In Helander, M. G., Landauer, T. K., and Prabhu, P. V. (Editors), Handbook of Human-Computer Interaction: Second Edition, Elsevier, Amsterdam (1997), 1433-1456.

Plaisant, C., Carr, D., and Shneiderman, B., Image-browser taxonomy and guidelines for designers, IEEE Software 12, 2 (March 1995), 21-32.

Plaisant, Catherine, Rose, Anne, Milash, Brett, Widoff, Seth, and Shneiderman, Ben, LifeLines: Visualizing personal histories, Proc. of ACM CHI'96 Conference: Human Factors in Computing Systems, ACM, New York, NY (1996), 221-227, 518.

Robertson, George G., Card, Stuart K., and Mackinlay, Jock D., Information visualization using 3-D interactive animation, Communications of the ACM 36, 4 (April 1993), 56-71.

Rohrer, R., Ebert, D., and Sibert, J., A shape-based visual interface for text retrieval, IEEE Computer Graphics and Applications 19 (Sept/Oct 1999).

Rose, Anne, Eckard, David, and Rubloff, Gary, An application framework for creating simulation-based learning environments, Univ. of Maryland Dept. of Computer Science Technical Report CS-TR-3907 (1998).

Rosner, Stanley and Abt, Lawrence E. (Editors), The Creative Experience, Dell Publishing, New York (1970).

Roth, Steven F., Lucas, Peter, Senn, Jeffrey A., Gomberg, Cristina C., Burks, Michael B., Stroffolino, Philip J., Kolojejchick, John A. and Dunmire, Carolyn, Visage: A user interface environment for exploring information, Proc. Information Visualization '96, IEEE, Los Alamitos, CA (1996), 3-12.

Schaffer, Doug, Zuo, Zhengping, Greenberg, Saul, Bartram, Lyn, Dill, John, Dubs, Shelli and Roseman, Mark, Navigating hierarchically clustered networks through fisheye and full-zoom methods, ACM Transactions on Computer-Human Interaction 3, 2 (June 1996), 162-188.

Shneiderman, Ben, Designing the User Interface: Strategies for Effective Human-Computer Interaction: Third Edition, Addison-Wesley Publ. Co., Reading, MA (1998a)

Shneiderman, B., Codex, memex, genex: The pursuit of transformational technologies, International Journal of Human-Computer Interaction 10, 2 (1998b), 87-106.

Shneiderman, B., Creating creativity for everyone: User interfaces for supporting innovation, Univ. of Maryland Dept of Computer Science Technical Report (1999).

Shneiderman, B., Dynamic queries for visual information seeking, IEEE Software 11, 6 (1994), 70-77.

Shneiderman, B., Feldman, D., and Rose, A., Visualizing digital library search results with categorical and hierarchical axes, University of Maryland Department of Computer Science Technical Report CS-TR-3992 (February 1999).

Sutcliffe, A. and Patel, U., 3D or not 3D: Is it nobler in the mind?, Proc. British HCI Conference (1996), 79-94.

Tufte, Edward, The Visual Display of Quantitative Information, Graphics Press, Cheshire, CT (1983).

Tufte, Edward, Envisioning Information, Graphics Press, Cheshire, CT (1990).

Tufte, Edward, Visual Explanations: Images and Quantities, Evidence and Narrative, Graphics Press, Cheshire, CT (1997).

Williamson, C., and Shneiderman, B., 1992. The Dynamic HomeFinder: Evaluating dynamic queries in a real-estate information exploration system, Proc. ACM SIGIR’92 Conference,

Copenhagen, Denmark, (June 1992), 338-346. Reprinted in Shneiderman, B. (Editor), Sparks of Innovation in Human-Computer Interaction, Ablex Publishers, Norwood, NJ, (1993), 295-307.