Multimedia is an art world term, often credited to designers Charles and Ray Eames, that describes the fusion of media such as painting, sculpture, photography, music, and video. Within the world of computers it is used broadly to describe almost any combination of media, ranging from simple text and graphics through to the Eames' vision (Nielsen, 1995; Preece et al., 1994; Shneiderman, 1992).
The diversity of multimedia user interfaces raises questions about
the origin of these varied species. We think Darwin would have
nodded his head knowingly. His theory of evolution by natural
selection through survival of the fittest elegantly explains the
huge diversity of organisms that occupy the numerous ecological
niches. But can this natural world theory explain the evolution
of widgets, interface styles and emerging genres of multimedia?
Why have some survived while others have not? Where do multimedia
design ideas come from and what determines their evolution?
Mutation, the raw material of evolution, does not exist outside
the natural world but the pull of new technology and changing
user demands stimulates novel designs. Then market forces determine
survival, as companies and products come and go. Darwin would
probably have been astounded by the rapid expansion of multimedia
within only a few generations of computers. Predicting change
is hard but we now know much about the strengths and limitations,
likes and dislikes of humans interacting with computers. Principles
can be drawn upon, which help to explain the survival of some
interface features and extinction of others. For example, interfaces
that are predictable and consistent, allow users to easily undo
their actions, protect against errors and provide useful help
at the right time tend to survive.
We identify seven multimedia eras: Nascent, Control, Construction,
Ubiquitous, Collaborative, 3-D Virtual, and Visualization, and
make tentative predictions for the future. As in nature, these
era co-exist and over-lap, and there are also sudden 'ice-age'
Around 1980, at the dawn of the personal computer age, the primordial
soup of multimedia consisted of green screens and videodisc images
on separate monitors. This Nascent Era produced many chaotic and
short-lived species with rigid interfaces that left the users
frustrated victims of machines that they could not control. Examples
include the five-minute video without a stop button or choice
sequences which could not be reversed or cancelled. These species
died out quickly because of their poor usability. As advancements
in high resolution displays and fast chips spread, still and then
moving images, animations, and sound flourished. The evolutionary
force coming from both technical development and the demands of
users, particularly video game and home computing enthusiasts
followed by advertising, films and education assured their future.
In parallel with these developments interface complexity grew
and users needed better and more direct ways of controlling them.
This gave rise to the Control Era in which direct manipulation
became the dominant interface form. Instead of modal dialogs and
rigid sequencing users could make choices selecting objects as
they saw fit, reversing, cancelling, reviewing, and confirming
their actions as they wished. 'What you see is what you get',
more commonly referred to as WYSIWYG, became a guiding principle
with the world of action displayed visually and keyboards giving
way to pointing, selecting, dragging, and stretching. The aim
was to make operations rapid, incremental, and reversible and
to prevent user errors by effective designs. For example, when
selecting a date on a calendar it is impossible to make a syntactical
input error. As direct manipulation interfaces became prevalent
and so did machines with better graphics variants prevailed in
which objects were designed with visual affordances (Norman, 1989)
suggesting how to use them. Buttons, for example, looked as though
they should be pushed.
In later generations of the Control Era during the late 80s and
early 90s, embedded menus in text and graphics kept user attention
on the contents and provided smooth hypertext linking (Koved &
Shneiderman, 1986). The mouse became the device of choice, but
trackballs, joysticks, and tablets with pens found successful
niches. High precision touch screens with lift-off activation
enabled the emergence of effective public access kiosks and creative
The Construction Era also developed in which an increasing number
of people became involved in some form of multimedia authoring
(Shneiderman & Kearsley, 1989). Numerous authoring languages
emerged in almost every ecological niche but were slow to spread
and regularly died out, because they were cumbersome to use. Robust
species support integration of text and images and construction
tools for individual media (music, photos, drawings, video capture,
etc.). On the other hand, simple and powerful tools to cut and
paste video with dynamic text overlays, create and alter animations,
synchronize music with images, or search multimedia are still
rare and beautiful to behold.
In the mid 90s we are witnessing the Ubiquitous Era dawning with
the growing availability of World Wide Web access with embedded
menus providing links across the world. These developments have
generated a frenzy of writing home pages and torrent of browsing.
The remarkable potency of access to the net has led to an unusually
rapid growth of web servers, applications, and usage. Isolated
computer users may soon find it difficult to survive. In the early
generations of this era the emphasis has been on surfing the net
(reading, browsing, navigating), but in later generations greater
facility in authoring web pages and delivering applications across
the web will emerge. The awkwardness of separate viewers for video
or external applications for animations will fade as integrated
layouts become dominant in future web browsers.
Predicting evolutionary developments is a risky venture, but a
new Collaborative Era seems likely. Email, once the delight of
computer junkies, is now so prolific that overwhelmed gateways
produce faltering businesses, raging children, cursing academics,
and wailing lovers. Only the telecommunications companies and
their shareholders smile. The prevalence of UseNet groups stimulate
an increase in electronic text-based communication, but the seduction
of video conferencing continues. Crude video conferences will
become smooth, and participants will conveniently integrate simultaneous
viewing and constructing of multimedia objects. Users will switch
from seeing each other to conferencing over photos, maps, videos,
documents, soundtracks, and animations with multiple cursors all
in motion. They will discuss changes and any participant will
make edits that are viewed by all.
Another likely prediction is the 3-D Virtual Era. While early
tools show cluttered displays, obscured data, slow updates, and
inadequate navigation, novel way finding techniques, better layouts,
and faster processors will help. Full immersion, although good
for a limited number of specialist tasks like fighter pilot training,
is unlikely to spread too far because the cumbersome helmets,
sweaty gloves and web of wires are intrusive. Desktop virtual
reality, where users replace ëbeing-iní with ëlooking-atí
seem more likely to flourish.
A final prediction is that the Information Visualization Era will
facilitate presentation and exploration of multidimensional, relational,
hierarchical, tabular, and temporal data. Information abundant
and perceptually rich displays such as treemaps, starfields, table
lenses, magic lenses, hyperbolic trees, fisheye views, timelines,
will be explored with dynamic queries widgets, even across the
Across the eras, the responsiveness of systems (response times,
display rates, transmission times, etc.) has increased, even while
the image resolution and sound quality has improved, and the size
of accessible multimedia databases has soared. While these trends
seem likely to continue, the key principles for survival are mainly
those of good usability and fulfilling a real need. However, market
forces can be cruel and fickle. Resistance to novelty can slow
down even robust worthy innovations with wealthy backing. But
the defense provided by intellectual property protection is only
sometimes a reliable shield against invaders. Stolen ideas, while
frustrating to originators, can promote their evolution and survival!
Another change is that the user community has not only expanded
but also diversified with a wider range of users and the distinction
between users and developers becoming increasingly blurred. Programmers
have been joined by graphic designers, filmmakers, historians,
teachers, musicians, artists, designers and poets in creative
teams. Content experts have gained exciting opportunities to tell
Evolution doesn't have a destination or a plan, each innovative
gene has a chance to prove itself. For those of us who design
innovations, the thrill is there every day as we create ever more
viable user interfaces. Ultimately the forces and whims of the
market place will drive evolution with history judging our success.
Koved, L. and Shneiderman, B., Embedded menus: Selecting items
in context, Communications of the ACM 29, 4 (April 1986), 312-318.
Nielsen, J., Multimedia and Hypertext: The Internet and Beyond,
Academic Press, Cambridge, MA, (1995).
Norman, D. A., The Psychology of Everyday Things, Basic Books,
New York, (1988).
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., and
Carey, T., Human-Computer Interaction, Addison-Wesley, Reading,
Shneiderman, B., Designing the User Interface: Strategies for
Effective Human-Computer Interaction: Second Edition, Addison-Wesley
Publishing Co., Reading, MA, (1992), 573 pages.
Shneiderman, B., and Kearsley, G., Hypertext Hands-On! An Introduction to a New Way of Organizing and Accessing Information, Addison-Wesley Publishing Co., Reading, MA, (1989).