Understanding Manufacturing Systems with a Learning Historian
for User-Directed Experimentation

Gene Chipman, Catherine Plaisant, Sean Gahagan, Jeffrey W. Herrmann, Sara Hewitt
University of Maryland

Lakeisha Reaves
North Carolina A&T State University

Abstract

This paper describes a learning historian to improve user-directed experimentation with discrete event simulation models of manufacturing systems.  In user-directed experimentation, an analyst conducts simulation runs to estimate system performance.  Then the analyst modifies the simulation model to evaluate other possibilities.  An important characteristic is the ad hoc nature of the experimentation, as the analyst forms and runs new trials based on the results from previous trials.  Through user-directed experimentation designers compare alternatives and students learn the relationships between input parameters and performance measures.  Recording and reviewing previous trials while using simulation models enhances their benefits, transforming trial-and-error into learning.  The learning historian combines a graphical user interface, a discrete event simulation model, and dynamic data visualization.  Usability studies indicate that the learning historian is a usable and useful tool because it allows users to concentrate more on understanding system behavior than on operating simulation software.

1. Introduction

Discrete-event simulation models provide a powerful tool for understanding manufacturing systems, for designing manufacturing systems, and for training and educational purposes.  The simulation literature has emphasized the application of discrete-event simulation to solve a wide variety of important problems in manufacturing and other complex systems.  In addition, simulation is a very useful educational tool.  Students learning about the behavior of manufacturing systems (and queueing systems in general) can use simulation to gain insight into system performance and the tradeoffs that occur between different performance measures (such as throughput and WIP).  In both settings (practice and education), user-directed experimentation is a key technique. 

In user-directed experimentation, the analyst (or student) conducts one or more simulation runs (also called trials) to collect some data points about the system performance.  Based on these, the analyst modifies the simulation model to explore other alternatives near the previous point or in a different region of the search space.  (Note that the user is not interacting with the simulation model during the run.)  The designer synthesizes the results of these various trials to determine the best alternative.  The student uses the results to understand the relationships between model parameters and performance measures.  An important characteristic is the ad hoc nature of the experimentation, as the user forms and runs new trials based on the results from previous trials.

In our experience, however, using current simulation software packages to conduct user-directed experimentation is a difficult task.  Though they support detailed modeling and impressive animation, simulation packages have limited capabilities to support experimentation.  Thus, one must tediously modify the components of the simulation model, view and summarize the output data, and record the results until the design or learning task is accomplished.  Some simulation packages do allow the user to define a set of trials.  The software will then execute all of the desired runs without user assistance.  Unfortunately, this batch process does not support user-directed experimentation.  The user interfaces for simulation software must support a wide range of activities that are not relevant to the experimentation process (see Figure 3 for an example).

This paper describes a new technique to improve user-directed experimentation with discrete event simulation models of manufacturing systems.  This technique is a learning historian that provides the needed user interface for defining and running trials and viewing the results with minimal effort. 

The learning historian gives users the ability to record and review their actions, edit or replay the history of their actions, and annotate the history and send it to peers or instructors.  We conjecture that these operations encourage users to reflect on their actions and promote collaboration between users, which supports the learning process.

Previous work (described in Section 2) developed learning environments and a learning historian for interactive simulation, such as those of manufacturing equipment.  In such settings human decision-making and interaction with the system or simulation is a major component of the simulated process.  The objective of such human interaction can be to model a process for which there is no good symbolic model by providing expertise, but it can also be used as a tool for training.  These goals are somewhat interchangeable.  For instance, a flight simulator can be used to train pilots, but it can also be used with pilots to test changes in aircraft design.  On the other hand, discrete event simulation models restrict the user interaction to setting the simulation parameters prior to execution.  Yet the user shares the same basic goals of gaining a better understanding of the simulated system and using this understanding to design or learn.

This remainder of this paper proceeds as follows: Section 2 describes previous work on learning environments.  Section 3 describes the new learning historian.  Section 4 describes the results of two usability studies.  Section 5 concludes the paper.

2. A Learning Historian for Interactive Simulation

Reviewing past events has been useful in many domains.  Videotapes and flight data recorders provide invaluable technological help to sports coaches or aviation engineers. Similarly, we conjecture that providing users with a readable recording of their actions may help them monitor their behavior, reflect on their progress, and experiment with revisions of their experiences.  Visual representations of user's actions have been used to show user's navigation through web pages (Hightower et al., 1998).

The University of Maryland Institute for System Research has been developing simulation-based engineering learning environments for a variety of domains such as semiconductor manufacturing, hydrology, transportation, and manufacturing.  A development environment called SimPLE (Rose et al., 1998) was developed to facilitate the preparation of such learning environments.  It combines a simulation engine with a graphical user interface, guidance materials, and other scaffolding features.

VacSim, a SimPLE module, includes a prototype learning historian for a time-based simulation of a vacuum pump, which is used in semiconductor manufacturing. This module taught the procedure for producing extreme low pressure in a reaction chamber, through a sequence of actions that turns pumps on and off and controls vacuum channels by opening and closing valves (see Figure 1).  The underlying simulation model was created in VisSim.  When learners use the simulation, the historian records their actions and displays on a timeline the correspondence between the users actions and the effect on the pressures in the different chambers (see Figure 2).  A small user study showed that students found the history mechanism valuable when they used the simulation and used the annotation feature to communicate with others (Plaisant et al., 1999).

Figure 1: VacSim, an operational SimPLE module for a vacuum pump.

Figure 2: The visual historian with annotations: The bottom half indicates user actions, while the top half shows corresponding pressure values.

3. The Learning Historian for User-Directed Experimentation

For user-directed experimentation with discrete event simulation models of manufacturing systems, we created a new learning historian, which we will refer to as the Learning Historian for Manufacturing Systems (LHMS).  The learning historian gives users the ability to define and run trials, view the results, record and review their actions, edit or replay the history of their actions, and annotate the history and send it to peers or instructors. 

The LHMS is designed for users who are interested in the results of multiple simulation runs and using these results to compare alternative designs or to understand the relationships between model parameters and performance measures.  Such users are not necessarily interested in viewing the location and status of parts and equipment in the manufacturing system.  Thus, animation during a simulation run is not necessary.

The LHMS is a Java application that combines a graphical user interface, a discrete event simulation model, and dynamic data visualization.  Section 3.1 describes the user interface, and Section 3.2 describes the visualization of the results.  The LHMS requires an Arena simulation model, exported as a SIMAN file to reduce the processing effort needed to conduct a simulation run, and a configuration file that gives the LHMS important information about the structure of the simulation model and the controls that are available to the user.

The LHMS is designed to work with any ARENA model.  Space in a user interface is a limited commodity, but an ARENA model can have a large number of input parameters and output performance measures.  The configuration file indicates which parameters and performance measures the user will control and view.  Configuration management allows an instructor to focus student attention on key issues for a given lesson and gives the instructor flexibility to construct different lessons by using different configuration files with the same simulation model.  It also allows a designer to concentrate on key design parameters.

Figure 3: The ARENA user interface.

3.1 Interface Description

The LHMS has two user interface frames.  The control frame has controls for the historian application and the execution of the simulation. The historian controls allow for saving and reloading previous histories.  The simulation controls provide for execution of new trials, by allowing users to adjust the input parameter values and pressing execute.  A numeric display of the input parameter values and performance measures for the current trial is shown. 

The second frame is a timeline frame that provides a temporal display of the history of the trials and the controls for selecting a specific past trial for review.  When a past trial is selected, the values of the inputs used are shown in the control frame and they can be adjusted to specify a new trial.  Finally, users can annotate any trial by entering comments in a small text entry window, saving those annotations with the rest of the history, and sending the history to a colleague or instructor.

The frames appear next to each other on the display, allowing the user to match the individual timelines and the corresponding parameters.  Figure 4 shows these two frames.  In this figure, the timeline frame (on the right) shows the values of the input parameters and performance measures for nine trials.

After a user defines a trial by specifying values for each input parameter, the LHMS modifies the simulation model and executes the needed simulation runs without any further input from the user.  When the simulation runs are finished, the LHMS reads the output text files that the simulation engine creates and finds the values of the relevant performance measures.  (These are the ones specified in the configuration file.)  The values of the performance measures are stored with the trial and displayed in the timeline frame.  The trial is added to the history.

  

Figure 4: The LHMS User Interface.

3.2 Visualization of the History

To enhance the display of the history the LHMS is linked to an interactive scatter plot visualization tool called StarDOM (Starfield Dynamic Object Miner) (Cailleteau, 1999).  StarDOM is based on earlier work on dynamic queries and information visualization (Ahlberg and Shneiderman, 1994; Shneiderman, 1999).  StarDOM allows users to review a data set by easily constructing two dimensional displays that can be filtered interactively using sliders and buttons built automatically to match the inputs and outputs used by the historian.  The StarDOM display provides an overview of the results, the ability to zoom and filter them, and accessible details about each point.

StarDOM appears as a third frame in the historian and is automatically loaded with the results of the trials.  Figure 5 displays an example of the StarDOM display.  The user chose to map “Arrival Rate” on the horizontal axis and “Net Profit” on the vertical axis.  Each trial in the history is represented as a point in StarDOM.  The color and shape of the markers can be used to represent other attributes of the trials.  Selecting a point displays the trial’s corresponding input and output values in the LHMS control frame and highlights the timeline accordingly, making it easy to identify useful trials and modify them.



Figure 5: The StarDOM Display.

4. Usability Studies

We conducted two usability studies to assess how well users could perform certain tasks with the LHMS.

4.1 Pilot Study One

The first pilot study was conducted without the StarDOM display.  The participants were graduate students studying mechanical engineering and computer science.  Users had to run a trial using the default values for the input parameters, manipulate the input parameters, execute new trials, and compare the results of their experiments using the history of the trials.  After completing these tasks, users answered questions about the exercise.  Some tasks required the user to change an input parameter to a certain value.  Another task asked the user to find the value of an input parameter that optimized a specified performance measure.

The users in this study were able to use the LHMS with little help and made many useful suggestions and comments (Reaves, 2000).  Some comments were directed towards labeling and organizing controls on the historian. Other users criticized the bars in the timeline frame.  Users who were familiar with simulation and manufacturing systems indicated that the LHMS displayed the input and output variables in a much clearer format than other simulation software they had seen and thus the relationship between the input and output variables were easier to discern.  Users who were not familiar with manufacturing systems stressed the need for descriptive materials about the manufacturing system being simulated.

Based on this input, the LHMS was revised.  Introductory material about the manufacturing system being simulated was prepared, and StarDOM was added to improve visualization of the results.

4.2 Pilot Study Two

The participants in the second pilot study were students recruited from a production management class.  (These students were senior undergraduate students majoring in mechanical engineering.)  Participation was not mandatory, and it was made clear to the participants that the exercise was not to be graded and that their feedback and suggestions would help improve the LHMS.  Users worked in pairs and were encouraged to think aloud, voice their opinions, and state what was clear or unclear in the data or user interface. Users read a one-page description of the manufacturing system being simulated and definitions of key terms like arrival rate or variability.  The system produced three types of cast iron parts using a series of drill presses, cleaning, welding, and testing stations.  After a very brief demonstration of the historian features (about one minute) each pair of users began a set of tasks.  The first four tasks were simple tasks requiring the users to adjust input parameter values and execute trials.  These helped the users become familiar with the interface.  The next three tasks were more complex tasks that asked the users to conduct additional trials and draw a simplified graph showing the relationship between performance measures and input parameters (e.g. throughput versus arrival rate, or cycle time versus arrival rate).  An observer recorded what the users did and administered a questionnaire when the users completed the tasks.  To ensure that the users did not leave the session with the wrong interpretation of the results of their experiments, the observer explained the correct answers after the exercise.  Each session lasted for approximately 45 minutes. 

With the new visualization frame all students were able to use the LHMS without serious problems.  We observed that users more frequently used the StarDOM display to analyze the data, as opposed to the timeline display, which was used only to go back to previous trials and modify the values of input parameters.  StarDOM’s larger display allowed better and more precise comparison of the results.

An encouraging result of the study is that all users commented very positively on the ease of exploring the manufacturing system behavior.  They said that using the simulation software directly would have required them to keep track of the input parameters and performance measures manually and calculating differences between values.  When asked to describe the relationship between two variables, students unfamiliar with formal experiment design started by trying a few randomly chosen values and then realized that they needed a more organized way of varying the parameter values.  Several teams mentioned that the LHMS should provide ways to specify ranges of values and run a series of trials.  This showed that some users saw a need for experimental design.

On the other hand we noted some potential pitfalls of this open-ended environment. Some users forgot to filter the trials to eliminate incompatible data points.  For example, when asked to look at the relationship between the variability of arrival rate and the cycle time, all users ran a series of trials nicely varying the variability for a given arrival rate, but one pair then analyzed the entire history, combining the new trials with previous trials that had used different values of other input parameters, which could have led them to a misleading graph.  Another pair tried to reach a conclusion about relationships with too few trials and constructed an incorrect answer.  These pitfalls would be lessened by incorporating integrated help and tutorials, especially if such help included design of experiments.

5. Summary and Conclusions

This paper described a new technique to improve user-directed experimentation with discrete event simulation models of manufacturing systems.  This technique (the LHMS) is a learning historian that gives users the ability to define and run trials, view the results, record and review their actions, edit or replay the history of their actions, and annotate the history and send it to peers or instructors.  The LHMS is designed for users who are interested in the results of multiple simulation runs and using these results to compare alternative designs or to understand the relationships between model parameters and performance measures.  The LHMS is a Java application that requires an Arena simulation model and a configuration file for input.  The LHMS user interface includes a control frame, a timeline (history) frame, and a visualization frame (StarDOM).

Based on the results of usability studies, we believe that the LHMS is a usable and useful tool for user-directed experimentation.  It supports the needs of designers who need to identify the best system configuration and the needs of students who are trying to understand the relationships between input parameters and performance measures.  The LHMS allows users to concentrate more on understanding the principles and to spend less time operating the simulation software.

The current LHMS is a prototype implementation.  We are currently designing the next version to overcome some technical problems and to add more features.  Improvements will include additional annotation capabilities, groupwork features that help users share history files, the ability to merge and split history files, and searchable repositories of histories.  We also need to explore the usefulness of creating experimental designs and the best user interface for this type of function.

We can also link the LHMS to heterogeneous simulation models of semiconductor wafer fabrication facilities (see Herrmann et al., 2000).  This would combine effective user interface and visualization technologies with powerful techniques to model complex dynamic systems.

Finally, the LHMS is designed to change input parameters of the simulation model, which implies that constructing the simulation model must make available input parameters that allow the user (through the LHMS) to analyze relevant design alternatives and interesting system behavior.  This is not always straightforward, since modifying complex production control rules or structural characteristics might involve many interrelated input parameters.  We are concurrently working on schemes to represent production control policies that can be adapted more easily.

Acknowledgments

This work was supported by the National Science Foundation through the Collaboration for Industrial and Systems Engineering.  The participants are North Carolina A&T State University, University of Maryland, and Georgia Institute of Technology.  Additional support was provided by Northrop Grumman through the Institute for Systems Research Industrial Fellowship Program.  The authors thank Anne Rose for her help with the design and development of the learning historian.

References

Ahlberg, C., Shneiderman, B. “Visual Information Seeking: Tight coupling of dynamic query filters with starfield displays,” ACM CHI '94 Conference Proc. (Boston, MA, April 24-28, 1994) 313-317, ACM, New York, 1994.

Cailleteau, L., “Interfaces for Visualizing Multi-Valued Attributes: Design and Implementation Using Starfield Displays,” HCIL technical Report, 1999. Available online via www.cs.umd.edu/hcil

Herrmann, J.W., B.F. Conaghan, L. Henn-Lecordier, P. Mellacheruvu, M.-Q. Nguyen, G.W. Rubloff, R.Z. Shi, “Understanding the impact of equipment and process changes with a heterogeneous semiconductor manufacturing simulation environment,” Proceedings of the Winter Simulation Conference, Orlando, Florida, December 10-13, 2000.

Hightower, R. R., Ring, L., Helfman, J., Bederson, B. B., & Hollan, J. D., “Graphical Multiscale Web Histories: A Study of PadPrints,” Proceedings of ACM Conference on Hypertext (Hypertext 98) ACM Press, pp. 58-65, 1998.

Plaisant, C., Rose, A., Rubloff, G., Salter, R., Shneiderman, B., “The design of history mechanisms and their use in collaborative educational simulations,” in Proc. of the Computer Support for Collaborative Learning, CSCL' 99, Palo Alto, CA, 348-359, 1999. Also online at http://sll.stanford.edu/CSCL99/

Reaves, Lakeisha A., “Designing a Learning Historian for Manufacturing Processes,” Technical Report U.G. 2000-4, Institute for Systems Research, College Park, Maryland, 2000.  Available online at http://www.isr.umd.edu/TechReports/ISR/2000/UG_2000-4/UG_2000-4.phtml

Rose, A., Eckard, D., Rubloff, G.W. “An Application Framework for Creating Simulation-Based Learning Environments,” University of Maryland Technical Report CS-TR-3907, 1998. Available online via www.cs.umd.edu/hcil

Shneiderman, B., “Dynamic queries, starfield displays, and the path to Spotfire,” available online at http://www.cs.umd.edu/hcil/spotfire/, February 4, 1999.