Evaluation of Platform Independent
User Interface Builders

Richard Chimera

Human-Computer Interaction Laboratory
University of Maryland
A. V. Williams Building
College Park, MD 20742-3255 USA
carm@cs.umd.edu
301-405-2757
fax 301-405-6707
March 1993


ABSTRACT

Platform independent user interface builders allow GUI applications to be created on one platform; that one set of specifications and source code can then be automatically compiled for several other platforms without modification (potentially). The implications for minimizing software development, porting, and maintenance issues are monumental. I evaluated four commercially available tools with rigorous and objective processes and evaluation forms, and provide those results along with opinionated insights. I determined which of these tools and methodologies are viable for creating portable, medium- to high-complexity GUI applications. This is a fast moving area and the impact of these technologies can shape the nature of creating software for open systems today and into the future.

1 Introduction

This document describes the first phase of an ongoing project to form GEIS' new, more efficient software development environment which is critical to maintaining a leadership position in the information services market. This was carried out at the University of Maryland Human-Computer Interaction Laboratory by faculty research assistant Richard Chimera and supervised by Dr. Ben Shneiderman. Phase I was the evaluation of commercially available user interface builders that support multiple hardware, operating system, and windowing system environments, or "platforms." Thus we have the term platform independent user interface builders, or PIUIB. (In clear context the words "platform independent" may be dropped or the words "builder" and "tool" interchanged for the sake of clarity through brevity.) Working with the User Interface Group and others at GEIS, the HCIL was charged to review available PIUIBs, select the elite few to evaluate rigorously and objectively, and to recommend if and which PIUIB GEIS should adopt.

1.1 Motivation

In GEIS' current software development environment, which is true of almost every software producer, each product supported requires a unique team of developers for each platform on which the software is delivered. It is very costly to maintain products when capability changes must be performed on multiple sets of code. This duplication of effort costs money and limits the number of products offered. A new development environment that supports platform independence will not only improve the quality, usability, and consistency of the interfaces GEIS designs for their products, it will help to significantly reduce costs and "time to market" and therefore allow them to offer a broader range of services to customers. By being a leader in this technology as it reaches viability, GEIS may significantly increase its advantage over competitors who do not inculcate PIUIBs into their software development environment.

There are more benefits of using today's PIUIBs. In contrast to early entrants into this field that were limited by what has become known as the "lowest common denominator" effect--provide only the functionality common to all platforms, the leading products today incorporate a superset of the functionality found on individual platforms. This is an advantage because a good concept found on only one platform is abstracted and offered on all platforms by the PIUIB. Another benefit is that new look-and-feel's or new releases to current ones are automatically supported by the PIUIB with no modification to application source code (though recompilation would be necessary).

1.2 Perspective

The most important aspects of a platform independent user interface builder concern software construction methodology/infrastructure and portability issues. Classic user interface issues, which transcend those first two, are next in importance. In practice, these three issues are often inseparable in that the salient features of a PIUIB are its user interface objects and construction methodology as supported by the underlying software engineering and infrastructure and porting paradigms. Phase I of this project looked at the PIUIBs in the large while keeping an eye to smaller concerns.

Evaluations were executed from the perspective of learning a tool's big picture and enough of the details in order to implement the front end to a typical organization communication application--a subset of GEIS' Business Talk 2000 product. The application required creating associated windows, dialog boxes, hierarchical menus, buttons, scrolling lists, editable textviews, and icons/graphic labels.

2 Project Components and Processes

2.1 Selection Process

The selection process started with collecting information about possible products from trade magazines, colleagues, and marketing material from vendors. We collected information on over 50 products, some of which did not cover all the platforms in which we were interested (appendix A). The products were rated via 19 selection criteria -- four "must-have" attributes, ten important attributes, and five attributes that would be major pluses.

Must-have Attributes

1. Generates interfaces for at least two platforms differentiated by hardware and operating system; at least one platform is not UNIX.

2. Has facilities for prototyping GUIs "easily" (we have in mind an interactive design tool with widget set).

3. "Chameleon" behavior, which is defined as the ability to conform to the various look & feel standards on each platform.

4. Company's track record. Issues include how long company has existed as software developer, how long current product has existed, installed user base, number of employees, revision frequency, etc. This is mostly to capture highly negative features of the company, and is more subjective and less researched than it should be.

Important Attributes

5. Ability to build own widget primitives from supplied widgets.

6. Can develop on HP/Unix platform.

7. Generates interfaces for Windows 3.X.

8. Prototypes not necessarily throw-away, they can be extended to be an "efficient" final application.

9. Uses direct manipulation as the main building process.

10. Can merge application code into iterative design versions easily.

11. Provides platform-specific industry-standard widget combinations for high-level objects, i.e. dialog boxes.

12. Ability to create own icons.

13. Maintains and uses user developed widget collections as templates or libraries.

14. Performance speed good; interoperability within platform such as cut&paste, access to operating system and native window system, etc.

Major Pluses

15. Uses OOP and can generate C++ code.

16. Allows for surface level end-user customization.

17. Updates for package itself are backwards compatible.

18. General user satisfaction with building and execution issues.

19. Reference checks.

The must-have selection criteria forced the bulk of the products to drop out of contention mostly due to a lack of platform coverage or weak GUI facilities. This narrowed the pool down to eight products worthy of further investigation: Aspect, CommonView, GUILD, JAM, Open Interface, OpenUI, WNDX, and XVT. Since this was still too large a group to evaluate deeply and they all satisfied the must-have criteria, we rated these eight tools on a four point scale for seven of the ten important criteria (appendix B). This, along with our desire to evaluate tools representing differing methodologies and to favor tools with object-oriented design and an interactive widget layout feature, led us to the tools we evaluated: Open Interface, XVT, GUILD, and OpenUI.

We knew this type of evaluation would carry on into the future and would be of importance to many divisions at GEIS and many professional communities, thus we realized the power of making the review process as objective as possible. Review objectivity would allow many things:

Of all the project components, the most important was the actual evaluation of a PIUIB and this is given its own section below.

2.2 Learning and Use

A two week period was allocated to learn and use each PIUIB. Only one at a time was attended so as to minimize confusion over PIUIB characteristic attribution. The author was the sole reviewer for all five tools. The various parts of the installation, tutorial, prototype creation, and porting were timed to the minute with a stopwatch and rated on various scales. In most cases, the first platform used was UNIX/Motif, and then the prototype was ported to the PC/Windows 3.0 platform.

The learning and use of the PIUIB consisted of these phases:

1. Open the shrink wrap of the "Read This First" document or installation instructions and understand them.

2. Install all software.

3. Read tutorial and follow interactively with the tool as requested.

4. If necessary read additional documents to achieve a comfortable level of general knowledge of the tool; this was the case for all tools. This could involve further issues on one or more of the interactive widget layout program (all PIUIBs reviewed provided one), user's guide, or programmer's guide.

5. The heart of the process -- build the application prototype. The prototype was specified by GEIS personnel via both a text description and a set of Windows look & feel screen snapshots of the existing Business Talk 2000 product. Functionality included:

6. Test the prototype within the interactive layout program itself, if possible.

7. Compile and run the prototype, go back to modify the prototype, and recompile to determine handling of iterative development.

8. Stop the building process with 2.5 days left. Install the PIUIB on the second platform. Transfer the prototype specifications to the second platform and load into the interactive layout program to corroborate its basic functioning and to check that the prototype specifications were parsed and laid out correctly. Then compile and run the prototype on the second platform to finish the port. One day was allocated for the installation and complete port to the second platform.

9. Spend the last 1.5 days filling out the evaluation forms and essays described in the next section.

2.3 Evaluation

We wanted to get answers to these characteristics of a PIUIB:

The primary difficulty with these questions is to find their answers in terms of objective definitions and measures. After much searching through the user interface, human factors, and software engineering and metrics literature, we discovered that Dr. Deborah Hix and her colleagues at Virginia Tech had created an evaluation tool very close to what we wanted. It is a tool for evaluating user interface development tools in general. It does not specifically address issues concerning platform independence or traversal. It was designed and validated as being a reviewer-independent form that could be used to compare different UI development tools running on different platforms reviewed by different people.

2.3.1 Dr. Hix's Evaluation Tool

Dr. Hix's evaluation tool is best described by her own account as reported in Communications of the ACM, March 1991. Hix's tool has been used by over 100 organizations since its inception two years ago. This section is not meant to replace her own introduction and descriptions, but to provide some context and explanation for how we have decided to use the tool in our evaluations. Dr. Hix's evaluation tool consists of 35 spreadsheet-like pages with issues as rows and evaluation criteria as columns. Rows are organized by user interface construction content areas such as menus, text, graphics, devices, etc. Columns are organized by implementation techniques for affecting the current issue (row), binary values for direct manipulation, text-coded language, and other techniques. There is an additional column organization set aside to rate the ease of performing the current task with the tool, which uses a three-level scale. The tool covers a wide range of topics in order to touch on as many areas as is tractable that are relevant to building user interfaces. Naturally, the tool does not fit perfectly with our needs; it covers some areas in which we are not interested as well as not providing enough coverage in some areas in which we have high interest.

Dr. Hix's evaluation tool constitutes a general methodology for quantitatively evaluating user interface development tools. Again, we feel an important part of this project is to create and use an evaluation process that is as objective as possible. This will be critical for future comparative or replicated research--if evaluations of other tools are performed by the same reviewer, if the same tools are evaluated by a different reviewer(s), if a new version of the same tools is evaluated by the same or different reviewer(s), etc. This quantitative approach can even be used to compare objectively these platform independent GUI builders with platform specific GUI builders, avoiding much of the concern as to whether apples are being compared to oranges.

We have shaped the Hix tool to better suit our needs. We have eliminated entire pages/forms that have little or no relevance to our current evaluation (issues of animation, multimedia, etc.). For pages of the tool in which we have interest but the form itself does not provide the appropriate perspective or coverage, we have instead composed small essays on the topic using the form as a structuring guide. We use the remainder of the pages in standard fashion. This involves assigning weights to each line item so as to specify the importance of that line item in relation to other line items in calculations of summary totals. We defined a weight of 1 to mean that we are not interested in that line item. A weight of 3 indicates that we are interested in that item, but we are not evaluating it specifically in building the prototype. We rated the item as well as we could from information (including users/references) provided about the tool by the company and other reviews of the tool. A weight of 5 indicates that we are interested in the item and are evaluating it in building the prototype. The pruned Hix tool we used appears in appendix C with its own glossary at the end. In a few cases, we have redefined a term as given in Hix's glossary, some only slightly and some with a related but arbitrary complete redefinition. Our new term definitions follow Hix's glossary.

The Hix tool uses a spreadsheet to input data and create summary reports using various macros they have defined. It is important to note that a current limitation of the Hix tool is that if any item on a page is used, the entire page is used in calculations. This may show up as low marks for a certain area simply because many items on the page were not weighted highly thus reducing the sum potential. Though not yet investigated, we could create our own macros to have the important items carry a more representative weight or eliminate unwanted items with respect to summary analyses.

2.3.2 Additional Evaluation Forms

The Hix tool doesn't address all our needs, so we used additional evaluation techniques. One was structured essay questions, most based on the Hix questions that did not cover deeply enough issues of which we were concerned. Another was a free-form catch-all essay of unusually good or bad aspects of the PIUIB. Another technique was a modified QUIS (Questionnaire for User Interaction Satisfaction) form, which collects subjective, emotional data in an objective manner. We added questions to the QUIS where it is weak with respect to this project, e.g., questions regarding objects and models supplied by the PIUIB, its use of direct manipulation, and porting procedures. These supplemental forms are in appendix D.

Hix's tool does not address time issues and we wanted measures of effort that were as objective as possible. Therefore we created a Hix-like form as another supplemental evaluation technique. This page was also spreadsheet-like with rows that covered the the majority of the install, learn, and build process and columns for preparatory time, performance time, and times for finding, reading, and applying help. Additional columns were for ease of doing the activity and for the porting issues of look, feel, doing, and help. Our Hix-like form is in appendix D, also.

As a pilot test of the critical phases of the building process and evaluation process and forms, the prototype building and evaluation phases were performed once with a non-platform independent user interface building tool with which the author was already familiar, Developer's Guide by Sun Microsystems. Minor changes were made to our Hix-like evaluation form.

The creation of the evaluation process and forms was the most difficult aspect of this project, and unfortunately was somewhat disappointing in terms of focused results and analysis. The current evaluation forms are an excellent start and provide data that are quite valid. There should be wider and deeper coverage for issues peculiar to platform independence as well as the interrelated roles of the interactive editor, resource language, and API, among other lesser deficiencies. The more useful evaluation results are the author's detailed experiences, essays, anecdotes, and recommendations.

3 Evaluation Results Summary

It is difficult to digest all the evaluation data collected during the reviews. Table 3.1 in section 3.1 attempts to summarize crucial aspects by combining relevant evaluation data, deriving numerical values for the twelve evaluation criteria, and weighting and summing them to produce a final rating value. Brief comparative analyses that support table 3.1 are in section 3.2. No statistical analysis has been performed on the evaluation data because five is too small a sample size.

The detailed evaluations in descriptive form for Open Interface, XVT, GUILD, and OpenUI are chapters 4 through 7, respectively.

The detailed, raw evaluation data in its original form for Open Interface, XVT, GUILD, and OpenUI are appendices E-1 through E-4 respectively.

3.1 Evaluation Results

I conclude that Open Interface's power, methodology, and maturity suits GEIS' technical needs and recommend its adoption. XVT is a solid tool with minimal surprises but is too conservative and lacks broad GUI coverage for full-scale use by GEIS. GUILD and OpenUI are not suitable for GEIS due to their esoteric flavor or narrow characteristics.

It is most interesting to note that the clear frontrunner, Open Interface, began as an internal effort and was later cleaned up and made into a commercial product. Perhaps the fact that their own personnel had to use their PIUIB to create a variety of platform independent client applications is a significant reason for the superior power, methodology, extensibility, and documentation of their tool compared to the other tools reviewed.

The rows of table 3.1 list items of general information, prototype building milestones and timing data, and the twelve evaluation criteria; the columns are the five tools. Most items should be self-explanatory. "Port Install & Load Time" is the time it took to install the PIUIB on the second platform and load the prototype specification file into the interactive layout editor to see how everything looks. Most PIUIBs didn't make it through a complete compile of the prototype on the ported platform for various reasons, among them that the HCIL has outdated compilers for the PC which are not supported by the PIUIBs. However, loading the prototype specification file into the interactive layout editor is a reasonable approximation under the circumstances.

3.2 Brief Comparative Analyses

This section provides supporting material for the values found in table 3.1.

Ease of Learning

XVT was the easiest tool to learn because it has the least amount of functionality which is presented in a straightforward manner. Open Interface and OpenUI use object-oriented techniques and are more difficult to learn than XVT. Open Interface is harder because it has so much functionality to learn. OpenUI uses its own version of message passing and further abstractions to invoke functionality which requires time to learn its specifics. GUILD is very hard to learn because it uses strict, pure object-oriented communication yet the documentation is so lacking that the mechanism specifics are basically unknown.

Ease of Use

Once learned and a certain comfort level is reached, the PIUIBs can be used similarly effectively and straightforwardly. XVT loses a point for not having any testing within their interactive design tool and forces the user to invoke a shell, compile, and then test. Both GUILD and OpenUI lose a rating point for not describing their message passing mechanisms clearly and for making certain actions needlessly obtuse. OpenUI loses an additional point for its abstract widget layout mechanism of using a logical grid system that, while not misguided, at least ended up in the wrong form.

Documentation

Open Interface's documentation is well organized, has excellent presentation style, and is voluminous. XVT had occasions where related issues were omitted or full disclosure was not provided or referenced. OpenUI's approach was rather simplistic and gives one the feeling there should be a further, large companion document for programmers. As noted earlier, GUILD's major downfall is its documentation which is sorely lacking purposeful direction, coverage, and explanation of the large (and probably powerful) object messages set.

Online Help Facilities

GUILD clearly had the best online help facilities but they weren't enough to make up for what the documentation lacks in high-level explanations. It was context sensitive, indexed, and had hypertext features. XVT's online help was good but not very context sensitive. Open Interface openly admitted there was no online help available nor is it (straightforwardly) producible for built applications. The curious thing about OpenUI is that it has what seems to be elaborate help mechanisms for the application authors, yet don't provide any online help for their own tool!

Technical Support

XVT's technical support was efficient and professional; the trouble call was received, delegated, and responded to within two hours. GUILD's technical support was even more conscientious, but one got the feeling there's only a handful of programmers who must also double-duty as technical support. OpenUI had competent and reasonably efficient tech support. Open Interface gave a cold, standoffish reception and took overnight to initially respond to the trouble call and yet another day to answer it.

Range and Capabilities of Supplied Widget Set

Open Interface has a vastly superior widget set compared to all others reviewed, including two hierarchy browsing widgets. XVT's widget set is slightly above average though more programmer intervention is necessary than should be to make everything work. GUILD also has a slightly above average widget set that functions reasonably with promising graphics features. OpenUI has the weakest widget set of the PIUIBs evaluated.

Prototype Evolves into Final Application Program

Open Interface handles evolution the best, most notably by leaving intact code modified by the user as it regenerates the same file in response to additions/changes made in the interactive layout editor. GUILD and OpenUI handle evolution well due to their extremely minimal use of C code. XVT has absolutely no mechanisms for incorporating evolutionary changes, all code merging must be done by hand. All PIUIBs treat their end product as the final application program not a throw-away prototype, whether or not the evolution aspect is handled well.

Conformity to Style Guide of Each Platform

The PIUIBs conformed to the platforms' various look & feel standards. The only significant departures were by Open Interface in the menu behavior. Menu items did not unhighlight properly in Motif or OpenLook leaving a rather ugly menu bar. It is because Open Interface emulates look & feels rather than use their APIs that this happens, but it should be easily fixed.

Briefly described, there are two ways to have an application to be in conformance with a look & feel standard. The obvious way is to use the look & feel's API which naturally guarantees perfect conformity for primitive objects. The other way is to use your own routines to emulate the look and the feel of a standard. This is a long and difficult process, but has one big payoff. There can be a significant gain in speed if the emulation routines are efficient and go down to the most elementary display routine level, therefore bypassing potentially many layers of library calls. For example, a Motif application using the Motif API is built upon many layers: Xlib, Xt, and Motif. Adding to that the UIB's API is yet another layer. But with emulation, the Xt and Motif layer are eliminated which can produce a significant speed gain.

Chameleon Ability

All PIUIBs have the ability to take on multiple look & feel's. Open Interface has the unique ability to change interactively to any look & feel it supports within the design tool (except for the Macintosh look & feel on non-Macintosh platforms due to legal concerns)! Open Interface emulates the look & feel standards rather than using their APIs so that it can do this.

Speed and Size of Final Application Program

No attempts were made to measure speed of execution of the prototype. Running applications were only completed on Unix which is less affected by application size than other platforms.

Interoperability: conformity to industry standard on each platform

All the PIUIBs' interactive design tools used platform standards to perform cut and paste routines. This was the only interoperability test conducted.

Subjective Opinion

It is this author's opinion that Open Interface is the best all-around PIUIB for creating typical office information and communication GUI programs that are robust, versatile, highly interactive, and maintainable. XVT is the second best tool limited by its conservative offerings. Both GUILD and OpenUI need to have their documentation overhauled and their message passing mechanisms made coherent. Until that time, they would be difficult to use to build medium- to high-complexity GUI programs.

4 Open Interface Evaluation

4.1 Executive Summary

Open Interface version 2.0 by Neuron Data is the frontrunner, enjoying a large lead over the other PIUIBs. Open Interface is strong in terms of software infrastructure, methodology, documentation, polished presentation, and completeness. Its strengths include advanced object-oriented design; the largest and most useful widget set; powerful and malleable interactive layout features; printing, string, and memory management support; and extensibility. Its weaknesses include menu display glitches and other small deviations from look & feel standards (due to its decision to emulate, not to use, the APIs of the standard window systems), and the company's consistently poor attitude towards customers.

4.2 Installation and Learning

The installation from tape was smooth and the instructions long-winded but clear. Open Interface has an extra step that requires one to run an install script that completes the environment and tool setup. They provided manual checks to make sure that the installation was completed correctly.

The tutorial ("primer") sections they provided were clear, organized, and followed well the principle of progressive disclosure. The documentation used a very convenient and informative presentation style for advancing the example tutorial code. For new code that was inserted, a bold typeface was used. Deletions from the previous code were printed with a strikethru typeface. When lines were moved, the old location used a strikethru, underline, and bold typeface; the new location used both an underline and bold typeface. With these presentation styles, the changes in the code necessary to create the new tutorial functionality were easily identified and the learning process was accelerated.

4.3 Documentation, Online Help, and Tech Support

When the product arrives, one gets a sense of completeness as the two volume, 2,000 page documentation set makes a resounding thud on the desk. Then one gets a sense of professionalism as the documentation organization and presentation are absorbed. The unique presentation style of the tutorial text, as mentioned earlier, was outstanding.

There is no online help facility in Open Interface. As for technical support, they receive a low rating for taking too long to get back to me after the initial call and were somewhat cold and uncaring in tone. The sales staff treated me in a similar manner.

4.4 Interactive Design Tool

OpenEdit is a powerful interactive design editor with which to build graphical user interfaces. It provides the largest widget set of the PIUIBs reviewed including a hierarchy browser and overview widgets. OpenEdit itself is written using Open Interface, a clever advertisement for the power of their tool. (Note: all PIUIBs reviewed wrote their interactive editor using their own tool!) There is an abundance of options and controls to choose among to further specify the behavior of various widgets. Indeed, sometimes almost an entire OpenEdit window changes form to accommodate specifying certain options and characteristics, and one can overlook the radio buttons that trigger this. Though after using OpenEdit for a while, this inconspicuousness is not a problem.

4.4.1 Windows

The module is at the highest level of Open Interface specification and corresponds directly to a window in an application. For every window a new module needs to be created, which is a slight design drawback since the module should be able to encapsulate any and all functionality for an abstract part of an application that may happen to need two or more windows. Below a module come the various encapsulation objects for menus, dialogs, buttons, strings, etc. There are check boxes in OpenEdit for window attributes such as zoom boxes, resize capabilities, etc. Automatic MDI functionality is possible with Open Interface.

OpenEdit takes navigation among objects in a window one step further than most PIUIBs. It supports the concept of a visit order that is independent of both geometry and creation order of objects, two common ways other tools set the visit order. OpenEdit provides a scrolling list of object names that is manipulated to specify the visit order. The last item wraps around to visit the first item, and navigation can go in reverse order with the back-tab (Shift Tab) key.

4.4.2 Menus

The way menus are defined is at once powerful and cumbersome. The power includes ways to specify hierarchical menus; checkable menu items, both independent and one-of-a-group items; menu mnemonics; menu separators; as well as allowing icons to appear in menu labels between the menu text and keyboard equivalent label. This is powerful functionality and should not be overshadowed by the longer explanation of the menu problems immediately following.

The cumbersome aspect is with keyboard equivalents. Firstly, for a menu item to have a keyboard equivalent involves defining two objects. Each keyboard specification (such as ^O or F8) must be defined as an object. Menu items take lists of keyboard objects so each keyboard object must be put into a list object that can be associated with a menu item. Though this is the most general approach, allowing any of N keyboard equivalents to invoke one menu item, most applications need only one keyboard equivalent so that the extra list object is a frustration. Secondly, the string that is displayed as the keyboard equivalent label must be supplied rather than automatically generated by OpenEdit. More than a nuisance, it is a look & feel portability problem because each displays the same keyboard equivalent differently. This can be worked around using the Open Interface string resource, but modifications would involve manipulations in two logical places rather than one. Thirdly, the checkable menu items and radio menu items only had their look specifiable within OpenEdit, the feel must be programmed. In other words, when any type of checkable menu item is chosen, several lines of code must be written to invoke the proper Open Interface functions that turn off an on check marks.

Finally, there were inexcusable display glitches of all menu bar items and menu items in the Motif look & feel. Whereas this is not a problem with the methodology like the first three issues, it is ugly and brings in to doubt Open Interface's ability to emulate every aspect of any look & feel.

4.4.3 Dialogs

Open Interface treats dialog boxes as just another widget. They offer good access to various attributes that control the behavior of a dialog box such as whether it is modal, its size, its location, of course its contents, etc.

4.4.4 Icons

OpenEdit has its own icon editor that allows an icon to be designed and built once and automatically ported to all platforms. The icon editor is fully functional and it took only ten minutes to create a mail in-box icon. Icons can also be embedded in menu item and button labels smoothly.

4.4.5 Other Notable Widgets

Open Interface provides a multiple row and column scrolling list object with variable size rows and columns. This object supplies all scrolling list functionality and handles textual or image data. The application prototype required the use of this type of object in single column mode only, thus its full functionality was only read.

Two more powerful widgets are a hierarchy browser and hierarchy overview widgets. Naturally they can be used together to provide a form of fisheye view that is very helpful for visualizing large hierarchies. These widgets are used quite appropriately in OpenEdit itself to view the application hierarchy being created. This is not only useful but almost necessary since Open Interface uses hierarchic object-oriented design to a large extent.

4.4.6 Notes on Building the Application

Everything encountered in the prototype specifications was straightforwardly handled in Open Interface and OpenEdit. Everything but menu item keyboard equivalents was handled quickly and practically effortlessly. Icons in menu items were a breeze. Arranging, grouping, and aligning widgets was simple. Password masking is a simple check box. "Combo boxes" are a supplied widget. A makefile can be generated for any platform, and the one I generated for HP/Unix compiled the first time without modification. It was a pleasure to use OpenEdit and Open Interface to build the prototype.

4.4.7 Miscellaneous

Another excellent aspect of OpenEdit is the ability to specify resize behavior for any object. This is done with the concept of lock bars. There are four lock bars external to an object--one on each side of the rectangular bounding box-- and two lock bars internal to an object--one vertical and one horizontal. The presence of a lock bar keeps display location (external) or display size (internal) constant. For instance, if the vertical internal lock bar is present and the horizontal internal lock bar is not present, a window resize will change the size of the object in the horizontal direction and not in the vertical direction. I could not specify that only one of the four external lock bars be present and I was not sure if this was a bug or a feature of OpenEdit. Objects could not be locked to one another, only to window or panel borders, but this combined with object grouping would seem to cover a majority of application needs.

OpenEdit was robust and never crashed or lost changes. This cannot be said of any other tool reviewed. The full evaluation data collected are appendix E-1.

4.5 Prototype Evolution

Code generation and regeneration capabilities are strong. OpenEdit generates stylistically clean code with good comments, and supplies appropriate switch statement cases for events of interest to an object. The regeneration of code is very carefully done so as not to wipe out additions to previously generated code that performs application-specific functionality. Complete makefiles are also generated, adding new files and dependencies when appropriate. Evolutionary development of an application is supported superlatively by Open Interface.

4.6 Look & Feel Conformity

This is one area in which Open Interface needs some work. They do a rather good job of emulating every popular look & feel standard to the point where one can see any look & feel on any platform! That's right, see Windows while the host computer is in OpenLook and vice versa. That is the good news.

However, there are two quite noticeable deviations from look & feel standards that were discovered. One was menu display glitches of the Motif style. Menu items both in the menu bar and pull-down menus themselves did not unhighlight properly, leaving ugly remnants of the 3-D effect in the menu items and menu bars. One would assume this will be fixed in future releases of Open Interface. The other deviation is the feel of the OpenLook submenus. Open Interface interprets the OpenLook guidelines differently than the popular OpenWindows implementation of OpenLook does. Open Interface forces the mouse to touch the triangle that represents the submenu rather than having a sweep right gesture of N pixels activate the submenu. While this may not violate the explicit OpenLook guidelines, it remains a curiosity as to why they would choose to do it differently than OpenWindows.

4.7 Extensibility

Extensibility is a key feature of Open Interface. They allow for subclassing of existing widgets with the API, and provide mechanisms for importing them to be first class objects in OpenEdit, as well.

4.8 Non-GUI Features

Open Interface provides strong abstractions for string, printing, and memory management support. The only file support is prepared file/directory choosing dialog boxes.

4.9 Porting Issues

Though we do not have a compiler that Open Interface supports on the PC, I was able to install Open Interface without a hitch and load my application specifications into OpenEdit and see that everything was there and looked as it should for the Windows 3.0 look & feel. When I interactively switched to the Motif and OpenLook standards, the color and shading palettes were a bit wacky and the screen was quite hideous and barely legible with red and greened checkered patterns, etc. There is probably a straightforward fix for that situation, but my evaluation time was used up by that point.

5 XVT Evaluation

5.1 Executive Summary

XVT version 3.0 by XVT Software, the grandfather of tools and companies in this field, falls in line behind Open Interface. It is a solid tool with good interactive layout features; file, printing, string, and memory management support; and an above average widget set. Some standards bodies consider some aspects of XVT in some decisions. It doesn't offer much in terms of methodology, iterative development, or extensibility.

XVT is the first commercially successful PIUIB, thus it seems to be the benchmark by which all other PIUIBs are judged. For low-complexity GUIs, this is okay and XVT should be commended; but almost no one cares about low-complexity GUIs anymore, sophisticated GUIs are becoming the norm for market leading companies. For the medium- and high-complexity GUIs, it is time to find a better benchmark. XVT's conservative offerings and lack of methodology and infrastructure are goals beyond which PIUIBs should aim.

XVT uses the greatest common denominator approach to platform independence. Only when all platforms have a similar functionality does XVT provide it. (Note: most others in the community name this the "lowest common denominator" approach because they just can't use the word "greatest" when describing a negative feature.) XVT leaves the methodology rather open for the programmer to do much using normal C code practices. There is no object-oriented design and only primitive structures for the programmer to handle user interaction.

5.2 Installation and Learning

The installation of XVT went very smoothly. The preparatory reading lasted only 45 minutes and there was no post tape-loading steps to take. The tutorial for the XVT system lasted 60 minutes and the tutorial for XVT-Design (the interactive editor) lasted only 90 minutes. In other words, the methodology is simple, the interactive editor is simple. An additional 50 minutes was spent becoming comfortable with XVT-Design and another 90 minutes was spent getting comfortable with XVT before embarking on prototype development.

5.3 Documentation, Online Help, and Tech Support

XVT's documentation was sufficient but not great. It lacked coverage for explanation of items in the proper places and didn't much mention associated objects or concepts when appropriate. The online help facility was utilized throughout XVT-Design, but was not very context sensitive. It provided what I was looking for about half the time. XVT's technical support was the best of any PIUIB evaluated. It was highly organized, the problem is assigned to an appropriate person and they respond immediately to tell you they're on the problem and to find out additional information if needed. I was contacted initially within two hours. My problem was discovered and fixed by the next day. They offered a patch to be sent out to me but I decided to work around my original problem.

5.4 Interactive Design Tool

XVT-Design version 1.1 is the interactive editor for XVT. It is the least powerful and most weakly coupled to the total PIUIB environment of the interactive editors evaluated. It has good WYSIWYG layout features and provides nice alignment capabilities. It does not provide interface testing capabilities and only a minimum of C code generation and regeneration. It also lacks access to event specifications and color control which must be done using XVT's Universal Resource Language (URL). Windows, dialogs, and menus are given priority treatment by XVT, one stubs file for each object type instantiated is generated and modified by the programmer.

5.4.1 Windows

Windows are one of the three major objects in XVT. The minimal attributes that are shared by all look & feel standards are provided.

All widgets are implemented as windows, which may slow down a complex application. There is no encapsulation from this fact for the programmer, window controlling routines are used to manipulate many aspects of widgets, I feel abstractions should be provided here.

5.4.2 Menus

Menus are the second of the three main XVT objects. Menu creation seems to be XVT-Design's best feature; ironically, the evaluated version crashed when twenty menu items were exceeded. The menu power was all there, including automatically generated labels for keyboard equivalents. The process of creating menus was clear and concise. Four default menus were provided with a menu bar: File, Edit, Font, and Help; these can be deleted. There was no ability to include a graphic in a menu item.

5.4.3 Dialogs

Dialogs are the third of the three main objects provided by XVT. Dialogs have minimal controls for themselves, only location, size, initial visibility, and whether they are modal. There is more functionality provided for the objects that go into a dialog. Alignment of objects is done rather well. The concept of a visit order among dialog objects is also handled well, it is a separate list with good editing features to control the list/visit order.

5.4.4 Icons

XVT defers the issue of icons to be created with platform specific graphic editors. This is not a very forward-thinking approach to building platform independent GUIs.

5.4.5 Other Notable Widgets

There really is not much extra to note about XVT because of their decision to provide only greatest common denominator objects.

5.4.6 Notes on Building the Application

XVT handles the look of widgets fine, but sometimes does not handle the feel. A case in point is radio buttons. They can be laid out with XVT-Design straightforwardly, but they won't function as radio buttons without several lines of source code created by the programmer. It is beyond my understanding why XVT does not provide for the entire radio button functionality without making programming necessary.

Events are handled at the window level. During the definition of an object, a unique identifier is issued automatically by XVT and the application author assigns a #define string for it. This is basically a low-level identification code that can be compared to part of an event structure that is passed during runtime. The other critical part of the event structure is the event type (window create, mouse click, menu, etc.). It is the responsibility of the application code to determine what kind of event it was, whose low-level identification code it is, and do the application-specific operation. This method is not elegant or well-structured. One potentially powerful and unique aspect of XVT's handling of events is that certain events can be sent to the application before getting sent to XVT's event dispatcher.

5.4.7 Miscellaneous

Mouse events are not handled perfectly. There are two situations that need extra special care and bookkeeping by the programmer: double clicks and mouse-down, mouse-up pairs when the mouse has left a window. It is possible for both these to be missed, misinterpreted, or for the code to become quite messy. It is not necessary to go into the details here; suffice it to say these should have been handled better by XVT.

5.5 Prototype Evolution

XVT-Design does not provide support for prototype evolution; however, it will not overwrite files with the same name as those it wants to (re)generate. Thus the burden is put onto the programmer to move any files that are expected to be regenerated, and then merge by hand the new code with the old code that has programmer modifications in it.

5.6 Look & Feel Conformity

XVT provides very good conformity with the various look & feel standards because it uses their APIs to create applications. Because XVT provides no extraordinary widgets for any particular platform, there is no situation that can arise that would put a look & feel standard issue in conflict with what XVT wants to do.

5.7 Extensibility

There is no extensibility capability with XVT's own mechanisms.

5.8 Non-GUI Features

The operating system features, file, printing, string, and memory resources, are all very strong in XVT. This shows that XVT understands there is more to a GUI than just what appears on the screen. These powerful capabilities are what keeps XVT near the top end of PIUIBs.

5.9 Porting Issues

Because XVT has been around so long, it was the only PIUIB that supported our out-dated PC compilers. It took me over 8 hours to compile and link my application due to the fact that I needed to specify that the link should be case insensitive which was mentioned nowhere in XVT documentation. Once running, the application fared well and conformed to the Windows look & feel.


6 GUILD Evaluation

6.1 Executive Summary

GUILD version 2.1 is produced and sold by Guild Products. It offers an all-encompassing object-oriented approach, good interactive layout and widget set features, strong database and variable substitution facilities, and dynamic handling of multiple types of very useful and powerful variables. However, the weaknesses of documentation; coverage by the object and message passing systems; and lack of file, printing, and memory management support relegate GUILD to its lower standing.

6.2 Installation and Learning

Installation of GUILD was straightforward and simple requiring 45 minutes to read the instructions and about 10 minutes to load from diskette. The first platform used for GUILD was PC/Windows; I intended to port to the Macintosh platform but time was used up before the port attempts.

Initial learning was quite difficult and I spent 7.5 hours getting comfortable. GUILD didn't supply a tutorial proper, rather it gave a brief description of most of the interactive editor functions that could be followed online. This approach lacked goals and at this point I still didn't know how to create a minimal application. I went next to reading about their resource language and object-oriented approach which took a long time. The approach is interesting and viable, but the documentation did not cover enough issues of use, caveats, and possible conflicts among messages. I had to go back to the documentation after starting the prototype build more often with GUILD than with any other PIUIB evaluated.

6.3 Documentation, Online Help, and Tech Support

As mentioned throughout this report, GUILD's documentation was lacking in coverage and direction. They may have put more effort into the online help facility for their interactive editor because the online help was superlative. It had a highly context sensitive providing the most specific information possible first. It also had hypertext features and a good index which allowed me to investigate associated topics as desired. However, the online help facility was not enough to explain everything I needed (online help probably shouldn't try to be everything), and thus didn't make up for the lacking hardcopy documentation.

GUILD's tech support was very conscientious. However, I got the feeling that I was being taken care of by one or two of the main developers. This could be good since I'd be getting the most correct information most quickly. This could be bad since my impression from one of their demos is that there are only a handful of developers in the first place and they are handling the tech support duties which can obviously take time away from forward progress on GUILD development. My impressions of the quickness and quality for tech support were positive.

6.4 Interactive Design Tool

GUILD has a mostly well-designed interactive editor (three actually, one each for windows, menus, and dialogs/controls). In addition to items covered in the immediately following sections, it provides a very strong database and variable substitution mechanisms that provide a higher level of flexibility for interface specifications than in the other PIUIBs evaluated. However, confusion was the norm for specifying which object is sending which message to which other object, I feel the interface could be done more clearly for this capability. GUILD also provides an interactive debugger, Snooper, that one can control to show various levels of detail during program execution including messages, variable modifications, etc. I found Snooper to be quite helpful in understanding what was going on during execution. GUILD also provides good graphics capabilities.

6.4.1 Windows

GUILD provided very good window attribute controls, though the interactive editor did not provide access to all that the resource language could do. Window objects are used to create both windows and dialog boxes.

6.4.2 Menus

Menu creation was clear and quick. There was good access to mnemonics and keyboard accelerators. It was also possible to put an icon into a menu item by declaring a background bitmap for it. All in all a powerful and simple menu generation mechanism.

6.4.3 Dialogs

Dialogs take a broader meaning within GUILD. In order to get any control into the interface, a dialog object must be used; in essence it is like a panel object. For a dialog box, one would create a window for it with a dialog object and controls.

There was a good widget set for GUILD with the exception of providing a multi-line text object which was still on the list of things to do. The widget alignment features was very good.

6.4.4 Icons

GUILD provides a good graphic editor that can be used for creating icons. GUILD uses a vector graphic image approach which allows for animation, arbitrary zooming, and good platform independence. Again, bitmaps can be used as the background for windows, dialogs, or menu items.

6.4.5 Other Notable Widgets

The only other item worth mentioning is GUILD's excellent online help creation capability. It provides features for hypertext links, many styles, and keyword and indexing of which end-users can take advantage. Integrating the help for any object is very simple which allows for easy creation of context sensitive help.

6.4.6 Notes on Building the Application

Compiling the application was reasonable. However, it took me a long time to discover a bug in my specifications in which I had conflicting flags or message passing constructs. It took the tech support personnel most of two days to figure out the problem. This points to the possibility of a design problem in these constructs if it is so easy to tell the interface to do conflicting things.

6.4.7 Miscellaneous

Message passing is viable, buts needs to be better documented for typical uses, conflicts with other typical and non-typical situations, and indexed.

The variable substitution mechanisms seem to provide a level of flexibility not possible with the other PIUIBs. Substituted strings are provided by all PIUIBs through one mechanism or another. GUILD provides six types of variables that allow for much more flexibility (straight from the documentation):

6.5 Prototype Evolution

No C code is generated, so prototype evolution is not made difficult. The more that functionality is performed with message passing rather than GUILD API calls, the easier will be prototype evolution.

6.6 Look & Feel Conformity

GUILD exhibited high conformance to the Windows look & feel which was the only standard in which GUILD was used. This makes sense since GUILD uses the look & feel APIs to create the interfaces.

6.7 Extensibility

GUILD does not provide any subclassing mechanism, so that kind of extensibility is not possible. GUILD does provide you with the GUILD specification of their own interactive editor such that this could be modified to provide additional or different functionality so long as it used legal GUILD constructions. It may be that C code would need to be added to certain functional additions or modifications performed in this way. This level of extensibility is cute, but not very powerful.

6.8 Non-GUI Features

GUILD provides no file, printing, or memory management support which detracts considerably from its usefulness. Though it should be noted that many file needs could be met by GUILD's strong database capabilities and memory management is only needed for C code which can be minimal in GUILD applications. As mentioned before, on the plus side GUILD does have strong database and variable substitution facilities.

6.9 Porting Issues

I did not attempt to port the prototype built with GUILD to another platform due to time constraints. However, similar to the case made for prototype evolution, it seems that in theory a port to another platform would be easy proportional to the amount of C code written.

7 OpenUI Evaluation

7.1 Executive Summary

OpenUI version 1.30 by Open Software Associates rounds out the evaluations. It uses an object-oriented approach that is reasonably powerful but can be extended in many circumstances. Its interactive layout mechanism is so abstract it is a negative feature. The lack of a strong widget set and file, printing, and memory management support compounds the negatives.

OpenUI seems to be the least grounded in a GUI and software engineering perspective. It is leaning towards database and OLTP domains.

7.2 Installation and Learning

It took 23 minutes to read and understand the installation instructions and another 20 minutes to load from tape. A small battery of tests and environment setup was completed in another 20 minutes. The initial learning period consisted of 1 hour 40 minutes following the tutorial and an additional 2 hours 20 minutes reading up on the interactive editor, ouibldr, and object-oriented methodology used by OpenUI.

7.3 Documentation, Online Help, and Tech Support

OpenUI's documentation seems usually to suggest or give the flavor of a description rather than to give direction, answer, or explain. Many times I would pick up the set of manuals and leaf through them recognizing none had the answer for which I was looking. OpenUI provided no online help for their tool, even though OpenUI itself provides a capable help authoring system with hypertext links, keywords, etc.

7.4 Interactive Design Tool

The interactive design tool, ouibldr, supplies reasonable power and a below average widget set. However, it uses a logical grid to lay down widgets in an attempt to be resolution independent. Instead, it is far too cumbersome to work with comfortably and creatively. Another strange choice of behavior in ouibldr is that placed objects are always in test mode except for the double-click event which will invoke the attribute editor. However, the actions associated with the objects do not execute while in test mode. A companion program, ouishow, is used for this purpose. This method does not make much sense to me, either do it or don't do it, but don't do it halfway. This methodology got me in to trouble when I created a modal dialog box but couldn't get out of it because the OK and Cancel buttons weren't specified with actions to close the window! It took me over an hour to find in the documentation that hitting the escape key will exit the window.

Another oddity about ouibldr is that it uses a textual form-based style that doesn't seem to "fit" well in a GUI-based application. All attribute changes are made by typing text or in a few cases choosing text items from a set of choices. This textual form-based approach takes longer than other methods, is somewhat frustrating and tedious, and leaves one wondering why.

There really is no point in following the structure of the previous chapters in explaining the various window, menu, dialog, etc. features of ouibldr because its methodology is quite limited compared to the other PIUIBs evaluated. Only if OpenUI was being compared to lesser competitors or platform dependent builders would such an exercise be worthwhile.

8 Acknowledgements

This project was jointly inspired by Ben Shneiderman of University of Maryland, Ren Stimart, Manager of the User Interface Group at GEIS, and myself. Ren also supplied much insight to criteria for selection, process, and evaluation of the PIUIBs. Cathy O'Donnell, Senior Specialist in the User Interface Group at GEIS, helped with much of the less glamorous but very important, voluminous details of this project, including reviews of this report. This project would not have been possible without the continued support and encouragement of Lew Priven, Vice President of Engineering and Application Development at GEIS, and Don Deutsch, Manager of Industry Applications and Services at GEIS.

I would also like to extend my gratitude to each of the five companies for supplying our lab with free, full-privilege copies of the tools evaluated (or extended evaluation periods).