PBD Interface Builders

David Wolber	Brad Myers
University of San Francisco	Carnegie Mellon University
wolber@usfca.edu	bam@cs.cmu.edu

Abstract
Programming by Demonstration (PBD) can be used not only to extend existing applications, but to build new interfaces starting from a blank canvas. Tools to interactively build applications, often called interface builders, are today limited to mainly the layout of the user interface, and require writing code for any of the behavior. Interface Builders that provide PBD allow the behaviors to be created by example. In order to do so, the designer must be able to demonstrate not only the operations to perform, but also the events that trigger those operations. This paper explores the area of interface builders that use PBD based on what we have learned in over a decade of building such systems.

1. Introduction
Although tools like Visual Basic significantly decrease the time and expertise necessary to define the layout of interfaces, the designer must still code its dynamic behavior. For instance, a designer can easily change the location of an object in the interface by dragging it. However, specifying that the object should move at run-time in response to some stimulus (such as a button click) requires coding. Unfortunately, many designers don't know how to program and don't have the inclination to learn.

Programming by Demonstration (PBD) interface builders can provide what is missing in standard tools. They allow both the layout and the behavior of an interface to be specified without coding. Perhaps most importantly, a designer is not restricted to building interfaces with standard widgets (e.g., menus, list boxes, buttons), but can draw arbitrary graphics and demonstrate their animated behavior. We call these living pictures application-specific widgets.

1.1 PBD: An Elaboration of Macro Recording
Many PBD systems elaborate on the familiar concept of macro-recording. The user tells the system to watch her perform operations so they can be recorded and played back later. The goal of macro-recording is to give a name to a sequence of operations so the user won't have to repeat those operations again and again.

PBD elaborates on the concept of macros by generalizing the demonstrated sequence of operations. For instance, with Mondrian [7], a user can demonstrate the drawing of three rectangles around a square. The system records the three rectangle operations, but generalizes from the example so that the macro can be used to draw an arch around any square of any size.

1.2 PBD Macro Invocation
Once a macro is created, there must be a way to invoke it. Mondrian creates a new item in the palette of operations--the user just clicks on the item to invoke the macro. Eager [2] invokes a macro automatically when it identifies that the user has explicitly executed its first few operations, that is, it finishes the task for the user. Other systems just allow the user choose a previously recorded macro from a list.

But a standard method for invoking recorded macros is not sufficient if the purpose is to build entirely new interfaces, not extend existing ones. The designer must be able to specify any triggering event for the activities that take place in the interface.

PBD interface builders solve this problem by allowing a designer to demonstrate not just macros, but the way macros are invoked. With this two-phase process, a designer can demonstrate behaviors such as "moving a node causes all lines connected to it to stay connected" or "when the monster sees a wall in front of it, it turns left." The ultimate goal is that complete applications can be created from the examples without writing any conventional code. It is a challenging goal, however, because the events that trigger activity are often more complicated than the activity itself.

1.3 Stimulus-Response Demonstration
Most systems we will discuss use some form of the two-phase demonstration process called stimulus-response demonstration [14] .Typically, these systems provide a drawing editor and a mode palette (see the control panel in the top right of Figure 1) with which the designer tells the system when she is drawing the initial interface, demonstrating a macro invocation (stimulus), demonstrating the macro itself (response), or testing the interface. Most systems also provide a second-level editor so the designer can view and edit a high-level representation of the inferred program. Some also generate code (e.g., a Java applet[15]). In this way, interface building is a combination of PBD and dialog-based editing, and, if necessary, coding.

1.4 A Quick Example
Consider how stimulus-response is used to create a Celsius-Fahrenheit converter in the Pavlov system [15]. The designer first draws two thermometers, as shown in Figure 1.

Figure 1. Pavlov Development of a Celsius-Fahrenheit Converter. The stimulus-response mode palette is the control panel in the top-right

After drawing the objects, the designer then uses the same drawing operations to demonstrate the interface's behavior. She first tells the system she is demonstrating a stimulus -- she selects Stimulus mode -- and drags the Celsius indicator (the small rectangle inside the Celsius thermometer) five pixels up. Then, in Response mode, She demonstrates how the system should respond to the stimulus by moving the Fahrenheit indicator 9 pixels up.

From the demonstrations, the system infers a generalized behavior, in this case that any time the Celsius bar is moved, the Fahrenheit bar should be moved 9/5 as much. When the interface is executed, and the end-user moves the Celsius bar, the Fahrenheit bar is moved the proportional amount.

1.5 Wait a Second!
An example like the Celsius-Fahrenheit converter inspires more questions than it answers. How does the system know to infer a proportion? Can the end-user move the Celsius indicator outside its enclosing box? How is such a restriction demonstrated? If the designer doesn't like what the system infers, how can she change it? Can animated behaviors and timing be specified? Can more complex stimuli be demonstrated, like the behaviors of a PacMan character?

Trying to answer such questions is the challenge of PBD research. This paper introduces what we have learned in building a number of PBD interface builders [8,9,10,11] over the past decade. Our goal is a system that allows the behavior of any interface to be defined without coding. There has been some significant progress: stimulus-response demonstration can be used to create such complex interfaces as diagram editors, driving simulators, shooting arcades, board games and even PacMan. However, these ideas have yet to appear in commercial interface builders, and much research is still needed to reach the ultimate goal. 

2. The Syntax of Stimulus-Response
The syntax of a stimulus-response system is the mechanism by which a user presents the "example" behaviors to the system. Providing a usable syntax is a challenge to systems builders because the target designer is a non-programmer, and because the system is providing the end-user with much more power than most software -- it is allowing her to create software, not just use it.

Conventional interface builders like Visual Basic have two modes--build and test. PBD interface builders add two more to the mix: record stimulus and record response. Our observations are that users have significant difficulty keeping track of the current mode when these additional modes are added.

2.1 Eliminating Modes
Some systems have attempted to simplify the approach by eliminating one of the modes. Chimera [6] effectively eliminates response mode by recording all actions in a history list and requiring the designer, after-the-fact, to signify which actions in the list should be part of a macro.

Gamut [8] eliminates stimulus mode by replacing the four-mode palette with two buttons: "Do Something" and "Stop That" (see Figure 2). When the user performs an action that is supposed to do something and it doesn't, she hits the "Do Something" button. The event just prior to pushing the button is assumed to be the stimulus. The system is then in response mode until the user hits "Done". "Stop That" is used to give negative examples when the system does something wrong. The distinction between build and test mode is eliminated by having the system always be in test mode even when editing is allowed.

Figure 2: A Gamut screen creating a board game. The "Do Something" and "Stop That" buttons are in the lower right.

2.2 Demonstrating Stimuli
Any of the following stimuli can trigger a response in an interface:

• An end-user manipulating the mouse or keyboard.
• The execution of some drawing editor operation, such as Move or Rotate.
• Objects in the interface exhibiting some state.
• The passage of time.
• The execution of some application event.

These are each discussed in the following sections.

2.2.1 Physical and Logical Operations
One key aspect of PBD interface builders, as opposed to PBD systems that extend existing applications, is that the base system, e.g., the drawing palette, menu and other mechanisms, do not appear at run-time. Thus, unless the designer explicitly demonstrates that the end-users can perform an operation in the target interface, they won't be able to.

For example, during development, the designer can rotate any object by choosing Rotate in the drawing mode palette. But if the designer wants the end-user to be able to rotate that object, she must demonstrate a stimulus-response behavior which maps a mouse-drag to a rotate operation.

Some systems require this physical (mouse-drag) to logical (Rotate) mapping to be explicitly demonstrated. In DEMO [14], when the designer clicks on an object in stimulus mode, she is prompted to choose between the mouse operations up, down, drag, enter, and exit. The designer selects one, then demonstrates the response which should be linked to the chosen stimulus.

Gamut provides special mouse operation icons that can be dragged onto an object to demonstrate the physical operation. These are visible at the left of Figure 2, below the drawing palette. Since there is an icon for each type of operation, no choice dialog is necessary.

If the physical->logical mapping is explicit, the designer needs two demonstrations for behaviors like the Celsius-Fahrenheit program: one to specify the physical mapping (drag -> Move), and one for the logical mapping (CelsiusBar.Move -> FahrenheitBar.Move).

Pavlov eliminates the need for two demonstrations in such cases. When the designer moves an object in stimulus mode, the system implicitly infers the drag->Move mapping, and also records that the Move should be mapped to the upcoming response operation(s). The disadvantage of inferring the physical-logical mapping is that sometimes the designer only wants to map a logical stimulus to logical response(s), and doesn't want the physical-logical mapping. For instance, in a shooting arcade game, the designer demonstrates that moving a bullet (and hitting a target), should cause a response of deleting the target. In Pavlov, when the bullet.Move stimulus is demonstrated, the system infers a physical-logical mapping, i.e., that the end-user can move the bullet. To restrict the end-user from doing this, the designer must use the editor to delete the mapping.

2.2.2 Context-Triggered Behaviors
Sometimes the activity in an interface should be executed not in response an operation, but in response to the interface reaching a particular context or state. For instance, in the shooting arcade game, a target should be deleted only when a bullet intersects the target, as shown in Figure 3:

Figure 3. Pavlov Development of an arcade game.

Graphical rewrite rule systems like StageCast Creator (the successor to KidSim [12] ) focus on context, but restrict object movement and context specifications to discrete grid coordinates (e.g., "trigger when a dog is one square to the left of a wall").

Both Pavlov and Gamut allow context to affect when a response is executed, and allow continuous object movement and context specification. Pavlov integrates the context into its stimulus-response model without adding a special "demonstrate context" mode. The idea is that an interface does not magically enter a particular state, but will only do so as a result of some operation. Thus, the designer always demonstrates a triggering stimulus operation, but as she does she can demonstrate a condition that must be true when the stimulus operation occurs, or no response is executed.

Consider again the shooting arcade game. In Pavlov, the designer demonstrates a move of a bullet as the stimulus, but completes the move (releases the mouse) so that the bullet intersects a target (see Figure 3). The system records the move stimulus and identifies conditions relating the objects in the interface. The system lists textual descriptions of the conditions it finds and the designer is allowed to choose which should be used as the stimulus. She then demonstrates the deletion of the target as the response. At run-time, every time a bullet moves, the system checks the intersect condition, and executes the response only if the condition is true.

The model is more efficient than traditional constraint or rewrite rule systems, because it only need check the condition when the triggering event is executed, while those systems check all conditions on every time-stamp. However, it has the disadvantage that a condition must be specified more than once if different stimuli can cause the same result, e.g., there are two objects and movement of either might lead to an intersection.

A general goal of Gamut is to eliminate the need for the designer to choose the correct generalization (e.g., condition), as is done in Pavlov. The reasoning is that the representation, whether textual or graphical, is inherently complex, so choosing can be difficult. Gamut's philosophy, similar to [5], is to allow the designer to perform multiple demonstrations of the same context. The system then uses AI techniques to infer the intended context from the samples. Gamut also asks the designer to provide hints by selecting objects that are relevant to the context. The hints allow more complex conditions to be inferred that are possible in other systems.

2.2.3 Time as a Stimulus
Sometimes activity should occur at a certain time, rather than in response to an external event. Gamut allows the designer to drag a special clock widget into the background window, and then demonstrate a tick as a stimulus. Pavlov provides a similar capability by incorporating a timeline editor into its system.

2.3 Guide Objects
A guide object is one the designer uses during development, but doesn't want to appear at run-time. For instance, the designer might use a directed line to show how a game-piece can move around the board. In Gamut, the designer simply sets an object property to specify that it is a guide object. It then appears in pastel colors during development, and disappears at run-time. In Figure 2, the yellow lines are guide objects to help show where the pieces can move.

Gamut also provides a back-stage area in which all objects are guide objects. The objects in this area generally store "computational" information in the form of text boxes or other widgets. By replacing information traditionally stored in program variables, the need to connect an interface to application code is reduced or eliminated.

2.4 Demonstrating Responses
A response is any action that creates, transforms, or deletes an object in an interface. Though most PBD research systems are based on fairly rudimentary graphics editors, the idea is that a commercial system would provide a full range of graphic and text editing capabilities.

Pavlov combines PBD with some animation mechanisms that allow timing to be specified on behaviors. Using Pavlov's timeline, the designer can specify a time-frame, relative to the stimulus, for a demonstrated response. Pavlov also provides a special mode, similar to one in MacroMedia's Director, called Real Time Response mode. When this mode is chosen, the system records a sequence of time-stamped operations (an animation path) as the designer demonstrates an operation. The designer can also specify animation by stipulating that a response be executed periodically at run-time,

Pavlov also allows a designer to specify the direction (nose) of an object using a guide object. Consider, for instance, the development of a driving simulator:

Figure 4. A driving simulator built in Pavlov. The arrow on the car is a guide object denoting the direction of the car.

After drawing the car, accelerator slider bar, and steering wheel, the designer specifies the "nose" of the car by manipulating its "direction" vector guide object. She then demonstrates a stimulus of moving the accelerator slider bar, and a response of moving the car, and marks the response as periodic. Because the car has a notion of direction, its movement is restricted -- the designer can only demonstrate that it moves forward or backward relative to the direction vector. In Pavlov, the response is recorded not as a Move, but as a (periodic) MoveForward operation. At run-time, the car goes faster or slower as the accelerator is manipulated.

3. The Semantics of Stimulus-Response
The goal of a generalization mechanism is to take the demonstrated stimulus-response examples as input, and output behaviors that, if they were represented as code, would appear something like:

On StimObjects.Stimulus1(<formal parameters>)
	{
	if (checkcontext)
		ResponseObjectA.Operation1(<parameters>)
		...
  	}

Some major challenges in interpreting the examples are:

• Which object(s) is the designer signifying with her demonstrations?
• What should the parameters of the response operation(s) be?
• How should context modify the inferred behavior?

There are two main methods of generalization, based on whether the system allows more than one example of a behavior. In the tradition of SmallStar[4] and Chimera [6], Pavlov generalizes from a single example. The philosophy of such a system is to keep the inferencing simple so the designer can understand it, and to provide a second-level editor that allows the designer to modify generalizations easily. Single example systems use as much domain knowledge as possible so that the single "guess" is as good as possible. Multiple example systems, like Gamut, can infer more complex behaviors and need not rely on domain knowledge. However, it is more difficult for the designer to understand what the system is doing.

3.1 Object Descriptor Problem
The first problem with generalizing has been named the object descriptor problem -- determining the object or objects that are intended at run-time when the designer demonstrates an action on an example object.

Sometimes no generalization need occur, as there is a one-to-one correspondence between the demonstrated object and the intended run-time object. For example, in the Celsius-Fahrenheit example, FahrenheitBar is the demonstrated response object, and it is the intended run-time object as well. FahrenheitBar is an example of a constant descriptor.

In some cases, however, the designer intends that the demonstration object be representative of some set of objects, or some single object that cannot be described with a constant. When the demonstration object is a dynamically allocated object, the object can't be described with a constant. Consider, for instance, the dynamically created bullets in an arcade game. A particular development-time bullet won't appear at run-time, because no bullets appear until a stimulus occurs. When a action is demonstrated on a bullet (e.g., movement), a non-constant descriptor must be inferred for it.

There are also cases when a non-constant descriptor is appropriate even though the demonstration objects itself is statically created. For example, in a board game, the pieces might be statically created, but the move piece operation may choose which object to move based on whose turn it is.

Because Pavlov is a single example system, it only infers non-constant descriptors when a response is demonstrated on a dynamic object. Pavlov includes a number of heuristics to guide its generalization of dynamic objects. The most simple inference is that the dynamic demonstration object represents all instances, at run-time, that have been created by the same stimulus. For example, suppose that the bullets in an arcade game are created by pressing the up-arrow key. If the designer demonstrates a stimulus of clicking a button, and a response of changing a particular bullet's color to blue, then at run-time, clicking the button will change the color of all bullets (created by an up-arrow).

Pavlov does identify special cases that use more distinguishing descriptors. For instance, if a transformation is demonstrated in the same response sequence as a creation (e.g., a bullet is created then moved), then the object descriptor will be "the same instance as the one just created". If a response is demonstrated on the same object as the stimulus, the inferred descriptor is "the same instance as the stimulus." Pavlov will also infer if a particular condition should identify which instances are chosen. In practice, Pavlov's special rules and condition identifier cover a wide range of behaviors. But no matter how many heuristics are used, the system will sometimes guess wrong using only a single example. In these cases, second-level editing or coding is necessary.

In contrast, Gamut uses multiple examples, both positive and negative. Gamut compares the properties of the objects participating in the examples to determine the correct properties for determining how to select the objects at run-time. The user is expected to repeatedly test the system and refine the behaviors that do not work by giving more examples. Gamut also reduces the number of required examples by requiring users to give hints about which of the many interface objects are important. With hints, the system can infer conditions that use objects that are not directly modified by the action itself (for example, when an off-screen toggle determines which piece should move).

3.2 Response Parameter Descriptors
In addition to the object parameter, PBD systems must infer the other parameters of an operation. For example, a move operation has both the object to be moved (as discussed above), and the position to which it moves. Again, the simplest inference is that the parameter is constant, but even in this situation there is ambiguity: did the designer intend the destination demonstrated, or the offset demonstrated (that is, move to this location, or move by this amount)? Single example systems must guess in this situation. Multiple example systems can distinguish after two examples.

Inferring constant parameters is hardly the most challenging issue. For instance, the position to which a board game piece should be moved in Figure 2 might be described as "take the number on the dice and move that number of spaces clockwise around the board from where the piece used to be." Such a parameter can be inferred by Gamut. The following sections discuss various techniques that can be used for generalizing non-constant parameters.

3.2.1 Linear Proportions
Pavlov takes advantage of the observation that many interface behaviors have response parameters that are proportional to the stimulus parameters. The Celsius-Fahrenheit converter is one example -- when one gauge is transformed, the other one should be transformed 5/9 (9/5) as much. Another examples is a diagram editor in which moving a node should cause all lines connected to it to be reshaped by the same amount.

Pavlov uses the heuristic that when a stimulus and response are both transformations (e.g., Moves), each response parameter is inferred to be proportional to the corresponding stimulus parameter. As with object descriptors, Pavlov also watches for special cases. For instance, if the designer demonstrated a Move(20,0) as a stimulus, and a Move(0,40) as a response, it maps the non-zero second parameter of the response to the non-zero first parameter of the stimulus.

InferenceBear [5] infers other linear combinations of parameter values from multiple examples of the desired behavior. This eliminates the need for hard-wired special cases, but requires the user to carefully choose the examples.

Furthermore, InferenceBear can take into account properties of the object that do not participate in the action (Pavlov only analyzes the demonstrated stimulus and response objects). For instance, in the shooting arcade game, Inference Bear can infer after two examples that the parameters of the bullet creation depends on the location of the gun, though the gun is not part of the stimulus or response.

3.2.2 Conditional functions of object attributes
Gamut adds even more inferencing power by using decision trees and other AI algorithms, and by allowing the designer to provide hints to help the system generalize the parameters. For instance, after the first example of a bullet creation was provided, Gamut would ask the designer to point out objects that are important. The designer would select the gun, and Gamut would use this to infer the correct create response parameters. Furthermore, Gamut can compute the parameters using values from many objects. Examples include setting the color of an object based on the color of a palette, or computing the position of the board game piece using the dice as mentioned above. As a result, Gamut can be used to create complete applications such as a Turing machine, Tic-Tac-Toe, or a PacMac game.

4. Feedback and Editing
When a PBD system infers a program from a designer's demonstration, it doesn't always infer the program that the designer intended. One solution is to allow the designer to demonstrate more examples until, hopefully, the system infers the intended program. Still, the designer can never be sure what program the system has inferred. Another approach is to ask the designer questions that will hopefully disambiguate the examples. However, experience with such systems [10,14] show that users have great difficulty answering such questions.

Therefore, we feel that the designer really needs to see some representation of the inferred program, and be able to edit it. Here lies the dilemma, called the PBD representation problem: since the goal of PBD is to allow people that aren't programmers to create programs, the system probably shouldn't represent a program as conventional computer code (C++, Java, etc.). Even special purpose scripting languages, like that generated by DEMO and Inference Bear, are difficult for end-users.

One solution is to present a graphical, story-board representation of the inferred programs [6,7,9]. Figure 5 shows one from Pursuit [9] that represents copying all the files edited TODAY in the papers folder that end in ".tex".

Figure 5. A storyboard showing a program created by demonstration in Pursuit.

Storyboards allow a designer to see what the system has recorded and what generalizations it has made. These "programs" can also be edited by moving tiles around, deleting them, or selecting them and re-demonstrating the particular operation.

Unlike Pursuit, stimulus-response systems must provide a representation which includes the triggering stimulus. In Pavlov, a dialog is provided that allows the designer to select an object and a stimulus. When the stimulus is changed, either with the dialog or by a stimulus demonstration, the corresponding timeline shows only the responses corresponding to that stimulus (see Figure 6). The designer can edit the parameters of each response (x and y in Figure 6) and can open a dialog to view and edit a graphical condition on the stimulus. Because there is a timeline for each stimulus, much of the scripting necessary in animation systems like Director [3] is eliminated, and objects can behave asynchronously.

Figure 6. The Pavlov Editor showing an animation path triggered by a button-click.

5. Summary
Many people use software, but few can create it. One of the only areas of creation open to most people is to create static web pages with one type of stimulus: the button click on a link. PBD interface builders can open up creativity by allowing people to build applications that come alive both for the Web and for the desktop.

References
1. Cypher, A.,Halbert,D., Kurlander,D., Lieberman, H., Maulsby, D., Myers, B. Turransky, A., eds. Watch What I Do: Programming by Demonstration. Cambridge, MA: The MIT Press, 1993.

2. Cypher, A., "Eager: Programming Repetitive Tasks by Example", Proceedings of CHI '91, ACM, New Orleans, May 1991, pp. 33-39.

3. Director 7.0, Macromedia, Inc., www.macromedia.com.

4. Halbert, D., "Programming by Example," Ph.D. Thesis, Department of Electrical Engineering and Computer Science, University of California, Berkeley.

5. Frank, M.R. and Foley, J.D. “A Pure Reasoning Engine for Programming By Demonstration,” in Proceedings UIST'94: ACM SIGGRAPH Symposium on User Interface Software and Technology. 1994. Marina del Rey, CA: pp. 95-101.

6. Kurlander, D. and Feiner, S., "Inferring Constraints from Multiple Snapshots", ACM Transcations on Graphics, May, 1991.

7. Lieberman, H., "Mondrian: A Teachable Graphical Editor", in Watch What I Do: Programming by Demonstration, Allen Cypher, ed., MIT Press, 1993.

8. McDaniel R., Myers, B., "Getting More Out Of Programming-By-Demonstration." Proceedings CHI'99: Human Factors in Computing Systems. Pittsburgh, PA, May 15-20, 1999. pp. 442-449.

9. Modugno, F., Corbett, A.T., and Myers, B.A., “Graphical Representation of Programs in a Demonstrational Visual Shell -- An Empirical Evaluation.” ACM Transactions on Computer-Human Interaction, 1997. 4(3): pp. 276-308.  

10. Myers, B.A., “Creating User Interfaces Using Programming-by-Example, Visual Programming, and Constraints.” ACM Transactions on Programming Languages and Systems, 1990. 12(2): pp. 143-177.

11. Myers, B., McDaniel,R.,Kosbie, D.,"Marquise: Creating Complete User Interfaces By Demonstration", Proceedings of INTERCHI '93, Amsterdam,April,1993,pp.293-300.

12. Smith, D.C., Cypher A., "KidSim: End-User Programming of Simulations", Proceedings of CHI ‘95,May 1995, pp.27-34.

13. Visual Basic, Microsoft, "msdn.microsoft.com/vbasic".

14. Wolber, D., and Fisher, Gene, "A Demonstrational Technique for Developing Interfaces with Dynamically Created Objects." Proceedings of UIST ‘91, pp. 221-230, November 1991.

15. Wolber, David, "An Interface Builder for Designing Animated Interfaces", Transactions on Computer-Human Interface( TOCHI), December 1997.