CS-TR-4323                                                                                       January 2002

UMIACS-TR-2002-05

HCIL-TR-2002-02

LAP-TR-2002-1

 

Conditional Branching in Computerized Self-Administered Questionnaires: An Empirical Study

 

Kent L. Norman and Timothy Pleskac

 

Laboratory for Automation Psychology and Decision Processes

Human-Computer Interaction Laboratory

Institute for Advanced Computer Studies

Department of Psychology

University of Maryland

College Park, MD 20742-4411

 

 

 

Abstract

 

Conditional branching is used in surveys to direct respondents to skip inappropriate questions or to answer additional follow-up questions.  When surveys are implemented on the World Wide Web, conditional branching can be automated in different ways.  This study compares three implementations: (a) a manual form which replicates the paper-and-pencil version in a scrollable browser window, (b) a semi-automatic form which also shows the whole survey but auto-scrolls to the next appropriate question, and (c) an automatic form that displays only one item per screen and implements all branching.  The surveys used for the study involved follow-ups of one, two, or three questions.  The three implementations were counterbalanced in a within-subjects design.  As expected it was found that completion times increased with the number of follow-up questions.  More importantly, the automatic item-by-item implementation proved significantly faster than either the manual or the auto-scrolling versions.  Respondents found the auto-scrolling to be disorienting.   These results suggest that automatic branching should be used but with graceful jumps that guide the respondents’ focus of attention without loosing it.

 

Keywords

 

Questionnaires, Surveys, Conditional Branching, User Interface, World Wide Web, CSAQ.


 

Conditional Branching in Computerized Self-Administered Questionnaires: An Empirical Study

 

Kent L. Norman and Timothy Pleskac

 

Laboratory for Automation Psychology and Decision Processes

Department of Psychology

University of Maryland

College Park, Maryland 20742-4411

 

 


Introduction

 

Many surveys involve conditional branching that directs the respondent to either skip questions that are not appropriate or to answer follow-up questions in response to a particular answer to a previous question.  In personal interviews, the surveyor is responsible for following the branching and the respondent is generally unaware of the logical structure of the survey.  In paper and pencil surveys however, the respondent must follow instructions to skip questions or to answer the follow-up questions. Such questionnaires are difficult to construct and often result in either unreliable responses or non-response (Dillman, 2000; Messmer & Seymour, 1983).

 

When a skip pattern is used in a survey, it means that one may be required to go from Question n to Question n + m, where m is the number of questions skipped over.  Although follow-up questions and skipping questions seem to be cognitively different, they are formally the same as shown in Figure 1.  One may think of Question 2 as being skipped when Alternative A of Question 1 is answered.  Or one may think of Question 2 as a follow-up to Alternative B of Question 1.  There are two parameters for conditional skips: the number of skip points and the number of questions skipped at each point.  An increase in either of these parameters is expected to increase the difficulty for the respondent following the instructions, the potential for problems completing the survey, and ultimately, the total time to complete the survey.

 

Figure 1.  Linear skip pattern.  (Ovals represent questions and squares represent the alternative answers to each question.  Alternative A causes  Question 2 to be skipped.)

 

When surveys are implemented on-line the conditional branching can be automated.   However, the design options for how to automate conditional branching and at the same time allow the respondent to browse the whole survey are many (Lazar & Preece, 1999).  Norman (2001) outlines the range of options from manual branching performed by the respondent to complete automated branching by the computer, from single item presentation of questions to whole survey presentation, and level of cognitive complexity required following instructions and focusing attention.

 

The present study compared three different implementations of a survey requiring a number of conditional branches.  The first was a baseline implementation that mimicked the standard paper-and-pencil form by presenting the whole questionnaire in a scrollable browser window (See Figure 2).  The questionnaire included all of the instructions on branching and the respondent was responsible for following them.  The whole-form presentation may be preferred over single-item presentation because it allows the respondent to view all of the items in context and to see items that would have been skipped.  Moreover, some research indicates that whole-form presentation is easier to use when the respondent must navigate back to previous items in a linear manner (Norman, Friedman, Norman, and Stevenson. 2000).

 

The second implementation also presented the whole questionnaire in a scrollable window but automatically scrolled to the next appropriate item when the respondent selected a branching alternative (See Figure 3).  Like the baseline, this implementation allowed the respondent to see the context of each question and the questions that might be skipped.  It also displayed branching instructions if the respondent wanted to manually navigate the questionnaire. 

Automatic scrolling positioned the next appropriate question at the top of the window.

 

The third implementation presented only one item on the screen at a time and all conditional branching was totally controlled by the software.  While branching instructions were not necessary, they were included in the text of the questionnaire to control for reading time. 

 

It was expected that the surveys implementing automated branching would be completed faster than the baseline implementation requiring manual branching.  It was not clear whether the automated scrolling or automated single item implementation would be faster.

 

 

 

Method

 

Design

 

Respondents were to complete the same questionnaire three times.  The first time they completed the questionnaire there was one follow-up question if they answered in the affirmative to a branching question.  The second time there were two follow-up questions (the one they had answered the first time and a new one).  The third time there were three follow-up questions (the two they had answered the first time and a new one).  The repeated questions allowed for a consistency check between the answers on each questionnaire. The three questionnaires varying in the number of follow-up questions provided a test of survey completion time. 

 

Three on-line implementations were varied across the three questionnaires completed by each respondent.  The implementations were (a) Manual shown in Figure 2, (b) Auto-Scroll shown in Figure 3, and (c) Auto-Item shown in Figure 4.  The three implementations were counterbalanced across the three administrations of the questionnaires varying in the number of follow-up questions as shown in Table 1.  This resulted in a randomized block partially confounded factorial design (RBPF-32) capable of testing the main effect of number of follow-up questions, the main effect of type of implementation, and a component of the interaction between the main effects (Kirk, 1995, pp 616-627).

 


 

Figure 2.  Whole-form presentation with manual conditional branching.

 

 

 

Figure 3.  Whole form presentation with automated scrolling used for conditional branching.

 


Figure 4. Item-based presentation with automatic conditional branching.

 


 

 

 

Table 1

Three Designs Counterbalanced Across the Number of Follow-Up Questions Asked

 

Group

One Follow-Up

Two Follow-Up

Three Follow-Up

1

Manual

Auto-Scroll

Auto-Item

2

Manual

Auto-Item

Auto-Scroll

3

Auto-Scroll

Manual

Auto-Item

4

Auto-Scroll

Auto-Item

Manual

5

Auto-Item

Manual

Auto-Scroll

6

Auto-Item

Auto-Scroll

Manual

 

 

 


Participants

 

Thirty-six undergraduates from an introductory psychology course at the University of Maryland, College Park, participated in the experiment for course credit.  The participants ranged in age from 18 to 21, with an average age of 19.17.  There were an equal number of males and females.

 

Materials

 

Survey. The survey consisted of two initial demographic questions and followed by 16 topic questions.  Each topic question was followed by 1, 2, or 3 follow-up questions if the respondent answered in the affirmative. The survey used is shown in Appendix 1.  In the manual form implementation and the auto-scroll implementation , the whole survey could be scrolled in a browser window.

 

Computer. The study was run in a Netscape browser window on a Mactinosh G4 computer with a 15 inch flat panel display.  Data was collected using FileMaker Pro 5.0 as the Web server.

 

Procedure

 

Respondents were asked to complete a series of questionnaires on the World Wide Web in a browse window.  After reading the informed consent form and agreeing to it, they completed a short questionnaire on prior use of computers. Then they completed the first questionnaire that involved a one-item follow-up, the second questionnaire that involved a two-item follow-up, and the third questionnaire that involved a three-item follow-up. Each questionnaire was the same as the previous one, but with an additional follow-up question added.  A different design was used for each of the three follow-up lengths so that each respondent was exposed to all three designs.  Table 1 shows the counterbalance design used. Upon completion of the last implementation of the survey, respondents were asked whether they liked the computer assistance in filling out surveys, which implementation they liked the best, and whether they had any comments and suggestions on the design of on-line surveys.

 

 

Results

 

Pre-Questionnaire.  Participants in this study rated themselves as having moderate to high computer skills.  On a 9-point scale (1 = no experience, 9 = very experienced) the mean rating of overall experience with computers was 6.88 (1.37 sd); use of the WWW, 6.97 (1.54 sd); use of email, 7.62 (1.43 sd).  They also rated themselves as having moderate experience in filling out surveys with a mean of 4.82 (2.35).

 

Completion Time.   There was a significant main effect for completion times for the type of questionnaire design (F (2,68) = 13.26, p < .01).  The Tukey HSD procedure indicated that there was no significant difference between the manual scrolling form and the auto-scrolling form.  However, the remaining pair wise comparisons indicated that the item-based automatic form took significantly less than the manual whole form or the auto-scrolling whole form, (p < .05).  Furthermore, there was a significant main effect for completion time for the number of jumps (F (2,68) = 44.08, p < .01). The Tukey HSD procedure revealed that all pair wise differences among means were significant,  (p < .05).  The partially confounded factorial design allowed for the testing of some within-blocks interactions that are not confounded (Kirk, 1995).  The interaction within blocks between number of jumps and questionnaire type was significant (F (4, 66) = 2.97, p < .01).   

 

Intra-Respondent Consistency.  Since respondents filled out surveys having the same questions from one implementation to another, it was possible to assess the consistency of each respondent in answering the questions in the same way.  Consistency was extremely high with the majority of comparisons being perfect and the average being .964.  Since consistency was assessed by correlating the answers of one survey implementation with another, no comparisons between implementations could be made. 

 

Post Questionnaire.  After completing the three questionnaires, participants were asked to respond to several questions.  When asked which survey type they preferred, four people responded that they preferred the manual scrolling, twelve people preferred the auto form scrolling, and nineteen people preferred the auto item form.   The frequencies were found to be significantly different from each other (c2 (2) = 9.65, p < .01).  Furthermore, 29 people responded that they preferred the computer assistance and only 6 people did not like the computer assistance, which is significantly different  (c2 (1) = 15.1, p < .01).


 

Figure 5.  Mean time to complete the survey as a function of type of on-line implementation and the number of follow-up questions.

 

 

 

Table 2

Mean Completion Time (Seconds) as a Function of Number of Follow-Up Items (Collapsed Across Type of Form) (Standard deviations shown in parentheses)

 

 

One Jump

Two Jumps

Three Jumps

Completion Time

109.34 (43.3)

148.51 (59.82)

206.26 (69.64)

 

 

 

Table 3

Mean Completion Time (Seconds) as a Function of Type of Form (Collapsed Across Number of Follow-Up Items) ) (Standard deviations shown in parentheses)

 

 

Manual

Item

Form

Completion Time

175.77 (66.3)

120.40 (48.82)

167.94 (81.07)

 

 

 

 


Discussion

 

The Manual and Auto-Scroll implementations seemed to take about the same amount of time for the respondents to complete the questionnaires.  However, the reasons for their taking longer than the Auto-Item implementation may be different.  In the Manual implementation, the respondents had to read and follow the branching instructions and periodically scroll down the browser window.  Additional time may also have been used answering questions that were supposed to be skipped.  One respondent admitted, “I answered some of the questions you were supposed to skip if you answered no to a previous question … there was no way to clear the button.” 

 

In the Auto-Scroll implementation, the respondents did not need to explicitly follow the branching instructions and scrolling was done automatically for them.  However, a number of respondents found the auto-scrolling to be disorienting.  One respondent noted, “Auto-scrolling was difficult to use with the screen jumping around.”  Another wrote, “I was confused and kept manually scrolling up until it dawned on me what was happening.”

 

Clearly the Auto-Item implementation was superior in terms of survey completion time.  It was about 25% faster than the Manual implementation with one follow-up question, 37% faster with two, and 26% faster with three.  On the other hand, the Auto-Scroll was only 17% faster than the Manual implementation with one follow-up question, 8% faster with two, the same with three, and overall, not significantly different from the Manual implementation.  It would seem that with one follow-up question, auto-scrolling helped a little; but with two and three follow-ups, it was too disorienting. 

 

Despite the clear superiority of the automatic branching using single item presentation, caution should be taken in using this implementation in general.  There are two basic problems with item-based surveys (Norman, et al, 2000).  First, they make it difficult to navigate back to previous items to review and change answers.  In the present experiment, this was not required; but in many surveys it is an important issue.  Second, in item-based surveys the respondents do not see the questions in context and do not see in advance the items that may or may not be skipped.  Again this was not an issue in the present experiment, but in many surveys a question asked out of the context of other questions can be misleading and result in unreliable answers.

 

Consequently, although auto-scrolling was a problem in the current implementation, it may be worth exploring other methods of automatically guiding the respondent to the next appropriate question but in ways that are not so abrupt and disorienting.  For example, one might use a slower, animated scrolling movement to the next question.  Alternatively, instead of scrolling, the follow-up items on the survey may be automatically highlighted and the items to be skipped might be grayed out.

 

Finally, it should be emphasized that automating conditional branching can lead to significant reductions in the time that it takes to fill out a survey, but the interface techniques used to accomplish this branching are critical to the success of automation.

 

In conclusion it is worth restating the design guidelines outlined previously (Norman, 2001):

 

1.  Reduce the branching instructions to a minimum to reduce reading time, confusion, and perceived difficulty of the questionnaire.

2.  Automate conditional branching when possible, but allow the respondent to override branching if there is a need or desire to do so on the part of the respondent.

3.  Hide inappropriate and irrelevant  questions to shorten the apparent length of the questionnaire and make such questions available only if the respondent specifically needs or wishes to view them.

4.  When the respondent is allowed to answer all questions, implement logic and consistency checks on conditional branches.

5.  Streamline forward movement through the questionnaire while allowing backtracking and changing of answers.

6.      When context matters, provide form-based views of sections to help to clarify the meaning of items and the interrelationships among items.

7.  Finally, it must be remembered that although good design seems intuitive, it requires empirical verification before final implemation.

 

 

References

 

Dillman, D. A. (2000).  Mail and Internet Surveys (2nd ed.), New York, N.Y.:  John Wiley & Sons.

Lazar, J. & Preece, J. (1999).  Designing and implementing Web-based surveys. Journal of computer information systems, 39, 63-67.

Messmer, D. J., & Seymour, D. T. (1983). The effect of branching on item nonresponse. Public Opinion Quarterly, 46, 270-277.

Norman, K. L. (2001). Implementation of Conditional Branching in Computerized Self-Administered Questionnaires. Laboratory for Automation Psychology, University of Maryland (LAP-TR-2001-1).

Norman, K. L., Friedman, Z., Norman, K. D., & Stevenson, R. (2000). Navigational Issues in the Design of On-Line Self Administered Questionnaires, Behaviour and Information Technology, 20, 37-45.

 

 

Acknowledgements

 

This work was funded in part by a grant from the Statistical Research Division of the U.S. Bureau of the Census (Contract #50YABC166008). We thank Kent Marquis, Beth Nichols, Betty Murphy, and others at the Census for their guidance and direction in this research.


 

Appendix

 

Text of survey showing two initial demographic questions and sets of three follow-up questions for 16 branch questions resulting in a total of 66 questions.

 


1. Are you

 

 Male

 Female

2. Are you

 Left handed

 Right handed

3. Have you seen a movie in the last two weeks?

 Yes

 No (If "No," go to Question 7.)

4. If you answered "Yes" to the previous question, did you like the last movie that you saw?

 Yes

 No

5. If you answered "Yes" to question number 3, would you recommend the movie you saw to your friends?

 Yes

 No

6. If you answered "Yes" to question number 3, what type of movie was it?

 Drama

 Action

 Comedy

 Other

7. Have you eaten out at a restaurant in the last two weeks?

 Yes

 No (If "No," go to Question 11.)

8. If you answered "Yes" to the previous question, did you like the last restaurant that you ate at?

 Yes

 No

9. If you answered "Yes" to question number 7, did you try an item from the menu that you have never had before?

 Yes

 No

10. If you answered "Yes" to question number 7, were you satisfied with the service of your waitress or waiter?

 Yes

 No

 

11. Have you taken a long trip within the last two months?

 Yes

 No (If "No," go to Question 15.)

 

 

12. If you answered "Yes" to the previous question, how did you travel?

 By airplane

 By car

 By train or bus

 By boat

13. If you answered "Yes" to question number 11, did you travel with others or alone?

 With others

 Alone

14. If you answered "Yes" to question number 11, did you travel within the United States or abroad?

 Within the United States

 Abroad

15. Have you changed jobs within the last six months?

 Yes

 No (If "No," go to Question 19.)

16. If you answered "Yes" to the previous question, are you happier or less happy with your new job?

 Happier

 Less happy

17. If you answered "Yes" to question number 15, did you change jobs because of the money?

 Yes

 No

 

18. If you answered "Yes" to question number 15, is your new job closer, further or about the same distance to where you live?

 Closer

 Further

 About the same

 About the same as the old job

19. Have you been ill with the last six months?

 Yes

 No (If "No," go to Question 23.)

20. If you answered "Yes" to the previous question, did you see a doctor?

 Yes

 No

21. If you answered "Yes" to question number 19, did you take a prescription drug?

 Yes

 No

22. If you answered "Yes" to question number 19, did you miss a day of work or school due to your illness?

 Yes

 No

23. Have you played any sort of sport within the last week?

 Yes

 No (If "No," go to Question 27.)

 

 

24. If you answered "Yes" to the previous question, where did you play this sport?

 Indoors

 Outdoors

25. If you answered "Yes" to question number 23, were you injured while playing this sport?

 Yes

 No

26. If you answered "Yes" to question number 23, did you play with a team or as an individual?

 With a team

 As an individual

27. Have you read a book within the last month?

 Yes

 No (If "No," go to Question 31.)

28. If you answered "Yes" to the previous question, was the book fiction or nonfiction?

 Fiction

 Nonfiction

29. If you answered "Yes" to question number 27, did you enjoy the book that you read?

 Yes

 No

30. If you answered "Yes" to question number 27, were you required to read the book?

 Yes

 No

 

 

31. Do you or have you ever smoked cigarettes?

 Yes

 No (If "No," go to Question 35.)

32. If you answered "Yes" to the previous question, have you stopped or do currently smoke?

 I have stopped

 I still smoke

33. If you answered "Yes" to question number 31, when did you start smoking?

 In grade school

 In high school

 In college

 Later

34. If you answered "Yes" to question number 31, does one of your closest friends smoke?

 Yes

 No

35. Do you have any brothers or sisters?

 Yes

 No (If "No," go to Question 39.)

36. If you answered "Yes" to the previous question, do you have any brothers or sisters that are older than you are?

 Yes

 No

 

 

37. If you answered "Yes" to questions number 35, do you get along with your brothers or sisters?

 Always

 Most of the time

 Sometimes

 Never

38. If you answered "Yes" to question number 35, have you ever shared a bedroom with a brother or sister?

 Yes

 No

39. Do you own a pet?

 Yes

 No (If "No," go to Question 43.)

40. If you answered "Yes" to the previous question, what kind of pet do you own?

 Dog

 Cat

 Fish

 Other

41. If you answered "Yes" to question number 39, what is the sex of your pet?

 Male

 Female

 Don't know

42. If you answered "Yes" to question number 39, do you or someone else take care of the pet?

 I do

 Someone else does

 

43. Do you own a computer?

 Yes

 No (If "No," go to Question 47.)

44. If you answered "Yes" to the previous question, what kind of computer do you own?

 IBM Compatible/Windows

 Macintosh

 Other

45. If you answered "Yes" to question number 43, do you have access to the internet from your computer?

 Yes

 No

46. If you answered "Yes" to question number 43, was the computer purchased within the last two years?

 Yes

 No

47. Do you drive a car?

 Yes

 No (If "No," go to Question 51.)

48. If you answered "Yes" to the previous question, how would you compare your driving to others?

 Better than average

 Worse than average

49. If you answered "Yes" to question number 47, have you ever received a speeding ticket?

 Yes

 No

50. If you answered "Yes" to question number 47, have you ever been in a car accident while you were driving?

 Yes

 No

51. Do you like to listen to music?

 Yes

 No (If "No," go to Question 55.)

52. If you answered "Yes" to the previous question, what kind of music do you like most?

 Rock

 Alternative

 Classical

 Country

 Easy listening

 Other

53. If you answered "Yes" to question number 51, have you ever been to a music concert?

 Yes

 No

54. If you answered "Yes" to question number 51, do you own mostly CD's or cassette tapes?

 CD's

 Cassette tapes

55. Have you ever bought a lottery ticket?

 Yes

 No (If "No," go to Question 59.)

 

 

56. If you answered "Yes" to the previous question, have you ever won?

 Yes

 No

57. If you answered "Yes" to question number 55, do you think buying lottery tickets is a waste of money?

 Yes

 No

58. If you answered "Yes" to question number 55, do you buy more lottery tickets when the prize money increases?

 Yes

 No

59. Were you concerned about the Y2K bug?

 Yes

 No (If "No," go to Question 63.)

60. If you answered "Yes" to the previous question, did you take any precautions?

 Yes

 No

61. If you answered "Yes" to question number 59, would you like to have been at Time Square in New York City on New Year's Eve?

 Yes

 No

 

 

 

62. If you answered "Yes" to question number 59, which did you think would be the biggest problem with Y2K?

 The banking system

 Computers

 The government

 Other

63. Do you tend to remember your dreams?

 Yes

 No (If "No," go to end.)

64. If you answered "Yes" to the previous question, do you remember a dream from last night?

 Yes

 No

65. If you answered "Yes" to question number 63, how often do you have scary dreams?

 Never

 Sometimes

 Often

66. If you answered "Yes" to question number 63, do you ever write your dreams down in a journal?

 Yes

 No