Usability and Effectiveness Evaluation of

Usability and Effectiveness Evaluation of a
               Course-Advising Chat Bot

Hyekyung Kim
Informatics Program, University at Buffalo, State University of New York
 hk67@buffalo.edu

Miguel E Ruiz
Department of Library and Information Studies, University at Buffalo, State
University of New York, 534 Baldy Hall, Buffalo, New York 14260
meruiz@buffalo.edu

Lorna Peterson
Department of Library and Information Studies, University at Buffalo, State
University of New York, 537 Baldy Hall, Buffalo, New York 14260
lpeterso@buffalo.edu

This research compares the usability and efficiency of a course-advising chat bot
with menu driven FAQs (frequently asked questions). Based on a survey and user
interviews, a text-based FAQ system was created and compared with a chat bot that
was developed to address library and information science (LIS) graduate student
course and program related questions. The students conducted tasks with both the
chat bot and FAQ systems. The usability and effectiveness of the functionality and
user-interface of both systems is assessed.

Introduction

A chat bot is an interactive computer program that enables users to converse with a computer
like a real person through natural language. Natural language processing (NLP), one of
several areas of artificial intelligence, is used to construct computer interfaces programmed
to understand natural human languages (Artificial Intelligence, 2003). A natural language
interface allows the user to type in his or her own words to form his or her inquiry. Natural
language interaction is defined as the operation of computers that people use as a familiar

natural language to give instructions and receive responses. The main advantage of using this
type of interfaces is that users do not need to learn any artificial language because the user
can use the natural language which they already know (Shneiderman, 1998).

Among chat bots, A.L.I.C.E. (Artificial Linguistic Internet Computer Entity) is one of the
strongest chatterbots. A.L.I.C.E. encourages users to converse by applying some heuristic
pattern matching rules to human’s input. A.L.I.C.E. uses a programming language AIML
(Artificial Intelligence Markup Language) specified in its program (Wallace, n.d.). AIML is an
XML-compliant language for programming chat bot (“AIML”). For example, Stella which was
built based on novomindIQ software is the library chat bot for Hamburg State and University
Library, Germany. (fn:
http://www.novomind.com/index_ht_en.html?press/2006/rel_124_en.html ) She helps
students use the digital library and find information in a position of all web pages. Stella also
eases the workload of librarians by providing research advice, informing people of open hours,
and assisting borrowers in extending their lending periods (novomind, 2006). Pandorabots,
which has been used for this research, is a software robot hosting service based on AIML.
Pandorabots was selected because it provides the services to create and publish one’s own
chat bot via the web. Chat bots created by Pandorabots have controlled answering and voice
response capabilities with an avatar. To launch a well-developed chat bot on the web, we also
used the SitePal software, providing commercial services to create animated characters and
artificial intelligence interface.

Chat bots have remarkable features regarding information search methods and feedback as
well as interactive user interface with the function stimulating human conversation. First,
users can search information not only by keywords but by using natural language with
complete sentences. Second, through typing questions on a form interface, users can actively
retrieve information that they want. Since web users of all levels are familiar and comfortable
with form, form interfaces have the advantage of being intuitive and easy to use (Meng,
1999). In addition, chat bots can increase understanding of information by providing text and
voice responses with an avatar.

Since chat bots offer convenience in the method and process of information search, this
research will explore two aspects of a course-advising chat bot so that it could be used as an
academic tool to address course-related questions.

      In terms of functionality, is a course-advising chat bot more effective than a text-based
      FAQ to address course-related questions?
      Does the interactive user interface of the course-advising chat bot positively affect the
      process of searching and understanding information?

Usability Test

This research conducts a comparative usability test to explore that a course-advising chat bot
effectively contributes to address course-related questions in terms of its functionality and
interface.

Materials

For the test, two systems have been created: One is a text-based FAQ interface and the other
is a LIS course-advising chat bot. The text-based FAQs interface was built based on a preferred
FAQs interface selected from a short web-survey and user interviews. 40 students participated
in the FAQ interface web-survey. 46% of participants preferred hierarchically organized menu
among four types of FAQ websites. They also preferred the FAQs interface providing answers
close to questions and less clicking. We conducted interviews with the LIS students and
analyzed the current contents of our FAQ to gather the content required to create both
systems. LISAbot, which is the LIS course-advising chat bot, was created by using
'Pandorabots'. The LISAbot was technically controlled for supporting LIS course-related
questions by adding potential questions and answers in the AIML database. By using the
SitePal software, we built an avatar and an artificial intelligent interface and published the
LISAbot on the web.

Participants

All participants for the FAQs interface web-survey, the FAQs interface interview and usability
test have been recruited among the LIS graduate students enrolled at the University at
Buffalo, State University of New York in the fall semester 2006. Forty LIS graduate students
participated in the web-survey. Six LIS students participated in the interview. Sixteen
participants for the usability test were voluntarily recruited.

Procedure

A set of eight questions were designed to assess performance, the information finding
process, and interaction behaviors of users on two systems. The presentation order of the
systems and questions are systematically controlled through a randomized block
experimental design. The test consists of two sets of task sessions with a ten minute break in
between. During each task session (20 min.), participants use a system in turn to answer a set
of four assigned questions.

Before beginning the first task test session, participants filled out a short questionnaire to

collect information regarding Internet usage and familiarity with chat programs and avatars,
information seeking attitudes, and a preference for animated characters. After each task test
session, they answered a system usage questionnaire for collecting data for usefulness,
effectiveness, satisfaction, and the perception of the time to complete tasks in each system.
After completing two task test sessions, they answered a post-test questionnaire about the
preference of the user interface and comparative functionality between both systems. We
collected data for the observation of the information finding process, performed tasks, logs
files, and user feedback.

Results and Analysis

Results are analyzed for all data collected in task test. The t-test method is used for data
analysis obtained from three kinds of questionnaires to compare two systems regarding
usefulness, effectiveness, learnability, satisfaction, and the perception of the time to
complete tasks.

Conclusion

Although this is a research in progress our initial results from the user interviews indicate that
the LISAbot could be a potentially successful interface in terms of easy accessibility of FAQs
and information search method. However, users have numerous open questions and various
expressions for questions on LISAbot. Our future work includes a formal evaluation of the
effectiveness of this interface in helping course-related questions. We plan to complete user
tests by the time we present the poster and include the data gathered at that point.

Reference
AIML (n.d). Retrieve June 30, 2006, from http://www.alicebot.org/aiml.html

Artificial Intelligence (2003).   Retrieved July 29, 2006, from the AccessScience database

Meng, F. (1999).    A natural language interface for information retrieval from forms on the
World Wide Web.      Paper presented at Proceeding of the 20th international conference on
Information Systems. Retrieved August 3. from      http://portal.acm.org

novomind (2006).      Hamburg University library wins government-backed IT award. novidmind
Press, November 21st, 2006. Retrieved March 15, 2007, from
 http://www.novomind.com/index_ht_en.html?press/2006/rel_124_en.html

Shneiderman, B. (1998).      Designing the User Interface (3rd ed.).   Harlow, England: Addison
Wesley Longman, Inc.

Wallace, R. S. (n.d.).   The Anatomy of A.L.I.C.E. A.L.I.C.E. AI Foundation   Retrieve July 29,
2006, from     http://www.alicebot.org/anatomy.html