Abstract - Thomas J. Watson ...

Managing the Mobile Inbox with MoMail
                                           Li-Te Cheng, Daniel Gruen
                                      Collaborative User Experience Group
                                        IBM T.J. Watson Research Center
                                             Cambridge, MA 02142
                            { li-te_cheng@us.ibm.com , daniel_gruen@us.ibm.com }

Abstract

Email, an indispensable communications tool on desktop computers, has a growing role on mobile devices.
However, the varied ways that desktop users use their email combined with known difficulties of designing
mobile applications pose a significant challenge to creating a mobile email user interface. In this paper, we
describe MoMail, a functional PocketPC prototype to explore a rich user email experience on a PDA. We
highlight MoMail’s chief user interface contributions that help users manage their email, discuss reactions from
prospective users, and reflect on lessons learned and suggestions for future work.

Summary

Email, an indispensable communications tool on desktop computers, has a growing role on mobile devices.
Desktop email users already face numerous difficulties that are only exacerbated by limitations of mobile devices.
Desktop users manage their email in ways that would be difficult to support on mobile devices. Furthermore,
mobile users have different goals when managing email. These problems pose a significant challenge to creating
a mobile email user interface and motivated us to develop MoMail, a functional PocketPC email client prototype.
MoMail focuses on exploring a rich user interface for email users, most notably in managing incoming messages
in an enhanced inbox view. Because many users spend much of their time in the inbox, and use their position in it
to situate their work, we created a number of user interface contributions directly within MoMail’s inbox screen.
These contributions include thread color highlighting to allow users to identify related groups of messages at a
glance, a thread mini-map to allow users to quickly jump directly to related messages, a gestural pop-up menu for
fast marking and deletion of messages, and a one-line “ticker-tape” message preview allowing users to read
multiple messages simultaneously without leaving the inbox screen. We presented our prototype to prospective
users and met with positive feedback, including suggestions to bring these ideas back to desktop email user
interfaces. Our future work includes further user interface explorations in mobile device form factors beyond
PDAs and formal user evaluation.

Keywords
Email, mobile user interface, information visualization, gestural pop-up menu, email threads

Introduction

Over the past 20 years, email has evolved from simple electronic letter writing to include much richer and varied
usage. Today emails serve as conversations, event notifications, sources of background information, contact lists,
records of past activity, and components of business workflow and electronic commerce transactions. Many
people describe "living in their email", with email playing a central role in how they organize, remember, and
conduct their work [3].

As the role of email has grown, the need for people to access their email while away from their desktop computers
has grown as well. This can be seen in the growth of tools such as the RIM Blackberry, Handspring Treo,
Symbian-based devices, Palm, Danger Hiptop, and connected Pocket PC devices that enable people to be
informed of new mail as it arrives regardless of where they are, and lets them send simple messages to others.

There are several challenges in developing an effective mobile email client. Many of these arise directly from the
physical form-factors involved. Small screens limit the number of items that can be visible at any one time and
make traditional windowing schemes unsuitable. Input mechanisms using styli, onscreen keyboards, or thumb
keypads make text entry difficult. Processors are also generally limited, providing less computing power than
desktop machines. Storage is often limited, and network costs and bandwidth limitations constrain the amount of
data that can be transported.

More fundamentally, most traditional mobile email clients are scaled-down, limited functionality clones of
desktop email applications. Because of this, these clients are useful tools for keeping abreast of incoming mail and
crafting brief responses, but are not well suited to many of the more complex ways people currently work with
email on desktop computers [3][9][7]. For example, people frequently read through a number of related messages
before replying. They consult one message – often after a search of some sort – for information or content to be
included in one they are writing. They copy from one, or from several messages, into a new one. People also use
their mail as a dynamic contact list, searching for previous messages as a way to locate a person’s name, phone
number, or email address. And people use their mail as a personal knowledge base, for example by searching
through old mail to find a resume someone sent them or to remind themselves when (and why) a decision was
made. Such activities are often hard or impossible to perform using current mobile email clients, which do not
allow people to keep several items open at once or to search old mail that may not be stored locally on the
portable device, and whose limited screen real estate can make navigation among items scattered through an
inbox laborious and clumsy. These practices become constrained by the needs of a mobile user, who usually is
“on the go” with less time to manage messages than a desktop user. Thus a mobile user needs faster ways to go
through messages and prioritize them, fewer screens to read through and reply messages, etc.

A number of trends support new approaches to deal with these challenges. Mobile device hardware is constantly
improving in graphical, networking, and processing performance. Network infrastructure continues to evolve,
supporting the delivery of rich multimedia email content. Packaging and form factor of new devices continue to
become more compact and ergonomic, as well as feature high resolution screens and different input mechanisms.
Software middleware, agents, and web services are growing in sophistication, being able to filter, transform, and
package data into formats and quantities that are more manageable to users of mobile devices.

Our approach focused on improving the user interface of the mobile email client itself, with the goal of supporting
more complex email activities. To explore the user interface possibilities, we built MoMail, a prototype
PocketPC based email client.

In this paper, we present MoMail’s user interface contributions around managing messages in the email inbox.
We conclude by discussing about related work, reactions from prospective users, and suggesting future areas for
research.

Introducing MoMail

Our prototype was created using Macromedia Flash to supply the user interface and housing the Flash interface
within a C++ application (making the entire application appear like a standalone program, not a web-based one),
all running on a Pocket PC 2002 PDA. Since our focus was on a user experience, this approach allowed us to
iterate through interface ideas quickly, given the relative ease of creating compelling interfaces in Flash. A
complete discussion of our implementation and design process, as well as features beyond the managing messages
in the inbox (e.g. composition) is the topic for another paper.

Like most email applications, MoMail’s default screen is the “Inbox screen” (see Figure 1). This screen appears
similar to the inbox shown in traditional email applications. It contains a list of email messages sent to the user,
with each message being identified by the sender, the date, and the subject.

Figure 1: The inbox with thread color highlighting and mini-map (a) A single tap on Scott’s email reveals two
replies (b) Navigating to the first reply (c) Navigating to the second reply (d) An example of how a reply in a
complex email thread gets highlighted

We gave MoMail the familiar appearance of a traditional email application, but our inbox screen is enhanced with
a number of features to examine message relationships and manage them, without leaving this screen. Thus, the
inbox becomes more than a browsing widget; it also becomes the mobile user’s workspace to triage email
messages. More time is focused on marking, deferring, replying, and deleting many messages in the queue at
once, and less time is spent reading entire messages one by one.

These features can be grouped into three parts, which are presented in the following sections:

    •   Threading: seeing messages in context of their replies (“email threads”) and navigating within this context
    •   Pop-up gestural menus: marking and acting directly on messages in the inbox
    •   Reading: previewing messages without switching screens and launching a dedicated view of the text of a
        selected message, and navigating to related messages

Threading

Email messages are often related by being part of the same discussion thread or activity, yet they often appear
scattered as separate items in list views. This can make it difficult for people to locate related items, to notice if
an issue raised in a message has already been handled by someone else, or to follow a discussion through to get
up to speed. This can be particularly difficult when using a portable device with a small screen as there is less
opportunity to see related items in a large on-screen list. Furthermore, due to the effort required to enter text on
many such devices, the cost of responding to a message that has already been handled by someone else is greater.
Our research group’s past work has addressed this problem through thread maps meant for desktop on screen use,
which depict all the messages in the thread and their hierarchical structure [8]. In MoMail, we use color
highlighting of thread items and designed a smaller, simplified view of a thread that would provide much of the
navigation and awareness of a full thread map.

Color Highlighting

Unlike a newsgroup thread, which tends to group threads around topics, email threads can have a strong sense of
temporal and personal urgency (e.g. responding to a high priority question from a manager before the end of the
day). A traditional inbox layout helps present some of this urgency using time-based ordering, but it scatters
messages from a thread around the inbox.

To keep the benefits of the inbox view and yet show thread relationships, MoMail’s inbox view uses color
highlighting. Color highlighting helps group messages into smaller sets of conversational context, allowing users
to skip over entire groups of irrelevant messages, and focus on the important discussions. Tapping any message
item in the inbox will “select it” (whereas double-tap “opens” it, as discussed later), which highlights the message
blue. All the message’s replies (direct replies and replies to other replies) are highlighted in orange.

For example, in Figure 1 (a), the user selects Scott’s email by tapping on it once with the stylus, which colors his
message blue. The two replies to Scott’s email (from Vladimir and Linda) immediately get highlighted in orange.
This shows that Scott’s email is at the beginning of the thread. Tapping on Vladimir’s message selects his
message in blue and highlights the preceding and following message in orange, as shown in Figure 1 (b). This
shows that Vladimir’s email is in the middle of the thread. Tapping on Linda’s message selects her message in
blue, and highlights the preceding two messages in orange, as shown in Figure 1 (c). This shows that Linda’s
email is at the end of the thread.

Mini-map

MoMail’s “mini-map” shows the existence of other messages in the thread before or after the current one. We
simplify the thread as a linear sequence of messages ordered by date, which lets the user step backwards or

forwards through the thread. While this does not reflect the real hierarchical structure of replies in an email
thread, the mini-map gives a general sense of how many items precede or follow the current message (none, one,
two, or many). This visualization allows the user to explore a message’s related responses directly as a single
entity instead of requiring scrolling to search for related messages. The mini-map is workable in a tiny amount of
space, and a user can jump immediately to threaded messages that could be far away from the current position in
the inbox.

The mini-map is best illustrated by reusing the same example about Scott, Vladimir, and Linda’s email thread.
Figure 1(a) shows a selected message from Scott (in blue) and two replies from Vladimir and Linda (in orange).
The mini-map is shown in the bottom right corner of the image, depicting the current position in this thread as a
blue circle connected to two circles stacked on top of each other. The currently selected message is represented
by the blue circle. The two replies are represented by the stack of two circles.

Tapping on the stack of two circles with the stylus advances to the next message in the thread (Vladimir’s), as
shown in Figure 1 (b). The mini-map now shows a blue circle preceded and followed by red circles. The blue
circle depicts Vladimir’s message being preceded by Scott’s original email and followed by Linda’s reply.
Tapping on the red circle to the left would navigate back to Scott’s email (the email prior to Vladimir’s).

Tapping on the red circle on the right selects the next message in the thread, Linda’s email, as shown in Figure 1
(c). The mini-map now shows the end of the email thread, with the last message (Linda’s) represented by a blue
circle preceded by a stack of two red circles (Scott and Vladimir’s).

The mini-map is only able to depict email threads with none, one, two, or “many” items. Figure 1 (d) shows a
long complex thread with many messages. Jorge’s email is currently selected (in blue), and the mini-map shows
his message preceded by one message (Dan’s) and followed by three or more messages. The one preceding
message is shown by a red circle, while the following messages are represented by a single “clump” of three red
circles.

Gestural Pop-Up Menus

In addition to browsing the inbox for messages, users typically perform actions on email in the inbox. These
actions can include flagging a message as urgent, deleting messages, forwarding messages, and replying to them.
Our Momail prototype supports these actions in the inbox screen through pop-up graphical menus that are
accessible next to each inbox message. These menus use careful icon placement to encourage memorization of
directional one-stroke actions. For example, a user going through a list of email messages might want to quickly
mark some for deletion, and others as important for later action. The user might also wish to invoke actions on
the items, such as copying a message to a calendar entry, replying to a message, or adding a notation.

We designed a “gestural pop-up menu” interface component allowing users to mark an item for deletion through a
quick dragging gesture (using the mouse or a stylus) to the edge of the display (corresponding to the notion of
"throwing the item out") or mark it as urgent by dragging upward (corresponding to the notion of "elevating it" to
"higher" importance). Additional possible actions are displayed and available as well. The component was
designed so options appear to left and above the initial stylus click, where they would be less likely to be blocked
by a hand holding a stylus than traditional desktop-style pop-up menus which appear below and to the right of the
mouse click.

Our menus are inspired from circular pie menus, also known as marking menus, which have been empirically
shown to have memorization benefits [6][2]. A distinction is that gestural pop-up menus are non-circular and
more like a hybrid of pie menus and a toolbar: “memorizable” elements are closer to the pen-tip to the immediate
left and top (“delete” and “urgent”), while the remainder of the strip is a horizontal layout of icons much like a
toolbar. Another distinction is that our gestural pop-up menus, which consist of six menu items, are carefully laid
out to minimize any obscuring of the inbox content, whereas traditional pull-down context menus and circular pie
menus with the same number of items would block more inbox content.

Figures 2 (a)-(d) show an example of marking a message for deletion. Figure 2 (a) shows the “Hot Deals”
message being selected by a single tap of the stylus. In Figure 2 (b), the gestural pop-up menu is invoked by
pressing the stylus down on the dot (The menu disappears as soon as the user stops pressing). In Figure 2 (c),
dragging the stylus to the left onto the “X” icon, marks the item for deletion. Selecting the “X” icon also puts up
the “Delete” tooltip over the pop-up menu. After dragging to the left, the stylus is lifted and the item is then
shown to be marked for deletion as shown in Figure 2 (d).

Similarly, in Figure 2 (e), pressing on the dot and dragging the stylus upwards to the “!” icon marks the message
as urgent. Figure 2 (f) shows the item after it is marked as urgent.

Reading

The user can sift through the MoMail inbox, using threading to identify relevant discussions and irrelevant ones,
and using gestural pop-up menus to mark up messages and remove them. After all of this, the user can choose a
subset of messages to focus on. MoMail provides one-liner previews to allow users to glance at the first few
sentences before committing to reading on a complete full-screen.

One-Liner Preview

Instead of taking up screen real-estate for a typical preview pane, we designed a control that allows the subject
areas in the list to be turned into a one-line horizontal scrolling field in which the body text of the message can be
displayed and scrolled, in a manner similar to an on-screen ticker tape. The user can control the scrolling of the
list in various ways, such as by clicking or pressing with a stylus on arrows at the side of the scroll area or
"grabbing" and dragging within the scroll area. The user can also dismiss the scroll area, returning the item to the
way it appears in the normal list.

For example, consider the snapshot of the inbox screen in Figure 3 (a), where Scott’s email is selected in blue.
Tapping the “eye” icon by the subject line (“Thoughts on the UIM Proposal?”) replaces the subject with the one-
liner preview, as shown in Figure 3 (b). The preview shows the beginning of the "body" of the message (“Please
send me your thoughts…”). The user can scroll the text by pressing on the arrows at either side of the scroll area,
or by grasping and dragging the text within the preview area, as shown in Figure 3 (c). This way of controlling
text in scrolling "ticker tape" style windows exists in other applications as well. Our contribution is having such a
ticker area appear in-place among other items in a list, without affecting the other items. It is possible to have
such scrolling text areas appear for multiple items in the list at the same time, as desired by the user, as
demonstrated in Figure 3 (d). Each one-liner preview is invoked and controlled separately. Clicking on the eye
icon again would dismiss the scroll area, and display the subject information as it initially appeared in the list.

Full Screen Preview

There will be times when a user will wish to see an entire email message in a larger region of the screen.
Double-tapping any message in the inbox will bring up a full screen view of the selected message, as shown in
Figure 4 (a). The full screen view contains elements found in standard email applications, i.e. subject, sender,
date, and body, but it also contains the gestural pop-up menu and the thread mini-map, so the user need not return
to the inbox to perform actions or navigate through a thread.

For example, on the bottom right of Figure 4 (a), the mini-map presents Scott’s message in the full-screen view as
the beginning of an email thread. The mini-map can be used to navigate to other messages in the thread like in
the inbox screen. So tapping on the red circle following the current message (blue circle), advances to the next
message in the middle of the thread, as shown in Figure 4 (b). Tapping again advances to the final message
(Linda’s), as shown in Figure 4 (c).

Figure 2: Using gestural pop-up menus (a) A single tap selects the message from “Hot Deals” (b) Holding the
stylus down on the grey dot by the message brings up the pop-up menu (c) Sliding the stylus to the left selects the
“Delete” choice (d) The message is now marked for deletion (e) Similarly, sliding the stylus up selects the
“Urgent” choice (f) The message is now marked as urgent

Figure 3: Inline preview in the inbox (a) A single tap selects Scott’s message (b) A single tap on the “eye” icon
beside the subject line “Thoughts on the…” replaces it with a preview of the body of message (c) Tapping the
triangles beside the preview scrolls the message (d) Several message bodies can be previewed at once in this
manner

Figure 4: Full screen viewer for email (a) A full screen view of Scott’s email with a thread mini-map on the
bottom right (b) Tapping on the next red dot in the mini map brings a reply from Vladimir (c) Tapping again on
the mini map goes to the last reply. A gestural pop-up menu can be brought up by holding down the stylus on the
grey dot by the subject (d) Linda’s message is shown marked as urgent

Navigating with the mini-map in this context brings up related messages in the full screen view, thus the user does
not need to jump back to the inbox screen. Also, each “opened” message, regardless if it was opened via the
mini-map or from the inbox, has a tab on the bottom of the screen. Tapping on the tab beneath the current
message in view “minimizes” it, bring up the last viewed message (if there is only one message, the inbox screen
is shown). Tapping on a tab on another message “maximizes” that message to full screen view. Tapping on the
“x” on a tab “closes” the message and removes its tab. Up to three tabs can be presented this way. This allows a
superficial semblance of a “multi-windowing” environment to the user, permitting the user to switch back and
forth between messages to compare, copy, paste, and edit a response. If more than three messages are open, the
first one that was opened is closed.

Figure 4 (c) and (d) show the gestural pop-up menu being used to mark a message as urgent. The grey dot beside
the subject allows the user to access the gestural pop-up menu without going back to the inbox. Providing a
gestural pop-up menu removes the space required by a menu bar, and remains consistent with the inbox user
interface.

Discussion and Conclusions
While a formal user evaluation is planned for the future, we did collect informal reactions from a large group of
prospective users. The prototype was shown to more than 50 people through demos during three consecutive
days in a public event, as well as during visits to our research lab over the course of several months. Feedback
was positive, and respondents felt MoMail’s user interface ideas as usable and desirable in commercial offerings.
Many people saw the need to directly address email user interface issues on mobile devices, and that it would
directly influence whether users will bother using mobile email on a regular basis. People saw the value of using
threading to manage mobile email. There was a great appreciation in the aesthetics of the user interface design,
and the integration of threading, navigation, gestural menus, and preview in the same inbox screen.
There were suggestions for improvements. Mini-map navigation could be more fine-grained than its simple one-
step forward and backward approach. The one-liner preview should provide automatic scrolling in addition to
manual scrolling. Folder management should be supported. We need to address how to clearly identify read and
unread messages. A number of people would like to see MoMail to step beyond email and consider integration
with other common functionality found on PDAs, such as calendaring, to-dos, and addressbook management.

Practically every PDA on the market has an email client available, as do many smartphone devices, and mobile
devices designed primarily for email. They all offer some view of messages in an inbox, and generally some
functionality to manage messages therein, usually through traditional context menus. While these applications
offer options to mark messages, delete, etc, or manually color highlight individual messages, we believe our
examples of enhanced support for thread displays, thread navigation, or multiple simultaneous one-line previews
are novel features for mobile devices.

On the desktop, there are examples of using email threads for visualization and navigation, such as those
presented by Rohall [8], whose work inspired our thread color highlighting and mini-map ideas. Microsoft
Outlook and newsgroup readers group messages by topic, effectively creating threads. Also, email threads need
not be restricted to just a chain of reply messages as what we used in MoMail. Bellottti et al’s “thrasks” [1] use
threads of related messages, files, and tasks, is an example of how the inbox can be tightly integrated with other
information.

Work exists in information visualization on mobile devices, which is related to the challenge of representing
email threads on a PDA, as well as development of mobile-specific user interface widgets. We describe two
notable examples here. For example, Kamba et al reviews the challenges of creating user interfaces and options
for visualizing and manipulating information on PDAs, and then presents semi-transparent widgets [5]. Although
such widgets provide a large area for user interaction without obscuring too much content on the screen, there are
some issues on how to select content behind the widget.

There are other mobile device form factors than just PDAs, such as smartphones. A smartphone design would
have to consider how to present different types of messages in the inbox than just email, such as voicemail and
“multimedia-mail” offered by some 3G phone services. Many smartphones still do not have touchscreens, and
thus gestural pop-up menus and mini-map navigation would have to be mapped against physical buttons.

Another direction is to apply our mobile user interface ideas back to the desktop, as was suggested by some of our
respondents. One useful application is when desktop screen real estate is constrained, such when web-portal
components are displayed in a small region on a larger screen. Another is for providing context on generic lists of
items: a gestural pop-up menu allows users to quickly mark or act on items without shifting their attention or
cursor from the item at hand.

In conclusion, as mobile email grows in usage, mobile email user interfaces need to support interactions that help
deal with the flood of incoming messages. MoMail shows that it is possible to create a user experience to tame
the inbox despite the limited screen size and input mechanisms of PDAs. Our approach is to instrument the inbox
with readily accessible tools to identify and navigate threads of related messages, preview multiple messages
inline, and gesturally mark up messages. Even when reading message in full screen, thread mini-maps and
gestural pop-up menus help save the user time in managing email. There remains more work be done. This
includes validating the user interface with a formal user study, supporting more ways how people cope with
email, and addressing non-PDA devices.

Acknowledgements
We would like to give special thanks to our interns, Devon Rueckner and Jorge Ortiz, who were the key
implementors behind the MoMail prototype.

References
[1] V. Bellotti, N. Ducheneaut, M. Howard, and I. Smith, “Taking Email to Task: The Design and Evaluation of a
Task Management Centered Email tool,” Proc. ACM CHI ’03, Ft. Lauderdale, FL, April 5-10, 2003, pp. 345-352.

[2] J. Callahan, D. Hopkins, M. Weiser, and B. Shneiderman, “An empirical comparison of pie vs. linear
menus”, Proc. ACM CHI ’88, Washington, D.C., 1988, pp. 95-100.

[3] N. Ducheneaut and V. Bellotti, “E-mail as Habitat: An Exploration of Embedded Personal Information
Management,” ACM interactions, vol. 9, no. 5, 2001, pp. 30-38.

[4] Dunlop, M., and Davidson, N., “Visual information seeking on palmtop devices,” Proc. HCI ‘00, vol. 2,
September, 2000, pp. 19-20.

[5] Kamba, T., Elson, S., Harpold, T., Stamper, T., Sukaviriya, P. “Using small screen space more efficiently,”
Proc. ACM CHI ‘96, Vancouver, Canada, 1996, pp. 383-390.

[6] Kurtenbach, G. and Buxton, W. “User learning and performance with marking menus”, Proceedings of the
SIGCHI conference on Human factors in computing systems, ACM Press, Boston, MA, 1994, pp. 258-264.

[7] W. Mackay, “More Than Just a Communication System: Diversity in the Use of Electronic Mail,” Proc. ACM
CSCW ’88, Portland, Oregon, 1988, pp. 344-353.

[8] S. Rohall, “Redesigning Email for the 21st Century Workshop Position Paper,” IBM Technical Report 02-17
http://domino.research.ibm.com/cambridge/research.nsf/pages/papers.html, (Current 2003).

[9] S. Whittaker, and C. Sidner, “Email Overload: Exploring Personal Information Management of Email,” Proc.
ACM CHI ’96, Vancouver, Canada, 1996, pp. 276-283.