вход по аккаунту


Design of an adaptive wayfinding system for individuals with cognitive impairments

код для вставкиСкачать
This reproduction is the best copy available.
Design of an Adaptive Wayfinding System
for Individuals with Cognitive Impairments
Alan L. Liu
A dissertation submitted in partial fulfillment of
the requirements for the degree of
Doctor of Philosophy
University of Washington
Program Authorized to Offer Degree: Computer Science L· Engineering
UMI Number: 3431611
All rights reserved
The quality of this reproduction is dependent upon the quality of the copy submitted.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.
Dissertation Publishing
UMI 3431611
Copyright 2010 by ProQuest LLC.
All rights reserved. This edition of the work is protected against
unauthorized copying under Title 17, United States Code.
ProQuest LLC
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, Ml 48106-1346
University of Washington
Graduate School
This is to certify that I have examined this copy of a doctoral dissertation by
Alan L. Liu
and have found that it is complete and satisfactory in all respects,
and that any and all revisions required by the final
examining committee have been made.
Chair of the Supervisory Committee:
Gaetano Borriello
Reading Committee:
A ^A^J2P,
Gaetano Borriello
****&Henry A. Káutz
Jacob O. Wobbrock
JW 2-?> ~z*>\o
In presenting this dissertation in partial fulfillment of the requirements for the doctoral degree at the
University of Washington, I agree that the Library shall make its copies freely available for inspection. I
further agree that extensive copying of the dissertation is allowable only for scholarly purposes, consistent
with "fair use" as prescribed in the U.S. Copyright Law. Requests for copying or reproduction of this
dissertation may be referred to ProQuest Information and Learning, 300 North Zeeb Road, Ann Arbor, MI
48106-1346, 1-800-521-0600, to whom the author has granted "the right to reproduce and sell (a) copies of
the manuscript in microform and/or (b) printed copies of the manuscript made from microform."
University of Washington
Design of an Adaptive Wayfinding System
for Individuals with Cognitive Impairments
Alan L. Liu
Chair of the Supervisory Committee:
Professor Gaetano Bordello
Computer Science & Engineering
Many individuals with cognitive impairments experience difficulty with wayfinding, that
is, the process of traveling from one place to another. This functional limitation decreases
their opportunity to live independently, maintain employment, and participate in their
community. This dissertation addresses the design of a system to help such individuals in
wayfinding by presenting personalized, multi-modal directions to users by mobile phone.
By creating prototypes that present directions to individuals with cognitive impairments
in real indoor and outdoor environments, I have observed a wide range of potential users'
reactions to different types of directions, both turn-based directions common in current
navigation aids and landmark-based directions that use photos augmented with path arrows.
As the use of landmarks in wayfinding has been found to be beneficial, and landmark-based
directions are becoming more widely available because of the growing availability of geotagged photos and other location information, my study results inform the creation of
criteria for choosing landmarks useful to wayfinding.
A recurring theme found in my studies has been the need to address the wide variation
in individual wayfinding abilities and preferences. Addressing this need is a challenge as
t manually adjusting an interface's design, tailoring it for each individual user, has required
significant cost in the past. To support such variation, I have developed a theoretical framework that creates customized and adaptable wayfinding directions to individual users. The
framework relies on a Markov Decision Process (MDP) problem formulation to create understandable sequences of directions for a given user. The MDP uses a model to describe the
user's likelihood of correctly following possible directions in different wayfinding situations.
We can improve model accuracy by incorporating new usage data over time, allowing the
system to adapt. Adaptation is an important capability as the physical and mental effort
of wayfinding limits how much observational data can be collected at once.
My user studies involving the framework generating directions show that individuals with
cognitive impairments can successfully wayfind when they have access to tailored wayfinding directions, demonstrating the design's potential to reduce a significant barrier to their
quality of living.
List of Figures
List of Tables
Chapter 1:
1.1 Explanation
1.3 Design Goals and Challenges
Chapter 2:
Related Work
2.1 Studies of Assistive Technology
2.2 Wayfinding Research
2.3 Design Discussion
· ·
Landmark Selection
2.5 User Modeling
Chapter 3:
Wayfinding Feasibility Study
3.1 Wizard of Oz Prototype
3.4 Design Discussion
Technical Discussion
3.6 Next steps
Chapter 4:
Outdoor Route and Landmark Studies
4.1 Outdoor Route Study
4.2 Outdoor Landmark Study
4.4 Next steps
Chapter 5:
Model-driven Wayfinding
5.1 Using a Markov Decision Process to Produce Directions
5.2 User Study
5.4 Next steps
Chapter 6:
Direction Difficulty Study
User Study
Results . . :
6.3 Modeling direction difficulty
6.4 Discussion and next steps
Chapter 7:
Wayfinding with Adapted Models
7.1 Incorporation of Adapted Models
7.2 User study
Chapter 8:
Future Work
Figure Number
A direction from Alice's wayfinding system
1.2 A potential automated wayfinding system diagram
Examples of augmented landmark-based photos
Sample images used in the interface
Sample iPAQ display
Screen shot of the navigation wizard GUI
Location wizard GUI running on a Tablet PC
System interaction diagram
Route la and lb
Route 2
Route 3
3.9 Example directions that confused or eased confusion
3.10 Example directions that had timing issues
Sample images used in the interface
4.2 Screen shot of the navigation wizard Tablet PC GUI
4.3 The three outdoor route study routes
4.4 An example of the client in the landmark analysis study
4.5 The landmark analysis study set-up
The Tablet PC Wizard of Oz GUI
5.2 Wayfinding system front-end on Nokia N95
5.3 Examples of turn-based directions
Wayfinding system front-end on Nokia N95
Route preview feature
Example directions used in study route
Study route
Frequency of incident types by participant
Frequency of incidents by direction
The most problematic direction in the study
Another problematic study direction
Observed participant result graph by train model prediction
Correlation of predicted and actual incident rate by model
Correlation coefficient by model
Cumulative incidents and nonincidents of directions by predicted difficulty of
train and train + 4% P models
Landmark selection criteria
7.2 Route used with the train + 4% P model for each returning participant. . . . 127
8.1 Examples of different techniques to visually show a path involving landmarks. 134
Table Number
Study route features
Participant demographics
User study results
Modality preference by participant
4.1 Outdoor route study participant demographics
4.2 User study results
4.3 Landmark study participant demographics
Participant demographics
Frequency of direction types in the study route
Participant demographics
Direction incidents by participant
Features used in linear regression models
Example difficulty models, trained for Participant 9
Correlation coefficents by model and participant
The number of directions presented to each participant in each condition. . . 126
ACTION: response by the system that transitions from one state to another state, typically to present a new message to the user
COST: a negative amount associated with an action that the system minimizes, repre-
senting physical and cognitive effort and/or preference
DIRECTION: a message that guides the user through some path
LOCATION WIZARD: a person who follows the study participant in a Wizard of Oz study
setup, simulating a system that tracks user location and orientation
MARKOV DECISION PROCESS: framework for simulating the behavior of an agent in an
environment, defined by a set of states, actions, transitions, costs, and rewards.
MESSAGE: any information that the interface intends to convey to the user
NAVIGATION WIZARD: a person in a Wizard of Oz study setup who sends instructions
controlling the prototype client interface output
OPTION: a sub-policy that replaces actions in the MDP framework so that reasoning
can be done over extended time steps
POLICY: a mapping of every possible state to a single action
PROMPT: a message that is triggered in order to help the user return to a path
REWARD: a positive value for states at the goal, otherwise negative values that are
accumulated after every state transition and thus minimized
STATE: a representation of what the wayfinding system knows of the user
TRANSITION: a change from one state to another due to a system action
WAYFINDING: any method for orienting and navigating in physical space from place to
WIZARD OF OZ STUDY: a user study were participants interact with a system prototype
that has aspects such as logic or output simulated manually by a researcher
I wish to express sincere appreciation to my advisors Gaetano Borriello and Henry
Kautz for their guidance and wisdom throughout my research efforts. I am grateful for
the invaluable help of. my collaborators Harlan Hile, Pat Brown, Mark Harniss, and Kurt
Johnson, who helped to make my studies possible, to Brian Ferris for his project groundwork,
and to Aditya Vaidya for his assistance in the early outdoor user studies. I am indebted
to the men and women who volunteered their time and effort to participate in the studies,
providing not only helpful and honest feedback to improve the system, but also motivation
for tackling the wayfinding problem.
I would also like to thank my other committee members, Jake Wobbrock and Julie
Kientz, and James Fogarty, who filled in during my defense on short notice. James and
Richard Ladner both helped me with their input and opinions on my research, lending their
experience in modeling and working in the area of assistive technology, respectively.
During my time as an undergraduate, I benefitted greatly from having James Landay
and the members in Berkeley's Group for User Interface Research as role models. I am
equally thankful to members of the dub group for fostering a communal atmosphere for
HCI research at UW.
Last but not least, I wish the thank my family and friends for their love and support,
with special thanks to my wife Jen and Sandra Fan, Craig Prince, and Tyler Robison, whose
presence helped make my journey joyful and rewarding.
to Jen
Chapter 1
In the United States alone, there are an estimated 22 million people who are affected
by cognitive impairments due to health conditions such as traumatic brain injury, cerebral
palsy, Multiple Sclerosis, and Autism Spectrum Disorder1. The functional limitations associated with these conditions can make it difficult for individuals to live independently and
maintain employment. One such activity that some individuals find difficult is wayfinding,
the act of orienting oneself and navigating from place to place. Individuals with sufficient
support systems to overcome the difficulties can be productive members of the workforce,
which then provides them economic freedom and a sense of pride and accomplishment [64] .
Unfortunately, when individuals cannot wayfind safely and/or independently, the burden on
caregivers and community services (for example, "paratransit" vans and other on-demand
services) increases [73] and opportunities to act independently and participate fully in the
community decrease.
New ways of representing and delivering wayfinding directions are being made possible
by advances in pervasive computing technologies. Location-sensing technologies are becoming more widespread and connected to other sources of data useful for wayfinding, such as
Geographic Information Systems (GIS). The ever-increasing capabilities and connectivity
of mobile phones makes them a logical platform for presenting location-based data. Initial explorations into viewing wayfinding information in an augmented reality framework
have shown promise, with ongoing research in computational photography that involve
assembling and transforming collections of geo-tagged photos further improving visual representation of directions.
The following section describes a hypothetical scenario that illustrates how we can incorhttp : //www . colemaninstitute . org/overview . php
L.^.J JM
Figure 1.1: A direction that Alice's wayfinding system presents to her showing a landmark
photo augmented by an overlaid path arrow, with associated text and audio messages telling
her to take the next right at this building and cross the crosswalk.
porate such advances to provide better wayfinding assistance to an individual with cognitive
1.0.1 A Wayfinding System Usage Scenario
Alice is commuting from her home to a new job at a university library. A job coach helped
train her to accomplish the tasks her job demands, but due to the long wait-list for his
services, could only help her for a few days in learning the new route to work. That wasn't
enough time for Alice to memorize the route, so she uses a wayfinding application on her
mobile phone to give her directions along the way. As she walks toward an intersection,
the wayfinding application alerts her by briefly vibrating and playing an audio alert. The
audio is played through a phone headset that Alice wears because she does not want to draw
attention to herself, especially in an unfamiliar area. Since the application runs on her mobile
phone, a device commonly seen and manipulated in public, she feels comfortable referring to
it regularly. After alerting her of the upcoming intersection, the application shows a photo
of a distinctive building where she should turn right (see Figure 1.1). Overlaid on the photo
is an arrow to emphasize the turn in relation to the building, and accompanying the image
is corresponding text. This same text is also converted to speech and played through her
headset. Alice glances down at the direction, then looks up to confirm that the landmark is
indeed the one she's heading toward. The application reminds her to cross at the crosswalk,
so she looks both ways for traffic before proceeding. She appreciates this reminder, as well
as the direction being presented using the combination of modalities, because she doesn't
need to worry about not understanding the direction if the screen is washed out, or missing
the direction due to the noisy traffic around her, or forgetting what she was supposed to do.
As she makes the turn, she admires the building and its brick exterior. When she has more
spare time, she will take her own photo of the building and ask her phone to give her more
information about it. However, today she needs to get to work, and is reminded by the
application to continue along the sidewalk. She doesn't need to look at the phone's screen
this time, so she holds it by her side and proceeds. Eventually, after a series of directions,
she reaches the library and gets to work.
At the end of her first day of work, Alice uses the wayfinding application to give her
directions home. Because she's tired from working, and because dusk is causing lots of
shadows, she has trouble identifying a building at which the application wants her to turn.
She presses the Help button and the application gives her a more basic direction, a simpler
step to take the next right ahead. She takes that right, and the phone gives her that style of
more basic turn directions the rest of the way. Since Alice doesn't always look at the screen
when given directions, she mistakenly takes a left turn instead of a right turn later on. The
application notices this and silently guides her along an alternate route that is about the
same distance. However, when she makes the mistake again that would cause her to go
down a path that eventually leads to the wrong side of the university campus, it alerts her
to stop and make her way back to the last intersection. Not only is the path long, it has
few exits where she could be rerouted, and is steep and contains multiple flights of stairs
that would needlessly exhaust Alice. Since the application tells her to backtrack, Alice does
so, but slowly because she's noticed that she's at a new part of campus. She presses Help,
which brings upa photo of a nearby building that assures her that she's going the correct
way. If it were too late at night to see the buildings well, she would have pressed Help again,
and the application would have asked whether she wanted to contact her boyfriend to get
directions or pick her up at her current location. This time she just needed confirmation of
the turn, so she feels confident enough to make it home independently.
As Alice uses the application through her first week at work, it gradually adapts by
reusing the routes and directions that she successfully followed, such as using the first
building she encountered on her first day as a landmark, while reducing the frequency of
directions that she didn't successfully follow, such as the landmark-based directions that she
found difficult to see at night. After several weeks, Alice no longer needs the application for
her daily commute, but she still uses it on occasion. When there's construction that blocks
off her regular route, she follows the detour a short way and the application helps reroute
her. Other times she wants to explore the campus and its surrounding areas. She uses the
phone or her PC to enter new destinations, such as the campus museum or an interesting
lunch spot that a coworker recommended. She can also pick from places that she noted
herself when she walked by them in the past, and travel between those places and work
either by actively following the directions given by the application, or only for reference
when she's in a hurry or feeling lost. Eventually, Alice feels more and more familiar and
comfortable with the area, so she decides to visit a completely new neighborhood on her next
long weekend. She'll enter a few places she'd like to visit into her wayfinding application
and bring it along so she can sight-see with confidence.
Alice is a fictitious character whose wayfinding needs are based on real individuals with
cognitive impairments that make it difficult for them to go from place to place easily and
confidently - between home, work, friends, and community. For people with cognitive impairments, learning and retaining new routes becomes a significant task. Some individuals
have reading problems, which mean the usual methods of wayfinding by street signs, written directions, or maps are impractical. Even when one memorizes a route, detours and
unexpected changes can lead to confusion and bewilderment. Getting lost is a common
occurrence because successful wayfinding involves simultaneously tracking one's location,
remaining oriented, and being aware of surroundings.
For the above reasons, current methods to aid in wayfinding are inadequate. Written
directions require reading ability, are difficult for individuals to track their progress and
know when to transition, and are not flexible when a mistake is made. Maps require
less reading, but some ability is still necessary to orient position and associate abstract
landmarks to the physical world. There's also stigma and safety concerns with maps since
they can indicate someone's unfamiliarity with the location. Global Positioning System
(GPS) navigation units are designed to help locate the user, and can update routes based
on detours, but still require association between abstract and physical worlds, demands
more attention at the expense of environmental awareness, and lacks functionality relevant
to pedestrian movement, such as a built-in concept of physical accessibility.
Written directions and current GPS devices also do not take into account individual
requirements for what constitutes a "good" route. A "good" route is distinguished from a
poor route based on low-level issues such as accessibility, and high-level issues such as ease
to follow and safety. In other words, these current modes assume a person will not make
mistakes and that everyone has the same "optimal" path. Maps are flexible because they
allow complete freedom to route, but they are limited in the amount of information that
can help an individual unfamiliar with the area judge what route to take.
In the scenario just presented, the wayfinding system has several key features not found
in conventional wayfinding aids: the knowledge of landmarks and the ability to display
directions relative to a landmark; a concept of safety so warnings can be included in directions; a choice between the higher-level landmarks and simpler turn-based directions;
help functionality that can display supplementary directions, alternate directions, or notify another person of the user's difficulty, either by user request or automatically; and the
ability to improve direction choice and routes by observing user success over time. The
inclusion of these features is a direct result of studying the wayfinding needs of individuals
with cognitive impairments.
Figure 1.2 shows one possible arrangement of components that the wayfinding system
would require to produce the above usage scenario and how they could be divided. A user's
Mobile client
Figure 1.2: A potential system diagram showing the different components of an automated
wayfinding system. The central component is a direction selection system that determines
a sequence of directions to provide a user, adapted to the user's abilities, health condition,
and other wayfinding concerns.
destination could be picked within the application or entered into a calendar application
that is shared with caregivers. The system adds accessibility information and topography
that could be supplied by publicly available sources and service providers to traditional map
data. Similarly, landmark and place information could be assembled from various sources
such as online repositories of geo-tagged photos. Combining these data sources, along with
the user's preferences and abilities encoded as a user model, a direction policy is generated
that determines the appropriate directions and route. Directions are given based on user
position, including orientation, which can be tracked by a system that uses a variety of
sensor technologies. The client application displays those directions and responds to user
input (help, repeat, etc.) using several text, audio, and images in combination.
In this dissertation, I focus on three major contributions to the design of a wayfinding system
for individuals with cognitive impairments: (1) insight into the range of direction types
that are understandable by our target users and their preferences to inform the creation of
individual user models [48], (2) criteria for choosing landmarks useful to wayfinding when
augmented with overlaid directional arrows [49] , and (3) a framework to produce individually
customized directions informed by a trial wayfinding session that can be adapted over time
given further use [47]. I will also briefly discuss some future work necessary to implement
the features described in the example usage scenario.
Design Goals and Challenges
Limited access to individuals with cognitive impairments poses a significant challenge to
designing and evaluating any system. There is a smaller population of potential user study
participants, and no such thing as a "typical" user as each individual has à unique combination of abilities and disabilities. Recognizing this brings to the forefront the importance of
producing a flexible design that can assist in wayfinding, as opposed to a "one-size-fits-all"
In attempting such a design, it makes sense to ask whether a technological solution is
appropriate or even possible. Chapter 2 describes work helping individuals with cognitive
impairments through assistive technology, research on improving wayfinding for both such
individuals and people without impairments, and several applications of similar decisiontheoretic approaches to related problems.
The following chapters describe the iterative process undertaken in the wayfinding system design, involving potential users, caregivers, and job coaches interacting with and
providing feedback on prototype designs. User studies heavily utilized the Wizard of Oz
simulation technique to produce a more realistic user experience without requiring a fully
implemented system as a pre-condition to testing. Both qualitative and quantitative data
were used to learn about important factors affecting direction understandability.
Chapter 3 describes the first design cycle, which showed that providing wayfinding directions from a handheld device to individuals with cognitive impairments inside a building
was feasible, and found that system customization was a highly desirable feature. An initial set of different direction types, consisting of iconic or photo imagery, text, and audio,
were manually created for three routes inside the Paul Allen Center, then delivered via a
portable digital assistant (PDA) to user study participants. Each participant was shadowed
to note their comments and reactions, where they encountered difficulty, the kinds of difficulty they had, and whether they expressed preferences over direction modality and timing.
Demonstrating the feasibility of using such directions, almost all participants were able to
complete the routes with only minor issues. The most notable finding, which would become
a recurring theme, was the wide variation of preferences that participants expressed over
the direction types and modalities. Another finding was that the wayfinding system needs
rich knowledge of the environment in addition to accurate location in order to effectively
guide users [50].
Chapter 4 describes the second design cycle, which further illustrated the importance of
being able to customize the wayfinding system to support user preferences. Taking place
outdoors, two user studies were conducted with a mobile phone prototype. The first user
study examined issues raised by the indoor study that might be exacerbated in an outdoor
environment, such as the effort needed in identifying complex visual features in images on
a small screen, difficulty making correct turns at less structured outdoor pathways, and
safety issues such as dealing with traffic. "Low-level" directions, while simple, can lead to
the same kind of attention-grabbing that GPS units exhibit. In order to keep attention
more focused on the actual environment, the design must not require constant attention,
strict adherence to a route, and therefore highly precise message timing.
The second study examines the use of landmarks to provide "high-level" directions as a
way to overcome some of the limitations of the directions used up to this point. Augmented
photos of landmarks can help lower the cognitive cost of associating symbolic directions
with the physical world, and also allow a user to pay more attention to the environment
instead of the device, thereby likely improving safety. However, since photos are more
complex to interpret than symbolic directions, landmarks need to be chosen carefully and
used appropriately. In contrast to previous studies, participants did not follow routes but
instead were shown multiple landmark-based directions at several starting locations, as a
way to collect enough useful data for analysis. The results of this study helped illuminate
how to differentiate and choose landmarks - i.e., the features describing landmarks that are
relevant to wayfinding.
Chapter 5 describes the design and initial evaluation of two parts of the wayfinding
system technology that make supporting different direction types possible: a landmark
selection component that produces landmark-based directions, and a direction-selection
component that uses a decision-theoretic approach to control routing and message delivery.
The landmark-selection component uses a database of geo-tagged digital photos (photos
containing landmark and GPS location metadata) to determine landmark visibility. It
selects the most appropriate photo of a desired landmark and augments the image with
an overlaid arrow. The direction-selection component uses the Markov Decision Process
(MDP) framework to formally represent individual wayfinding preferences and capabilities.
In addition, MDPs provide a way to continuously adapt to the user, so that customizations
can be adjusted over time. This adaptation is useful for adjusting system behavior, since the
number of observations that can be used to learn a user model is initially small due to the
amount of effort required to follow directions. The initial evaluation involved studying user
reaction to two different preset user models, showing that users can notice the difference in
the resulting wayfinding experience.
While the initial study showed that the system could work using contrived models,
learning models directly from observing individuals wayfind remains a challenge due to
the limited number of data points that can be collected with reasonable effort. Chapter 6
discusses the first of a two-part study that investigated the performance of models trained
on different mixtures of an individual's observations with all participants' observations. The
models were used to predict the notion of direction difficulty, the likelihood that the direction
would be problematic for the user to follow. The first part of the study collected training
and test data through a static series of directions presented to each study participant. It
was used to determine the appropriate amount of observations from other users to use
in augmenting a model trained on the relatively scarce amount of individual observations
Chapter 7 details how the MDP incorporated the individually adapted models to generate directions. It also describes the second part of the model performance study, which
used those adapted models to guide returning participants. The study also included a small
contrasting condition where participants were given directions that were more or less dim-
cult according to their individual model than models trained with only other participants'
The work in this dissertation makes important steps towards a usable wayfinding solution, with a theoretical framework at its core and an implementation backed by evidence
supporting each of its the design decisions. Chapter 8 discusses how the work could be
extended. On a basic level, the work here omits how locations are entered into the system,
or waypoints chosen to break one route into subcomponents that would require less com-
putation. Another interesting area of future research would be in automating the detection
of conditions such as confusion, as people with cognitive impairments can become stressed
and momentarily lose the meta-cognitive skills necessary to identify when to request help.
Other conditions, such as physical well-being that can vary day-to-day for people with M.S.,
warrant the development of more sophisticated user models. Beyond enhancing the user
model representation, the system could be improved by finding more relevant and predictive features to describe wayfinding difficulty. Other path and landmark features that were
absent from our campus-only environments need to be explored. Technological advances in
distributed sensor networks could provide more metadata to the system, such as dynamic
traffic conditions and real-time pedestrian detours. Another important area of work would
be in streamlining the adoption of a system to new users. For example, techniques related
to collaborative filtering might be borrowed to seed user models with training data based
on users with similar impairments. Finally, there are remaining technical issues that need
to be addressed, such as finding the appropriate rate at which models adapt and scaling the
system to support computational limitations inherent in mobile phone platforms.
Chapter 2
Current methods for aiding people with wayfinding are labor-intensive. For example, a
job coach from an employment agency, who works with individuals with cognitive impair-
ments to support them in learning new jobs and maintaining paid employment, may work for
months helping a person learn how to travel to and from work, and even then, the individual
may at times still require assistance [61]. As a result, the majority of otherwise-employable
persons with cognitive impairments remain unemployed [10], rarely access appropriate community services, and are too often socially isolated [82].
Wehmeyer outlines the potential for personal computing to enable individuals with men-
tal retardation to accommodate for cognitive impairments in [92] . Specifically, he argues
that adapting software that are visually uncluttered, providing graphical and audio complements to text, having few decision points, presenting information sequentially, and reducing
the reliance on memory are key features to the success of using computing applications.
LoPresti et al. survey the state of the art of assistive technology for cognitive rehabilita-
tion, which they collectively refer to as assistive technology for cognition (ATC) [52] . They
note that many ATC interventions are developed for individuals whose cognitive impairments limit how much they can contribute in each stage of design and evaluation. Our work
has taken these limitations into account by including people who regularly interact with
potential users, such as family caregivers and job coaches, to participate in brainstorming
design ideas and pilot testing phases of the implementation. The design draws from usability studies of interfaces by people with cognitive impairments, psychological models of
spatial navigation [24, 55], and interviews we conducted with individuals with impairments
and their caregivers regarding wayfinding [6].
Studies of Assistive Technology
LoPresti et al. also argue that, as ATC interventions include more ubiquitous technologies
and begin to be useful out in the community, researchers should establish context generalizability or ecological validity [87]. This holds true for the wayfinding application, which by
its very nature is designed for use out in the community. Therefore, the wayfinding application needs to be designed with realistic situations and tasks in mind. Also, research to
determine the usability of designs need include evidence that they are usable by individuals
from the target user-base to accomplish realistic tasks under realistic situations. I will use
the research surveyed in this section to show what related work used for tasks, situations,
and measures for usability analysis.
In [40], Kjeldskov and Graham reviewed 102 publications on mobile human-computer
interaction research and categorized them under the definitions taken from Wynekoop and
Conger [95]. The following studies discussed will be categorized according to those definitions.
Case studies
Among assistive technology studies, several researchers have used the case study approach to
come up with design implications. Cole and Dehdashti summarized their multidisciplinary
group's research on assistive technology for enabling everyday activities for people with
traumatic brain injury or stroke in [16]. They used a single-subject case study approach
to designing successful software for use as "cognitive prostheses," for example, to aide
in check-writing, word processing, and scheduling. They describe each as a one-of-a-kind
hardware/software system customized to each user's combination of priorities, cognitive and
physical abilities and deficits. These systems were designed, implemented, and installed for
use in the users' homes. Extended use led to users' feedback regarding usability issues and
feature requests. Success was determined based on a user's increase in functioning after
the introduction of a system after the control phase, where spontaneous recovery was not
medically expected. Cole et al. attribute part of their success to matching the application
to each individual's priority activities, which was a motivator to learn new interfaces.
In another case study, Paradise et al. designed and evaluated a prototype cognitive aid
for the home to help pace an individual with traumatic brain injury through her morning
activities [66]. Involving both the user and caregivers, they created two alternative designs
and evaluated their paper prototypes. The evaluation consisted of showing the paper prototypes to the user and caregivers, eliciting feedback via a questionnaire asking what they
thought of the interfaces along design dimensions that came about from initial discussions
with all parties.
The above case studies are similar in their single-subject approach to designing interfaces.
The high time demands and lack of generalizability are fundamental drawbacks to the case
study approach, and even more pronounced when the context of a wayfinding application is
taken into consideration. There are many design decisions and implementation details that
need to be fleshed out before a functioning wayfinding application could be used by potential
users in their daily lives. On the other hand, a paper prototype of a wayfinding application
would be useless without researchers to simulate its functionality. Therefore the studies
that need to be carried out investigate the design with partially functional prototypes, lying
between the two extremes.
2.1.2 Ethnographic field studies
Lepistö and Ovaska studied the use of an Internet-based learning environment with students
from a boarding school for young people with special needs [42] . They found from pilot tests
that a "think aloud" method was not effective. Instead, they chose to interview students
to get overview of the target user population and their computer skills, observe real use
situations, and performed expert evaluation using heuristic analysis [63] of the applications.
They also chose to do informal walkthroughs with the applications rather than usability
tests because varying degrees of intervention were needed to explain tasks to the student
participants, which would make measures difficult to objectively quantify. The informal
walkthrough let participants freely explore the interface. A checklist was kept of tasks and
features that were under evaluation. A study moderator asked students to show how they
used application (which they were already familiar using). Interviews were used to get
subjective information such as student motivation for use. Interestingly, expert evaluation
uncovered different usability problems than the walkthroughs and observations, because
they missed "the big picture" and problems between moving across different parts of the
Lepistö' and Ovaska's work is interesting because they not only studied users in their
actual environments, but also because they found challenges in applying traditional usability
methodologies such as the usability test and heuristic evaluation to the design of assistive
technology. Researchers may have trouble communicating task instructions with individuals
with cognitive impairments, and such difficulty affects the ability to control experiments,
complicating data collection. At the same time, relying on "experts" is insufficient, because
it takes "triple experts", people who are familiar with usability issues, the tasks, and the
impairment conditions, to provide the highest quality feedback.
2.1.3 Field experiments
Wu et al. evaluated an orientation aid for individuals with anterograde amnesia that was
designed through a participatory design process [94]. The application was designed to
present situational information to help an amnesiac recover from disorientation. They found
that such an application has the opportunity to aid in reorientation because amnesiacs are
self-aware of their limitations and can recognize when they are outside their comfort zone.
To evaluate their tool, they had members of the design process participate in a two-hour
evaluation at a local shopping mall, with one participant acting as the control. Confederates
were paired with each participant and the pairs were scheduled to meet within two hours of
the beginning of the experiment. Wu et al. measured the number of times the orienting tool's
alarm was actually heard, number of spontaneous uses of tool, how many pairs made it to the
meeting location within the time window, and number of times participants used the wrong
application for meeting scheduling. A second study lasting three weeks involved getting
feedback from users, their caregivers, and their family. Measures recorded in the second
study were number of application uses, which they combined with recollections from the
user, caregiver, or family member of the situations when the application was used. Rather
than numerical analysis, the researchers decided, based on positive qualitative feedback,
that the application was successful. Interestingly, they found that the tool was used more
for short-term reminders than orientating and decided to refocus their design. .
This example of a field experiment again demonstrates the difficulty of rigorously ana-
lyzing experimental data. However, the method was successful at getting a large amount of
qualitative feedback in realistic use cases. Wu et al. chose to let potential users drive the
design of the application, and even its purpose, which might have been overlooked had they
studied the use of the tool only in highly controlled situations.
2.I.4 Laboratory experiments
Sutcliffe et al. applied traditional usability methods to study the design of an email application for people with cognitive impairments in a controlled setting. Participants were tasked
with using several different interfaces to compose certain types of email to their caregivers
[84]. The researchers categorized and counted usability errors by probable causes, and solicited feedback from participants on problems they had encountered and suggestions for
improvement. Participants were asked to rate each condition based on clarity, naturalness
and completeness, and to pick which interface they preferred most. Participants' caregivers
were then shown the emails and asked to rate them based on the same metrics. Among
the eight participants, no common errors were encountered, which Sutcliffe et al. suggested
meant that support for user customization was needed.
Davies et al. studied a handheld visual and audio prompting system to guide individuals with mental retardation through the vocational tasks of assembling a pizza box and
packaging software [17]. They utilized a two-group within-subjects design and measured
independence by number of prompts required for each step and accuracy by number of er-
rors made for each task. To analyze the data they used paired t-tests and found significant
differences in average errors per task between using and not using their prompting system.
The difference in number of help prompts requested were significantly different between the
two cases.
Lancioni et al. performed two studies with participants with severe developmental dis-
abilities performing tasks concerned with cleaning, table settings, and food preparation
using a computer-aided system or card system [41]. The first was a within-subjects and
counterbalanced study, which included training, maintenance period, crossover test, then
participant preferences. The baseline condition was to perform the tasks without computer
or cards. Performance was measured as the percentage of correct steps. Participants in the
study showed significant improvement in performance using the computer-aided system.
Participants who were especially successful were asked to participate in the second study,
where the computer-aided system was adjusted to show two instruction formats: 1) a format
combining two pictorial instructions by replacing the first instruction, then the second after
one second, and 2) a format where the system only showed the image associated with the
second instruction in the pair. The results showed that performance was always inferior
with the non-dynamic (latter) format.
Unlike the other two studies, Sutcliffe et al. 's laboratory experiment did not use statistical analysis to show significant findings in their study results. In fact, statistical analysis
was not needed because of clear variation in participant performance and preference. Using
controls and statistical analysis do allow Davies et al. and Lancioni et al. to provide a basis
for demonstrating that there was a benefit to using their systems, so there is a place for the
laboratory experiment, provided the tasks and task context can be controlled. The implications of Lancioni et al. 's study indicate that the wayfinding application might be able to
leverage animation when it would be more beneficial than showing directions step-by-step.
2.1.5 Methodological discussion
The type and sequence of the studies we conducted were determined by what we wished to
learn to guide the design of the wayfinding system. McGrath discussed research methods
and their inherent flaws, presenting the dilemma of desiring to gather research data that
maximizes generalizability, precision, and realism, which no one study can do [57]. Every
study that increases one of these features reduces the extent of one or both of the other two.
A controlled laboratory experiment optimizes for precision at the expense of the realism of
the situation and setting. Field studies in the context of the actual setting for our system
lack precision because there are many uncontrollable factors. Conducting either of the above
types of studies limit our ability to get data from a broad and representative set of people
from the user population. Therefore, our studies varied in the degree to which conditions
were controlled. We wanted to see whether it was feasible at all for potential users with
cognitive impairments to follow directions indoors, which is why our initial study was not in
a highly controlled setting but a real environment, and we did not collect data for rigorous
statistical analysis. In contrast, in a later study where we we needed to understand the
impact of various landmark-based directions, the need to collect such observational data led
us to deliver more directions at fixed locations rather than range over complete paths that
presented limited observational opportunities of each type.
Wayfìnding Research
Wayfinding for people without impairments
In the past decade, more attention has been paid to utilizing location systems to provide
navigation directions. For example, Baus et al. described a framework supporting both
indoor and outdoor navigational instruction given heterogeneous location technologies and
client hardware.
In the indoor setting, Look et al. studied the efficacy of automatically generated walking
directions, comparing them to human-generated directions solicited from people without
impairments who worked in the building where the study took place [51]. Their study
consisted of seven volunteers given a set of three route descriptions automatically generated
by the Stata Walking Guide. After completing the routes, participants were asked to rate
the quality of the directions, then to compare them to the equivalent written directions.
The results were generally favorable, with participants unanimously preferring the Statagenerated directions to the written directions in all but one route, which was due to the
Stata database containing too few landmarks for that route (resulting in the generated
directions consisting of many simple turn descriptions). The existence of several possible
turns at one intersection along the route compounded the problem. Look et al. 's findings
demonstrate the technological feasibility of automatically generating directions that are as
close to understandable as human-generated directions, and mirrors our own results that
step-by-step directions may be inappropriate when there are close successive turns. One
approach to deal with this situation might be to incorporate animation, such as was done by
Lancioni et al. [41]. The approach we take is to combine two steps together with a reminder
to follow the second step, and avoid routes that contain such short successive turns when
creating a policy for users who have more difficulty following compound directions.
Chittaro and Burigat looked at audio and visual presentation of outdoor directions
and found that a map combined with photographs or directional arrows with photographs
significantly improved user performance [14]. They measured "orientation before walking
time" (the time between a user stopping at preset points and continuing once the user
clicked to continue), the total walking time, and the number of errors. In a comparison
of mean times, the researchers found that using directions with maps were significantly
worse for reorientation, while the number of errors across conditions showed no significant
differences among the conditions. Participants were asked to pick their favorite condition,
and a Chi-Square test found that there was a significance in difference among users' preferred
condition, and that the combination of map and photograph was highly preferred. Again,
it is unclear whether the results are generalizable to people with cognitive impairments,
because many individuals with cognitive impairments cannot interpret maps easily, but
Chittaro' and Burigat's results suggest that there may be some merit to studying a map
condition as a complementary component in wayfinding directions.
More recently, Walther-Pranks and Malaka compared a traditional map-based view to
two views using augmented photographs [91]. The two augmented photograph views were
dubbed "explicit" and "implicit." The "explicit" mode featured an overlaid arrow directing
the user, while the "implicit" mode featured a path consisting of lines connecting waypoints.
They conducted a between-subjects evaluation of the 3 views using the same route with 15
participants, aged 24 to 33, counting wayfinding incidents during the study, then eliciting
participant comments and subjective quantitative ratings of usability afterward. Participants in the two augmented photograph conditions rated usability more highly than the
participants who used the map condition. They did not investigate the statistical significance of their findings.
May et al. analyzed directions created by participants without impairments familiar with
an urban area to gain an understanding of the popularity of different types of directions and
how it might vary based on the context in which directions were created [56]. Participants
either created directions by dictating into a recorder while walking through the route (walkthrough) or using a schematic map of the route in an office (cognitive map) . The types of
directions were coded (landmark, distance, junction, road type, street name/number) and
the purpose for each (preview, identify, confirm) and found that the majority of directions
were landmarks for identification, and that previewing occurred infrequently. May et al.
conclude with some suggestions for when to use certain types of directions. However, there
was no evaluation of the actual direction quality with people unfamiliar with the area or
the routes, and no indication that the results could generalize to people with cognitive
Wayfinding for people with visual impairments
Golledge et al. surveyed people with visual impairments, asking them what types of input
and output methods they preferred in a personal guidance system [25] . Survey participants
were also asked about the methods that they employed for pretrip planning and the types
of information that they needed. Results showed that speech input and output were most
preferred and headphones or vibrating collars or bands were least preferred. As a survey,
input from a broad group of people was gathered, the results might have indicated subjective
biases in participants, who may not have actual experience with all the modalities they were
questioned about. This possibility is bolstered by the next study, which had contradictory
Ross and Blasch studied the efficacy of sound and tapping feedback in street crossing
experiments with blind participants [71] . Crossing time, off target error, out of crosswalk
errors, hesitations, and apparent subject confusion were all measured. The baseline condition was the average of pre- and post-tests, where no system was used for crossing. The
ratio of a measure in the instrumented condition to its equivalent in the baseline condition
was used as a metric of the degree of performance and t-tested for significances. Comparing
participants' best and worst conditions, performance-wise, to the baseline, variation was
found among participants, indicating that different participants had better performance in
different conditions. Ross and Blasch also attempted to measure confusion but did not
have a sufficient number of occurrences for analysis. Contrary to Golledge's findings, the
tapping interface was found to be usable under all conditions. This is an example showing
that survey results may be unreliable at learning preferences when the technology surveyed
may be unfamiliar with participants. While it may be the case that there are users who
would greatly prefer not to use the tapping interface, it may also be the case that the people
surveyed by Golledge were simply unfamiliar with the interface and dismissed it without
realizing that it may be dependable and usable. Though such tactile feedback may be useful
to sighted users, we opted to focus only on the modalities both familiar to our potential
users and widely available.
In later work, Ross and Lightman [72] evaluated the Talking Braille system for wayfinding in indoor public spaces. Objective measures of performance were time and distance,
measured with a surveyor's wheel. The blind participants' performance was compared
against that of participants without impairments. A subjective questionnaire asked participants about their perception of the system's ease of use and their feeling of safety when
using the system. Participants' traveled distance were within 4% of sighted norms and
differences between the with-prototype and without-prototype conditions were significant.
However, the importance of this measure may be highly dependent on the wayfinding task,
situation, or user motivation. The measures pertaining to feelings of safety are more closely
aligned to the issues that people with cognitive impairments also face when wayfinding, but
the ability to convey the question and get answers from future wayfinding study participants
with cognitive impairments may be limited.
Wayfinding for people with cognitive impairments
The growing recognition that assistive technology can be developed to aide individuals with
cognitive impairments has led several research groups to prototype wayfinding systems. The
Nursebot project [69] demonstrated a robot that would physically guide an elderly person
within an assisted living home. Researchers at the University of Colorado have designed an
architecture for delivering manually created (for example, by a caregiver) transit directions
in a just-in-time manner to a PDA carried by bus users, using GPS and wireless technology
installed on the buses [9] . The Assisted Cognition Project at the University of Washington
has developed algorithms that learn a user model in order to infer when the user needs
help [45]. A prototype system, Opportunity Knocks, demonstrated the feasibility of using
machine learning in an assisted cognition system, but the parts of the user interface that
involved delivering prompts were not well developed [68]. Researchers have also developed
novel technologies for localization [8, 89] to support wayfinding. These efforts focused less
on the design of directions for users to follow and evaluations of their usability.
Goodman et al. [26], projects from Chung Yuan Christian University by Chang et al. [11]
and Chen [12], and Hagethorn et al. [27] are notable exceptions, including trials with potential users as part of their evaluation. The directions in their systems were statically created,
with a fixed policy for presenting directions. Goodman et al. studied the use of landmarks
to help older people with navigation, finding that they reduced the cognitive load required
to wayfind. Like our initial prototype, the system tested by Chang et al. used augmented
photos to direct potential users with cognitive impairments. Chen's system was text-only,
with preset messages triggered depending on user location within a subway system. Hagethorn et al. studied the use of directions via audio and visual landmarks among individuals
with mild dementia, finding that fewer errors and moments of hesitation were made with
landmarks (no statistical significance was measured). Like our studies, not all participants
in Hagethorn et al. 's studies found landmark directions to be easier than turn-based, demon-
strating that there are drawbacks to relying solely on any single direction type. The system
we designed can choose between such "low-level" or precise directions and "higher-level"
landmark-based directions, making it capable of more than one type of direction. Another
difference is how we use the wayfinding situation, such as the type of intersection or the
distance from a turn, to determining direction difficulty, while all the related work considers
only the type of direction given.
Design Discussion
Dawe interviewed young adults with cognitive disabilities and their families about their
use of assistive technologies, learning the kinds of barriers to use [18]. Design implications
included the need to include multiple groups involved in the adoption process. Portability,
simple functionality with low learning curve that could be extensible, customizable, and/or
incrementally configured, ability to upgrade and replace easily, and failure recovery were all
considered important. Lewis also argues for customizability, because simplicity in a user
interface is not a unified concept but rather must be defined in terms of an individual's
cognitive capabilities [43].
Kintsch and DePaula propose a framework for facilitating the adoption of assistive technology [38]. They split up the process into four phases, the first two of relevance here. In
the Development phase, they emphasize the need to support customization as each person's
abilities and disabilities create a "universe-of-one" condition, also noted by Cole and De-
hdashti [16] and Jacko and Vítense [35]. Consideration must be made for user preference to
the device's aesthetics. In the Selection phase, assessment is necessary to learn aspects of
the user's personality relevant to choosing the right assistive technology. One aspect of an
individual's personality is their tolerance for the frustration inherent in first using a device.
Frustration is a factor also mentioned by Fischer [21], Paradise et al. [66], and Schulze [78],
with respect to the balance that a system must strike in giving useful and timely prompts
versus the impression of being overwhelming or nagging.
Technological solutions that use prompting to guide people with cognitive impairments
through tasks are useful because they aid in the initiation and completion of those tasks
[19, 37]. However, as a cautionary point in the design landscape, Nimwegen et al. 's study
of the effect of interface styles on problem-solving found that argued that if the making
errors comes with a cost, an interface that externalizes information could be actually be
disadvantageous [90]. This might suggest that over-reliance on a wayfinding application
could impede the learning of routes and locations, not to mention putting a user at risk if
they trust the application in situations such as crossing a street. Effort is needed to ensure
that the wayfinding application is designed to remind users not to immerse themselves to
the point of getting into such dangerous situations. This immersion is one reason why there
have been numerous reports in both research and news outlets of people, without cognitive
impairments, following GPS navigation systems unsafely by nearly walking into traffic or
driving dangerously off paths [91, 75, 62].
Our approach to creating a truly usable interface on top of technology like Opportunity
Knocks addresses wide variation in wayfinding preference and ability across individuals with
cognitive impairments while considering qualitative aspects to usability, such as participants'
reactions to different direction types. We developed a Wizard of Oz [76] infrastructure
so that potential users could walk through a realistic experience of using a wayfinding
system without all parts of the system being fully implemented. This prototyping approach
is particularly important in collaborative, iterative design with individuals with cognitive
impairments where "thinking through" designs on paper is rarely effective.
Aside from noting wayfinding errors, we consider signs of confusion that occur even when
participants wayfind correctly. We have also adapted ideas from commercial automobile
navigation systems, such as displaying arrows in perspective view (for example, products
by Garmin or Magellan), and GPS systems for blind users that employ voice prompts (for
example, products by VisuAid or Pulse Data). Although our work focuses on a system
for people with cognitive impairments, it is likely that a design that requires low cognitive
overhead will also be attractive to many users without impairments. The lessons learned
from our research may useful, for example, in tour guide systems [13] and general locationaware applications [30].
Landmark Selection
Based on positive findings about landmark usefulness in wayfinding, we incorporate visually augmented landmark directions into the wayfinding system. In the past, integrating
landmarks into wayfinding directions was not practical on a wide scale due to the lack of a
system to provide landmark metadata and imagery. Recently, systems to choose appropriate
landmarks for directions and generating user-specific directions based on familiar landmarks
have been studied. Hile et al. describe their implementation of a system that can provide
scalable and ubiquitous access to such landmark information [33, 31], which we incorporate
into our system.
Bell et al. used the input from people playing a mobile phone game to create a set of
photos where landmarks are more easily found, compared to randomly chosen, nearby geotagged photos from online photo sites. Like their system, our current system requires an
initial seeded set of images with landmark name and location tagged. Unlike their system,
our system can immediately use those photos without a second phase of user input, because
it uses heuristics such as landmark popularity to choose appropriate landmarks. Chung and
Schmandt also developed and studied a system that produces landmark-based directions,
finding that the careful use of familiar landmarks may be beneficial to wayfinding [15].
However their directions are generated for a route beforehand, and only as text, rather than
delivered visually in a just-in-time manner to users who may not be familiar at all with
their route.
Automatically generating landmark based directions requires selecting an appropriate
landmark and an image of that landmark. Our landmark selection system leverages existing
collections of geo-tagged images (images with location and landmark metadata) to retrieve
suitable images of landmarks. This makes it possible to select an image from the database
that relates to the user's current location and intended direction, for example, to select
an image of the building they should walk toward in a perspective close to their current
position. Additional aspects of the image database make it possible to choose landmarks
by popularity and to choose a quality representative view from the possible choices. These
images can also be augmented with arrows. See Figure 2.1 for some example images that
were automatically constructed by the system and
marwebapi/marwebapi/apiindex for a Web-based front-end.
User Modeling
Our studies showed that individuals with cognitive impairments would benefit from a
wayfinding system that is capable of supporting customizable and adaptable direction selection. Moffatt et al. discuss the tension between producing customized solutions for in-
dividual users and identifying generalizable results [59]. Our approach is to design around
(a) An example landmark and (b) The same direction as in 2.1a (c) A example showing a jointed
overlaid path arrow
but from a different location and arrow to denote an upcoming
Figure 2.1: Examples of augmented landmark-based photos.
a decision-theoretic framework where system actions are chosen based on knowledge from
customized user models, thus providing a generalizable solution that can still support individual requirements.
In many ways, the determination of an appropriate sequence of directions to give a person
is similar to the path planning problem in the robotics community. Various techniques for
path planning under uncertainty have been developed by that community [88], which we
can apply toward creating an automated wayfinding system. A key concept in this context
is the Markov decision process (MDP), which provides techniques for generating navigation
plans even when observations and the outcome of navigation actions are uncertain [85] . For
example, partially-observable MDPs have been used to assist persons with dementia through
tasks such as hand-washing [4]. The framework we use builds upon on these techniques to
model uncertainty in whether a person will follow the guidance provided by our system, but
to simplify our model and reduce the state space necessary to solve our MDP, we rely only
on observable action results.
Patterson et al. [68] showed how machine learning could be used to build models able
to predict users' intended destinations, based on observations of regular destinations. They
proposed using photos taken by users themselves in the user interface to lessen the cognitive
load required in recognizing locations. They focused mainly on the learning aspect, so no
user study was performed to verify their hypotheses about direction difficulty.
Other work demonstrating the usefulness of building user models include Gajos et al.'s
research on user interfaces that adapt to motor-impaired users [23] and Ziebart et al.'s
research using data collected from taxi drivers to predict future navigation decisions [96].
Ziebart et al. did not study variations between potential individual users. Gajos et al.
modeled such variation by using a set of one-time motor performance tests from each user.
We also generate individualized user models using collected training data, however the
complex interaction between direction and location makes it more challenging to do so with
relatively little observational data. Intille et al. [34] use case studies as the background to
discuss this tradeoff between combining training data for generality and trying to determine
individualized models. We also study whether there is a benefit to combining training data
from other users with an individual's own training data given data collection limitations.
Chapter 3
WAYFINDING feasibility study
This chapter describes a study evaluating various configurations of a wayfinding user
interface for accuracy of route completion, time to completion, and user preferences among
three indoor routes in the Paul G. Allen Center for Computer Science and Engineering. We
operated the system using the Wizard of Oz technique, which allowed us to experiment with
various guidance strategies and interface modalities without the need to fully implement
We chose to examine wayfinding indoors because we believed that it is a safer environment for initial study, and because indoor wayfinding assistance can be just as important to
potential users. Individuals with cognitive impairments are likely to encounter times when
they must remain oriented in indoor spaces, for example, in an office building, shopping
mall, or hospital. Many indoor environments, such as the Computer Science and Engineering building, also pose some unique challenges, such as a relative uniformity in layout
and lack of landmarks in much of the building. Such an environment, lacking in "spatial
differentiation," has been shown to pose navigational difficulties even for people without
cognitive impairments [I].
Wizard of Oz Prototype
We designed an interface suitable for use on a handheld device, such as a PDA, to send directions and prompts to the user. We will use the term message to describe any information
that the interface intends to convey to the user, direction to describe a message that guides
the user through some path, and the term prompt to describe a message that is triggered
in order to help the user return to a path.
Both directions and prompts consisted of a subset of images, audio, and text messages.
The design was a result of several rounds of pilot testing involving members of our research
Figure 3.1: Sample images used in the interface. Clockwise from top-left: plain photographs,
directional symbols, photographs with highlighted areas (for example, room number), and
photographs with overlaid arrows.
group and job coaches from a community based rehabilitation program. We included job
coaches because of their experience with clients and ability to consider their needs when
evaluating the system. Based on the pilot tests, we refined the types of images presented
and the wording of the audio and text messages.
Images: We used four types of images: photos, arrows and other generic symbols, photos
with overlaid arrows, and photos with an outlined area (see Figure 3.1). Photos were
of landmarks and other interesting features. Arrows and other generic symbols were
used to tell a user to turn or stop and can be used at times when appropriate photos
are not available or distinctive enough. Overlaid arrows on photos were intended to
disambiguate where to go as well as provide additional indication of where a user
should go next. Some photos contained a highlighted area (for example, a room
number or elevator button). The need to include arrows and outlines became clear as
a result of pilot testing. In particular, with small images of indoor landmarks, it can
be difficult for a user to know where to focus.
Audio and text messages: Text messages were brief messages displayed in large font.
The text and audio messages shared the same wording, in order to minimize the
complexity of directions with both text and audio. See Figure 3.2 for an example
New message alert: An alert chime preceded new messages and acted as a prompt to get
the user's attention, alerting the user that a next step was being indicated.
Acknowledgment message: Besides directions and prompts, the interface also had a
simple message, "Good," which was intended to tell the user when a prior direction
was completed successfully and has been cleared from the screen. The device did not
play an alert chime before this message because there was no need for participants to
switch their focus to the device. This addressed an issue in pilot testing where users
would check their display and see prior, irrelevant messages that they had already
3.1.1 Prototype implementation
We implemented the prototype client interface in Java using the Standard Widget Toolkit
(SWT). It runs in the Windows Pocket PC 2003 environment on a Hewlett-Packard iPAQ
handheld with a 802.11 (WiFi) connectivity. The software supports display of images up
to 240x320 resolution. We used images with 240x180 resolution in order to leave room for
text to be displayed as well. Users chose to use headphones or the built-in speaker to hear
the audio. Figure 3.2 shows the device displaying a sample direction with image and text.
#«?«:44 f
Stop at room 522 on
our right.
Figure 3.2: Sample iPAQ display. Participants received a combination of image, text, and
audio directions to follow when navigating a route.
We overlaid arrows and highlighted regions on the photos manually. As will be discussed
later, it is possible to automate this image processing step, but the intent behind this study
was to first understand which modalities would work best for directions. Only after finding
that such a feature would be useful did we choose to invest effort into supporting it.
The device acted as a client to a remote server controlled by the navigation wizard,
a person who sent instructions to the client on what to display and play based on the
participant's location and heading. To gather location and orientation information, we used
a location wizard, a person who followed study participants and transmitted their location
and orientation to the navigation wizard in real-time using a simple map-based GUI that
ran on a Tablet PC.
We chose to use a location wizard to act as a substitute for an accurate location system.
WiFi-based localization systems were close to providing the resolution that we needed [30],
but we also required orientation, which would have necessitated a separate sensor. Fig-
ure 3.3 shows the server control program and Figure 3.4 shows the map GUI. We divide
study responsibilities between two wizards (in addition to other observers) in order to more
effectively operate the multi-modal simulations. Figure 3.5 shows the system diagram.
We preload all images and audio on the client device to maximize responsiveness of
the initial prototype. With the high bandwidth of WiFi connectivity, these objects could
be easily transferred in real-time from a locally-deployed server with negligible latency.
However, we also want to support caching on the client for those situations where WiFi is not
available. We pre-record the audio messages, matching the wording of the text directions,
to avoid potential text-to-speech conversion issues confounding results pertaining to the
We used a within-subjects, partially counterbalanced study design where the interface
guided every participant through three routes of differing complexity using three different subsets of modalities. We varied the order of both routes and modalities presented to
o ir" ß
¦? ramtsol filiases
·* Äißfei «* Text r »nage
t A
OîiiiiJÎ aiKíOHS:
Atra« alert
aerrt Prewece
Au&rc fêtera
«wage: S PSÎ^O
ùelay: o
Tas© tftç met iaft down this
Figure 3.3: Screen shot of the navigation wizard GUI running on a desktop. Green marks
on the map show where pictures were taken from, and are used to send directions to client.
Prompts are delivered using the buttons on the center left panel. Solid red mark shows
information from location wizard.
Figure 3.4: Location wizard GUI running on a Tablet PC. Red mark shows participant
location and orientation.
Participant follows
directions on PDA
Location wizard
Wizard sends
directions based on
location information
shadows participant
Figure 3.5: System interaction diagram. Location wizard provides location information on
the participant (x, y, floor, and orientation) (1) sent over WiFi (2), while the navigation
wizard uses that location information to decide which messages to send (3). Messages (to
display images, text, and play audio) are sent to the client device over WiFi and acknowledged for robustness (4). Participant then follows the direction or prompt displayed on
device (5).
each participant. We conducted the studies in the University of Washington's Paul G. Allen
Computer Science and Engineering building, which was unfamiliar to all participants. Two
researchers from the Department of Rehabilitation Medicine followed each participant in
order to take notes, get feedback from the participant, and provide assistance in case the
participant became confused or uncomfortable. All participants gave permission to audio
record their session. Our procedures were in accordance with institutional guidelines.
At the end of each study, we asked participants a series of questions about what they
thought of the interface and how they navigate when in unfamiliar locations. We also asked
them to look at alternative interfaces that displayed maps with arrows, identify where they
would be, according to the map, (for example, in a room or a hallway), and explain what
the map was telling them to do (for example, move forward and turn right). Specifically,
we asked each participant the following questions:
1. Please rate your modality preferences (text, audio, picture)
2. Was the chime helpful?
3. Were the arrows helpful?
4. How do you typically get directions and find your way around (for example, using a
map, asking someone)?
5. Do you have a preference between a cell phone or PDA form factor?
6. Did you like using the earbuds?
7. Can you identify areas on this map (when shown map on computer)?
8. Is there anything else we should know about using the device or usefulness of this
Table 3.1: Route features. Turns: intersections along the route where participants had to
change their heading. Intersections: all intersections along path.
Route la
Route lb
Route 2
Route 3
Routes and modalities
Participants were shown the device and given examples of each of the subset of modalities.
They were led to the starting location of each route by the location wizard and given the
task of following the device's directions to a set destination. Participants were told their
destination and what modalities would be used for that route.
Routes: We chose three routes that traversed through different parts of the building in
order to minimize any learned familiarity. Route 1 (Figure 3.6) involved no floor changes,
while Route 2 (Figure 3.7) involved using an elevator, and Route 3 (Figure 3.8) involved
taking the stairs down one flight. Each route involved five turns, but differed in the number
and type of intersections along the route. Route 1 had two variations, la and lb, which
were functionally identical except they traversed different floors. Most participants were
given Route la, but two participants traversed Route lb because they did Route 2 first,
which would have given them the opportunity to familiarize themselves with Route la's
destination. See Table 3.1 for a breakdown of route features.
Modalities: We used three combinations of modalities to get feedback on which combinations participants preferred and to gain insight into how effective they were for navigation.
Combination 1 used all three modalities for messages (all). Combination 2 used only text
and audio (-images). Combination 3 used text and images (-audio).
Figure 3.6: Route la and lb. Intersections are denoted by shadows around the path. Both
routes had five turns across a single floor of the building. Route lb had one fewer 3-way
Figure 3.7: Route 2. Intersections are denoted by shadows around the path. This route
had five turns across two different floors, which involved using the elevator.
Figure 3.8: Route 3. Intersections are denoted by shadows around the path. This route
had five turns across two different floors, which involved using the stairs.
3.2.3 Participants
The selection criterion for participants was purposive. We recruited a pool of adults with
cognitive impairments who were receiving employment services from a community-based rehabilitation provider. From the pool of interested participants, we selected two individuals
whose primary disability is traumatic brain injury (TBI), two participants whose primary
disability is mental retardation (Downs Syndrome) , two diagnosed with Pervasive Developmental Disorder (PDD), and one with cerebral palsy (CP) with cognitive impact, ranging
in age from 26 to 46. See Table 3.2 for more participant demographics.
Of the participants, only two (Participants 5 and 6) had some experience using a device
similar to ours. Participant 5 uses a smart phone and a computer on a regular basis but
does not use a PDA. Participant 6 uses a computer on a regular basis. Both use computers
in their work and personal life; Participant 5 also uses a computer to plan routes prior to
making trips into the community. Participant 6 uses the computer routinely for email and
is familiar with several search engines but does not use the computer for navigation.
Participant 1 uses a mobile phone on occasion; he does not own nor use a computer.
Participant 2 uses a computer at home for email; he also uses a mobile phone with prepro-
Table 3.2: Participant demographics. CP: cerebral palsy, TBI: traumatic brain injury, DS:
Downs Syndrome, PDD: pervasive developmental disorder.
Health Condition
grammed phone numbers. Participants 3 and 4 rarely use a computer (Participant 3 uses
a computer in the library for email on occasion); neither use mobile phones. Participant 7
does not use a computer and does not have a mobile phone.
Participant 1 has a mobility impairment requiring the use of a power wheelchair and was
unable to complete the route that involved stairs, so his route was slightly modified. Participant 3 required a different and relatively minor modification because she could not reliably
distinguish right from left. We taped a small note to her right hand as a reminder. However,
she only completed two routes because she struggled with the modality that included only
audio and text.
All other participants were able to follow the directions to their destinations. Table 3.3
summarizes the performance of each participant and Table 3.4 summarizes each participant's modality preferences. Since the intent of the study was to learn how suitable the
directions were for our target user-base, we encouraged feedback from participants as they
were following the directions. Thus, the measured times includes the times when participants were showing researchers what was on their screen, and are provided only to give a
general idea of the how long each participant took.
Table 3.3: User study results. Modalities are the subset of image, audio, and text used for
that route. Time is duration of task with elevator wait times omitted. Errors are situations
when participant took a wrong turn or passed their destination. Directions, Prompts, and
Good are the types of messages as described in Section 3.1. Routes are ordered consistently
for the table, although participants did not perform them in the same order. See Section
3.2.3 for details on participants and Section 3.3 special cases marked with asterisks (*).
Table 3.4: Modality preference by participant.
Text and Audio (tie)
System usage
The pool of participants had three general ways that they used the device. The most
common was to keep the device visible at chest level for the whole session, and switch their
attention between the device and their surroundings as needed, walking at a consistent pace
and only altering course when they received directions. To keep the screen at a visible
angle for Participant 1, who uses a powered wheelchair, we attached the device dock to his
wheelchair tray. Participant 2 kept the device at his side the majority of the time, and only
look at the screen occasionally.
3.3.2 Direction clarity
Participants liked directions that communicated a sense of responsiveness by the system.
The system needed accurate location information in order to send directions at appropriate
times. Those directions also needed to convey a sense of how long they were applicable,
otherwise participants could get confused.
Directions containing more information would have been beneficial to our participants
in some situations. A challenge would be simplifying longer directions containing both the
turn and the destination. We tried an alternative method for informing participants when
to expect additional directions by adding a message to wait (see Figure 3.9, right). Some
Follow the arrow to
floor 5.
Exit the elevator on
floor 6 and wait for the
next direction.
Figure 3.9: Left: A detailed but complex arrow guiding the participant off the staircase (see
also Figure 3.8, C). This confused participants more than simple directional arrows. Right:
Direction telling the participant to exit the elevator and wait for the next direction. This
helped overcome confusion from delays in WiFi outages in the elevator.
participants commented that it was helpful to them. However, some participants did not
wait and began walking or looking around after exiting the elevator. Some participants
also wanted to know when to expect subsequent directions when given directions such as
'Walk down' or 'Walk to the end of a hallway or walkway. When given the latter, some
participants were unsure whether they should walk past intermediate intersections.
3.3.3 Modality preferences
Participants varied widely in their preferences of modalities, but found the combination of
modalities useful. Some participants found text most useful, while others found images more
helpful. Details of these ratings are shown in Table 3.4. Participants gave different reasons
n£674, the
econd door
on voj
voúr left
aoor on
Walk to room 674, the
rived at destination
second door on your
Figure 3.10: Left: Direction telling users to proceed to their destination and stop. This
direction very closely followed the direction to turn right, and sometimes came late. In
addition, Participant 1 thought the skewed arrow was confusing. Right: Map showing
spacing of turns and the path of a participant with difficulty locating the destination. Small
yellow dots indicate participant's path, red dot indicates the final destination reached.
for their rankings. Text was highly rated because it let participants process messages at
their own pace, and refer back to them as necessary. Some participants said audio was
useful in combination with other modalities, but audio by itself was not preferred (probably
due to its fleeting nature), although all participants said the audio was of good quality
and clear. Images were reviewed positively because they also gave the participants more
time to consider the message and make the visual associations. Several participants noted
the arrows overlaid on the images were helpful because they "tell you where you're going"
(Participant 7). Only Participant 2 stated he preferred images the least, because he had
concerns that photos might not always be clear and accurate in a real setting.
Analysis of modalities
Images were helpful for disambiguation when the text and audio messages were inadequate
at describing directions. It was important to keep images simple and give participants more
time to study them by sending directions earlier. Participant 3, who was unable to reliably
differentiate the audio/text messages "Turn left" and "Turn right," also had difficulty with
the left/right turn arrows, but was not able to complete a route when given messages that
contained no images. Most participants were able to follow the arrows on photos and
reported that they were useful. Participant 1 had difficulty interpreting an arrow at an
atypical orientation (see Figure 3.10, left), mentioning that it "zig-zags" and gave him some
trouble. Another arrow containing a series of turns, meant to show where participants were
to go once they took the stairs down a floor, caused several participants to hesitate (see
Figure 3.9, left).
Directions with photos increased the cognitive load on participants, who needed to interpret features in the photos, which were small and of a relatively uniform building environment. Participants suggested that photos might be more useful if they showed more
easily seen landmarks and objects, such as bright red fire extinguishers. Some noted during
the study that they liked when landmarks, such as a painting, appeared in the photos.
Text was a well-liked modality for providing messages due to the lower cognitive load associated with reading. It was important to use familiar vocabulary. This was evident with
the use of the word "walkway" to describe the catwalk structure that was part of Route
1. Participants who were unfamiliar with the term or unsure that it applied to the catwalk
asked for confirmation from researchers.
Although the wording of audio messages were the same as complementary text messages,
they were not as useful as text for participants. Audio messages were better when used in
conjunction with messages in other modalities when participants wanted to check against
the others, provided they were delivered inconspicuously and at an appropriate (and possibly
adjustable) rate. Participant 6, after receiving directions, would pause often in the middle
of intersections and only proceed when sent short prompts, but indicated that he preferred
taking his time with directions containing text and images at his own pace as the audio
"went a little too fast."
Timing of directions
Participants liked directions that communicated a sense of responsiveness by the system.
The system needed accurate location information in order to send directions at appropriate
times. Those directions also needed to convey a sense of how long they were applicable,
otherwise participants could get confused.
One situation where the timeliness was important to participants was during the elevatorriding portion of Route 2. Participants could not receive messages while they were in the
elevator because the device would disconnect from the WiFi network and lose the ability
to receive messages from the navigation wizard. Our pilot testing discovered that this gap
was problematic, so we implemented delayed message handling so the navigation wizard
could send a message to the device before losing connectivity, and have the device process
the message after a fixed delay. We could then send a direction reminding participants to
exit the elevator on their destination floor. However, this work-around was not entirely
successful. Participant 3, clearly having difficulty determining when to exit the elevator
when other riders were exiting on different floors, reported that, "It was confusing; I wasn't
sure what to do." Additional sensors (e.g., barometer) could aid in determining where
participants were and when they should exit the elevator.
Directions containing more information would have been beneficial to our participants
in some situations. The destination in Route 2 was close to a previous turn and caused
two participants to walk by, because they received the direction telling them to stop at
their destination room (see Figure 3.10) only after they had already passed by at their
normal walking pace. The system should have informed them earlier that the destination
was close. A challenge would be simplifying longer directions containing both the turn
and the destination. We tried an alternative method for informing participants to expect
additional directions by adding a message to wait (see Figure 3.9, right). Some participants
commented that it was helpful to them. However, some participants did not wait and
began moving or looking around after exiting the elevator. When the next direction finally
arrived, Participant 3 had just turned away from the orientation that the direction was
meant for, which caused her to go the wrong way. Some participants also wanted to know
when to expect subsequent directions when given directions such as "Walk down" or "Walk
to the end of" a hallway or walkway. When given the latter, some participants were unsure
whether they should travel past intermediate intersections. Participant 7 asked, "All the
way? Part of the way-walk down this hallway?"
3.3.6 Confirmation
We found that the navigation wizard intuitively began sending more "Good" messages
after participants followed correction prompts correctly, and used them less when standard
directions were followed. Sending feedback rather than simply the next direction when
participants were on path tended to slow down the pace. When the "Good" feedback was
received, Participant 5 even replied out loud [whimsically], "of course." When participants
were less sure and had to take more care to follow directions, this feedback became more
important. Several participants thought this message was helpful because they received
important feedback when trying to make corrections to get back on path. Participant
1 even stated his preference for the audio modality because the text-only version of the
"Good" feedback was less noticeable when it occurred, as it did not have an audio alert
preceding it.
3.3.7 Device usage and form factor
Our pool of participants had three general ways that they used the device. The most
common was to keep the device visible at chest level for the whole session, and switch their
attention between the device and their surroundings as needed, moving at a consistent pace
and only altering course when they received directions. To keep the screen at a visible angle
for the wheelchair user, we attached the device dock to his wheelchair tray. Participant
3 gave the device the majority of her attention, and would walk briskly after receiving a
direction until she reached an intersection where she would stop and wait for more directions.
Participant 2 kept the device at his side the majority of the time, and only look at the screen
occasionally. He would listen to the audio first and then check the screen, though at times
he missed the audio cues.
Although the other two modes are reasonable for short routes, Participant 2's mode of
occasionally referring to the device seems more practical for routes of a larger scale. Using
vibration feedback in addition to audio could help avoid missed messages, and address one of
the participants request for a 'silent mode.' One participant also expressed a preference for
a wireless headset, saying he would be less self-conscious. Although none of the participants
had complaints about the size of the iPAQ, one thought it was too heavy, and another was
concerned there would be a social stigma attached to using one. When asked whether they
would prefer using a mobile phone instead of a PDA, some participants felt the text may
be harder to read, but that images would probably be large enough.
3.3.8 Debriefing comments
Most of the participants expressed positive attitudes toward the development of the wayfinding system after using the prototype for the study. One recurring theme was their belief
that such a system would be beneficial to a wide variety of people.
Pl: I think this would be a very good device for anybody that's lost or doesn't
really know how to get to a place... People would go out more because they can
find their way
P2: What I see and what I experienced, I think it is a very well designed and
well planned instrument... Really beneficial for everybody.
P3: I want to keep [the device].
Pl: This will be very, very helpful to people.
Design Discussion
Many factors come into play when evaluating appropriate interface modalities for individuals
with cognitive impairments. In this usability study employing qualitative methodology, we
have studied the experiences of seven individuals with disabilities as they use interfaces
for wayfinding devices to better understand their preferences. Because individuals with
disabilities in general, even with similar disabilities, may have significant differences in their
preferences and functional abilities and limitations [81], we cannot draw conclusions here
about the potential match between disability and interface. Rather, we propose issues that
may guide such a discussion for future research.
3.4-1 Individual skills, experience, and comorbid impairments
Cognitive impairments are not a uniform construct. For example, individuals with acquired
brain injury may have patterns of specific and global deficits based on their specific type of
injury and very specific cognitive losses related to the location of trauma, whereas individuals
with mental retardation will commonly have more generalized deficits. As a result of the
differing etiologies of cognitive impairments and varying functional limitations, we assume
individuals will possess different prerequisite skills and knowledge that might impact the
use of assisted cognition devices. For example, individuals with acquired brain injury may
have been proficient technology users prior to the onset of their condition and may be able
to learn to use new technology with more ease than individuals with mental retardation
who might not have previous experience.
Designers must also consider the prevalence of other physical impairments [53]. For
example, in our group two of the participants had other impairments that impacted their
ability to use the system. One participant used a wheelchair and was unable to hold the
device. The other user had a hearing impairment and preferred to not use hearing aids,
thus making the auditory prompting more difficult to use. In addition, cognitive impairments are not a uniform construct. For example, individuals with acquired brain injury may
have patterns of specific and global deficits based on their specific type of injury and very
specific cognitive losses related to the location of trauma, whereas individuals with mental
retardation will commonly have more generalized deficits. As a result of the differing eti-
ologies of cognitive impairments and varying functional limitations, we assume individuals
will possess different prerequisite skills and knowledge that might impact the use of assisted
cognition devices. For example, individuals with acquired brain injury may have been proficient technology users prior to the onset of their condition and may be able to learn to
use new technology with more ease than individuals with mental retardation who might not
have previous experience.
3.4-2 Cognitive load and environmental effects
Participants' executive function, receptive language, attention to detail, and memory capacity no doubt impact their performance with the system. In addition, participant's performance will differ based upon the cognitive demand of the task being attempted and
individuals who function quite well under some conditions will function less well when the
demands of the task or environment change. Cognitive load theory acknowledges that the
"cognitive architecture" of the brain (e.g., working and long-term memory, attention) has
limits that can be overloaded [65]. For example, working memory can handle only a small
number of interactions at one time. When cognitive limits are reached, an individual becomes overloaded and unable to process more information.
3-4-3 Individual expectations and preferences
In the study there were differences between how individuals used the system. Some users
were systematic in their use of the device. They followed each direction, and then halted
waiting for the next direction. Sometimes these individuals indicated frustration when the
system prompted too slowly. For example, Participant 3 said "Come on now!" to the device
when it failed to give a prompt at the expected time. Other individuals were less systematic,
more distractible, and lost focus on the device and its directions.
Individuals had definite preferences about the modalities through which they wanted to
receive information, as shown by the lack of consensus among their ratings. Preferences
are likely a result of the issues discussed above, but may be a result of familiarity with a
modality as well. In other words, text may be a more common form of receiving directions
for some participants. And it should be noted that all participants were at least basic
Technical Discussion
The study made clear that any wayfinding application needs rich knowledge of the environment in addition to accurate location in order to effectively guide users. This information
needs to be presented in a clear and familiar way on a device with a practical form factor.
3.5.1 Location and orientation accuracy
An underlying location system should have location accuracy of approximately 2-3 meters
to adequately distinguish between rooms. In turn, a location system would need to produce
at least 1 update per second while a user is traveling at walking speeds and there are
upcoming turns or stops. The exact frequency is dependent on the individual user, because
of variations in movement speed and attentiveness, and the indoor environment, because
of variations in the distribution of turns and possible destinations. In environments where
turns are spaced further apart, less accuracy would be acceptable.
Knowing whether a user is facing the correct way is more important than fine-grained
orientation, so 30-45° orientation accuracy can be satisfactory. However, direction of motion
does not provide sufficient orientation information. Users may stop and look around when
they are confused, and it is important for the wayfinding application to give directions
appropriate to their current orientation. Additionally, orientation information can allow the
wayfinding application to recognize signs of confusion - like stopping and looking around and respond accordingly.
A suitable location technology would have to combine and extend a variety of existing
location technologies in order to provide the level of coverage, accuracy, and ubiquity needed
for our wayfinding application to be of practical value to our user population.
GPS is a good solution for localization in outdoor areas where there is coverage, but
performs poorly indoors as well as near tall buildings and other obstructions. Using WiFi
radio beacons to localize has the benefit of leveraging existing WiFi covered areas, such as
indoor environments and dense urban areas, where GPS is typically inadequate. Scalable
WiFi solutions are the focus of several indoor localization techniques [20, 60]. However,
WiFi techniques lack the accuracy to reliably provide orientation.
WiFi localization also does not work in dead-zones, where there are no WiFi signals.
Elevators are one type of area that is highly likely to lack WiFi coverage, but they are
also an area where we found that location information was most needed. Sensors such as
barometers and gyroscopes could be used for dead reckoning by smoothing over transients
in WiFi signal strength due to dead-zones, other people walking by, the user turning, etc. A
particle filter-based localization algorithm could leverage these extra sensors by using them
to improve the accuracy of its motion model [22] .
Computer vision could also enhance user location and orientation estimation [80, 54].
Photographs taken by a user with a camera phone could be sent to a server for location
estimation. Annotations, such as arrows, could then be added automatically to those photographs, and might be easier for the user to understand quickly.
3.5.2 Maps of indoor environments
In order to produce directions to guide users, the wayfinding application needs the location
system to provide a map that contains the structure of the environment, such as the position
of hallways, rooms, stairs, and elevators. Such maps would have to be created for each area
where users want to travel. As a bonus, location systems that provide this information can
leverage this encoded knowledge to produce more accurate location estimates [20].
Potential users of our wayfinding application may find themselves in places outside their
routine paths. Therefore, the underlying location system must be scalable and easy to deploy
in a wide area. Infrastructure systems could provide information about the environment
(such as maps, photographs, and labels) [36]. However, it is desirable for a wayfinding
application to work well even when this information is not available.
Current WiFi localization systems share this undesirable dependency on existing mapping and infrastructure information. Specifically, most WiFi localization techniques require
a training set of signal strength readings labeled against a ground truth location map, which
is prohibitive to collect and maintain as maps grow large. Recent research has attempted
to address this problem by solving the simultaneous localization and mapping (SLAM)
problem for WiFi: given an unlabeled sequence of signal strength readings, reconstruct the
underlying beacon locations and localize the scanning device within the resulting topological
map. Techniques for accomplishing this generally use a form of dimensionality reduction
to reduce the highly-dimensional WiFi readings into low-dimensional latent location. The
resulting method allows an agent to easily build localization maps for new areas, allowing
wayfinding at a truly large scale [5].
Place names
Using understandable place names along routes is a crucial feature for any textual or audio
directions that a wayfinding application might provide. One approach to determine the
labels of common route features might be to use machine learning to train an algorithm to
automatically label locations as hallways and rooms [46]; this could be used in conjunction
with SLAM. A complementary approach using manual labeling could produce labels for
nonstandard environmental features, such as walkways or unique landmarks, or names of
places with special meaning to specific users [29].
Next steps
The results from this study showed that it is feasible to deliver multi-modal wayfinding
directions to individuals with cognitive impairments via a mobile device, though much work
remains in both interface and technology to support such a solution. In the following
chapter, we discuss follow-up studies with participants following the same type of directions
outdoors. The pitfalls are due to differences in the complexity of the environment, both
from a geographic standpoint and from additional distractions.
Chapter 4
Outdoor Route Study
The indoor study showed that images with overlaid arrows, combined with text and audio
messages, could be used by people with cognitive impairments. This chapter discusses two
follow-up user studies. The goals of these user studies were to study the effects that the
outdoor environment have on wayfinding. Specifically, we investigated these aspects:
• The usability of images. Visual features tend to be less uniform and more complex outdoors, so recognizing photos might be more challenging for users, especially
recognizing details (including text) on a mobile device screen outdoors (for example,
due to the small size of the screen or glare from the sun) . Changes in weather and
season can affect the appearance of environmental features. Would participants have
issues with photos that were taken in conditions different from that in the study?
• Turn precision. In the indoor study, some participants had trouble with arrows that
directed them to turns that were not at 90° angles. Outdoors, paths may wind and
not meet at precise four-way intersections, so we investigated whether participants
could make correct turns given the lack of precise angles of paths and intersections.
• Finding a precise location. Indoors, most rooms are labeled and ordered by number. Could the prototype's set of directions guide someone to an unlabeled building
entrance? How accurate must the wayfinding system's estimate of a user's location
• Outdoor distractions. In a more dynamic environment, a user has more varied
distractions from pedestrian and traffic activity, which could make paying attention
to directions from a device harder to maintain.
• Baseline vs. prototype. We also included a baseline condition to compare with
the prototype, giving us insight into how a just-in-time system affects how our target users find their way to new locations. Specifically, we wanted to find out what
methods of wayfinding were currently used (for example, written, verbal, or map directions) and why they preferred that method. We also wanted to observe any problems
users encountered and whether the prototype improved confidence and comfort while
wayfinding compared to methods used during the baseline data collection.
Wizard of Oz prototype
We used the same HP iPAQ handheld interface to deliver directions and prompts to the user
that Chapter 3 first described. Both directions and prompts consisted of a subset of images,
audio, and text messages. Images were photos, arrows and other generic symbols, photos
with overlaid arrows, and photos with an outlined area (see Figure 4.1). Text were brief
messages displayed in large font. The text and audio messages shared the same wording.
Users chose to use headphones or the built-in speaker to hear the audio.
The client was remotely controlled by the navigation wizard, a person who determines
what to display and play based on the participant's location and heading. The navigation
wizard interface is shown in Figure 4,2. To simulate location and orientation sensors, we
again used a location wizard, a person who follows study participants and transmits their
location and orientation to the navigation wizard in real-time using a simple map-based
GUI. Both wizards used WiFi-enabled Tablet PCs to communicate. Study responsibilities
were divided between two wizards (in addition to other observers) in order to more effectively
operate the multi-modal simulations.
4.I.2 Method
The study involved every participant wayfinding through three routes of differing complex-
ity (see Figure 4.3). We chose routes that traversed through different parts of the campus in
order to minimize any learned familiarity, and varied the order of routes presented to each
participant. For the "baseline" case, participants were asked to choose the mode they would
! ?
Figure 4.1: Sample images used in the interface. Clockwise from top-left: plain photographs,
directional symbols, photographs with highlighted areas (for example, room number), and
photographs with overlaid arrows.
Wîzâfd Otntrat
Centrai Atlases
B raT 'S
if Audio * TeKt ^ Image
Send action
Gleba! actions
Audeo alert
W ask fonward
Turn around
3. IL* 1-1.» t ,,.fi?
Client Preview
Name: OefasA nsroe
Audio: Reccrd:nc5I av
tmaqe: p???-orc
Delay: o
Walk to the street and
tum oght
Figure 4.2: Screen shot of the navigation wizard Tablet PC GUI used in the outdoor route
study. Green marks on the map show where pictures were taken from, and are used to send
directions to client. The prompt buttons are on the right side for easier access while the
tablet is cradled in the left arm.
Figure 4.3: The three routes used in the outdoor route study. Each starts outdoors and
ends at a specific location inside a building on the University of Washington campus.
Table 4.1: Route Study participant demographics. CE: cerebral encephalopathy, TBI:
traumatic brain injury, DS: Downs Syndrome.
Health Condition
typically use to find their way to a novel location (for example, map, written directions,
or verbal guidance). Participants used the prototype for the other two routes. Researchers
followed each participant and took notes, obtained feedback from the participant, and provided assistance when the participant chose to use verbal directions or became confused or
Six participants were recruited from a pool of adults with cognitive impairments who
were receiving employment services from a community-based rehabilitation provider (see
Table 4.1 for participant demographics). Participants 3-6 had participated in the previous indoor study, but had little familiarity with the outdoor routes or recollection of the
4-1.3 Results
While all but one participant struggled to complete their baseline route, they had few to no
issues following directions given by the prototype to navigate. Participants had noticeably
less trouble transitioning between steps when using the prototype. See Table 4.2 for a
summary of results.
Table 4.2: User study results. Baseline routes are followed by chosen baseline in parens
(w: written, wrverbal, m: map). Time is approximate time from start to end of route
(m:ss). Errors are mistakes in navigation that required prompting or intervention to correct.
Problems are pauses, stops, or hesitations that did not result in incorrect turns. Routes are
ordered consistently for the table, although participants did not perform them in the same
left/right, crowds
determining exact location
message timing
Notable Issues
initiation, left/right
recognizing landmarks
initiation, left/right
message timing
image detail
turn ambiguity
matching landmarks to map
The usability of images
Participants did not have trouble viewing photos most of the time, although it was more
problematic when they had to pick out details in photos without distinctive visual features.
Participant 1 was not able to match a destination room inside a building to a photo with
that room's number highlighted. Participant 5 was not able find some landmarks by photo
and thought that it might have been too sunny to see the screen clearly.
Inconsistencies due to changes in season, weather, lighting conditions, and other influences did not cause any observable impact on participants' wayfinding. In a notable example
of such inconsistency, brightly colored poles that were the most visible aspect of a photo
had been removed from a path. Participant 6 was the only person to note some of the
inconsistencies, pointing out that a trash can had moved and that some plants and trees
were in different stages of blooming.
Turn precision
Because outdoor paths are typically less constrained than indoor paths, we were interested in
seeing whether atypically oriented arrows (for example, slight turns) would cause problems
for users. We did not observe any wrong turns caused by such arrows, however there were
times when the overlaid arrow was ambiguous. One intersection where two nearby paths
both went off to the right caused some participants to take a different path than the intended
Finding a precise location
Even with a person as the location wizard, there were still times when location errors caused
directions to be sent too early or late to participants. Participant 2 passed the doors to a
destination before the interface prompted her to turn and enter it. Confused because she
could not turn at her location, she became frustrated.
Outdoor distractions
We noticed that some participants focused their attention on the device and reduced their
awareness of the environment. Some participants had to be reminded to watch for traffic
even in the baseline scenario, but this was an even greater concern when participants were
using the wayfinding device. However, on the positive side participants did not exhibit signs
of being overwhelmed by crowds or noise.
Baseline vs. prototype
In addition to a reduction in wayfinding errors, we noticed that participants had less trouble
making decisions and initiating action when using the device. Participant 2 demonstrated
the most difficulty initiating each step in her baseline navigation trial. She consistently
stopped at the end of each step and waited for a verbal prompt (for example, "What does
the next step say?") before proceeding. Participant 3 also had difficulty knowing when to
proceed to the next step. For both, the device provided the necessary prompts to move to
the next step without the need for additional prompting by the accompanying researchers.
The results from the user study showed that while just-in-time directions can provide a large
benefit over current wayfinding practices available to people with cognitive disabilities, there
are still issues to address. Relying on photos with overlaid arrows had several drawbacks.
The system would need highly accurate knowledge of the user's location in order to present
photos that align to the user's perspective, otherwise turns could be missed. Since systems
that can provide both indoor and outdoor location and orientation are not yet readily
available, we simulated the location capabilities during the study. This simulation cannot
satisfactorily answer questions such as, How accurate does the localization have to be? How
successful would user be at following directions given realistic location inaccuracy? Although
there exists some support for simulating location error in Wizard of Oz prototyping tools
[44], it cannot be assumed that there will be some "typical" error distribution without
empirical evidence through the use of specific hardware.
Besides the current design requiring highly accurate location to avoid sending incorrect
directions, another problem is that the user's perspective does not always contain distinctive
visual features, so matching the photo to the environment can become a cognitively higheffort task that takes attention away from the surroundings. For these reasons, we next
examine using landmarks as a complementary approach to providing directions for users
and situations where the current types of direction may not be ideal.
Outdoor Landmark Study
In the second outdoor user study, we wanted to better understand whether there were
aspects of landmarks that could affect the usability of directions for this population. Could
users recognize landmarks on a mobile device? What kinds of landmarks might likely to be
known and usable by a system? Are there landmarks that are easier or harder to recognize
in general, or are useful landmarks heavily dependent upon the individual? Do users "plan"
farther when given a landmark rather than a turn-by-turn direction such as the ones used in
the previous studies? Thus, we designed a repeated measures study in a realistic setting, the
University of Washington campus, to explore multiple dimensions of landmark directions.
Directions were classified along dimensions informed by empirical findings on the cognition
of geographic space [39] :
Landmark type: Typical landmarks used in directions include buildings, sculptures, roads,
etc. Landmark information could be derived from Graphical Information System
(GIS) databases or the Web, where geo-tagged collections of landmarks are populated
by web users (for example, Google Earth, Flickr). Sometimes there are multiple landmarks to choose from at a location, so we wanted to learn whether our users would
be more successful at recognizing certain types of landmarks.
Uniqueness of the landmark: Unique landmarks often have names and might be pho-
tographed, while generic landmarks that users might recognize without photos include
bus stops, roads, parking lots, etc. While a unique landmark might be less ambiguous,
if a user is unfamiliar with it, he or she may have to rely on recognizing its visual
features. On the other hand, generic landmarks may be more commonplace and familiar, but without associated photos, variations in their appearance might make them
difficult to recognize.
Landmark distance: Landmarks that are far away from a user's current location may be
more useful for longer-range directions, as they may be visible for a longer duration as
a user is moving. Also, their location with respect to the user's location do not change
as quickly, thus minimizing any problems that could occur due to location inaccuracy.
However, there is also a chance that there might be more obstructions that block the
view of a distant landmark.
Orientation in relation to user: Landmarks may be in front of, behind, or on the left
or right of a user. Intuitively, users should recognize landmarks that are directly in
their view more quickly, but there are situations when no such landmark exists.
Alignment of landmark to path: Landmarks can be used as a goal for a user to move
toward, to move away from, to cross (a road, for instance), or to keep alongside the
user's side.
Perspective of photo: A landmark might be photographed from near the user's location,
containing context surrounding a landmark to aid in recognizing it. However, the
landmark itself may not as apparent as it might be in a close-up photo or a more
"canonical" shot that emphasizes the distinctive features of the landmark.
For the study, we ported the application that ran on the iPAQ to a Nokia N80 mobile
phone with 802.11 (WiFi) (see Figure 4.4). The new platform is capable of vibration
feedback, a frequently requested feature to alert the delivery of a direction. It also provided
a smaller form factor as well as button input capability. The latter allowed us to introduce
a "Help" button to the study, which study participants were informed that they could press
to request help rather than directly ask a researcher for help during the study. While we did
not implement any functionality in the button, we used it to observe whether participants
Figure 4.4: An example of the client in the landmark analysis study.
could remember to request help, and if so, under what circumstances. We also used it as a
basis to later discuss what kinds of help they would want in a real situation wayfinding on
their own.
We recruited 9 participants1 with cognitive impairment through the University of Washington Center on Outcomes Research in Rehabilitation and an outpatient rehabilitation clinic.
Participants ranged in age from 21-60 (mean 41) with 4 women and 5 men (see Table 4.3
for more demographics).
Researchers met with the participants and explained the goals of the study, showed the
interface used, and led them to 4 locations on a university campus. At each location, partici1As an example of how real the problem of wayfinding can be, a tenth participant was recruited for the
study but due to difficulties in wayfinding, could not reach our campus.
Follow the path,
keeping this building
to your left
Follow the path,
keeping this building
to your right
Look for the two
towers in the distance
and head towards them
Turn and follow the path
away from the tennis court
Figure 4.5: The landmark analysis study set-up. A participant starts at one location marked
by the blue cross and is asked to follow different directions to different destinations. Example
directions and expected destinations are shown.
Table 4.3: Landmark Study participant demographics. MS: multiple sclerosis, TBI: traumatic brain injury, MD: Duchene's Muscular Dystrophy.
Health Condition
pants were tasked with following 5 separate directions that used surrounding landmarks; see
Figure 4.5 for examples of directions at one such location. The order of locations was varied
and the order of directions at each location was randomized. Participants were encouraged
to talk aloud, while researchers shadowed and tracked navigation errors, participant confusion, etc., that were part of a set of predetermined categories of observable behaviors.
At the end of the outdoor portion of the study (1-2 hours in duration), a semi-structured
interview was conducted that asked whether they liked different aspects of the directions,
what made directions easy or hard to follow, and other features they would like such as a
"help" mode.
In order to focus on landmarks and reduce the number of factors in the study, arrows
were not overlaid on directions.
4.2.2. Results
We collected and labeled 180 observations of participants following the set of directions.
Participants correctly followed directions 150 times. Several factors are likely to have played
a part in the 30 times participants incorrectly followed the directions. Misunderstanding
the direction (for example, mistakenly moving toward rather than away from a landmark)
was noted in 13 observations. Misinterpretation due to direction ambiguity (for example,
moving toward a street that bordered campus rather than an on-campus road several meters
away) was noted in 15 observations. Participants showed signs of confusion (for example,
circling around, pacing) 16 times, though that did not always result in choosing an incorrect
path. Despite being confused or unsure, participants requested help only 11 times.
Because we recruited people who had not participated in the earlier studies, we ended
up with different representation of health conditions. Unlike participants in the previous
studies, these participants were not as likely to confuse their left and right sides, suggesting
that they may have had a higher capability for wayfinding. However, they had other difficulties that shows the need for a wayfinding system that can support a range of individual
wayfinding capabilities.
Directions using photos taken from the participant's perspective were less likely to lead to
difficulties and were not involved in any incidences of misinterpretation caused by ambiguity
(?2=2.8409, df=l, p<0.01). We initially thought that users would have difficulty clearly
seeing the features of a landmark when the photo was in perspective, given a mobile phone's
limited screen size. In a close-up photo, landmark features occupy more of the screen at
the expense of the surrounding context. The results and participant comments suggest that
this additional context was key to increasing the understandability of directions, because it
lessened the cognitive effort required to identify landmarks.
Participants made fewer errors when directions featured a road or sculpture (?2=8.6154,
df=l, p<0.01) rather than a building or miscellaneous landmarks (for example, parking lot,
flagpole). They also made fewer errors when tasked with traveling toward rather than
to the right/left or away from a landmark (?2=12.1333, df—4, p<0.05). While we chose
landmarks that had distinctive features, we believe that landmarks such as buildings, not
photographed in perspective, were more challenging for participants to clearly identify. This
was highlighted in several sessions, when a participant would focus on finding the reference
landmark first, then forget where the direction told him/her to go in relation to it, or vice
P6: The first step... was to have the instruction of turning to the left or whatever, so because of that I focused. That was my priority, and hence I didn't
always go to the second step on the process.
When a referenced landmark was located behind them, 6 of the 9 participants went
the wrong way at least once. Even though they were reminded that the directions could
reference landmarks anywhere around their location, participants commented that they
expected to have landmarks in their field of view, and missed things such as a flagpole
because it was behind them and taller than they expected.
Pl: If I wasn't facing in a particular direction, like I wasn't sure about the
flagpole. I thought about it and twirled around a bit... I thought that in the
direction that I'm looking, that's where it's going to be, so I just looked there.
Our qualitative observations and participant feedback illuminated some more issues to
Effort vs. time
Individuals with MS often experience fatigue and our participants with MS mentioned that
one of their concerns when traveling is knowing about the effort needed to complete a route.
Specifically, they noted that they might carefully plan a route that includes rest stops, or
choose longer but easier routes versus shorter, more difficult routes (for example, with lots
of stairs) in order to conserve their energy. The system has the potential for minimizing
such effort, and also providing routes that include wheelchair or ramp access for those in
wheelchairs or prefer not to take stairs.
P8: I mean I could go, I would go [along a] shorter [length route] if I had to do
two flights of stairs maybe, as opposed to, like a longer [length route] if I had to
go five flight of stairs or so.
Pre-existing knowledge of places
Some participants were familiar with the campus or surrounding area, while others were
not. We did not use any place names in the study, but noticed that once some participants
recognized a landmark, they would often mention it by its name. By using more familiar
landmarks in directions when possible, the system could lessen some of the cognitive effort
needed by a user when identifying visual features in a photo. In some cases, it may not
even be necessary to present a photo, or even detrimental if it makes a user hesitate and
verify the landmark's location.
P8: I couldn't see it because it was obscured by the Safeco building. I think that
is what [that building] used to be called and so it is now. I just knew where that
was but I'm not sure I would have spotted it among the trees.
Cardinal directions
Several participants were aware of their orientation with respect to the cardinal directions
(north, south, east, west). In some situations, referring to those directions might have been
easier or less ambiguous to them.
Level of detail
Some participants did not think of the route in terms of following path segments. Instead,
they would interpret directions literally. For example, unless a crosswalk was mentioned in
the direction, they would cross the street from their starting location. Other participants
interpreted the directions as general guidelines, so they remained on paths or chose alternate
but equivalent paths that they knew would reach the same destination. The system needs
to provide appropriate directions to individuals on either side of this spectrum to avoid
potentially dangerous situations while also not bogging down the user with too many shortrange directions.
Error detection
When participants went in the wrong direction, some checked as they moved and realized
their mistake, but others committed to their choice and did not reevaluate it. The common
behaviors we observed when participants could not find a landmark were repeatedly turning
in place or taking a "best-guess" and moving in that direction.
P2: I actually considered asking for clarification using the help button. . . But I
didn't, because there's something about momentum that once you start moving,
it's way easier to keep moving than it is to make everything stop.
Many participants did not press the Help button on the phone during the study. Besides
not wanting to "make everything stop," it is possible that they could not decide when they
needed help. As one participant noted, such an event often causes some level of stress and
impedes problem-solving and meta-cognition. This effect of cognitive tunneling [93] can
be severe enough to significantly impact wayfinding, and is one that must be considered
seriously in designing for people with cognitive impairments.
However, if the system were able to determine that help was needed, the kinds of help
that participants suggested included revealing more detail about the landmark via text or
animated zooming, providing a different set of directions, or calling another person. Calling
another person was widely considered an action that would be taken as a last resort, but
potentially necessary at times when totally lost.
P4·' It would give you the direction in a different way... You could have a GPS
function and a person [elsewhere] to find out where you are and then, when you
press the help button, it will call a certain person who knows that area. They
would be able to see where you are located and maybe look on the map to help
you to find where to go?
P5: I know it's around here someplace, so should I take a left or a right? That
would be one level, the other level would be I'm on campus but I have not a clue
where I'm going...
Device issues
The vibration notification was a welcome feature to study participants, although if a user
was not holding the device (for example, when the device was resting on a wheelchair tray)
then its usefulness would be diminished. Most participants said they could view the text
on the screen without trouble. Unexpectedly, overcast conditions caused more problems for
viewing the screen than sunny conditions, because the screen would reflect the cloud cover
and could not be easily moved to shield away from the source of the glare. Under these
conditions more care needs to be made in selecting landmarks, potentially including the use
of animation (as was suggested) to zoom in on the visual features of the landmark that a
user could look for.
Situational issues
While participants might have preferred or been successful with certain types of directions,
several mentioned that their situation could have a significant impact on how they wanted
the system to behave.
P2: For me when I'm on a big relapse, I wanna know how to get to where I
need to go as quickly and as easily as possible.
P6: I think who cares, you know, I just went. But if I wanted clarity. . . because
I was really nervous about finding a place. . . it depends upon how well I know
the area, how comfortable I feel being in the area...
4-2.3 Summary
These results suggest several considerations when choosing the appropriate landmark and
its photo representation when providing directions to help guide individuals with cognitive
impairment. Nearby landmarks that are in the user's path should be preferred, and should
be shown with a photo of the landmark from the user's current view. Identifying a landmark
can be a cognitively challenging task, and if an individual does not find the landmark
immediately, they may become stressed or confused, making it even more difficult for them
to perform the problem-solving necessary in navigating. The best photos are the ones that
lower such cognitive effort by providing features that are evident to the particular individual.
While certain types of directions did not match the majority of the participants' expected
usage model, many directions elicited more varied responses. Only 4 of the 20 directions
were misunderstood by more than 2 participants, while 7 directions were misunderstood
only by a single participant. These findings suggest that the ability to adapt the photo
selection algorithm to individual users is a crucial requirement for the system.
Our studies provide further evidence that that both customization and adaptation in a
wayfinding system are needed. Supporting customization involves incorporating individual
users' health conditions, preferences, ability to handle detail, error behavior, safety concerns,
and place familiarity, among others. Adaptation involves adjusting system behavior when
initial customization is not sufficient, because of changes to the situation, due to user stress
or energy levels, the environment2, or users' own preferences [83].
Current navigation systems are limited in their ability to support customization and
adaptation. For example, Global Positioning System (GPS) navigation devices give users the
ability to choose between quickest and shortest routes, but every user that chooses the same
route will receive the same type of direction, without regard to user preference. GPS devices
also do not adapt based on user behavior - if a user does not wish to follow the device's
proposed route, the device may create a new route, but the next time the user is traveling to
the same destination, the device will revert back to the original route. Because GPS units
do not have alternative methods for delivering directions, they cannot produce different
In the landmark study, the large "Broken Obelisk" statue in Red Square that had not been moved for
decades was taken down for repairs. Such changes are difficult for any system to predict, but should be
handled by producing alternate directions to avoid a breakdown in wayfinding.
levels of help that a user may desire. Finally, current devices do not support incorporating
landmarks into directions, despite the utility of landmarks in pedestrian wayfinding.
Next steps
To produce a wayfinding system that better supports the needs of individuals with cognitive
impairments, we must enhance the user model that control the system's routing and message delivery. The next chapter describes a system to automatically generate customized
directions. The central piece of the system is a decision-theoretic framework for choosing appropriate directions and adapting to user success over time. The system will automatically
generate directions that previously required manual creation. The chapter also describes the
incorporation of a landmark selection system that can retrieve photos of landmarks based
on criteria that represent what is best suited for the individual user, such as a photo that
shares the same perspective or highlights a visual aspect that the user tends to recognize
more easily.
Chapter 5
This chapter discusses the framework we use to automatically produce a personalized sequence of wayfinding directions, then a user study of the system that investigated individual
preferences and abilities to follow the different types of supported directions.
Using a Markov Decision Process to Produce Directions
The robotics community has developed various techniques for path planning under uncer-
tainty [88]. A key concept in this context is the Markov decision process (MDP), which
provides techniques for generating navigation plans even when observations and the outcome of navigation actions are uncertain [85]. For example, partially-observable MDPs
have been used to assist persons with dementia through tasks such as hand-washing [4].
The framework we chose builds upon these techniques to model uncertainty in whether or
not a person will follow the guidance provided by the system, but to reduce the state space
necessary to solve the MDPs, we rely only on observable action results.
MDPs are defined by state and action sets and one-step transitions. States have
associated rewards, actions have associated costs, and a solution to a MDP is a policy
that maps states to actions in order to maximize expected reward. A key aspect of MDPs is
that they can be used as a framework for learning and adaptation - transition probabilities
may be approximated at first and then updated given observed behavior. Techniques for
solving MDPs have been shown to enable the generation of navigation plans in robotics,
and we believe they map well to the problem of choosing directions along a route for an
individual with cognitive impairment that maximizes the chances of success.
There are some differences between previous applications of MDPs to robot navigation
tasks versus producing wayfinding assistance to a person. One noteworthy alteration to
traditional MDPs that we are using are options, which we use to represent each direction
instead of actions [86] . Traditional MDPs rely on fixed time slices and actions take only a
single time step, but obviously users may take a variable amount of time when following
directions. Options can be considered to be sub-policies that dictate when certain directions
can be given, what expected transitions will occur within the option, and when they should
terminate so that a new option may be selected. Incorporating options allows us to reason
over temporally-extended actions with a lower computational cost. There is a trade-off in
terms of the optimality of the eventual solution, however this trade-off is reasonable because
finer-grained control over system actions does not make sense, given that users expect only
a single direction at any give choice point.
A MDP state consists of variables that determine available options and affect transition
probabilities. The most accurate model would incorporate every valid, observable variable
into the state, but at the expense of a huge state space. Our initial design of the model
contains only a very small set of state variables, with the intention of investigating whether
it is sufficient, and if not, what other variables would be the most beneficial to incorporate
into the model.
The state consists of user position (location and orientation), the current option selected
by the system, and a options blacklist. User position is a critical variable. Location is
represented as a node in a graph network, while orientation is represented as either a
heading toward an adjacent node or from such a node. We consider the current option as
part of the state for the sake of simplicity, because the choice of options is determined by
a combination of the current option and the (other) state variables. The options blacklist
provides a way to temporarily avoid using directions that have been unsuccessful in the
recent past. It is represented as an array of decrementing counters, where an option is
given a positive value when a user incorrectly interprets that option, and that option is
not available for that many subsequent future states. A new model is computed when the
blacklist is changed, generating an alternative set of directions.
5.1.2 Option
An option is a sub-policy with its own set of termination states. To simplify the model we
define the set of termination states of an option to be any state where the user has changed
to an adjacent location, changed orientation, or not moved at all after an abnormally long
time. The sub-policy determines when a state change has occurred. This determination
is not yet implemented in the system, so we simulate that functionality manually in the
Wizard of Oz style.
The options available in the system are:
Straight: This option is given to make a user move forward in the direction that they
are facing. It might be used to provide further assurance to the user that they are
moving in the correct direction, or to prompt a user to continue moving in case the
user expects a new prompt from the system.
Turns: This category of options covers making a turn. There are different turn options
depending upon the type of intersection. A regular left/right turn is available if there
is a standard intersection, while slight turns are usable when the angle of the turn is
closer to straight. First and second turns are supported in the case where there are
more than one possible path on that side. We choose not to support hard turns to
keep options as simple as possible for users.
Turn around: This is the U-turn option. It may not be as effective, as we have found
that turning around can be disorienting for some, but it can be used to correct a user
moving in an undesirable direction.
Landmark: Based off the landmark analysis study in Chapter 4.2, we found that a number
of users are capable of identifying and moving toward a landmark provided a photo.
Because users did significantly better moving toward a landmark in front of them, we
choose to make this option available only when landmarks are visible and in front of
a user.
Stop: This option is given to make a user stop. Reasons for stopping include notifying the
participant of reaching a destination, or introducing a pause in wayfinding when close
to traffic for extra caution, or to prevent a user from moving too quickly away from a
"better" (from a wayfinding standpoint) location.
Once states are defined, transition probabilities are assigned to map initial state-option
pairs to subsequent states. In other words, if the system were to choose an option while
the user is in a given state, how will the user react? This is where user customization
plays a major role. Transition probabilities can be seeded based on general intuition of
wayfinding (for example, the findings in the landmark analysis study that directing a user
toward a nearby landmark has a higher likelihood of success) can be defined manually if
enough information is known about the user's preferences and capabilities. Otherwise, the
user might go through some evaluation routes to collect initial data on their tendencies.
Health conditions with impact on wayfinding can dictate an initial distribution over the
probabilities. For individuals with visual impairments, directions that use audio and text
may be preferred over those that involve detailed landmark photos, so the chances that the
former type of directions are successfully followed should be greater than the latter's.
Costs and rewards
Ultimately reaching the goal location is a highly desirable state, while giving a new direction
has a cost both in terms of cognitive load, distraction, and time. Some of the study participants mentioned that reaching a destination in the quickest amount of time or taking the
shortest path is less important to them than being more aware of their route ör using less
physical effort. Some users benefit from "high-level" directions that apply beyond a single
turn, because they can plan their own routes farther, while others interpret directions more
literally, so directions for them may need to be step-by-step or include additional wording
to avoid ambiguity.
5.1.5 Producing a policy
A policy is a mapping from every state to an option that produces the highest-valued
expected reward. It is calculated by iteratively backing up state values. Each state's value
is the sum of its reward and a discounted expected value of all next states of the best option.
The discount factor puts a preference on nearer-term rewards. We use value iteration to
determine the policy, with a termination threshold that stops iterating when no state's value
changes between iterations by more than a small delta.
User Study
To be useful, generated directions for an individual requires an accurate user model. We
therefore conducted a study to learn key individual preferences and abilities that would
suggest the types of customizations needed to be supported by the system. Using these
findings, we would be able to support system customization for future users, who might
initially either go through similar trial usage sessions and/or answer a set of pre-defined
questions to "bootstrap" their system's model.
The study involved participants wayfinding through two routes on the University of
Washington campus. We defined two different static user models to create policies for these
routes. The client was implemented on a Nokia N95 model and is shown in Figure 5.2. We
chose the N95 because it has a larger, newer screen than the N80 model previously used,
and because it has GPS onboard, though we did not use the reported coordinates for the
Similar to our methodology in prior user studies, user position and message timing were
still controlled by a wizard, a person who shadows the participant with a Tablet PC (see
Figure 5.1 for the wizard/system interface). Unlike our prior studies, all system prompts
were automatically chosen by the system calculating a policy using the active model. In
addition to the wizard, two researchers shadowed participants to note reactions to the
system, and to intervene if there were safety concerns. Using system interaction logs and
researcher notes, we tracked success and difficulties wayfinding due to choice of prompt,
wording, timing, etc.
File Edit View Actions Made Options
• OK
Current statt·:
' .st: IJO From: 131 4~,657 23S4K>4284, -K
Curr-nt opt: JTRiJSHT
Froi-:jr>es!5: tasi*
Vaiò« ;~4.ì-:'ulìKCC-33254
• Best ??-st opt: PO!
B*st FOI: Sia=.·?;.
E»3¿ id SS
Figure 5.1: The Tablet PC Wizard of Oz GUI used in the user study. The left pane shows
the graph network with Route 2 in our study from the start (top-left node in red) to the
destination (south of the Art building on the right, in yellow). Metadata for the route
segment ahead of the user's position (the dot and arrow) is shown in the upper right table
and can be used by the model or by the prompting system. The lower right pane shows the
current state variables and available options.
60 along the path toward the
Student Union Building (HUB)
Figure 5.2: Wayfmding system front-end running on a Nokia N95 mobile phone. A
landmark-based direction is shown with a selected photo, overlaid arrow, and text. Au-
dio equivalent to the text is also produced using the built-in text-to-speech function. The
phone is used in landscape mode to utilize a larger portion of the screen for images. The
screen remains slid out to expose the Global Positioning System (GPS) antenna and keypad.
Any of the top row of keys can be pressed to repeat a previous direction. The center face
button is used as a Help button that requests a different direction.
Initial models for study
We defined two models to study that use the same state variables, but have different transition probabilities and costs.
1. The Naive Model: The first model incorporates few assumptions about wayfinding.
It assumes that every option will be correctly followed 100% of the time, and that all
options have the same base cost. An additional cost for distance traveled is added
so that the distance of a route is also factored into the policy. A high reward is
given for reaching the goal location, and no discount factor is applied to future state
values. When more than one option has the same expected value, one is picked
pseudorandomly based on a distribution seeded by the state. This ensures that there
is no variation in the policy across different users.
2. Landmarks for Longer-range Movement Model: The second model adds some
additional assumptions about wayfinding to move closer to a model suggested by
our previous study results. First, it penalizes the Stop and Turn Around options
by assigning higher costs to those options, since users tend to prefer that the system
adapt to their chosen route rather than be corrected after every minor route deviation.
Second, it assumes that lower-level options are more likely to be followed correctly, but
gives an additional bonus to use a Landmark option when the same landmark was just
followed correctly. This models the assumption that a landmark option potentially
requires more cognitive processing by a user at first, but that there is an advantage to
using a landmark for longer, continuous stretches of a route. All other positions are
given a small, non-zero probability, and the total weights are normalized to represent
the transition probability of the options. A discount factor of 0.98 is applied to future
state values to reflect the uncertainty of future options and slightly prefer nearer-term
5.2.2 Participants
We recruited 7 participants with cognitive impairments through the University of Washington Center on Outcomes Research in Rehabilitation and an outpatient rehabilitation clinic.
Participants ranged in age from 21-49 (mean 34) with 1 woman and 6 men (see Table 5.1
for the demographics). We also conducted the same study with 6 participants without
impairments, ranging in age from 28-46 (mean 36.7) with 2 women and 4 men. As the
results from the participants without impairments were similar to the results found from
the participants with impairments, this chapter focuses on the findings from studying the
participants with impairments.
Due to the modest number of participants for the study and the high variability in the
directions that each received due to differences in where wayfinding errors occurred, we focus
more on the participants' notable actions and comments during the study and debriefing
session. Of the 7 participants with cognitive impairments, 6 were able to follow the directions
with only minor errors such as wrong turns or other issues due to our prototype system's
limitations. Participant 4 had some minor problems interpreting landmark photos along his
first route when using Model 1 because he did not notice the overlaid arrows on the photos,
while on the second route using Model 2, he needed a significant number of rerouting
directions from the system to recover from several wrong turns.
Landmark-based directions
Landmark-based directions were well-received by some but found less useful than the turn-
based directions by others. Participants 1, 2, 3, and 5 all expressed the usefulness of using
the photos to wayfind. Participant 3, in particular, would often press the Help button
when given other types of directions in order to receive a photo-based direction. Suggesting
the usefulness of Model 2, all of these participants were able to wayfind flawlessly when a
follow-up direction used the same landmark (for example, "Continue along the path toward
Table 5.1: Participant demographics. *: Uses powered chair
Health Condition
Traumatic brain injury
Traumatic brain injury
Cerebral palsy*
Traumatic brain injury
Cerebral palsy*
Multiple sclerosis
Pl: "When you have an actual picture of what's in front of the person, that's
excellent. "
P2: "I can coordinate [matching the picture to what he can see] pretty well. "
Participants 4, 6, and 7 preferred the lower-level, turn-based directions. Participant 4
and 7 had difficulty seeing the images on the phone, while Participant 6 simply found the
turn-based directions to be easier. During the debriefing session, Participant 7 confirmed
that the lighting conditions outside made it difficult for him to see the images, but under
better conditions he was able to see the images more clearly, suggesting that a better (larger,
brighter, and/or higher-contrast) screen might have made a difference in his wàyfinding
The other notable usability issue with the landmark-based directions occurred when
the provided photo was taken from a different vantage point than the participant's actual
location. In some cases, a photo would be used from the other side of the street. In these
cases, participants would either trust their own judgment and choose a direction to go in,
or request a different type of direction.
Pl : "The towers [of the Campanile] were behind trees, but I trusted myself to
go forward. "
P2: "I think [the directions were] pretty clear... if I made a mistake it could tell
me to turn around. "
P7: "I couldn't see the photos, there was too much glare, but the façade and
context helped in certain cases. "
Using landmark names
In our study, we chose to include the name of the landmark in every landmark-based direction. Participants expressed different opinions on using the names. Some participants,
though unfamiliar with the campus, stated that they would try to match the names to signs
as a way of confirming their direction, and that it could be helpful in remembering the
landmarks for future use. Other participants stated that they ignored the names. In one
case where using the landmark name was detrimental, Participant 6 made an incorrect turn
based on an incomplete knowledge of where a certain landmark was on the campus.
P2: "Some of the signs on the buildings are tricky, you just have to figure out
where the signs are. "
Participants suggested that only referring to a name in limited cases, such as to describe
their destination, would allow them to focus on finding the landmark carefully only when
it was necessary.
Using compound directions
In our previous studies we found that sequences of turns could be problematic when they
occurred close together. Some of our directions were therefore compound directions such as
"Go along the sidewalk toward <landmark> and take the next left.", followed by a reminder
to turn once the participant came closer to the intersection. This caused some confusion
for participants, who interpreted the second turn direction as a separate instruction to be
completed after the first, compound direction. Participants noted that they felt that they
were falling behind while the directions were delivered too quickly.
Stop. You ve
(a) Straight and standard
left/right turns use solid ar-
(b) Stop and U-turns use standard street signs
Take the second right and follow
the sidewalk.
(c) Modified street sign for
non-standard intersections
Figure 5.3: Examples of turn-based directions.
At other times, the compound directions were misinterpreted. For instance, Participant
6 would sometimes perform the straight and turn segments in reverse order.
Other direction issues
Some participants had a few issues with non-standard turns (slight turns, or "second left"style turns such as in Figure 5.3c). Participants 4 and 6 missed a segment of the second route
that the system attempted to direct them towards when they did not spot the accessible
ramp to the side of a small flight of stairs.
Using help
Participants 3 and 4 used the Help function often to get different directions, for opposite
reasons. Participant 3 preferred the landmark-based directions and would use the button to
request them, while Participant 4 did not find them helpful. The other participants used the
help function sparingly, or not at all, which matches our previous observation that potential
users may be better served with some form of automatic assistance.
Attention between device and environment
We observed that Participants 2, 3, 5, and 6 were able to balance their attention between
the device and the environment. Participant 4 heavily relied on the text-to-speech audio
and did not pay attention to the environment at most times, though unlike in our previous
studies, no participant crossed a street at an unsafe location.
Participant 1 explained that some combination of being nervous and unfamiliar with the
device might have contributed to her focusing too heavily on the device, while Participant 7
also mentioned that increased familiarity with the device would have changed how he used
Pl: "I would have enjoyed the walk a lot more if I just would've relaxed. [I]
know it's going to buzz in [my] hand so then [IJ can look at it, but I looked at it
the whole time. "
Pl: "I was thinking for a while, maybe I should stick this in my pocket and react
when the thing goes off. I didn't do it because [I thought], 'Well, by the time I
take it out, will I miss the instruction?' but obviously not, the instruction stayed
on which was good, so I should have. "
Hardware usability
We observed some drawbacks to using the N95 as the sole device for this study. As previously
noted, some participants found the screen on the N95 to be insufficiently bright for displaying
images, though others were able to see them. Participants 4 and 6, who have cerebral
palsy and use motorized wheelchairs, found simultaneously holding the device and pressing
buttons to be uncomfortable. Participant 4 suggested that an arm attachment would allow
him to detect the vibratory new direction alert, while Participant 6 suggested a swivel
mount that could be attached to one armrest of his chair.
Another issue during the study were accidental keypresses that switched phone applications. As we were unable to disable many of the keys on the N95, this caused issues where
miscellaneous applications would take the foreground and had to be exited manually by a
Aside from these issues, the size and form factor of the N95 was deemed acceptable.
Participants felt comfortable with using it out in the public without stigma.
P3: "I don't find [the device] obtrusive. Nowadays everybody's carrying something. "
Overall participant impressions
The overall impressions that our participants had were highly positive. Participants expressed the desire to have the system as soon as it becomes available, with most volunteering
to participate in further studies as the system is enhanced.
Pl: "I'd take it to a new city. That would be fantastic!"
P2: "Especially in my case, since I do a lot of performing and I need to get
directions, because sometimes I go places I haven't been before. "
P5: "I use my iPhone for directions, but if I had this, I would definitely prefer
to use it for the images. "
The results showed that the system, with a few adjustments, could be suitable for potential
users to independently wayfind. It also confirmed that there would be individual, disparate
preferences and interpretations of the directions. Below are some of the customizations that
could be needed to produce high quality directions suitable for the user.
Individual preference for types of direction
Solving a MDP produces a policy that chooses the direction that the model believes most
likely produces the highest reward. The transition probabilities that the model relies on to
calculate this policy can be initially populated with approximate values based on our user
studies. For example, the probability that a user will successfully follow a direction that
involves moving toward a landmark ahead of her would be higher than that of a landmark
behind her. A direction that uses a photo with an overlaid arrow might have an even higher
chance of success, but only lead the user to a nearby location, while the landmark-based
direction could allow the user to make more progress towards the destination. In addition,
some study participants have stated that they preferred to have a better idea of where they
are going, so landmark-based directions might again be preferred and states involving their
use would be assigned higher rewards.
Leveraging user familiarity
Landmarks familiar to a user should be used more often than those less familiar. Similarly,
users comfortable using cardinal directions might be given them more often. The model
can reflect this by boosting the probabilities of success for options that use familiar places
or terms.
Level of detail in a single direction
Our studies showed that while minimizing the cognitive load necessary to interpret a direction is beneficial, some users may benefit from more "high-level" directions that cover more
than one route segment, because they can plan their own routes farther. Some individu-
als interpret directions more literally, so directions for them may need to be more in the
step-by-step fashion or include additional wording to avoid ambiguity.
Health conditions that impact route planning
One critical aspect to customization is that people with cognitive impairments often have
other health conditions that impact their device use and the appropriateness of routes.
Visual or hearing impairments would affect the efficacy of landmark or audio directions,
respectively. For individuals who may be concerned about fatigue, such as Participant 7,
the MDP can increase the weight of distance to cost so as to minimize the total effort
involved. For those who require accessible pathways and entrances, options involving stairs
or other inaccessible paths can be heavily penalized.
Detecting errors and intervening
Different users make different errors. Some find that they cannot follow a direction, stop to
look around, and become frustrated. Others will pick a direction even when unsure. Some
want to be corrected right away if they are taking a path that makes it difficult to recover
later (for example, people with multiple sclerosis, who get fatigued). Others are fine with
their decision and just want to proceed rather than backtrack if it's unnecessary. Encoding
these errors as states involves identifying the behaviors based on the individual's tendencies
and assigning large penalties to them, while also terminating the option that caused the
error so a new option can be selected.
Customizing for safety
We observed that some individuals tend to be more careful and maintain an awareness of
their surroundings, whereas others do not always remember to look out for vehicles and
other hazards. While teaching individuals about traffic safety is outside the scope of our
design, our system must be able to be customized for the range of user behavior in regards
to safety. For users who are not in the habit of crossing only at crosswalks or do not always
look out for traffic when crossing streets, the system might explicitly direct the user to a
crosswalk and instruct them to wait until the street is clear of traffic before having them
cross. For users who tend to be more careful, an occasional safety reminder might suffice.
Next steps
Solving an MDP involves computing the expected value of every state and producing a
policy that decides on the highest-valued option to take at each state. The quality of the
policy is only as good as the accuracy of the underlying model at predicting user behavior.
The following chapter discusses our use of a short trial evaluation to define a model's initial
transition probabilities. Given the large amount of effort required to gather wayfinding
observations from one individual, we investigate the pooling of training data as one method
of fitting a customized model. We then discuss a complementary approach, which is to use
reinforcement learning to improve model accuracy as usage time increases.
Chapter 6
The results from the previous study showed that automatic generation of directions is
possible given a user model of direction difficulty. It also showed that potential users can be
sensitive to differences in wayfinding experience, motivating the need to create user models
that reflect individual wayfinding abilities and priorities. This chapter describes the first
steps toward determining whether it is possible to learn those models through observations
of individuals wayfinding.
User Study
We conducted a user study involving participants following the same sequence of directions
around the University of Washington campus. Each direction consisted of an initial vibratory and audio alert, followed by an image, audio, and textual message. Directions were
displayed on a Nokia N95 8GB model mobile phone.
We assigned functions to four areas of the phone keypad, as seen in Figure 6.1. Participants could press the Help button to receive the alternate form of direction than the one
they initially received. Pressing the repeat button would repeat the audio message, in case
participants wanted to hear it again because they could not see the screen clearly or pri-
marily relied on the audio to follow the direction. Pressing the "1|2" buttons would switch
between the initial view of a landmark based direction involving a turn (Figure 6.2a), and
the view at the intersection that would be seen if the turn were correctly followed (Figure
Directions were either landmark^based (directing the participant in relation to a landmark such as a building) or turn-based (messages with few details except the turn to take).
Go along the path toward the
Student Union Building (HUB)
Figure 6.1: Wayfinding system front-end running on a Nokia N95 mobile phone.
landmark-based direction is shown with a selected photo, overlaid arrow, and text. Audio equivalent to the text is also produced using the built-in text-to-speech function. The
phone is used in landscape mode to utilize a larger portion of the screen for images. The
screen remains slid out to expose the Global Positioning System (GPS) antenna and keypad.
Any of the top row of keys can be pressed to repeat a previous direction. The center face
button is used as a Help button that requests a different direction. The functionality of the
two button areas at the bottom of the keypad are described in Figure 6.2.
Figure 6.2: Route preview features, (a) The initial photo-based direction containing a turn
that is shown on screen, (b) By pressing "2" on the N95, a user can see the view at the
intersection that would be seen if the turn were correctly followed.
(a) Follow the path straight
(b) Follow the path straight
ahead, then take the next left
(c) Take the path slightly to
(d) Turn around
your right
(e) Follow the path straight
(f) Follow the path straight
ahead, then take the second
right you come to
ahead toward the fountain
(g) Go toward this building,
(h) Follow the sidewalk toward
then turn right
the lawn, then take the next
left and cross the crosswalk
Figure 6.3: Example of directions used in our study route. Figures (a) to (d) show shorter,
"low-level" directions, while Figures (e) to (h) show more complex directions that contain
additional information, such as more specific turn descriptions, landmarks, and compound
Table 6.1: Frequency of direction types in the study route.
Direction type
Straight ahead
Turn around
Left or Right
lst/2nd Left/Right
Slight Left/Right
The various types of directions used are shown in Figure 6.3 with the corresponding text
direction underneath.
We created a single fixed route around our university campus that used a variety of
turns, intersections, types of landmarks, and distances. The route consisted of a sequence
of 50 directions and, unlike a typical route, contained many consecutive turns and U-turns.
Figure 6.4 shows the entire route, while Table 6.1 shows the frequency of each type of turn
when broken down into a few basic categories. We aimed to have good coverage of the
different types of turns possible on campus, though the scarcity of intersections that had
multiple outgoing paths on a single side limited the number of such "lst/2nd Left/Right"
directions. Similarly, few intersections had only a single path veering off slightly to one
side, resulted in only 3 such directions in the route. Many directions involving landmark
photos were compound because they involved first acquiring a landmark and then turning
in relation to it.
A researcher acting as the Wizard of Oz simulated the location tracking system, and
directions were triggered as each participant neared an intersection. Whenever participants
made a wayfinding error, we verbally guided them back to the correct path.
We recorded any comments voiced during the study by participants who gave permission
to be recorded, otherwise we took note of their comments. In addition, we followed and noted
when participants made errors or exhibited signs of confusion indicative of a direction being
* M?mor·*',,
s- 5 Levels
Gen -al Plaza
P&rtór ? Gara«
Theater and
\xGarclen ,
^ J^^i
Figure 6.4: The study route that each participant followed, beginning on the northeast
section of the map near the Husky Union Building (HUB) and ending in the southeast
section in front of More Hall.
unclear to them, while we also logged button presses on the phone to track when alternate
directions were requested.
After the route was completed, we conducted a semi-structured interview and debriefing
session with each participant, collecting feedback on the aspects of the user interface they
liked and disliked, their preferences over the different directions types, and other comments.
Each study lasted about 1 to 1.5 hours. Participants were told to travel at a comfortable
but leisurely pace during the wayfinding, as we were not evaluating them based on time to
completion of the route.
6.1.2 Participants
We recruited ten participants with cognitive impairments for the study. Individuals who
participated in previously conducted research studies and expressed interest in participating
in future studies were contacted via email. Of these, four subjects were able to participate in
this study. The remaining participants with cognitive impairments were recruited through
the University of Washington Traumatic Brain Injury Model System program. All of those
participants are receiving or have received services from the TBI Model System. We also
recruited three participants without impairments among staff of our university. Due to
the small sample size of our pool of participants without impairments, we chose not to
thoroughly analyze their results, though their experience following the directions were qualitatively similar to participants' with impairments. Table 6.2 contains the demographics of
our participants with impairments.
Due to minor technical problems, Participant 2 received only 48 directions and Participant
3 received only 49. For each participant, we tallied:
• Errors: directions that were not followed correctly
• Confusion: directions that caused hesitation noted by the researchers following the
Table 6.2: Participant demographics. *: Uses powered chair
Health Condition
Multiple Sclerosis
Cerebral Palsy*
Asperger's Syndrome
Multiple Sclerosis
Traumatic brain injury
Autism Spectrum Disorder
Traumatic brain injury
Multiple Sclerosis
Traumatic brain injury
Traumatic brain injury
• Help: directions that led to the participant pushing the "Help" button to request an
alternate direction
• Incident: a term we use to denote a direction that led to any of the previous three
The results from the study are tabulated in Table 6.3.
Qualitative evaluation
In general, participants were positive about their experience following the directions in the
study, with only Participant 6 expressing strongly negative opinions about the wayfinding
system. Participant 6 requested help for many more directions than the other participants,
in large part because he did not find the overlaid arrows on photos clear and struggled with
the dual arrows denoting first or second turns.
Landmark-based directions
Consistent with our previous study findings, participants varied in how they responded to
different direction types. For instance, Participants 1, 3, 5, 7, and 9 preferred the augmented
photos of landmarks. A common reason for this preference was because the photos provided
Table 6.3: Number of directions that led to an error, observed confusion, help (alternate
direction) request, or any of the three, by participant.
more context for wayfinding (for example, Figures 6.3f-h), giving them more confidence as
they traveled.
Pl : "[Landmark directions] gave me that feeling of confidence, that 's where they
mean to go. . . Those made sense to me every time I saw them. You know you 're
exactly on the right path. "
P5: "I think the images worked best because I could study them on the screen and
compare them to what I was looking at. The image is specific, icon is general.
[A landmark direction] gives you a lot of detail, shows you surrounding area to
confirm on that. Shows you the building structure and you can confirm that... I
know absolutely that's where I'm going [because there are] details I can confirm.
And now I'm happier, relaxed, and pleased, so it eliminates doubt. "
Three participants did not have a strong preference between landmark-based and turnbased directions. Participants 2 and 4 did not have a clear preference between landmarkbased and turn-based directions, both saying that it would depend on the wayfinding sit-
uation. They found landmarks useful when the photos were clearly visible on screen, but
glare on the phone screen made the photos difficult for them to see at times. Participant
10 said she mostly relied on the audio because the discrepancies in landmark appearance
between the photos and what she saw led to them being less useful to her.
P2: "What I'd like to see is both [direction types] as an option. Say it was
cloudy one day, sunny the other, you have a choice between what [type]."
P'4: (during route) "This [photo] is nice to have now because I see I've done the
right thing, so I don't have to worry about whether I got that right, and I can
see my destination. I feel so much better when I can see my destination. "
P4: (during debriefing) "My feeling about [direction type] was situational. I
like the simplicity of the arrow, that's real clear what you want people to do,
but I really enjoyed the pictures because, for reassurance, that I was doing the
right thing. I do not have problem seeing the icons, so when we get out into the
brighter sunshine, I often had problems with the pictures, finding the [overlaid]
arrows. "
PlO: "I mainly went off of audio, because the pictures, sometimes I noticed it's
a different season, so sometimes the pictures looked much different than what
the actual building looks... But also you could see similarities. You can get
landmarks on the buildings."
Participants 6 and 8 commented that they found the photos difficult to see or that
buildings in the environment often looked too similar to distinguish. Besides the normal
visibility issues that other participants also noted, a specific issue they both had was the
lack of contrast of the overlaid arrow, especially when the ground was of a similar color, for
example when drawn over the reddish brick in the UW Red Square area in Figure 6.3g.
Turn-based directions
Participants were also split on the usefulness of the double-arrowed icons that signified
taking the first or second turn at an intersection (see Figure 6.3e). Participants 6 and 9
were confused by which arrow to follow, the bolded dark arrow or the shaded gray arrow.
Participant 9 had a problem determining which turn to take initially but said afterward it
was clear that he should follow the darker arrow.
In contrast, Participants 5, 7, and 10 praised the icon, telling us that it made it more
clear that there was a path they should avoid taking by accident. Participant 7 mentioned
that he felt they were useful as an alternative direction when he needed help.
Pl: "¡The double arrow] would be good for help, if you see something like this,
go to help, and that 'Il be more understandable]. For just cruising around or so
forth, this would work good if it doesn't get complex. "
Participants who were more comfortable with the double arrow also mentioned seeing
similar graphics on traffic signs. Participant 9 was an exceptional case, as he said that he
had never encountered such signs while driving, which contributed to his confusion.
Requesting help and correction preferences
As evident from Table 6.3 and Figure 6.5, participants varied quite a bit in the number of
directions to which they requested help. Some participants had little difficulty during most
of the route, or consciously chose not to request help. Participants 3 and 10 made very few
errors and showed little confusion during their sessions. Participants 5 and 9 misinterpreted
the directions more but rarely requested help. Participant 9 preferred to avoid requesting
help in the same way he avoids doing so when wayfinding independently.
P9: "My way of thinking was, soon as I used the button, Ifailed. I did something
wrong or, 'Why don't I get this?'"
However, Participant 9 was not against a system informing him of a mistake, as long as
it was subtle and not alarming. Participant 4 also felt that corrections of a certain manner,
such as being prompted to turn around, could be considered negatively, though during this
study she approached it more casually.
Incidents by Participant
£? 15
errors D confusion ¦ help
Figure 6.5: Bar chart showing the frequency of occurrences of different types (errors, confusion, and help) by participant.
P4: "[The turn around direction] made me feel naughty (laughs). Uh-oh, did
the wrong thing!"
P9: "Maybe if it buzzed when I did something wrong, that would be the easiest
way. No alarms, because alarms would set off a panic. "
Other participants differed in how they wanted the system to behave in situations where
they deviated from a route. Some preferred to stay on the previous route and be told to
turn around. Others thought they should be given the option to proactively decide on the
correction strategy.
P2: "I would like for it to tell you to turn around. I would like there to be a
button to recalculate [the route] if you don't like where it's telling you where it's
turning around or if it 's telling you where to go and it 's not wheelchair friendly. "
P3: "Like you could either choose most efficient way or scenic way, you get
different options. "
P4: "I'd rather not meander, when it first told me to turn around, I felt chastised. If I were to continue along the longer path, I might be annoyed, because
I might have figured out that it was kind of a long way, so personally I would
prefer to be efficient about it. "
Pl O: "If I could get to this location without too much difficulty as opposed to
turning around. "
Compound directions
In this study, we found another difference between participants was how they dealt with
compound directions, such as those that directed them to turn at an intersection and follow
a path or those that directed them to head toward a landmark and then turn. Some
participants preferred simpler and shorter directions, such as those in Figures 6.3a-d,f,
while others had no issue with the lengthier directions.
A related direction design choice is whether to include upcoming turns. A common
difficulty we noticed that participants had with directions occurred when the turn was a
significant distance away. For example, in directions such as 6.3g, participants varied in
how far they moved toward the landmark before turning. The open space of Red Square
exacerbated the problem because it did not constrain where they could turn.
Previewing turns
We added the "1|2" buttons to the prototype because previous participants suggested having
a feature to preview wayfinding directions ahead of their current location. Since the "2"
button shows the view after making a correct turn, it can be used to clarify the current
landmark-based direction, since the overlaid arrow is not always exactly overlaid correctly.
However, there were almost no uses of the function during the study. Participants either
forgot its functionality, or stopped trying to use it after trying to trigger the function on
other directions types.
There are two approaches to addressing the variation in participant opinions in our wayfinding system design. We can make certain design decisions in reaction to usability problems
encountered across many participants, or based off of choices that we ask each user to
make. Our study results suggest that avoiding compound directions, when completing the
first part of the direction takes a relatively long time, is a good design choice. The results
also suggest that we should ask each user for their preferred default correction strategy,
whether the system should tend toward correcting route deviations or avoid doing so when
Other choices in directions are less straightforward because they are often based on the
particular wayfinding situation. Some participants were less vocal or descriptive in what
directions they found easy or difficult to follow, and therefore it was only possible to uncover
their priorities and capabilities through the trial route of the study. Others gave examples of
situations where they would want different direction types, such as under different lighting
Incident Rate by Direction
I 8
Incident rate
Figure 6.6: Histogram showing the frequency of incident rates (errors, confusion, and help)
at each of the 50 directions presented in the study route.
conditions or depending on how recent a landmark photo had been taken. Again, the study's
trial route served to elicit these opinions.
6.2.2 Direction difficulty scope
Figure 6.6 is a histogram of the frequency of different incident rates at each of the 50 directions, showing the disparity across different directions. A large proportion of the directions
caused little difficulty for participants, with 14 directions presenting no difficulty to anyone.
Unsurprisingly, 10 of those directions instructed participants to go straight ahead, such as
the one in Figure 6.3a, or turn around, such as the one in Figure 6.3d.
The other 4 involved basic (not "lst/2nd" or "slight") turns. Three were variations
Figure 6.7: ]
The most problematic direction, which incurred a wayfinding incident for 8 out of the 10
study participants. The associated text and audio message was "Go toward this building
and then turn right."
on the direction of the form shown in Figure 6.3b. The fourth was the direction shown
in Figure 6.3h. Participants considered that direction to be clear because it mentioned
specifically crossing at the next crosswalk, and some were familiar with the Rainier Vista
lawn it references.
Another 11 directions were difficult for only a single participant, while 13 more were
difficult for two, again showing how there were many cases of individual differences in experience. In contrast, only 5 directions out of 50 were considered difficult by a majority
of participants, though never unanimously so, as every direction had at least two participants who were able to follow them without incident. The most difficult direction was one
instructing participants to go toward Gerberding Hall in Red Square, then turn right (see
Figure 6.7). Reasons that participants had for finding it difficult included being unsure how
far to go toward the building before turning, having trouble seeing the building or arrow
Figure 6.8: Another problematic direction, which incurred a wayfinding incident for 7 out
of the 10 study participants. The associated text and audio message was "Take the first left
you come to."
clearly on the screen, and being confused by the orientation of the arrow, which points at
a sharper than 90° angle toward the next waypoint, the Campanile stacks in Red Square.
Only slightly less difficult, causing a wayfinding incident for 7 participants, was a direction to take the first left at an intersection with two outgoing paths on that side (see Figure
6.8). One reason that participants had difficulty with the direction was its proximity to
their previous turn, meaning they had little time to react. Another was some participants'
not understanding the double-arrow icon.
Modeling direction difficulty
Given the results from the study, our next step is learning user models that predict direction
difficulty - i.e., how likely a direction will cause an incident. As we can obviously expect
users to want wayfinding assistance at locations where they have never visited before, we
need a model that can predict direction difficulty at new locations. One approach is to
extrapolate from previous wayfinding behavior observed from similar directions. While the
observed outcome of a single direction is a binary value (incident or nonincident), a difficulty
prediction is not, because our definition of difficulty addresses the question "How often will
this type of direction lead to a wayfinding incident?" , rather than the alternative question
"Will this direction lead to a wayfinding incident?" Our assumption is that minimizing
potential incidents along a path results in routes more suitable for potential users with
cognitive impairment.
6. 3. 1 Direction features
Besides treating the observed outcome of a direction as a binary value, we describe each
direction using a set of features informed by our qualitative observations and also prior work,
which isolated factors that made wayfinding experiences differ across individuals. Since we
are limited to the directions and locations used in the trial route, the models we build should
not be considered to be generalized or suitable for use in all possible locations. Instead,
the set serves as the starting point of our effort, and we use it to investigate the feasibility
of modeling wayfinding on an individual user basis. We expect that further studies would
uncover other factors that impact wayfinding for individuals. Table 6.4 summarizes the
features used to describe each direction. Note that a direction's feature values are both a
function of its inherent wording and also of the location at which the system would present
Turn type (nominal): Obviously the type of turn can impact the difficulty of a direction.
We observed very few instances of trouble when participants were supposed to go
forward or make a U-turn. We also observed errors and confusion when they were at
complex intersections, where more than one path goes off on the same side (left or
right). We combine symmetric directions since we did not observe any participants
differing in their wayfinding behavior between left and right directions and because
there were too few occurrences of some turn types.
Image type (nominal): As participants often differed in both their preference for turnbased or landmark-based directions and their ability to follow the two separate styles,
we consider each to be a separate feature value.
Landmark (nominal): Participants reacted differently to the various landmark types in
Table 6.4: Features used in linear regression models. All features were Boolean, with binary
values (1 if present/true, 0 if not present/false) except the numeric features, which were
non-negative integers. Note that other possible features (for example, landmark=fountain
as seen in Figure 6.3) were omitted because there were too few instances in the total data
Go straight ahead
Make a left or right turn
Make a slight left or slight right turn
Take the first left or first right turn
Take the second left or second right turn
image type
Contains iconic imagery
Contains a photo of the environment
Uses a building as a landmark
Does not use any landmarks
Directs user toward a location
Directs user to travel in front of a location
Does not refer to landmarks at any alignment to path
Turn around
As user receives direction, approaching immediate turn
or choice point
Turn or choice point several seconds travel away
Turn or choice point far away
Direction to maintain trajectory from previous direction,
Contains more than a single instruction
Number of paths on same side as a turn
Number of paths on the opposite side of a turn
our studies. The value of this feature represents whether a direction contains a land-
mark that is a building, another type of landmark, or no landmark at all. Participants
varied on how easily they were able to identify buildings in photos, with some finding
them distinctive while others felt that they lacked detail. There were not enough occurrences of other types of landmarks in the directions, such as sculptures and other
structures, so we do not consider them separately.
Alignment (nominal): Directions that used landmarks used them in one of two ways either indicating that the user should travel toward one, or that they should make a
turn in front of one. The landmark analysis study results in Chapter 4 suggest that
alignment can have a significant impact on wayfinding difficulty, and other alignments
are more incident-prone.
Distance (nominal): We noticed that the distance to an intersection at which point a
user would need to make a wayfinding decision played a part in some incidents. When
participants received a new direction, they tended to continue along their previous
trajectory while interpreting it. Sometimes participants could not process the direction
by the time they reached the next intersection. Compound directions, which are meant
to give participants an indication of such quick successive turns, did not fully solve
this problem. Some participants were not as likely to remember the second part of a
compound direction when the intersection was relatively far away from their position
at the time they received it. Due to GPS accuracy limitations, we do not use numeric
values to represent distance. Instead we treat distance as a nominal feature with
three possible values based on the length of the path segment where the participant
is traveling, from their last intersection to the upcoming one.
Continue (binary): We treat continuing along a path or toward a landmark as a special
case of wayfinding, since it relies on the previous direction and trajectory. Study feedback supports our intuition that presenting such directions makes wayfinding easier
for users because they contain less new information to process.
Compound (binary): Compound directions are longer and can be more involved as users
must track their own progress and determine when to transition from the initial step
to the following step. However, compound directions can also be useful for preparing
users for quick turns in succession, or to provide more information about the upcoming
route to users who want to know what to expect ahead. The value of this feature should
therefore affect wayfinding difficulty and may possibly differ across individuals.
Choices in the same and opposing sides (both numeric): Both intuitively and suggested by our studies, users are less prone to wayfinding incidents when there are
fewer choices at an intersection. We consider two ways to represent the complexity of
direction at such an intersection: the number of outgoing paths to the same side of
the direction, and the number of outgoing paths to the opposing side of the direction.
6.3.2 Model training
We choose to use linear regression to create our initial user models because it is a well-
studied and extensively used algorithm that produces a function that outputs rational (not
binary) prediction values. Since we do not expect users to always follow one direction
correctly or incorrectly, the range of values output by such a model are more useful in
indicating the degree of difficulty than a binary prediction. Linear regression is also not
very computationally intensive, so could be feasibly performed on a mobile device. Finally,
the learned model's feature weights can be interpreted in a straightforward manner, as a
larger associated weight indicates that a feature has more impact on difficulty.
Linear regression has some limitations that could affect a model's predictive power.
A key requirement is that the true difficulty model should be linearly related to its feature values. While this assumption does not hold for our chosen features, given the small
observational data set size, it would be too early to determine exact relationships between
features. We view the regression models we create as prototypes that would warrant further
We use the data mining tool WEKA [28] to learn difficulty models for each participant.
First we preprocess the data into a format operable by WEKA, then we split the data into
training and test sets, and finally we run the linear regression algorithm to produce models.
1 . We first preprocess each observation. We remove the direction number and participant
values, and prune feature values with too few occurrences in the route. We then apply
an unsupervised nominalToBinary transformation to the remaining feature values.
The conversion is necessary as the linear regression algorithm accepts only numeric
and binary inputs.
2. Next, we split each participant's data into a training and test set. We randomly split
directions that have similar feature values into the two sets. Specifically, we split
directions grouped as followed:
• lst/2nd turn, icon
• 2nd turn, photo
• straight, photo, continue
• straight, photo, not continue
• not straight, photo, compound
• not straight, photo, not compound
• straight, icon, compound
• not straight, icon, compound
• not straight, icon, not compound, not continue
3. We then perform linear regression to the training set(s), generating several different
models for each participant:
• only p: A model trained with only the training data of that participant
• train: A model trained with the training data from all other participants
• train + Ix p: A model trained with both of the above two training sets
• train + 2,4,8,10,2Ox ?: Models using all other participants' observations, plus
that participant's own training data added a certain number of times in order to
weigh it more heavily for the linear regression algorithm
Our rationale behind training models using different combinations of participant obser-
vations was to study the benefit (if any) of combining observations from an individual and
those of other users. Our research has found a large amount of evidence indicating that
individuals with cognitive impairments may vary quite a bit in both their preferred wayfinding method and also in their abilities to follow different wayfinding direction types. At the
same time, it is unclear how many observations from an individual user is needed in order
to predict direction difficulty, especially given the large number of possible directions and
locations. Using other users as part of the training data set to "bootstrap" the model is a
technique that Intille et al. have used and recommend in cases where there is an insufficient
amount of individual data [34]. Their case studies suggest that combining training data
can produce better models when there are commonalities across potential users. In our
study experience, the degree to which individuals differ in their tendency to find directions
difficult varies, with some directions considered similarly difficult across individuals, while
others are considered much differently. Therefore, we alter the weight of the individual's
training data to study whether we can fine-tune a model to reflect both commonalities and
We performed linear regression using WEKA's default parameters except the attribute
selection method. We turned off attribute selection due to the default M5 method failing
to output a usable only ? model. See Table 6.5 for example models output by WEKA.
We chose to illustrate the difference in Participant 9's train and only ? models to show
how unique an individual can be in his or her wayfinding tendencies. While all incident
values were binary (1: incident, 0: nonincident), the range of the resulting models' outputs
sometimes were not between [0, 1] because the training sets would not always include all
representative combinations of feature values. Therefore, the model outputs should not be
interpreted as exact likelihoods, but relative likelihoods.
Figure 6.9 contains a histogram of the occurrences of both incidents and nonincidents
Table 6.5: Two difficulty models for Participant 9, which use two disjoint training sets,
train and only p. Larger positive weights denote features that caused the direction to lead
to more incidents, such as when directions required walking a significant distance before
making a turn (distance = far). Smaller negative weights denote features that were more
easily followed, such as when the direction was to go straight ahead (turn = straight). The
two models differ greatly on many of the feature weights, with the only ? model being more
predictive of the participant's incident rate, implying that Participant 9 was unique in his
wayfinding tendencies.
Participant 9 train model
Participant 9 only ? model
difficulty =
difficulty =
* turn=left_right +
-0.0475 * turn=turnaround +
-0.0217 * turn=straight +
0.0585 * turn=sleft_sright
0.1657 * tum=lright_lleft
-0.2333 * turn=2right_21eft
-0.0851 * type=icon +
0.0851 * type=photo +
* poi=none +
poi=building +
alignment=none +
alignment=to_toward +
alignment=front +
-0.0244 * continue +
0 . 1597
* distance=near +
* distance=far +
* distance=mid +
* compound +
* choices +
* oppchoices +
0.0043 * turn=left_right +
-0.0663 * turn=turnaround +
-0.1683 * turn=straight +
0.3119 * turn=sleft_sright
0.2399 * turn=lright_lleft
* turn=2right_21eft +
0.0736 * type=icon +
-0.0736 * type=photo +
0.0448 * poi=none +
0.394 * poi=building
* alignment=none +
* alignment=to_toward +
* alignment=front +
* continue +
* distance=near +
* distance=far +
* distance=mid +
* compound +
* choices +
* oppchoices +
Observed Participant Result
model trained w/out each participant
c 5°
H fffffffl
60% 1
g 40
50% g
r 40% ^
* 30
r 30%
il IbI p I
^ 20%
1 10%
predicted difficulty
e=) incidents
¦¦¦ nonincidenfs
incident rate
Figure 6.9: Graph showing occurrences of incidents and nonincidents grouped by the predicted difficulty rating given by the train models, which were generated for each participant
and trained on all other 9 participants' observations. The percentage of incidents in each
bin is plotted on the secondary Y axis, showing that directions with relatively high pre-
dicted difficulty (difficult for the other 9 participants) were often not difficult for the tested
over all participants using the train model, our baseline performance model. Overlaid on
the secondary Y axis is the percentage of occurrences in each bin that were incidents.
6.3.3 Model testing
Given a set of directions, we want a model that outputs difficulty predictions such that
the incident rate among directions increases as the predicted difficulty increases. Under
the same simplifying assumption that the relationship between the features and difficulty
are linear, we therefore calculate a correlation coefficient between predicted difficulty and
remainder incident rate, where higher correlations suggest better predictions.
Given the sparseness of the test data sets, we bin the directions into equal-sized ranges
and calculate the expected incident rate within each bin. We calculate the correlation
between these expected incident rates and the actual incident rates that occurred within
the bins. The correlation calculation depends on a reasonable bin size, because a bin that is
too small will be overly sensitive to the presence or absence of even a single incident, while
a bin too large will create too few bins to be meaningful. We empirically found that setting
the bin size to 0.15 balances between grouping too many directions and too few. Figure
6.10 plots the correlation coefficient values of the eight models for all the participants except
Participant 3. Participant 3's test set contained no incidents, so the correlation coefficient
is undefined. Figure 6.11 plots the spread of performance by model.
One caveat to using fixed size bins is that directions types are not evenly distributed
in our test set. On the other hand, participants found a large number of directions not
difficult, so we already expect that directions will be skewed toward the low end of the
difficulty scale. Most of these directions are not interesting to model because they involve
going straight ahead, so the fact that more of them will be binned together and treated as
a single point in the calculation of correlation is not a major concern.
We tabulate all correlation coefficients broken down by model and participant, and
the mean performance and standard deviation by model, in Table 6.6. The only ? model
shows the worst prediction performance, and also varies the most across participants. The
train model shows much better average performance, but also with large variations across
participants. Not surprisingly, for many participants where one model predicts poorly, the
other does better, because the only ? model can easily overfit to its small training set, while
the train model can ignore specific individual tendencies. Combining the (disjoint) training
data used to generate each model results in the set used to produce the train + Ix ? model,
which is much improved over the aforementioned models, though it still performs poorly
for a few participants. We see that adding multiple instances of participant training data
improves the performance and decreases the deviation, however at train + 8x ? we see the
deviation begin to increase again.
Correlation of predicted and actual incident rate
by user model
? only ?
strain+ 1xp -a-train + 2xp ¡> train + 4xp
< train + 8xp ¦+?•train + 10x ? * train + 2Ox ?
Figure 6.10: Graph showing the correlation coefficient of predicted difficulty values to actual
binned incident rate across all users for all trained models. Higher correlation values suggest
better predictions.
Correlation coefficient by model
predicted vs actual incident rate
only ?
train + train + train + train + train + train +
1Ox ? 2Ox ?
Figure 6.11: Modified box and whisker plot showing the correlation coefficient of predicted
difficulty values to actual incident rate across each trained model. Box shows 1st quartile,
median, and 3rd quartile values of the correlation coefficient over all participants except
3. The error bars show minimum and maximum values, while the plotted lines show the
mean values and ±1 standard deviation. From the graph we see that the only ? and train
models exhibit a large deviation in performance, while the combination models show signs of
improvement until overfitting causes performance to be less reliable again by train + 1Ox p.
Table 6.6: The correlation coefficient between the values of predicted difficulty and the
binned incident rate by model and participant, as well as the mean and standard deviation
of the value for each model.
only ?
+ Ix ?
+ 2xp
+ 4x ?
+ 8xp
+ 1Ox ?
-I- 20? ?
Looking specifically at the difference between the train and train + !¡.? ? models, Figure
6.12 graphs the cumulative occurrence of incidents and nonincidents on the primary Y
axis as the predicted difficulty. The percentage of incidents of all occurrences < predicted
difficulty is plotted in the same graph on the secondary Y axis. Note how the incident rate
is more unstable in the train model, and that there are spikes when low predicted difficulty
values have more incidents occurring, suggesting that some individual participants had more
difficulty on directions that weren't as troublesome to other participants.
Our results suggest that while the train + 4? P model is not definitively the best performing model for all participants, it may be the most reliable. The train and train + Ix ?
models take little advantage of observations from individual participant tendencies, there-
fore doing well most of the time but poorly in several occasions. In contrast, the only ? and
train + 8/10/2Ox ? models possibly over-fit the individual participant's training data.
Discussion and next steps
It is important to reiterate that this chapter describes only an initial attempt at modeling
user wayfinding behavior in the context of the observations that we collected in the user
Oooifrenœof incidentsami iMHimadents
model trained wfthout participant
^t* ??
Occurrencesof incidentsand noninddents
model trained with participant 4x
Jfß * *¦**
UK 1 §
? noninddents
(a) w/out participant
re 'w
Predicted difficulty
(b) w 4x participant
Figure 6.12: Graphs of the cumulative number of incidents and nonincidents as predicted
difficulty of (a) the train model (trained on all participants' observations except the one
tested) and (b) the train + 4^P model (trained on all participants' observations except
the tested participant's test set, with the tested participant's training set included in the
training set 4 times) on the primary Y axis. The percentage of occurrences < the predicted
difficulty that were considered incidents are graphed on the secondary Y axis.
study. We are more interested in determining whether it is even possible to model direction
difficulty on an individual basis, given the evidence that differences exist across individuals in
our target user population. Our study provided some starting features to use in describing
directions, and linear regression showed that it is possible to combine training data in
differing quantities to create models to reflect general direction difficulty and some individual
One problem we face when creating and evaluating individual difficulty models is the
scarcity of data from each user. We split training and test sets equally because too few test
data would make calculated incident rates too coarse-grained. We did not perform a "leaveone-out" style fitting analysis because our goal is to produce a model usable to generate
directions in a follow-up study. Such a step will be useful to determine how well our chosen
features actually predict direction difficulty, however given that all the observations are
highly dependent on the study route, such features would only give an incomplete view on
true difficulty and could be premature.
The next chapter discusses the incorporation of the train + 4% P models for actual use
in a follow-up user study. We use the individual models to generate personalized directions
guiding five returning participants on campus. We also present participants with a set of
participant-specific static directions. The set is comprised of directions that differ the most
in predicted difficulty between the participant's train + 4% P model and the non-customized
train model, and we include them to observe whether there is a noticeable difference in
wayfinding experience.
Chapter 7
WAYFINDING with adapted models
This chapter describes the current wayfinding system implementation and a follow-up
user study conducted with participants from the previous study, who returned to interact
with directions generated by adapted models.
Incorporation of Adapted Models
The Wizard GUI is the application that runs the bulk of the wayfinding system in its current
form. We use the Wizard GUI for two primary functions, (1) to create the path graph that
defines the area for a route in preparation for a user study, and (2) to send directions to
participants during the study.
Route map and path graph
We use a bitmap to represent the backing map appearance. A properties file defines map
metadata, such as the latitudinal and longitudinal coordinates of the bitmap corners. We
use the Wizard GUI to add intersections as graph nodes over the backing map, and connect
the intersections with edges to form paths. We manually add a name describing each path
edge, for example "sidewalk," "crosswalk," or "stairs." The system uses this metadata in
the written and verbal directions. For participants who use powered chairs, the system also
uses this metadata as accessibility information to avoid generating inaccessible routes.
The properties file defines a route as a starting position and a goal location. User position
is a combination of location (graph node) and orientation. As a simplification, we represent
the orientations possible at a given location to be either facing toward (at node X, facing
node Y) or away (at node X, with back to node Z) from another graph node, rather than
by angle. A small threshold prevents the system from considering two similar orientations
separately, where one is defined as facing a node, while the other is defined as away from
another node.
7.1.2 Direction options
Once we define the graph and generate all possible user positions, we determine the set of
possible directions that a system can give. In the MDP, these directions are options and are
first discussed in Chapter 5.1.2. For users with mobility impairments, we exclude options
that require traveling along inaccessible path segments.
One change to the options in that chapter is the expanded use of landmarks to support
not only going toward them, but also turning in relation to them. Landmarks can accompany
a valid turn direction under several conditions. We consider a landmark to accompany a
slight turn direction if it is between 20° and 60° to the same side as a non-landmark slight
turn segment1. We consider a landmark turn direction if the landmark itself is within the
"quadrant" formed by the angle between the desired path segment and 0° (straight ahead)
as in Figure 7.1b.
Once we determine all the possible landmarks that can be used in directions, we perform
a batch request to the landmark photo database, generating all possible augmented photos.
We download the augmented photos onto the N95 for the user studies. In a real deployment,
photos could be grouped by region, downloaded from a service provider on demand, and
cached for quick access.
User state in our MDP consists of these variables:
Position: Location (graph node) and orientation (facing toward or away from another
graph node)
1A limitation of our current landmark criteria is that angles are calculated between a graph position and
the centroid of the landmark. Therefore, a building with a large façade may not be considered for use with
a turn, since the calculated angles could exceed 60°. One alternative would be to calculate the angle from
the point along the façade closest to a graph location, rather than the centroid. Before doing so, it should
be important to study whether potential users can identify a large landmark from a photo potentially
cropping the façade due to the proximity of where the photo was taken.
First point along
shortest path
to landmark
Current location
(a) Landmark for straight option
First point along
shortest path
to landmark
Current location
(b) Landmark for turn option
Figure 7.1: Landmark selection criteria. The area where landmarks are eligible for use in a
landmark-based direction is shaded in gray.
Last option: Knowing the last system option, the system can determine whether the next
option will be a continuation rather than an all-new direction. The system also uses
the last option for short-term adaptation, which Section 7.1.7 will describe.
Last landmark: The landmark (if any) used along with the last system option
Progress: Progress is a Boolean that indicates whether the user requires help or is confused,
and can be set by the user pressing the Help button on the phone client or by the
navigation wizard using the Wizard GUI.
Correct: The system tracks and determines the correctness of following the last option
outside the MDP, so the state only needs to include a Boolean indicating whether the
user followed it correctly, rather than store the last position, which would necessitate
creating states for all possible last positions.
Reward: States also have a reward, which is the immediate numerical value that is a
function of the values of the previous variables. For example, the reward at a state
where the location is the goal would be a high positive value, while the reward at a
non-goal state would be some small negative value to encourage shorter routes.
Blacklist: Each state contains a blacklist, which is a paired list that matches each option
to a counter value. Section 7.1.7 describes how the system uses the blacklist for short
term adaptation.
Value: Every state has an associated value, which is an approximation of the the maximum
expected value of the reward plus future state values. The MDP implementation
calculates the values using value iteration.
Cost model
The system associates each option with a cost, defined by the cost model, that is subtracted
from the accumulated reward. The cost is a value that reflects user preference and effort.
For example, for a user who prefers to continue along a path rather than be interrupted
and corrected, the cost of the Turn Around option would be greater. A user who prefers
to wayfind visually with landmarks would have a cost model where landmark directions
are less costly than a model for someone who dislikes them or expresses no preference. To
reflect physical effort, we factor the distance between the current user location and the new
location into the cost of the option. To reflect cognitive effort, we add the difficulty value,
as output by the linear regression model, to the base cost. Since the regression model can
output values less than 0 or greater than 1 when its training set does not contain all possible
combinations of feature values, we limit the range of difficulty values to [0.1, 0.9].
Transition probability model
We consider each option taken from a given state to have a single correct next state. The
system calculates the probability of correctly transitioning based on the learned model of
direction difficulty, again limited to the range [0.1,0.9]. Algorithm 1 shows the steps the
system takes to calculate the transition probabilities of all reachable states given a state
and option pair.
Algorithm 1 Algorithm for calculating transition probabilities to all reachable states.
totalP *- 0
d <— calculated difficulty
if d > 0.9 then
d^ 0.9
else if d < 0.1 then
end if
for all nextState do
if nextState. corred then
nextState.transitionProb <— 1 — (0.2 · d))
nextState.transitionProb <— 0.2 · d
end if
totalP <— totalP + nextState.transitionProb
end for
for all nextState do
nextState.transitionProb <— nextState.transitionProb/totalP
end for
We empirically use a minimum robability value for a correct transition to reflect the
scarcity of actual wayfinding mistakes, due to difficulty including confusion and help requests. As a simplification, we assign the same probability value to every incorrect state.
The final step is to normalize all the values to get the actual probabilities for any state
7.1.6 Discount factor
We used a discount factor of 0.95 to make the algorithm slightly greedy in picking options
that result in higher expected value in the near-term. This preference for near-term reward
is helpful when there is more uncertainty in the results of future options.
7.1.7 Adaptation
The prototype does not yet support on-line, long-term adaptation of the user model through
reinforcement learning from new wayfinding observations. Off-line adaptation can be done
by re-running the linear regression classifier on new training data.
The prototype does support short-term adaptation behavior. By including a blacklist
variable in every state, the system can prevent the use of unsuccessful options for some
number of subsequent steps. Each state transition decrements the counter associated with
an option in the blacklist. In this chapter's study, we set the behavior of the system to
blacklist an unsuccessful option for only a single step. This simplication limits the size of
the state space and allows us to precompute all states, so during the study, the system can
respond immediately to a state update by executing the next option according to the MDP
User study
We were able to schedule five participants from the previous user study described in Chapter
6 to return for two route conditions. The first purpose of the study was to observe their
reaction to the models adapted based on their previous study observations. The second was
to observe their reaction to directions that were either considered easier or more difficult for
Table 7.1: The number of directions presented to each participant in each condition.
Nonincidents Incidents
Nonincidents Incidents
them than other participants, according to differences between their adapted models and
the non-adapted models. Therefore, we had two route conditions for each participant:
1. Model: Route using the participant's train + 4X P model from a starting position to
the north of Kane Hall on the University of Washington campus and ending at the
James J. Hill bust in front of More Hall (see Figure 7.2).
2. Static: Route consisting of directions where the participant's train + 4? P model
differed the most with the train in predicted difficulty. We constructed a static route
for each returning participant.
Participants 2, 3, 5, 7, and 9 returned (see Table 6.2 for their demographics). Each
participant first used the wayfinding system in the first condition, then followed the static
directions in the second condition.
We again recorded wayfinding errors, confusion, and help requests in wayfinding, as well as
participant comments during the conditions and in the debriefing session. Table 7.1 shows
the incidents among the total directions for illustration purposes. We focus on the more
qualitative reaction to the wayfinding directions since the routes were short and not suitable
for quantitative evaluation.
File Edit View Actions Mode Options
'{'/¦,.,¦/Mi/ ?
! ,¿WH&i
Current state: O
¦At: Sl From: S3 47.6505246-:
Current opt: STOP
Progress: true
Value: -1.9800021375471SoIi
Best nest opt: POI
,.Best n*xt POI: STRAIGHT Susi
¦Soto Map Point id 238 47.65*:
100 STOP
100 POI
it: 61 From: (53 4"-.656¿*2463318452, -122.3*898i>~
Figure 7.2: Route used with the train + 4x ? model for each returning participant.
7.3. ? Experiences with adapted model condition
Participants were able to follow 95 of the 107 total directions generated by their train + 4x ?
model. No participant experienced major difficulty with the directions that prevented them
from reaching the destination. There was only one incident that required a significant
rerouting by the wayfinding system, when Participant 3 misinterpreted a direction to keep
going straight toward Gowen Hall to mean walking directly to its wall. The system successfully rerouted him through the UW Quad because the short-term blacklist adaptation
feature caused the policy to calculate that the new route would be more likely to succeed
than an attempt to reorient him and try the original path again.
7.3.2 Experiences with static condition
Participants expressed mixed reaction to the second, static route that consisted of directions
thought to be more or less difficult for them to follow compared to other users. On one
extreme, Participant 2 had much more trouble following the static directions because they
tended to contain landmark photos.
P2: The first set [of directions] was just fine. I wasn't expecting the only option
[in the second set] to be pictures. If it 's not set to an address, it 's hard to signify
which building because the pictures are not absolutely matching. The picture was
not registering in my brain.
Participant 9 did not have much difficulty with the directions in the second route, however he noted that the wayfinding experience felt different to him. While Participant 9 held
a more negative opinion of the directions given in the second route, he did mention the
value of the system presenting landmark photos to him if it did so only occasionally.
P9: I felt like a tourist because I'm looking around for the buildings instead of
the directions. I felt on the first set, I was fine, because it gave me the warning,
I could go ahead and look around and gawk and, you know. Because I knew that
the device was going to vibrate and make a noise so a new direction was coming
up, so I didn't have to walk with it right in my face, so I felt more at ease with
the first set. Because the second made me stop and made me look around and
learn where I was. It may be nice to throw a couple in, but if I had my druthers,
I would have took to the first set.
Participant 5 served as a contrasting example. He had noted his preference for visually
wayfinding using landmarks in the past, but stated that while he still preferred them, their
absence in the second route experience was not problematic.
Due to the short length of this follow-up study, we cannot conclude that the customized
and adapted models provide a superior wayfinding experience to our participants. Had all
participants been able to return for this study, we could compare the correlation performance
of the adapted and non-adapted models, calculated between the actual incident rate to the
expected incident rate.
However, our participants' wayfinding successes in the first condition suggest that the
personalization of user models fairly matches their expectations for route guidance. The
issues they ran into show that there is room for improvement, for instance in supporting the
inclusion of occasional landmarks even when a user usually prefers turn-based directions.
In practice, the effect of the cost and reward values can vary depending upon the graph
structure and other MDP settings, such as the discount factor. We chose the values used
in the MDP for this chapter's study through empirical testing. Further trials in different
contexts would serve to provide more design feedback to determine how to set the values
Chapter 8
This dissertation has presented my work in the design of a wayfinding system for individuals with cognitive impairment. I used an iterative process that involved cycles of design,
prototyping, and evaluation, resulting in these main contributions:
Understanding the extent of user abilities and preferences: By using a Wizard of
Oz study setup, I studied how potential users with cognitive impairments wayfind in
indoor and outdoor environments without needing to fully implement location sensing
systems or navigation logic. The feasibility studies described in Chapters 3 and 4
showed that people with cognitive impairments can follow multi-modal, just-in-time
wayfinding directions, though not without usability issues due to the wide variation
across individuals, both in terms of their ability to follow different direction types,
and their expectations of system behavior.
Criteria for incorporating landmark-based directions: While initial studies showed
that participants with cognitive impairments could follow wayfinding directions, the
turn-based nature of the directions caused many to focus on the mobile device and
lose awareness of their surroundings. As this cognitive tunneling effect leads to safety
concerns, I investigated whether users could interpret landmark-based directions. The
second study in Chapter 4 informed the implementation of a landmark selection system, which uses a collection of geo-tagged photos [33] to incorporate augmented landmark photos into the wayfinding experience.
Framework for customizing and adapting directions: Chapter 5 described the framework used to account for the differences between individuals. By representing the
problem of choosing the wayfinding directions to maximize expected wayfinding success as a Markov Decision Process, customized cost and transition probability models
can be created to reflect individual abilities and preferences. The results of the study
described in Chapter 6 showed that combining an individual's observations with other
participants in different proportions can be a viable solution when data from individuals by themselves are insufficient to learn models of direction difficulty. Finally,
the study in Chapter 7 found that such adapted models can improve wayfinding experience, though more work is needed to refine the features that describe possible
directions and wayfinding situations in the MDP.
Future Work
The work presented in this dissertation has demonstrated the potential for technology to
help people with cognitive impairments overcome barriers to wayfinding independently. It
describes important steps towards a usable wayfinding solution, but there are many more
needed before such a system can be realized as an off-the-shelf product available to all. The
following are some areas in which this work could be extended:
Model improvements
Many factors can affect wayfinding beyond the direction type or physical layout of an intersection. The study experiences suggest numerous features that could be incorporated
by the user model. Collecting more extensive wayfinding observations to determine feature
relevance is an important next step. One likely set of features are those to describe user
health condition, such as mobility impairment, reading abilities, and vision. Knowing relevant wayfinding commonalities found in people with similar health diagnoses could shorten
the initial customization step required for a new user of the system. This form of collabo-
rative filtering could be implemented using mixed effects modeling, which can capture the
different degree of impact that other user observations have on the individual model to a
finer degree.
Environmental conditions may also play a significant role in direction difficulty. Lighting
can affect image visibility, but can also change rapidly. The system should adapt to such
changes in the short-term. Other changes have more extended effects on wayfinding. Time
of day will likely affect landmark usability through changes to shadows and visibility range.
Seasonal differences can affect foliage, which some users rely on to match photos in context.
More observations collected through further studies, under a wide variety of conditions, will
provide insight into what potential features should be added to cover such changes.
The addition of new features increases the search space of the modeling problem because
each new feature adds another potential dimension, further increasing the amount of data
needed to fit models. As new features are considered for inclusion in the user model,
the feature set should be kept small through feature selection. Features that consistently
have low weights in the models or those are highly correlated with other features should
be removed. Finding the appropriate feature set size will depend on making a trade-off
between predictive capability and computational efficiency.
In addition to identifying more relevant features, examining the benefit of other machine
learning algorithms would be worthwhile. Linear regression is useful because it outputs
weights that are understandable, but it has its drawbacks, such as its output range not
contained within the range [0, 1]. Logistic regression has similar advantages but can also be
directly used to create models of risk. I have begun preliminary experiments fitting logistic
regression models to the study observations gathered in Chapter 6, but more data is needed
with higher incident rates to produce good models. A data gathering study that focuses on
more difficult directions can provide a better understanding of which features tend to have
similar weights and those that differ more across. different users.
Tuning adaptation
There are several factors to consider when determining the appropriate speed at which
the system should adapt to new observations. While newer observations should generally
be more useful than old observations, the system does not have complete knowledge of
the reasons behind success and failure, so care needs to be taken when deciding how to
react. Ideally, the system needs to distinguish between a mistake - an intentional error,
from a slip - an unintentional error, especially if it only has partial knowledge of the
environment. This is because conditions in the environment could have caused a user to
make a mistake by making an option undesirable or impossible. For example, a path blocked
due to construction might prevent the user from proceeding. Short-term adaptation in the
form of temporary blacklisting of options is one way to deal with the situation. Other
methods that can be investigated include querying the user to indicate whether there is a
problem following the current option, and using additional sensing to detect confusion.
Designing the dialog queries involves considering when they should appear, their language, and how users would interact with the device to input an answer. One condition
could be inactivity, when a user does not move for an extended period of time, suggesting
hesitation. However, some users may dislike being prompted repeatedly, so individual differences should be taken into account when setting a timed trigger. Some users may not
be able to answer questions about their meta-cognitive state, so alternative system actions
should be explored as well.
Investigating whether confusion or hesitation can be detected and distinguished from
normal wayfinding movement could be done by collecting labeled data using a device such
as the Multi Sensor Board (MSB) unit, which contains onboard GPS, 3-axis accelerometer,
gyroscope, magnetometer, and various other sensors [7]. Coupled with observations from
a study involving more difficult wayfinding directions, we can build classifiers to recognize
lack of motion, pacing motions, or other physical indications that a user is in a state that
deviates from their normal response to directions. Bio-medical sensors, such as galvanic
skin response and heart-rate sensors, could be another interesting type of measurement to
use for confusion detection.
Another approach to classifying behavior deviation would be to instrument the environment or utilize sensor readings from other users. For example, traffic sensors could indicate
whether traffic is crossing the user's path, while aggregated location logs from many users
could could indicate pedestrian detours.
8.1.3 Landmark improvements
Aside from model enhancements, producing better augmented images can also improve
system usability. There are multiple methods for dealing with discrepancies between a
user's location and where a photo was originally taken, an issue that caused some confusion
(a) A desired path seg- (b) A photo with overlaid (c) "Zoom-out" view that (d) View rendered using
ment shown on a satellite arrow showing the desired warps original photo to a lightweight 3-D model
align to user's view
made from a small set of
Figure 8.1: Examples of different techniques to visually show a path involving landmarks.
for study participants. Figure 8.1 shows alternative images synthesized by the landmarkselection system. Preliminary evaluation of these images with participants without cognitive
impairments suggest the potential for improved understandability of the directions, especially Figure 8.1d, where a high quality 3-D perspective is composed from multiple photos
in a light-weight manner [32]. Further studies are needed to gauge the usefulness of these
images with people with cognitive impairments. The simplified 3-D view might be useful
for users with autism by reducing visual distractors that could overwhelm them, but might
be less useful for those who primarily rely on context to identify the view.
Other enhancements to landmark usage include the incorporation of animation to clearly
mark the path or emphasize distinctive features of a landmark. Scaling support of the
latter would involve research into how to find those features, either automatically using
computational photography techniques, or manually using crowd-sourcing systems where
many users can contribute labels to image regions.
Finally, algorithms to identify the view of a photo taken by the user while in the process
of wayfinding can be used to support augmenting the user's view directly. Currently, such
capability requires a high density of photos, so an interesting approach could be to integrate
photo collections taken from many sources. Photos tagged by users provide the benefit of
indicating landmark relevance, while services such as Google Street View provide for greater
photo coverage.
8.1.4 Expansion of studied environments
The work in this dissertation can be extended to cover other locations where people need
wayfinding assistance. More grid-like environments, such as those downtown, pose different
issues with landmark selection. In environments with many tall buildings, users may focus
only on ground-level appearances, which might not have as many distinguishing features as
buildings found on the University of Washington campus.
Statistics on the relative frequency of different intersection types in urban, suburban,
and rural environments would serve to indicate the degree of wayfinding difficulty typical in
these environments and would guide the composition of representative directions to try in
user studies. Wayfinding data from people familiar with locations, such as taxi cab drivers
[96], could indicate high-quality routes, while wayfinding data of a system user's actual
travel patterns could indicate preferred routes, and both could be used to inform the usage
of subroutes or waypoints [67]. These might not only increase wayfinding success, but also
simplify the MDP formulation by breaking down a larger state space into smaller state
spaces that can be solved more efficiently.
8.1.5 Cognitive aid integration
Wayfinding occurs within the greater scope of one's daily activities. A natural extension
to this work would be to integrate the system into a personal planner so that destinations
need not always be entered explicitly by the user. The planner could receive destinations
from caregivers or determine them based on other scheduling constraints [58] . Destinations
could also be automatically inferred, as they were in the Opportunity Knocks research [68] .
The system could learn places during a training phase, augmenting the current practice of
a job coach training an individual in traveling between places such as home and work.
Beyond wayfinding, user modeling has the potential to provide customized and adaptive
assistance in other domains for individuals with cognitive impairments, enabling them to
overcome further barriers to their independence and quality of life.
[1] Aysu Baskaya, Christopher Wilson, and Yusuf Ziya Ozcan. Wayfinding in an unfamiliar
environment: Different spatial settings of two polyclinics. Environment and Behavior,
36(6):839-867, 2004.
[2] Jörg Baus, Antonio Krüger, and Wolfgang Wahlster. A resource-adaptive mobile navigation system. In IUI '02: Proceedings of the 7th international conference on Intelligent
user interfaces, pages 15-22, New York, NY, USA, 2002. ACM Press.
[3] Marek Bell, Stuart Reeves, Barry Brown, Scott Sherwood, Donny MacMillan, John
Ferguson, and Matthew Chalmers. Eyespy: supporting navigation through play. In
Proceedings of CHI '09, pages 123-132, New York, NY, USA, 2009.
[4] Jennifer Boger, Goeff Fernie, Pascal Poupart, and Alex Mihailidis. Using a POMDP
Controller to Guide Persons With Dementia Through Activities of Daily Living. Adjunct Proceedings of Ubicomp '03, 2003.
[5] Michael Bowling, Dana Wilkinson, Ali Ghodsi, and Adam Milstein. Subjective Localization with Action Respecting Embedding. The International Symposium of Robotics
Research, 143, 2005.
[6] Pat Brown, Mark Harniss, and Kurt Johnson. Cognitive support technologies: Implications for design and user interface. In International Conference on Technology and
Persons with Disabilities, 2006.
[7] Waylon Brunette, Jonathan Lester, Adam Rea, and Gaetano Bordello. Some sensor network elements for ubiquitous computing. In Jfih International Symposium on
Information Processing in Sensor Networks, pages 388-392, 2005.
[8] David T. Burke, Steven B. Leeb, Rodrick T. Hinman, Elmer C. Lupton, Joel Burke,
Jeffrey C. Schneider, Brian Ahangar, Kristen Simpson, and Eric A.K. Mayer. Using
talking lights to assist brain-injured patients with daily inpatient therapeutic schedule.
Journal of Head Trauma Rehabilitation, 16(3):284-91, 2001.
[9] Stefan Carmien, Melissa Dawe, Gerhard Fischer, Andrew Gorman, Anja Kintsch, and
James F. Sullivan. Socio-technical environments supporting people with cognitive disabilities using public transportation. ACM Trans on Computer-Human Interaction,
12(2) :233 - 262, Jun 2005.
[10] Temple University Developmental Disabilities Center. The final report on the 1990 nati
consumer survey of people with developmental disabilities and their families, 1990.
[11] Yao-Jen Chang, Shih-Kai Tsai, and Tsen-Yung Wang. A context aware handheld
wayfinding system for individuals with cognitive impairments. In Assets '08: Proceedings of the 10th International ACM SIGACCESS conference on Computers and
accessibility, pages 27-34, New York, NY, USA, 2008. ACM.
[12] Wei-Hsun Chen. Mguider: mobile guiding and tracking system in publictransit system
for individuals with cognitive impairments. In Assets '09: Proceedings of the 11th
international ACM SIGACCESS conference on Computers and accessibility, pages 271272, New York, NY, USA, 2009. ACM.
[13] Keith Cheverst, Nigel Davies, Keith Mitchell, Adrian Friday, and Christos Efstratiou.
Developing a context-aware electronic tourist guide: Some issues and experiences. In
Proceedings of CHI 2000, pages 17-24, 2000.
[14] Luca Chittaro and Stefano Burigat. Augmenting audio messages with visual directions
in mobile guides: an evaluation of three approaches. In MobileHCI '05: Proceedings of
the 7th international conference on Human computer interaction with mobile devices
& services, pages 107-114, New York, NY, USA, 2005. ACM Press.
[15] Jaewoo Chung and Chris Schmandt. Going my way: a user-aware route planner. In
Proceedings of CHI '09, pages 1899-1902, New York, NY, USA, 2009.
[16] Elliot Cole and Parto Dehdashti. Computer-based cognitive prosthetics: assistive technology for the treatment of cognitive disabilities. In Assets '98: Proceedings of the third
international ACM conference on Assistive technologies, pages 11-18, New York, NY,
USA, 1998. ACM Press.
[17] Daniel K. Davies, Steven E. Stock, and Michael L. Wehmeyer. Enhancing independent
task performance for individuals with mental retardation through use of a handheld
self-directed visual and audio prompting system. Education and Training in Mental
Retardation and Developmental Disabilities, 37(2):209-218, 2002.
[18] Melissa Dawe. Desperately seeking simplicity: how young adults with cognitive disabilities and their families adopt assistive technologies. In CHI '06: Proceedings of the
SIGCHI conference on Human Factors in computing systems, pages 1143-1152, New
York, NY, USA, 2006. ACM Press.
[19] Jeffery N. Epstein, Melissa G. Willis, C. Keith Conners, and Diane E. Johnson. Use of
a technological prompting device to aid a student with attention deficit hyperactivity
disorder to initiate and complete daily tasks: An exploratory study. Journal of Special
Education Technology, 16(l):19-28, 2001.
[20] Brian Ferris, Dirk Hähnel, and Dieter Fox. Gaussian processes for signal strength-based
location estimation. In Robotics Science and Systems, 2006.
[21] Gerhard Fischer. User modeling in humancomputer interaction. User Modeling and
User-Adapted Interaction, ll(l-2):65-86, 2001.
[22] Dieter Fox, Jeffrey Hightower, Lin Liao, Dirk Schulz, and Gaetano Bordello. Bayesian
filtering for location estimation. IEEE Pervasive Computing, 2:24-33, 2003.
[23] Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces.
In CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 1257-1266, New York, NY, USA, 2008. ACM.
[24] Reginald G. Golledge. Wayfinding behavior: Cognitive mapping and other spatial processes. John Hopkins University Press, 1999.
[25] Reginald G. Golledge, James R. Marston, Jack M. Loomis, and Roberta L. Klatzky.
Stated preferences for components of a personal guidance system for nonvisual naviga-
tion. Journal of Visual Impairment and Blindness, 98(3):pl35-147, 2004.
[26] Joy Goodman, Phil Gray, Kartik Khammampad, and Stephen Brewster. Using Landmarks to Support Older People in Navigation. In Mobile Human- Computer Interaction
- MobileHCI 2004, pages 38-48. Springer, 2004.
[27] F. N. Hagethorn, B. J. A. Kröse, P. de Greef, and M. E. Helmer. Creating design
guidelines for a navigational aid for mild demented pedestrians. In Ambient Intelligence,
pages 276 - 289. Springer, 2008.
[28] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and
Ian H. Witten. The weka data mining software: An update. SIGKDD Explorations,
11(1), 2009.
[29] Jeff Hightower, Sunny Consolvo, Anthony LaMarca, Ian Smith, and Jeff Hughes. Learning and recognizing the places we go. Proceedings of the Seventh International Confer-
ence on Ubiquitous Computing (Ubicomp 2005), pages 159-176, 2005.
[30] Jeffrey Hightower and Gaetano Bordello. Location systems for ubiquitous computing.
IEEE Computer, 34(8):57-66, 2001.
[31] Harlan Hile, Radek Grzeszczuk, Alan Liu, Ramakrishna Vedantham, Jana Kosecka,
and Gaetano Borriello. Landmark-based pedestrian navigation with enhanced spatial reasoning. In Pervasive '09: Proceedings of the 7th International Conference on
Pervasive Computing, pages 59-76, Berlin, Heidelberg, 2009. Springer-Verlag.
[32] Harlan Hile, Alan Liu, Gaetano Bordello, Radek Grzeszczuk, Ramakrishna Vedantham,
and Jana Kosecka. Visual navigation for mobile devices. IEEE MultiMedia, 17:16-25,
[33] Harlan Hile, Ramakrishna Vedantham, Gregory Cuellar, Alan Liu, Natasha Gelfand,
Radek Grzeszczuk, and Gaetano Borriello. Landmark-based pedestrian navigation from
collections of geotagged photos. In MUM '08: Proceedings of the 7th International
Conference on Mobile and Ubiquitous Multimedia, pages 145-152, New York, NY, USA,
2008. ACM.
[34] Stephen S. Intille, Ling Bao, Emmanuel Munguia Tapia, and John Rondoni. Acquiring
in situ training data for context-aware ubiquitous computing applications. In CHI '04:
Proceedings of the SIGCHI conference on Human factors in computing systems, pages
1-8, New York, NY, USA, 2004. ACM.
[35] Julie A. Jacko and Holly S. Vítense. A review and reappraisal of information technologies within a conceptual framework for individuals with disabilities. Universal Access
in the Information Society, l(l):56-76, 2001.
[36] Jong Hee Kang and Gaetano Borriello. Harvesting of location-specific informa-
tion through wifi networks. 2nd International Workshop on Location- and ContextAwareness (LoCA 2006) (at Pervasive 2006), pages 86-102, 2006.
[37] Kathryn G. Karsh, Carol M. Dahlquist, and Alan C. Repp. A comparison of static and
dynamic presentation procedures on discrimination learning of individuals with severe
or moderate mental retardation. Research in Developmental Disabilities, 15(3): 167-86,
[38] Anja Kintsch and Rogerio DePaula. A framework for the adoption of assistive technology. SWAAAC 2002: Supporting Learning Through Assistive Technology, 3:1-10,
[39] Rob Kitchin and Mark Blades. The Cognition of Geographic Space. I.B.Tauris, 2002.
[40] Jesper Kjeldskov and Connor Graham. A review of mobile hci research methods.
Human- Computer Interaction with Mobile Devices and Services, pages 317-335, 2003.
[41] Giulio E. Lancioni, Mark F. O'Reilly, Philip Seedhouse, Frederick Furniss, and
Bernardo Cunha. Promoting independent task performance by persons with severe
developmental disabilities through a new computer-aided system. Behavior Modification, 24(5):700-718, 2000.
[42] AuIi Lepistö and Saila Ovaska. Usability evaluation involving participants with cognitive disabilities. In NordiCHI '04-' Proceedings of the third Nordic conference on
Human- computer interaction, pages 305-308, New York, NY, USA, 2004. ACM Press.
[43] Clayton Lewis. Simplicity in cognitive assistive technology: a framework and agenda
for research. Universal Access in the Information Society, 5(4):351-361, April 2007.
[44] Yang Li, Jason I. Hong, and James A. Landay. Design challenges and principles for
wizard of oz testing of location-enhanced applications. Pervasive Computing, IEEE,
6(2):70-75, 2007.
[45] Lin Liao, Dieter Fox, and Henry Kautz. Learning and inferring transportation routines.
In Proc of the 19th Natl Conf on AI, 2004.
[46] Benson Limketkai, Lin Liao, and Dieter Fox. Relational object maps for mobile robots.
In IJCAI '05: Proc of the 19th Intl Joint Conf on Artificial Intelligence, 2005.
[47] Alan L. Liu, Harlan Hile, Gaetano Borriello, Pat A. Brown, Mark Harniss, Henry
Kautz, and Kurt Johnson. Customizing directions in an automated wayfinding system
for individuals with cognitive impairment. In Assets '09: Proceedings of the 11th
international ACM SIGACCESS conference on Computers and accessibility, pages 2734, New York, NY, USA, 2009. ACM.
[48] Alan L. Liu, Harlan Hile, Henry Kautz, Gaetano Borriello, Pat A. Brown, Mark Harniss, and Kurt Johnson. Indoor Wayfinding: Developing a Functional Interface for
Individuals with Cognitive Impairments. In Proceedings of Assets '06, pages 95-102,
New York, NY, USA, 2006. ACM.
[49] Alan L. Liu, Harlan Hile, Henry Kautz, Gaetano Borriello, Pat A. Brown, Mark Harniss, and Kurt Johnson. Informing the Design of an Automated Wayfinding System
for Individuals with Cognitive Impairments. In Proceedings of Pervasive Health '09.
IEEE, 2009.
[50] Alan L. Liu, Harlan Hile, Henry Kautz, Gaetano Borriello, Brian Ferris, Pat Brown,
Mark Harniss, and Kurt Johnson. Implications for location systems in indoor wayfinding for individuals with cognitive impairments. In First Workshop on Location Based
Services for Health Care, LOCARE 2006, November 2006.
[51] Gary Look, Buddhika Kottahachchi, Robert Laddaga, and Howard Shrobe. A location
representation for generating descriptive walking directions. In IUI '05: Proc of the
10th intl conf on Intelligent user interfaces, pages 122-129. ACM Press, 2005.
[52] Edmund F. LoPresti, Alex Mihailidis, and Ned Kirsch. Assistive technology for cognitive rehabilitation: State of the art. Neuropsychological Rehabilitation, 14(l):5-39,
[53] R. Lubinski, B. Steger Moscato, and B. S. Wilier. Prevalence of speaking and hearing
disabilities among adults with traumatic brain injury from a national household survey.
Brain Injury, 11:103-114(12), 1997.
[54] Patrick Morris Luley, Lucas Paletta, and Alexander Almer. Visual object detection
from mobile phone imagery for context awareness. In MobileHCI '05: Proceedings of
the 7th international conference on Human computer interaction with mobile devices
& services, pages 385-386, New York, NY, USA, 2005. ACM.
[55] Kevin Lynch. The Image of the City. MIT Press, 1960.
[56] Andrew J. May, Tracy Ross, Steven H. Bayer, and Mikko J. Tarkiainen. Pedestrian
navigation aids: information requirements and design implications. Personal and Ubiquitous Computing, 7(6):331-338, 2003.
[57] Joseph E. McGrath. Methodology matters: doing research in the behavioral and social
sciences. Human-computer interaction: toward the year 2000, pages 152-169, 1995.
[58] Joseph Modayil, Rich Levinson, Craig Harman, David Halper, and Henry Kautz. Integrating sensing and cueing for more effective activity reminders. In AAAI Fall 2008
Symposium on AI in Eldercare: New Solutions to Old Problems, 2008.
[59] Karyn Moffatt, Leah Findlater, and Meghan Allen. Generalizability in research with
cognitively impaired individuals. In CHI 2006 Workshop on Designing Technology for
People with Cognitive Impairments, 2006.
[60] Navizon. Cell phone location, gsm and win triangulation for mobile devices, http:
//, June 2010.
[61] E. D. Newbigging and J.W. Laskey. Riding the bus: teaching an adult with a brain
injury to use a transit system to travel independently to and from work. Brain Injury,
10(7):543-550, 1995.
[62] BBC News. Drivers on edge over cliff route, http : //news . bbc . co . uk/1/hi/england/
north_yorkshire/4879026 . stm, April 2006.
[63] Jakob Nielsen. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993.
[64] President's Committee on Mental Retardation. The mentally retarded worker: An
economic di$covery, 1983.
[65] Fred Paas, Alexander Renkl, and John Sweller. Cognitive load theory and instructional
design: Recent developments. Educational Psychologist, 38(1) :1 - 4, 2003.
[66] Jessica Paradise, Elizabeth D. Mynatt, Cliff Williams, and John Goldthwaite. Designing a cognitive aid for the home: a case-study approach. In Assets '04'- Proceedings
of the 6th international ACM SIGACCESS conference on Computers and accessibility,
pages 140-146, New York, NY, USA, 2004. ACM Press.
[67] Kayur Patel, Mike Y. Chen, Ian Smith, and James A. Landay. Personalizing routes. In
UIST '06: Proceedings of the 19th annual ACM symposium on User interface software
and technology, pages 187-190, New York, NY, USA, 2006. ACM.
[68] Donald J. Patterson, Lin Liao, Krzysztof Gajos, Michael Collier, Nik Livic, Katherine
Olson, Shiaokai Wang, Dieter Fox, and Henry Kautz. Opportunity knocks: A system to
provide cognitive assistance with transportation services. UbiComp 2004: Ubiquitous
Computing, 3205:433-450, 2004.
[69] Martha E. Pollack, Laura Brown, Dirk Colbry, Cheryl Orosz, Bart Peintner, Sailesh
Ramakrishnan, Sandra Engberg, Judith T. Matthews, Jacqueline Dunbar-Jacob, and
Colleen E. McCarthy. Pearl: Mobile robotic assistant for the elderly. In AAAI Workshop on Automation as Eldercare, pages 85-92, 2002.
[70] Cliff Randell, Erik Geelhoed, Alan Dix, and Henk Müller. Exploring the effects of
target location size and position system accuracy on location based applications. In
Pervasive Computing, pages 305-320. Springer, 2006.
[71] David A. Ross and Bruce B. Blasch. Evaluation of orientation interfaces for wearable
computers. In Proceedings of IEEE International Symposium on Wearable Computing
(ISWC 2000), pages 51-58, 2000.
[72] David A. Ross and Alexander Lightman. Talking braille: a wireless ubiquitous computing network for orientation and wayfinding. In Assets '05: Proceedings of the 7th
international ACM SIGACCESS conference on Computers and accessibility, pages 98105, New York, NY, USA, 2005. ACM Press.
[73] Meredeth A. Rowe. People with dementia who become lost: preventing injuries and
death. American Journal of Nursing, 103(7) :32-39, 2004.
[74] Meredeth A. Rowe and Judith C. Glover. Antecedents, descriptions and consequences
of wandering in cognitively-impaired adults and the safe return (sr) program. American
Journal of Alzheimer's Disease and Other Dementias, 16(6):344-352, 2001.
[75] Enrico Rukzio, Michael Müller, and Robert Hardy. Design, implementation and evaluation of a novel public display for pedestrian navigation: the rotating compass. In
Proceedings of CHI '09, pages 113-122, New York, NY, USA, 2009.
[76] Daniel Salber and Joëlle Coutaz. A wizard of oz platform for the study of multimodal systems. In INTERACT '93 and CHI '93 conf companion on Human factors in
computing systems, pages 95-96. ACM Press, 1993.
[77] Matthew R. Sanders and Janette M. Parr. Training developmentally disabled adults in
independent meal preparation: Acquisition, generalization, and maintenance. Behavior
Modification, 13(2):168-191, 1989.
[78] Hendrik Schulze. Memos: an interactive assistive system for prospective memory deficit
compensation-architecture and functionality. In Assets '04: Proceedings of the 6th
international ACM SIGACCESS conference on Computers and accessibility, pages 7985, New York, NY, USA, 2004. ACM Press.
[79] Will Seager and Danae S. Fräser. Comparing physical, automatic and manual map
rotation for pedestrian navigation. In CHI '07: Proceedings of the SIGCHI conference
on Human factors in computing systems, pages 767-776, New York, NY, USA, 2007.
ACM Press.
[80] Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo tourism: exploring photo
collections in 3d. ACM Transactions on Graphics, 25(3):835-846, 2006.
[81] McKay Moore Sohlberg, Laurie A. Ehlhardt, Stephen Fickas, and Alistair Sutcliffe.
A pilot study exploring electronic (or e-mail) mail in users with acquired cognitivelinguistic impairments. Brain Injury, 17(7):609-629, 2003.
[82] McKay Moore Sohlberg, Bonnie Todis, Stephen Fickas, Pei-Fang Hung, and Rik
Lemoncello. A profile of community navigation in adults with chronic cognitive im-
pairments. Brain Injury, 19(14):1249-1259, Dec 2005.
[83] Constantine Stephanidis. Adaptive techniques for universal access. User Modeling and
User-Adapted Interaction, 11(1):159-179, March 2001.
[84] Alistair Sutcliffe, Steve Fickas, McKay Moore Sohlberg, and Laurie A. Ehlhardt. Investigating the usability of assistive user interfaces. Interacting with Computers, 15(4):577602, Aug 2003.
[85] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction.
MIT Press, 1998.
[86] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps:
A framework for temporal abstraction in reinforcement learning. Artificial Intelligence,
112:181-211, 1999.
[87] John C. Thomas and Wendy A. Kellogg. Minimizing ecological gaps in interface design.
IEEE Soßware, 6(l):78-86, 1989.
[88] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics. MIT Press,
[89] Shih-Kai Tsai. Wader: a novel wayfinding system with deviation recovery for individuals with cognitive impairments. In Proceedings of the 9th International ACM
SIGACCESS Conferenceerence on Computers and Accessibility. ACM, 2007.
[90] Christof C. van Nimwegen, Daniel D. Burgos, Herré H. van Oostendorp, and Hermina
H. J. M. Schijf. The paradox of the assisted user: guidance can be counterproductive.
In CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing
systems, pages 917-926, New York, NY, USA, 2006. ACM Press.
[91] Benjamin Walther-Pranks and Rainer Malaka.
Evaluation of an augmented
photograph-based pedestrian navigation system. In Smart Graphics, LNCS, pages
94-105. Springer Berlin / Heidelberg, 2008.
[92] Michael L. Wehmeyer. National survey of the use of assistive technology by adults with
mental retardation. Mental Retardation, 36(1):44-51, 1998.
[93] Christopher D. Wickens, John D. Lee, YiIi Liu, and Sallie E. Gordon Becker. An
Introduction to Human Factors Engineering. Pearson Prentice Hall, 2004.
[94] Mike Wu, Ron Baecker, and Brian Richards. Participatory design of an orientation aid
for amnesics. In CHI '05: Proceedings of the SIGCHI conference on Human factors in
computing systems, pages 511-520, New York, NY, USA, 2005. ACM Press.
[95] Judy L. Wynekoop and Sue A. Conger. A review of computer aided software engineering
research methods. In H. Nissen, H. K. Klein, and R. Hirschheim, editors, Information
Systems Research: Contemporary Approaches and Emergent Traditions, pages 301-326.
North-Holland, New York, 1989.
[96] Brian D. Ziebart, Andrew L. Maas, Anind K. Dey, and J. Andrew Bagnell. Navigate like
a cabbie: probabilistic reasoning from observed context-aware behavior. In UbiComp
'08: Proceedings of the 10th international conference on Ubiquitous computing, pages
322-331, New York, NY, USA, 2008. ACM.
[97] Craig Zimring and Ruth C. Dalton. Linking objective measures of space to cognition
and action. Environment and Behavior, 35(1):3-16, 2003.
Alan Liu grew up in Chicago, Illinois before attending the University of California,
Berkeley, where he received his B.S. in Electrical Engineering and Computer Sciences in
2003. He then joined the University of Washington Computer Science L· Engineering Ph.D.
program. Advised by Gaetano Bordello and Henry Kautz, he received a Ph.D. in 2010.
Без категории
Размер файла
6 730 Кб
Пожаловаться на содержимое документа