An embodied conversational agent as bartender - Peter
Type
Research Project
Team
Lars Wintermans
Rayhan B
Sabin K Lopez
Rishi Gurrala
Duration
6 Weeks
My Role
User research, conversational design, ethical research practices, user testing followed by interpretation.
Problem Statement
Employees in hospitality sector such as waiters and bartenders are often mistreated by customers
and due to the nature of their management, they often have to bear such mistreatment.
The use of machine interfaces can be used to alleviate some of the issues faced by these employees. With use of voice interfaces, the process of placing an order at a bar/restaurant can be as fluid and functional as ordering with a person.
With the advent of humanoid robots such as Furhat, the implementation of such embodied conversational agents as bartenders can provide a rich experience to the customers as well as promoting a safe working environment for the employees
Furhat
Furhat is a social robotics platform developed by Furhat Robotics. It features a customizable,
human-like robotic head with a projected animated face that can display a range of expressions.
Furhat is designed to engage in natural, interactive conversations with humans, utilizing speech
recognition, natural language processing, and computer vision technologies. Its modular design and open SDK allows developers to customise its appearance and behaviour.
Design Methods
Research through design - Building design artefacts that addresses research gaps
Furhat provides an open SDK which makes it easier to build and test artefacts which can accelerate further research in the human robot interaction field. The furhat SDK comes with a remote furhat which works on desktop. We were able to use the desktop version for initial phases and the physical furhat for further iterations
Scenario Based design
We outlined scenarios in which we would like to employ Furhat for the specific application.
- Where the customer intiates and continues a purely transactional flow
- Where the customer is being more social
Dialogue Flow
Non-Verbal Expression
The non-verbal expressions were designed to be cohesive with the
transactional and social flow. Research shows how gaze can influence
communication to a great extent.
Non-verbal expressions were designed in terms of gaze, eyebrow
movements and lip-movements of Furhat. These were mapped
accordingly to transactional flow and social flow. In case of
transactional flow the expressions were gaze aversion, frown like
brows and lack of smile. This coupled with dialogue of Peter feeling
like he’s being treated like a Robot.For the case of social flow furhat’s
facial expressions were gaze into customers eyes, with eyeborws
neutral and lips smiling when not talking.
Integration of GPT-3
In case of social interaction , ChatGPT form OpenAI was integrated for the case when the customer
asks Peter for suggestions and peter prompts a choice of custom cocktail as an option based on the ingredients of choice of the customer.
User Testing
User testing was conducted on a conversational agent named Peter the Bartender, aimed at assessing its performance and user experience. Five participants interacted with Peter, attempting to order a drink and engage in conversation. Observations revealed mixed results, with only two participants successfully ordering a drink on the first try. Issues included limited vocabulary recognition, difficulties in handling user-initiated conversation shifts, and sub-par error handling. Survey results indicated that participants appreciated Peter's facial expressions but were frustrated by repetitive error messages and felt the agent didn't always understand them.
Future Implications
Recommendations for improvement include expanding vocabulary recognition, refining error management, addressing unexpected bugs, and further developing nonverbal communication features. Overall, while participants generally enjoyed interacting with Peter, improvements are necessary to enhance the agent's reliability and effectiveness.