Engagement in Human Robot Interaction

PDF
Instructed by: Dr. Selma Sabanovic
Video Presentation: https://drive.google.com/file/d/1N7uyq7MoxKjMLL-W0WoX-6f122RIlKgx/view?usp=sharing
Bartender Robot Source https://www.youtube.com/watch?v=d-8ZlLIs8fg

1. Abstract

This paper explores the definitions and research methods of engagement in human robot interaction (HRI) research. Then, it introduces three different bartender robots that have different behaviors and strategies to increase user engagement. Increasing the user's level of engagement with a robot is known to produce positive outcomes in many fields and thus has been the goal of many HRI researchers.

keywords - Human Robot Interaction, Engagement

2. Background

Measuring and assessing users’ engagement with a robot is essential. Researchers have found that a high level of engagement enhances learning outcomes in both learning environments without robots [Pon09] and with robots [Gle17]. Another reason why HRI engagement research is crucial is that children with autism show higher engagement with robots than humans. Robots are used as service providers and practical companions such as study mates and weight-loss coaches. When designing these robots, engagement plays a vital role because they are interacting with humans instead of working in an assembly line.

3. Definition of Engagement in Human Robot Interaction

Castellano et al. provide an overview of engagement in human-agent(robot) interaction [Oe20]. According to them, diverse papers attempt to define and measure engagement in different ways. They grouped researchers into two primary groups: those that consider engagement as a process and those that consider it as a state. One example that treats engagement as a complex process defines engagement as initial contact, negotiating a collaboration, checking on other's interaction, evaluating whether to stay involved, and deciding when to end [Ca04].

Other researchers categorize engagement based on the level of engagement to better investigate its role. For example, Sabanovic et al. categorize engagement into four levels: present, attending, engaged and interacting [Mi06].

This paper focuses on the user's measurable social engagement state in which the user is paying attention to the robot. Treating engagement as a state instead of a process gives a better chance to measure the level of engagement because it is less complex. The level of engagement can be perceived and measured by the robot's camera and sensors. The analysis of the level of engagement can be used to assign different instructions to the robot. Then, the robot can improve its behavior which ultimately maximizes the user’s level of engagement.

4. Measuring the Level of Engagement and Models to assess Engagement

Many researchers use pre-trained models such as Affdex [Mc16] and OpenPose to measure user engagement with a robot. However, those models are not strictly based on HRI data thus, they may not tell us the accurate result. To solve this problem, some researchers proposed their own model using the HRI data set. For example, Hanheide et al. developed and validated a novel model to measure user engagement from a robot's perspective \cite{De20}. They used The TOur GUide RObot (TOGURO) annotated image data set collected from the two cameras mounted on the robot’s body and head during a guided tour in a museum. Their research produced an HRI data-based regression model that predicts the engagement of a user with a robot. They claim their model is generic enough so it can be used in other HRI engagement research. Some researchers collect their own data using an experimental setting with a virtual assistant and cameras that record users. Sabanovic et al. proposed a model to assess user engagement towards a robot receptionist based on the user's movement, the timing of interactive behavior, the user's spatial position, and predefined areas around the robot [Mi06].

5. Bartender Robot Brillo #1

Rossi et al. introduce Bartending Robot for Interactive Long-Lasting Operations (Brillo) project focusing on how the robot's interaction with humans can increase the level of the user's social engagement (Figure 1) [Ro20]. This robot has three strategies to enhance the level of engagement: arm gestures, eye colors, and the tablet. Unlike a typical service robot focused on functional features, Brillo starts a natural conversation with a customer by first asking how the day was and recommending several drinks. While the order is being processed, Brillo brings up Google News with the tablet attached to its body. During these interactions, researchers used Affdex to detect facial expressions and measure the level of engagement. The results revealed that entertaining behavior using its arm gesture and emphatic behavior implemented with sociable with right-arm-up gesture stimulated more engagement than neutral behavior. However, no significant difference was observed between entertaining and emphatic behavior.

This paper would like to recommend the robot have changing eye shapes to express the emotion of the robot better. It is controversial whether the eyes' color is enough to show the robot's emotional state accurately. If Rossi et al. had a way to express the emphatic behavior better, emphatic behavior could be more effective than entertaining behavior at increasing engagement.

6. Bartender Robot Brillo #2

John et al. present the personalized behavior of Brillo on different users, and its ability to engage with multiple users simultaneously (Figure 2) [Jo22]. They personalize Brillo's behaviors upon the customers’ level of engagement, moods, purchasing preferences, and previous interaction methods. User detection, sentiment analysis, emotion recognition, and engagement analysis are strategies they use. They use the OpenPose library to determine whether Brillo perceives the customer as engaged. Depending on this state, Brillo will interact differently with the customer. Brillo will remember each customer's level of engagement and provide personalized service. Moreover, Brillo recommends drinks and uses sentiment analysis on the customer's reaction. Brillo forms each customer's preferences with the result of this procedure. 

John et al.'s effort to give the robot personalized behavior for each customer to resemble the real-life bartender's behavior is valuable. This paper recommends them to apply different methods to express that the robot recognizes the user. If such a method increases engagement, it will make Brillo more human-like.

7. Bartender Robot Rob

Bartender robot Rob successfully served as a robot bartender on the MSC Virtuosa Cruise Ship in 2021 (Figure 3) [Ma21]. Frances Marcellin interviewed Rob's project head Luca Pronzati to discover the goal that researchers had with Rob. Researchers gave a voice, sympathetic ear, diverse facial expressions, and a human-like personality to Rob on top of the state-of-art function to mix cocktails.

In a video demonstrating the process of Rob mixing a cocktail for a customer, we can observe his dynamic and interactive facial expressions and gestures. Although it is not documented elsewhere, we can observe a certain logic for the changes in his facial expressions. On top of that basic logic, this paper recommends more advanced possible interactions related to HRI engagement. In the video, when Rob starts mixing the cocktail, he verbally expresses, "I love mixing cocktails," and his eyes turn to stars. Instead, Rob should make an eye-contact with the customer while mixing the drink, and if he recognizes that the customer's engagement is high, he can make that star-eyes reaction as a reward interaction. This will increase the level of engagement of that customer and other customers that witnessed that reaction. Moreover, in the video, while Rob gazes at the tools to mix the drink, he makes eye contact with the customer by looking out of the corner of his eyes. This corner-eyes facial expression should happen anytime when Rob assesses any previous or current customer's engagement is high. These recommended changes will make Rob act more human-like.

8. Conclusion

Researchers who design robots use various ways to understand user engagement and experiment with diverse methods to increase the level of engagement. This is because the higher level of engagement achieves the ultimate goal of the HRI research. The goal can be enhanced educational output, maximized cocktail sales, and gifting a user an unforgettable experience on a cruise ship.

References: 
[Can04] Cory K. Christopher L. Neal L. Candace, S. Where to look: A study of human robot engagement. 2004 International Conference on Intelligent User Interfaces, pages 78–84, 2004.

[DDF20] Hanheide M. Del Duchetto F., Baxter P. Are you still with me? continuous engagement assessment from a robot’s point of view. Front. Robot. AI, 7:116, 2020.

[Gle17] Greenhow C. Gleason, B. Hybrid education: The potential of teaching and learning

with robot-mediated communication. Online LearningJournal, 21:4, 2017.

[Joh22] Rossi A. Rossi S. John, N. E. Personalized human-robot interaction with a robot

bartender. Adjunct Proceedings of the 30th ACM Conference on User Modeling,

Adaptation and Personalization July, 2022.

[Mar21] Pronzati L. Marcellin, F. Mine’s a screwdriver: meet the robot bartender serving

up drinks at sea. Future Cruise, 2021.

[McD16] Mahmoud A. Mavadati M. Amr M. Turcot J. Kaliouby R. McDuff, D. Affdex sdk: A

cross-platform real-time multi-face expression recognition toolkit. CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2016.

[Mic06] Sabanovic S. Simmons R. Michalowski, M. P. A spatial model of engagement for

a social robot. 9th International Workshop on Advanced Motion Control, pages 229–240, 2006.
[Oer20] Castellano G. Chetouani M. Nasir J. Obaid M. Pelachaud C. Peters C. Oertel, C.

Engagement in human-agent interaction: An overview. Front. Robot. AI, 7:92, 2020.
[Pon09] C. Rimm-Kaufman S. E. Grimm K. J. Curby T. W. Ponitz, C. Kindergarten class-

room quality, behavioral engagement, and reading achievement. School Psychology Review, 38:102–121, 2009.

[Ros20] Dell’Aquila E.-Maggi G. Russo D. Rossi, S. What would you like to drink? engagement and interaction styles in hri. HRI ’20: ACM/IEEE International Conference on Human-Robot Interaction. 2020.