Nathan Tsoi

I am a computer scientist working at the intersection of robotics and machine learning. My research aims to advance the effectiveness and robustness of robotic systems that interact with humans, with the ultimate goal of making robotic platforms more useful in the real world. Currently, I am a PhD student at Yale University in robotics and a member of the Interactive Machines Group advised by Marynel Vázquez. Previously, I have done research at Stanford University in the Stanford Vision and Learning Lab under Silvio Savarese and have worked on machine learning and data engineering at Sequoia. For fun, I enjoy designing hardware and embedded systems programming.


Improving the Robustness of Social Robot Navigation Systems
Nathan Tsoi, Marynel Vázquez
Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2021
Our aim is to advance the reliability of autonomous social navigation. We have researched how simulation may advance this goal via crowdsourcing. We recently proposed the Simulation Environment for Autonomous Navigation (SEAN) and deployed it at scale on the web to quickly collect data via the SEAN Experimental Platform (SEAN-EP). Using this platform, we studied participants' perceptions of a robot when seen in a video versus interacting with it in simulation. Our current research builds on this prior work to make autonomous social navigation more reliable by classifying and automatically detecting navigation errors.
SEAN-EP: A Platform for Collecting Human Feedback for Social Robot Navigation at Scale
Nathan Tsoi, Mohamed Hussein, Olivia Fugikawa, JD Zhao, Marynel Vázquez
We introduce the SEAN Experimental Platform (SEAN-EP), an open-source system that allows roboticists to gather human feedback for social robot navigation at scale using online interactive simulations. Through SEAN-EP, remote users can control the motion of a human avatar via their web browser and interact with a virtual robot controlled through the Robot Operating System. Heavy computation in SEAN-EP is delegated to cloud servers such that users do not need specialized hardware to take part in the simulations. We validated SEAN-EP and its usability through an online survey, and compared the data collected from this survey with a similar video survey. Our results suggest that human perceptions of robots may differ based on whether they interact with the robots in simulation or observe them in videos. Also, our study suggests that people may perceive the surveys with interactive simulations as less mentally demanding than video surveys.
Challenges Deploying Robots During a Pandemic: An Effort to Fight Social Isolation Among Children
Nathan Tsoi, Joe Connolly, Emmanuel Adéníran, Amanda Hansen, Kaitlynn Taylor Pineda, Timothy Adamson, Sydney Thompson, Rebecca Ramnauth, Marynel Vázquez, Brian Scassellati
ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2021
The practice of social distancing during the COVID-19 pandemic resulted in billions of people quarantined in their homes. In response, we designed and deployed VectorConnect, a robot teleoperation system intended to help combat the effects of social distancing in children during the pandemic. VectorConnect uses the off-the-shelf Vector robot to allow its users to engage in physical play while being geographically separated. We distributed the system to hundreds of users in a matter of weeks. This paper details the development and deployment of the system, our accomplishments, and the obstacles encountered throughout this process. Also, it provides recommendations to best facilitate similar deployments in the future. We hope that this case study about Human-Robot Interaction practice serves as inspiration to innovate in times of global crises.
A Heaviside Function Approximation for Neural Network Binary Classification
Nathan Tsoi, Yofti Milkessa, Marynel Vázquez
Neural network binary classifiers are often evaluated on metrics like accuracy and $F_1$-Score, which are based on confusion matrix values (True Positives, False Positives, False Negatives, and True Negatives). However, these classifiers are commonly trained with a different loss, e.g. log loss. While it is preferable to perform training on the same loss as the evaluation metric, this is difficult in the case of confusion matrix based metrics because set membership is a step function without a derivative useful for backpropagation. To address this challenge, we propose an approximation of the step function that adheres to the properties necessary for effective training of binary networks using confusion matrix based metrics. This approach allows for end-to-end training of binary deep neural classifiers via batch gradient descent. We demonstrate the flexibility of this approach in several applications with varying levels of class imbalance. We also demonstrate how the approximation allows balancing between precision and recall in the appropriate ratio for the task at hand.
SEAN: Social Environment for Autonomous Navigation
Nathan Tsoi, Mohamed Hussein, Jeacy Espinoza, Xavier Ruiz, Marynel Vázquez
Proceedings of the 8th International Conference on Human-Agent Interaction
Social navigation research is performed on a variety of robotic platforms, scenarios, and environments. Making comparisons between navigation algorithms is challenging because of the effort involved in building these systems and the diversity of platforms used by the community; nonetheless, evaluation is critical to understanding progress in the field. In a step towards reproducible evaluation of social navigation algorithms, we propose the Social Environment for Autonomous Navigation (SEAN). SEAN is a high visual fidelity, open source, and extensible social navigation simulation platform which includes a toolkit for evaluation of navigation algorithms. We demonstrate SEAN and its evaluation toolkit in two environments with dynamic pedestrians and using two different robots.
Improving Social Awareness Through DANTE: Deep Affinity Network for Clustering Conversational Interactants
Mason Swofford, John Peruzzi, Nathan Tsoi, Sydney Thompson, Roberto Martín-Martín, Silvio Savarese, Marynel Vázquez
Proceedings of the ACM on Human-Computer Interaction
We propose a data-driven approach to detect conversational groups by identifying spatial arrangements typical of these focused social encounters. Our approach uses a novel Deep Affinity Networkto predict the likelihood that two individuals in a scene are part of the same conversational group, considering their social context. The predicted pair-wise affinities are then used in a graph clustering framework to identify both small (e.g., dyads) and large groups. The results from our evaluation on multiple, established benchmarks suggest that combining powerful deep learning methods with classical clustering techniques can improve the detection of conversational groups in comparison to prior approaches. Finally, we demonstrate the practicality of our approach in a human-robot interaction scenario. Our efforts show that our work advances group detection not only in theory, but also in practice.
Prompting Prosocial Human Interventions in Response to Robot Mistreatment
Joe Connolly, Viola Mocz, Nicole Salomons, Joseph Valdez, Nathan Tsoi, Brian Scassellati, Marynel Vázquez
ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2020
Inspired by the benefits of human prosocial behavior, we explore whether prosocial behavior can be extended to a Human-Robot Interaction (HRI) context. More specifically, we study whether robots can induce prosocial behavior in humans through a 1x2 between-subjects user study ($N=30$) in which a confederate abused a robot. Through this study, we investigated whether the emotional reactions of a group of bystander robots could motivate a human to intervene in response to robot abuse. Our results show that participants were more likely to prosocially intervene when the bystander robots expressed sadness in response to the abuse as opposed to when they ignored these events, despite participants reporting similar perception of robot mistreatment and levels of empathy for the abused robot. Our findings demonstrate possible effects of group social influence through emotional cues by robots in human-robot interaction. They reveal a need for further research regarding human prosocial behavior within HRI.
Generalized Intersection over Union
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, Silvio Savarese
Computer Vision and Pattern Recognition (CVPR) 2019
Intersection over Union (IoU) is the most popular evaluation metric used in the object detection benchmarks. However, there is a gap between optimizing the commonly used distance losses for regressing the parameters of a bounding box and maximizing this metric value. The optimal objective for a metric is the metric itself. In the case of axis-aligned 2D bounding boxes, it can be shown that $IoU$ can be directly used as a regression loss. However, $IoU$ has a plateau making it infeasible to optimize in the case of non-overlapping bounding boxes. In this paper, we address the weaknesses of $IoU$ by introducing a generalized version as both a new loss and a new metric. By incorporating this generalized $IoU$ ($GIoU$) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, $IoU$ based, and new, $GIoU$ based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.


HAI 2020 Best Poster Award - Runner Up
For the work: SEAN: Social Environment for Autonomous Navigation by N. Tsoi, M. Hussein, J. Espinoza, X. Ruiz, and M. Vázquez
Alan J. Perlis Graduate Fellowship Recipient
This fellowship was established at Yale in 2006 through generous gifts from various donors in honor of Professor Alan J. Perlis (1922–1990), a pioneer of programming language research, the first winner of the Association for Computing Machinery’s (ACM) Turing Award, and the founding chair of Yale’s Computer Science Department.


Robots For Good: Fighting Social Isolation with Robots
Robotic telepresence for elementary-age children during social distancing.
Multi-sourced 2D and 3D Sensor Fusion and Person Tracking Pipeline
For research in imitation learning, creating motion policies for social navigation
A Tensorboard-like visual interface for Darknet, available as part of g-darknet

Service and Activities

YHACK 2019 Final Judge