Current Research Project Proposal Summaries
Toyota Research Institute/U-M Partnership
Current Research Project Proposal Summaries
Personalized Driver Alertness with Multimodal Holistic Models of Drivers
Project Abstract/Statement of Work:
According to a survey run by the National Sleep Foundation (2005), 60% of U.S. adult drivers drove while being fatigued, and as many as 37% admitted to have fallen asleep at the wheel. Moreover, distracted driving caused by engaging in a different activity can cause fatal accidents. The U.S. Department of Transportation estimated a total number of 421,000 injuries in vehicles crashes due to distracted driving.
The goal of this project is to construct holistic models of drivers, covering a multitude of channels, including vision, physiology and language, as well as background information (demographic and psychological) and affective information of the drivers. We will use these holistic representations to build effective personalized multimodal models of driver alertness, aiming to identify patterns associated with two main driving states: alertness (alert vs. drowsy) and attention (attentive vs. distracted).
Previous research (including ongoing TRI projects) has focused on the detection of either alertness or distraction, but not their joint co-existence. Yet, we expect that distractors would have a different impact on a driver in a drowsy (tired) state than on an alert driver, and therefore our goal is to build joint models that take into account this interdependency.
Our team spans three areas of expertise in sensors, vision, and language, and it will be a unique collaboration across all three U-M campuses.
Don’t Bite the Hand that Feeds You: Soft Robots for Eldercare
Project Abstract/Statement of Work:
In order to create a means for robots to enter the home and begin caring for the elderly, we envision a small robot modeled after a household pet. Like a pet, such a robot could offer companionship and fetch slippers, but also expand these roles to include bringing medicines, assisting in self-feeding, and even operating appliances. A small robot could also support health monitoring, telemedicine, and communication with family and caretakers. Such a robot should be inherently soft to pose minimal risk to its user and be widely accepted. In this project, we strive to develop the underlying technology and the necessary mathematical methods to design and control such a versatile soft robotic system. In particular, we seek to bridge the established world of rigid robots with the world of soft robots by reasoning about constraints, compliances, and energy flows in an analogous manner, thereby introducing the soft equivalents of Jacobians, operational spaces, and spatial impedances. We will implement and test these methods in a multi-dimensional manipulator.
PI:
Brent Gillespie
Collision Avoidance Guardian at the Dynamic Limits of Handling
Project Abstract/Statement of Work:
This project aims to solve the problem of collision avoidance when collision can be avoided only if the vehicle is maneuvered at its dynamic limits of handling. In this context, dynamic limits refer to events that can cause loss of control over the vehicle, such as excessive sideslip or tire lift-off.
Maneuvering a vehicle at its dynamic limits becomes necessary if the driver is unable to react to a situation fast enough (e.g., due to limitations of human response times, or distraction) and the safe braking distance is too long to avoid collision with an obstacle by braking alone (e.g., due to high speed, or low friction). In these situations, an autonomous collision avoidance algorithm with the ability of maneuvering the vehicle at its dynamics limits can serve as a guardian to avoid collision. The goal of this one-year pilot project is to develop this guardian feature and perform pilot studies with driver-in-the-loop simulations to evaluate its effectiveness.
Current obstacle avoidance approaches can be classified into four categories: graph search based methods, virtual potential and navigation function based methods, meta-heuristic methods, and mathematical optimization based methods. The first three categories cannot guarantee obstacle avoidance at the dynamic limits, as they do not account for the dynamic limits of the vehicle explicitly. Mathematical optimization provides the formalism needed to explicitly include dynamic limits; however, the existing collision avoidance formulations do not consider all the constraints needed to be able to maneuver a vehicle at its dynamic limits.
To solve the above mentioned problem, we will use a model predictive control framework. In this framework, a low-order model of the vehicle will be used to predict the vehicle trajectory for a short time horizon into the future for given steering, throttle, and brake commands. A mathematical optimization will be set up with this vehicle model as part of the dynamic constraints. Additional constraints will include actuator and safety constraints (e.g., no tire lift-off). The goal of this optimization will be to find the optimal steering, throttle and brake commands to avoid collision with a maximum safety margin. This optimization will be repeated every few hundred milliseconds with a prediction horizon of a few seconds to refresh the optimal control commands according to the new measurements from the vehicle and environment (e.g., vehicle position, sensor data, etc.). The resulting steering, throttle and brake commands will not be applied to the vehicle unless a specified minimum safety margin is reached. At that point the algorithm will take over the control of the vehicle to avoid the collision and then give control back to the driver once the vehicle is safe again. A driver-in-the-loop simulator with a high-order vehicle model will be used for evaluation purposes.
PI and Co-PI:
Tulga Ersal
Jeffrey Stein
How People Make Tradeoffs: Developing New Models
Project Abstract/Statement of Work:
Understanding when collaborators are likely to make mistakes is a key way in which people mitigate the impact of errors in team settings. Between people, this understanding of when and where to trust a collaborator — called a mental model — is formed through repeated interactions and communication about uncertainty. Machine Learning (ML) systems have tried to convey similar information on reliability through confidence scores, but prior work suggests the rules by which people judge possible outcomes is based on heuristics that can fail in artificial settings, much less with artificial agents.
We propose to study i) the impact of agent stated uncertainty (over a range of uncertainty values) on user mental model formation in humanAI teams; ii) the importance of reliability in communicating uncertainty (i.e., the negative impact of incorrectly assessing uncertainty), and iii) the impact of team size and complexity on people’s ability to form accurate mental models under uncertainty. We will extend our existing platform (which is inspired by our work in [1]) for controlled measurement of mental model formation in a joint decision-making environment (classification) to include estimates of the AI agent’s confidence. We will then vary the agent’s ability, the accuracy of the reported uncertainty, method for conveying uncertainty (e.g., number, visualization, visualization type), and the size of the team. This will provide an interactive setting in which users/participants can collaboratively complete tasks of differing complexity and length. We will measure collaborative performance in terms of total time taken to complete, and accuracy on, the given shared classification task. This will provide a uniform measure across all task complexities and configurations. Participants will primarily be recruited from Amazon Mechanical Turk.
PI:
Rich Gonzalez
Causal Modeling of Human Drivers
Project Abstract/Statement of Work:
Autonomous vehicles (AVs) have potential for both improving safety and efficiency as well as detracting from them. In our view, both traffic flow and safety will be maximized by self-driving cars implementing what we are calling the Minimum Disruption (MD) Principle, i.e., causing the smallest change possible to the trajectories of other vehicles (and other traffic participants). Under this principle, the path planning problem can then be formulated as follows: 1) determine a set of reasonable possible actions that satisfy basic safety constraints and are consistent with the goal of the AV; 2) calculate the disruption caused to other drivers by each possible action; 3) execute the action that causes the least disruption. The goal of this research is to enable part 2) of this planning problem by developing a model of the way a human-driven vehicle responds to an action taken by a neighboring vehicle.
PI and Co-PI:
Johann Gagnon-Bartsch
Richard Frazin
Multimodal Sensing of Human Behavior
Project Abstract/Statement of Work:
Home robotic companions provide support, entertainment, and interaction. This requires that an agent is user-aware: that it can accurately sense and interpret user state. We focus on three aspects of user state tied to the home environment: user comfort (i.e., thermal comfort), affective state (i.e., ambient and expressed valence/activation), and stress. The primary goal of this proposal is to create innovative approaches that are accurate and robust for user state recognition.
PI and Co-PIs:
Mihai Burzo
Rada Mihalcea
Emily Provost
VehiQL-Query Processing for Visual Data Streams
Project Abstract/Statement of Work:
The ubiquitous availability of inexpensive digital cameras has profoundly changed the world. Video is being produced at unprecedented rates. Continuous video processing and acquisition lies at the core of transformative advances in autonomous vehicles. As the number of cameras in each vehicle grows, autonomous vehicle researchers are acquiring voluminous libraries of in-vehicle video streams (e.g., the repository at Michigan’s transportation research institute). Our objective is to create an infrastructure for performing real-time analysis and information retrieval from such rich video libraries.
PI and Co-PIs:
Mike Carefully
Jia Deng
Thomas Wenisch
It’s the Transitions…!: Supporting Shared Control in Vehicle Steering Across Routine and Off-Nominal Conditions
Project Abstract/Statement of Work:
Experience with increasingly autonomous systems in aviation and other complex domains has shown that performance breakdowns tend to occur at transition points and in off-nominal conditions, rather than during routine operations. In particular, operators experience ‘automation surprises’ during transitions between levels of automation, in cases where the system acts in unanticipated or unexplained ways, and when it transfers control to the operator without adequate advance warning [1]. Further, a lack of transparency regarding the capabilities, limitations, and strategies of highly automated systems has been associated with trust miscalibration and the failure to intervene when necessary. Keeping operators involved, informed, and engaged on a continuing basis has proven critical for handling unexpected events and preventing such events from turning into accidents.
PI and Co-PI:
Brent R. Gillespie
Nadine Sarter
Highly Dynamic Bipedal Locomotion in Unknown, Loosely Structured Environments
Project Abstract/Statement of Work:
Bipedal robots offer a more versatile solution to traversing uneven outdoor terrain and cluttered indoor environments compared to their wheeled counterparts. The state-of-the-art control algorithms for dynamic biped locomotion are capable of providing stabilizing feedback for a biped robot blindly walking across uneven terrain with height disturbances of about 10 cm per step; however, without perceiving the environment, the application of legged robots remains extremely limited.
The ultimate objective of this project is the development of an integrated perception and planning systems that can operate in conjunction with available stabilizing walking control algorithms. We aim to integrate robot perception onto the Cassie-series biped robot to enable: autonomous navigation through a forest at “real-life” speeds (walking at ∼ 0.5 m/s) walking across highly uneven terrain while maintaining stability (UM’s Wave Field) planning and safely maneuvering around obstacles in complex indoor environments autonomously walking up and down flights of stairs found in homes. These goals will be achieved by extending state-of-the-art SLAM algorithms to include leg odometry, developing novel path-planning algorithms, and designing feedback controllers that make use of terrain information by utilizing machine learning and trajectory optimization techniques.
PI:
Jessy W. Grizzle
Development of a “Primary Other Test Vehicle” for the Testing and Evaluation of High-Level Automated Vehicles
Project Abstract/Statement of Work:
It is critical to thoroughly and rigorously test and evaluate an Automated Vehicle (AV) before it is released as a production vehicle. Recent crashes involving Google and Tesla vehicles brought significant attention to the safety of AVs. The Tesla Autopilot, in particular, was criticized of being released too early and the consumers were used as beta testers. Currently, there is no federal standard or guidance on the testing and evaluation of AVs. Many companies choose to test their AVs on the public streets. A well-known problem of this approach is its inefficiency due to the low exposure of safety critical events in daily driving. Therefore, the concept of accelerated evaluation was proposed by the PI, with the goal of exposing the AV with higher driving risk, mainly from more aggressive behaviors of the primary other vehicle. This concept was found to accelerate the evaluation process by 2-5 orders of magnitude. [1]
In this proposal, we focus on developing two practical elements for experimentally realizing the concept of accelerated evaluation on a test track: a) building a stochastic library of conflict driving scenarios; b) developing a test vehicle that can flexibly and safely emulate the motions of the Primary Other Vehicle (POV).
PI and Co-PI:
Huei Peng
Ding Zhao
LifeQA: Holistic Visual-Linguistic Scene Understanding for Real-Life Question Answering
Project Abstract/Statement of Work:
The main goal of our project is to lay the foundations for a new generation of in-home multimodal question answering systems, which can answer day-to-day questions by jointly leveraging language and vision. At the core of our approach are scene semantic graphs, a novel holistic representation of scenes that we propose, which combines pixels and words into a common symbolic space. Semantic graphs are abstractions whose nodes represent instances or entities, while the edges represent relationships. For example, “person cutting cake with knife” can be represented by a graph that has four vertices (“person”, “cutting”, “knife”, “cake”) and three edges (“person-cutting”, “cutting-with-knife”, “cutting-cake”). The edges connect “cutting” with its components—“person” is the subject, “knife” is the instrument, and “cake” is the patient. We note that because the edges can represent arbitrary relations, such graphs are expressive enough to capture a wide range of semantics, be it causal, spatial, or temporal. We will develop algorithms to convert video scenes into a semantic graph that represents a scene at a semantic and cognitive level.
The result of this process will be a large dataset of real-life multiple-choice questions that can be used for the purpose of training and evaluating our methods.
PI and Co-PI:
Rada Mihalcea
Jia Deng
Human Augmented 3D Computer Vision for Robust Simulation of Rare Events
Project Abstract/Statement of Work:
Research in robotics and autonomous vehicles (AVs) suffers from a lack of realistic training data and environments in which to test new approaches. The sparsity of rare and unusual events in real settings—which may occur only once every few years in a home, or every few million or billion miles driven—results in standard collection techniques (like instrumented vehicles on real roads) encountering exceptionally few such events. These events occur several orders of magnitude less frequently than is needed to collect large enough training and testing sets over the timespan of less than a decade using current methods, presenting a fundamental bottleneck in the research and deployment of such systems.
Simulation is a mechanism for overcoming this bottleneck. However, generating realistic simulations, especially of rare and unusual events, is a challenge. This project envisions a future in which publicly available videos from individual users (i.e., general in-home/office footage from television) and municipal sources of visual traffic data (i.e., traffic cameras) can be used to enable generation of simulated environments containing rare events.
We plan to use a crowdsourced human-in-the-loop approach to guide computer vision algorithms to extract measurement information from large video corpora, allowing us to create simulations of scene dynamics for training and testing (including vehicle speed, object orientation, etc.) The proposed work creates a new, improved pipeline for scene reconstruction and deploys it on real data to generate useful training and simulation datasets.
PI:
Jason J. Corso
Intelligent and Automatic Motion Planning for Self-Driving Vehicles
Project Abstract/Statement of Work:
Motion planning is a fundamental task for self-driving cars.
Connected automated vehicles (CAVs) deployed in real-world traffic can be surrounded with many moving/stationary obstacles. In many cases, the pre-defined path was obtained by having the vehicle driven manually once or more than one time—rendering this approach not scalable for deployment in the real world.
This proposed work aims to develop, implement and test algorithms for intelligent and automatic motion planning, for both RTK-centric and Sensor-centric self-driving cars. Being “intelligent” means that the path will be generated by considering a wide range of factors, road/lane constraints, dynamic interactions with other road users, traffic rules, vehicle dynamics, and environmental constraints. Being “automatic” means that human inputs are only required at the initiation of the process or during algorithm calibration, and not afterwards. We will start from traditional deterministic model/theory based approaches or AI based algorithms such as reinforcement learning, and will extend to consider the stochastic nature of dynamic self-driving.
PI and Co-PI:
Huei Peng
Shaobing Xu
Building Mental Models Under Uncertainty in Human AI Teams
Project Abstract/Statement of Work:
Understanding when collaborators are likely to make mistakes is a key way in which people mitigate the impact of errors in team settings. Between people, this understanding of when and where to trust a collaborator — called a mental model — is formed through repeated interactions and communication about uncertainty. Machine Learning (ML) systems have tried to convey similar information on reliability through confidence scores, but prior work suggests the rules by which people judge possible outcomes is based on heuristics that can fail in artificial settings, much less with artificial agents.
We propose to study i) the impact of agent stated uncertainty (over a range of uncertainty values) on user mental model formation in humanAI teams; ii) the importance of reliability in communicating uncertainty (i.e., the negative impact of incorrectly assessing uncertainty), and iii) the impact of team size and complexity on people’s ability to form accurate mental models under uncertainty. We will extend our existing platform (which is inspired by our work in [1]) for controlled measurement of mental model formation in a joint decision-making environment (classification) to include estimates of the AI agent’s confidence. We will then vary the agent’s ability, the accuracy of the reported uncertainty, method for conveying uncertainty (e.g., number, visualization, visualization type), and the size of the team. This will provide an interactive setting in which users/participants can collaboratively complete tasks of differing complexity and length. We will measure collaborative performance in terms of total time taken to complete, and accuracy on, the given shared classification task. This will provide a uniform measure across all task complexities and configurations. Participants will primarily be recruited from Amazon Mechanical Turk.
PI:
Matthew Kay
Building and Reasoning about Fully 3D Representations
Project Abstract/Statement of Work:
Humans have a deep understanding of the physical environment around them that they use to move through and interact with the world. Their knowledge is fully three dimensional: upon entering an unfamiliar building, they know that the floor continues behind furniture even though it is hidden by the furniture and can make sensible inferences about the layout of the nearby parts of the building given limited observations. The main goal of this project is to enable computers to learn to extract such a 3D representation from ordinary images and to connect this ability with tasks and settings that are relevant to autonomous systems, such as service robots indoors and autonomous vehicles outdoors.
The goal of this project is to work towards giving computers the ability to infer a full 3D understanding of the world from a conventional image and to demonstrate how to apply this understanding to a variety of tasks. This proposal aims to better connect efforts with robotics by working with natural images and applying it towards tasks and scenarios that are more closely related to robotics and robotics tasks. We intend to explore projects in three main directions towards achieving this goal: better handling natural data from ordinary cameras (not synthetic data), integrating robotic sensors with learned systems, and reasoning for robotic tasks on top of the predicted 3D.
PI:
David Fouhey
Learning Visual Representations Via Language
Project Abstract/Statement of Work:
Currently, most computer vision systems rely on large labeled datasets such as ImageNet or COCO for training. This paradigm has been hugely successful, but the necessity of employing human annotators limits the scale of data on which we can train.
Recent developments in natural language processing such as BERT have shown that large quantities of raw textual data downloaded from the web can be used to learn high quality representations of text that transfer to many downstream language tasks.
We aim to build on these recent advances in NLP, and use large quantities of image and text data in order to jointly learn representations of images and text that transfer to many downstream vision and vision+language tasks.
We believe that the supervisory signal provided by language will be particularly useful for learning visual representations that generalize to the long tail of objects, and the at capture the compositional structure of scenes. If rare or unusual objects appear in images, they are likely to be mentioned in corresponding text; this provides natural supervision for objects in the long tail of the category distribution that is not present when training on datasets labeled with fixed categories, such as ImageNet or COCO. Text associated with images often describes properties of objects, or relationships between objects; as such it provides natural supervision for the compositional structure of complex images, without resorting to expensive and explicit manual annotation of compositional structure as in datasets such as Visual Genome.
We imagine that language supervision will be useful both in weakly and strongly supervised regimes. Compared to traditional human annotation pipelines that rely on complex category hierarchies and marking exact segmentation hierarchies, we believe that asking people to annotate images with language will require orders of magnitude less annotation time per image, resulting in cost savings.
PI:
Justin Johnson
Modeling and Understanding Human-Machine Teaming and Decision Making
Project Abstract/Statement of Work:
We propose a system to support human-machine teaming with the goal of augmenting existing Wikipedia team members’ abilities in managing Wikipedia projects by automatically coordinating communications and task assignment to optimize opportunistic team performance. We propose a Reinforcement Learning-based system that can automatically reason about and forecast the quality of an article using information about the article topic and current team working on the article (e.g., team roles, team member experience). When the system predicts team performance that could potentially negatively impact the quality of the article, it will automatically act to suggest changes to the current team or process to improve the quality of the article. There are four main phases in the project.
PI:
Nicola Banovic
Semi-Smooth and Variational Methods for Real-Time Dynamic Optimization
Project Abstract/Statement of Work:
Model Predictive Control’s capability to enforce safety constraints online makes it a strong candidate for autonomous vehicle control. Deploying real-time optimization (RTO) and Model Predictive Control (MPC) on real systems is dependent on the availability of fast and reliable methods for solving the underlying optimal control problems (OCPs), which may be constrained and/or nonlinear, at each sampling instance. This is a significant technical challenge as onboard computing power is limited and sampling rates for automotive and robotic systems are typically in the range of 10 – 1000 Hz. General purpose optimizers may not be able to meet these requirements and hence it is necessary to exploit structures inherent to RTO problems in developing solvers tailored for real-time applications.
An important class of optimization problems are convex quadratic programs (QPs). The optimal control problem arising from MPC for systems with linear dynamics and quadratic cost functions can be written as a convex quadratic program. In addition, many methods for nonlinear optimization, in particular sequential quadratic programming and its real-time variants, require the solution of a sequence of convex QPs. As a result, the ability to efficiently solve QPs is a key enabling technology for RTO.
Research Objective:
The research objective is to develop, implement, and validate novel and effective numerical methods for solving real-time optimization problems in support of TRI’s Guardian thrust. In particular, the algorithms developed in the proposed research will support TRI in applying MPC to control of vehicle dynamics as part of their autonomous vehicle development programme. The PIs research group is developing optimization techniques based on semismooth calculus and variational methods which combine the best aspects of Active Set and Interior Point methods. In particular, these techniques can exploit both sequential and internal structure of the associated problems. The implementation of MPC at kHz sampling rates using FBRS, an early version of these methods, has been experimentally demonstrated for an engine control application. In addition, a preliminary version of the FBRS algorithm has been successfully implemented and demonstrated on TRI’s autonomous vehicle by TRI.
PI:
Ilya Kolmanovsky
Developing and Simulating Pedestrian-Related Corner Case Scenarios for Testing Self-Driving Cars
Project Abstract/Statement of Work:
In this proposal we aim to develop pedestrian-related corner case scenarios and Carla-simulations for testing self-driving cars using existing large-scale naturalistic driving data, crash data and observation data.
PI and Co-PI:
Shan Bao
Aditi Misra