Situational Awareness from Multimodal Input (SAMI)

Project Lead: N. Ashish (UCI)
Project Participants: UCI - C. Butts, R. Jain, D. Kalashnikov, S. Mehrotra, P. Smyth, N. Venkatasubramanian, U. Westermann, UCSD – R. Hegde, B.S. Manoj, S. Park, B. Rao, M. Trivedi, ImageCat - R. Eguchi, C. Huyck
Other Project Members:  8 Students, 3 Post-docs, 1 Programmer

Visit the SAMI Project Website

Project Summary:

Our objective in SAMI is to design and develop technologies that can create actionable situational awareness from the avalanche of heterogeneous multi-modal data streams (audio, speech, text, video, etc.) including human-generated input (e.g., first responders’ communications, field reports, etc.) during or after a disaster.  Such technologies are of profound importance to first responders since response activities that occur as the disaster unfolds are decision-centric and decisions in our view depend directly on the situational awareness available. Awareness of the situation (past, present, and predicted future) which constitutes information about people (their vulnerabilities, location, demographics), resources (food, water, shelter) and progression of the event and activities (plume spread, storm track, evacuation progress) as well as implications of actions or inactions are amongst the most important factors that influence the quality of such decisions and hence efficacy of the response.  From a technology perspective, we see limitations in two major areas for situational awareness, namely information and data management technology and in signal analysis, interpretation, and synthesis; we aim to significantly advance these technologies in SAMI. Our approach is based on the notion of events as fundamental building blocks in situation awareness applications. Our research and development efforts in SAMI will cover 3 areas - situational information management, signal analysis and synthesis of situational information, and also an analysis environment for SA applications.

Grand Challenge:

The grand challenge in this project is to develop general-purpose tools/technologies/methodologies for building situational awareness applications across a wide variety of domains. Today, such applications are built in-house using a variety of data and knowledge management technologies and signal analysis techniques integrated in an ad-hoc way.  Most existing situational awareness systems do not adequately separate "media-specific" analysis from "application-specific" analysis in a straightforward way.  Applications operate directly on signal inputs leading to complex designs, and rigid systems that cannot be easily extended with additional analysis or input from other modalities. Furthermore, the approach inhibits exploiting multimodality data or domain knowledge and context for situation understanding in any principled way.  The data model and abstraction provided by existing data management systems is at too low a level for representing and reasoning about real-world activities - steps necessary for building situational awareness applications.

Project Focus:

Our effort in SAMI focuses on three interrelated components of a situational awareness system. A system for data ingest that extracts, fuses, synthesizes, situational information from multimodal input; a situational information management system that models, represents activities and supports queries; and an situational analysis and visualization system.  In SAMI, we are undertaking an event-oriented approach to building situational awareness. Such an approach has several advantages: events provide a natural way to abstract situational information from lower-level signal data; it supports a clean separation between media-level and application level (semantic) events; it enables incorporation of semantics and context when analyzing multimodal data and reasoning about situations; and it provides a generalized abstraction for situation representation that can be used to build data management technology for situational awareness applications. Our research will explore how events can be used as a fundamental abstraction for each of these component systems. In the data ingest component, the focus will be on integrated analysis of text, video, and speech inputs. An event-based approach will be exploited to incorporate contextual information and domain knowledge in signal analysis. An event model and corresponding query language and analysis tools that can serve as general purpose technologies for building a variety of situational awareness applications will be explored. Using the above research, we will develop two situational awareness artifacts. A smart reconnaissance system will be developed that realizes the “humans-as-sensors” concept from multimodal human-generated input.  Such a system will be demonstrated on data from specific disasters (e.g., Hurricane Katrina) for which we have collected datasets and data streams (including speech input from field-level observers). The other system is an integrated information dashboard that supports monitoring and analysis of dynamic & evolving large-scale crisis activities by providing seamless access to situational information spread over variety of information sources (human-as-sensors, field observations, news, simulations, crisis site input, etc.). Incorporating speech input from the public in the form of incident reports, including models to associate reliability with such input, will be an important component of such a system.  We envision multiple uses of such a system such as in an emergency operations center or in the form of a public information portal. 

Expected Results and Artifacts:

We expect two major scientific achievements: (1) an event-oriented situational data management system that seamlessly represent activities (their spatial, temporal properties, associated entities, and events) and supports languages/mechanisms/tools to build situational awareness applications, (2) a robust approach to signal analysis, interpretation, and synthesis of situational information based on event abstraction.  We expect to develop two artifacts -- an information reconnaissance system for disaster data ingest, and an integrated situational information dashboard that aids decision making.  We expect to gain valuable insight into disaster response information and awareness needs through such application development.

Plans for Broader Impact and Outreach: 

SAMI has the potential to significantly improve crisis response by providing decision-makers access to accurate, timely and reliable information about crisis. The biggest impact of SAMI is to first-responders and response organizations in the form of a decision-aid tools that provide better situational awareness.  The next generation situational awareness tools incorporate multimodal inputs, in particular, human generated inputs from crisis workers and citizen journalists. Our primary plans are to engage first responder organizations in the context of artifacts. We have begun interactions with the City of Ontario Fire Department who is in the process of building new state of the art Emergency Operations Center for the City of Ontario. In partnership with the City of Ontario officials, we will be co-developing a portal based information dashboard for specific EOC personnel (e.g. information officers).  Our initial step at this is to develop a portal-based dashboard suited for general public.  An early adopter of the smart reconnaissance system described earlier is Caltrans, California’s state government entity for transportation.  Finally, ImageCat, a key institution in the SAMI effort is involved in several field work related efforts to obtain reconnaissance subsequent to the disaster including recent disasters such as Hurricanes Katrina, Rita, and Charley, the Southeast Asian Tsunami and the Bam earthquake in Iran.  The smart reconnaissance system will be used in several future disasters to enable easier ingest of situational information.  The technology we are aiming to build is general purpose and we see numerous potential uses for this general situational awareness technology in other domains besides disaster response.