UC Berkeley

Technological advances in communication and computing coupled with low costs of sensors have enabled large-scale deployment of video camera networks in our environment nowadays. They are increasingly being used by decision-makers for perceiving elements of an environment in real-time, comprehending their meaning and predicting their status in the near future. Such situational awareness is crucial in dynamic scenarios that arise in military command and control, facility security and emergency services such as policing and fire-fighting. While human perception driven situation awareness has worked well in constrained settings, it is unfortunately not scalable. Furthermore, scenarios such as security and surveillance commonly involve highly complex cognitive tasks that can quickly become monotonous and mentally taxing for human operators. In this thesis, we present new frameworks for automating the perception stage of situation awareness.

We begin this thesis with the development of a system that is capable of categorizing objects and landmarks in an efficient and distributed manner. Our system is designed to operate with net- works of wireless smart cameras for local perception, and a central station for global inference. We demonstrate that this decoupling of the algorithm pipeline can drastically minimize the power and bandwidth consumed by the wireless cameras. Further, we experimentally validate that our multiple-view inference framework can significantly improve the performance of object and land- mark categorization over traditional single-view settings.

In the second part of the thesis we extend our distributed object categorization framework to address the problem of automatic human activity detection and categorization. We are particularly interested in the development of rich representations for human motion that are invariant to perspective, scale and the speed at which actions are performed. We propose a generalized framework to perform spatiotemporal fusion of dynamic imagery from multiple wireless smart cameras, and validate the efficacy of our fusion framework on both publicly available, and novel datasets.

In most realistic scenarios that require situation awareness, objects and people occur in cluttered scenes and exhibit immense variability in their appearance, shape and pose. In the final part of this thesis, we analyze the interplay between computer vision tasks such as segmentation and categorization and present joint frameworks that significantly improve the performance of each task. Our experimental analysis demonstrates that detection and categorization hypotheses help provide good segmentation results, and that segmentation can be used to prune errors in the hypothesis.