UC Riverside

Recently, there has been a surge of interest in modeling contextual information for various computer vision applications. Use of inter-object/inter-activity context has ushered in significant performance improvement over most classical video analysis approaches that independently estimate multiple variables of interest, which are, in fact, interrelated. In this work, we explore the problems of multi-camera data association, spatio-temporal cell tracking and activity forecasting, where the contextual relationship models could be extremely useful but are little studied in literature.

Most existing data association techniques focus on sequentially matching pairs of data-point sets and then repeating this process along space-time to achieve long-term correspondences. However, in multi-camera problems, simply combining the local pairwise association results often leads to inconsistencies over the global space-time horizons. We present an optimization framework to combine all pairwise data-association results over a network, which not only establishes global consistency but also improves pairwise accuracies. The proposed 'Network Consistent Data Association' (NCDA) method is also capable of handling the problem of variable number of data-points across different sets of instances in the network and its application is shown in the classic person re-identification problem.

In spatio-temporal cell tracking problem, 3D cells within tightly clustered tissues are imaged over time at various depths of the tissue. The objective is to associate these 2D cellular projections that often lack good feature quality. We exploit the tight spatial topology of the cells in a CRF model and obtain robust pairwise similarity measures between the 2D cell segments, which are further combined together via the NCDA method to yield consistent and accurate cell lineages. The estimated lineages are utilized in the proposed tessellation based reconstruction method for generating 3D structures of individual cells.

We also explore the activity forecasting problem in continuous videos. We model the simultaneous and/or sequential nature of human activities as contextual information on a graph and combine that with the interrelationship between static scene cues and dynamic target trajectories. The forecasting problem is then solved as inference problem on an MRF model defined on this graph and high accuracy is observed throughout our experiments.