UC Riverside

We present a multi-disciplinary framework for motion modeling and recognition in machine vision. Building upon the neurobiological model of motion recognition, we propose computational equivalents for the Motion Energy and Form Pathways. We derive the Hamiltonian Energy Signature (HES) from first-principles in physics as the basis for the Motion Energy Pathway. The Form Pathway is modeled using existing low-level feature descriptors based on shape, appearance, and gradients. We propose an Integration methodology to combine both pathways using a variant of the Feature Integration and Biased Competition neurobiological models, which we implement via statistical hypothesis testing using the bootstrap. We also show the extensibility of our physics-based approach by proposing a physically-significant, compact representation for the gait of a person called the Gait Action Image (GAI), which is based on core physics principles employed in the HES formulation. We then show the generalizability of our neurobiologically-inspired integration framework by casting the GAI within this infrastructure.

Since Motion and image analysis are both important for activity recognition in video, we also develop a new approach that extends the Hamiltonian Monte Carlo (HMC) to allow us to simultaneously search over the combined motion and image space in a concerted manner using well-known Markov Chain Monte Carlo (MCMC) techniques. For motion analysis in video, we use tracks generated from the video to calculate the Hamiltonian equations of motion for the systems under study, thus utilizing analytical Hamiltonian dynamics to derive a physically significant HMC algorithm which can be used for activity analysis. We then use image analysis to help explore both the motion energy space and the image space by integrating the Hamiltonian energy-based approach with an image-based data-driven proposal to drive the HMC, thereby yielding a Data Driven HMC (DDHMC). We reduce the enormity of the search space by driving the Hamiltonian dynamics-based MCMC with image data in this DDHMC. We also develop the reverse algorithm, which uses motion energy proposals to search the image space. While HMC has been used in other contexts, this is possibly the first work that shows how it can be used for activity recognition in video, taking into account the image analysis results and using the physical motion information of the system. In addition, the DDHMC framework has potential application to other domains where statistical sampling techniques are useful, as we outline in the section on future work.

Experimental validation of the theory is provided on the well-known KTH, Weizmann, and USF Gait datasets with very promising results.