Skip to main content
eScholarship
Open Access Publications from the University of California

UC Santa Barbara

UC Santa Barbara Electronic Theses and Dissertations bannerUC Santa Barbara

Compressed Training for Uncertainty-Aware Compact Neural Networks

Abstract

The rising computational and memory demands of machine learning models, particularly in resource-constrained edge-device settings, motivate us to develop compressed models that reduce both training and inference costs. In this dissertation we develop several algorithms that advance the compressed training capabilities of low-rank tensor models and the uncertainty quantification capabilities of general compressed nonparametric models. First we introduce compressed low-rank tensorized training with rank reduction for the Tensor-Train format. This enables compressed tensorized training with further compression during the training phase. Then we improve our approach in two ways. First we improve scalability by introducing a variational inference algorithm tailored to the tensor learning problem and demonstrate its effectiveness compressing larger-scale models. Second we improve the generality of our approach by extending our tensorized training with rank reduction to all popular low-rank tensor formats. Finally, we introduce a general-purpose algorithm for compressed nonparametric Bayesian learning that can automatically determine the appropriate complexity of nonparametric distributional representations. This algorithm improves the computation/accuracy and storage/accuracy Pareto frontiers over state-of-the-art Bayesian learning approaches.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View