UC San Diego

In interactive machine learning, learners utilize data collected from interacting with the environment or with humans to better achieve their goals. Real-world applications often involve heterogeneous data sources, such as a large pool of human users with diverse interests or preferences, or non-stationary environments with distribution shifts. In this dissertation, we investigate interactive machine learning in the presence of heterogeneous data. In particular, we study when and how provably efficient learning can be achieved when the heterogeneous data exhibit structure.

In the first part, we study transfer learning in sequential decision-making. We consider a setting where learners are deployed to perform tasks in similar yet nonidentical multi-armed bandit environments. We study when and how knowledge acquired from one environment can be robustly transferred to others so as to improve the collective performance of the learners. We present two provably efficient algorithms that properly manage data collected across heterogeneous environments: one uses upper confidence bounds and the other is based on Thompson sampling. We then generalize the setting and certain results to multi-task reinforcement learning in tabular Markov decision processes.

In the second part, we study metric learning from crowdsourced preference comparisons. In particular, we consider the ideal point model in preference learning, where a user prefers an item over another if it is closer to their latent ideal point. While users may have individual preferences and distinct ideal points, our goal is to learn a common Mahalanobis distance, which provides a more accurate measure of "closeness" that aligns with human values, perception and preferences. We study when and how such a metric can be learned if we can query each user a few times, asking questions in the form of "Do you prefer item A or B?"