UC Santa Barbara

The rapid development of the current availability of advanced hardware and software is opening up new opportunities for digital creation every day. This situation provides great freedom for new artistic expressions with advanced audio, graphics, interface, and algorithms, including machine learning. However, while our nature is multimodal, these modalities in the digital domain are genuinely separate, and the computational platform allows innumerable varieties of linkages among them. For this reason, the holistic multimodal experience is highly dependent on the design and connection of different modalities. This dissertation explains the properties of coherent digital multimodal instruments and discusses their creative opportunity from the process of music composition and performance. The chapters introduce the related projects with their design process, demonstrating the role of crossmodal correspondence in scientific simulation and proposing a numerical method to evaluate the crossmodal correspondences using the correlation coefficient between the modalities. This dissertation aims to contribute to reorganizing the design process of multimodal instruments beyond the old and recent customs.