BGMGen (2024-01 Software Capstone Project)

Music Generation / Visual Language Models / Multimodal Learning

We present BGMGen, a novel framework for generating background music that seamlessly aligns with video content. BGMGen leverages multimodal learning models (CLIP/MusicGen) to conditionally create music based on the visual information in videos, specifically focusing on the emotional tone and human actions depicted. Our framework evaluates the congruence of the generated music with the video using the VAD (Valence-Arousal-Dominance) space, a vector space that effectively represents emotional states. By integrating visual cues and emotional context, BGMGen aims to enhance the overall audiovisual experience through precisely tailored background music.

Human Body Reconstruction (during internship in Innoyard)

3D Scanning / Depth Estimation / RGBD Odometry

Conducted research on 3D reconstruction using photos taken from mobile devices and contributed to developing a human body 3D scanner app. Additionally, I created an application for detecting plagiocephaly, designed for both marketing and data collection purposes.