DBS Data Synchronization with Continuous Audio and Video for Psychiatric Biomarker Discovery

WSSFN 2025 Interim Meeting. Abstract 0026

Resumen

Introduction: Obsessive-compulsive disorder (OCD) and treatment-resistant bipolar depression (TRBD) constitute some of the most challenging psychiatric disorders to treat, as a significant proportion of patients demonstrate poor response to pharmacological strategies alone or in combination with psychoterapeutic approaches.1 Deep brain stimulation (DBS) is a potential therapeutic approach for select patients, although clinical outcomes remain inconsistent. To improve treatment efficacy, clinicians need to gain a deeper understanding of the neural activity and associated behavioral outcomes that correspond to distinct symptom profiles.2 One of the major hurdles in any effort to explore psychiatric neuromodulation is linking neural activity with behavior that we can observe. Examining correlated neural features with behaviorial information may provide biomarkers to inform DBS interventions.3-5 The present study introduces a platform that integrates video, audio, and neural data, including theta-band activity and spectrograms, all collected during real-world observations. This multimodal data approach enables clinicians to examine brain-behavior interactions with greater precision, facilitating the development of more individualized and effective treatment strategies.
Objective: to validate the feasibility of frame-level synchronization of video, audio, and intracranial signals during real-world clinical sessions, and to demonstrate its potential to support behavioral–neural biomarker discovery for DBS in OCD and TRBD.
Method: We implemented a hardware-based synchronization system designed for ecological clinical environments, that interfaces continuous high-definition video (FLIR Blackfly), high-fidelity audio, and direct neural recordings from implanted DBS systems. As shown in Figure 1, an Arduino-based controller generates a unique frame ID that increases sequentially, embedding it into one audio channel. This ID serves as the common temporal reference to align video and audio streams, which are later synchronized with the neural recordings. This platform aligns: Patient video recordings capturing spontaneous behavior and facial expressions; multichannel audio, including verbal responses and environmental context; intracranial electrophysiology from DBS electrodes. By extracting the frame ID from the audio channel, all modalities can be aligned with sub-frame accuracy. This approach remains robust to environmental noise and brief recording outages, which is crucial for long term clinical monitoring. The resulting output is processed and merged into a single synchronized outputs for subsequent behavioral annotation and biomarker analysis.
Results: In recent recordings, the system was able to constantly align continuous video, audio, and neural signals over sessions lasting up to one hour. The system was validated across five patient sessions, consistently achieving stable synchronization throughout. Each dataset contained thousands of precisely matched frame identifiers, ensuring accurate temporal alignment across all modalities. The resulting synchronized outputs (Figure 2) allowed frame-by-frame correlation of patient behavior, speech patterns, and neural activity without perceptible desynchronization. Across recordings, alignment error remained below one frame (<33 ms), ensuring temporal precision sufficient for clinical and research analysis.
Discussion: Synchronizing neural, behavioral, and contextual data enables real-time insight into brain–behavior relationships, helping identify biomarkers that can inform and refine DBS treatment. In the future, synchronized datasets could be integrated into clinical routines by allowing physicians to review neural activity alongside patient behavior and speech, providing a more objective complement to standard psychiatric assessments. One limitation is that the system has so far been validated in sessions of about one hour, which is well suited for proof-of-concept but more challenging to scale to recordings lasting several hours.
Conclusions: This platform demonstrates the feasibility of precise multimodal synchronization in DBS patients, providing clinicians with richer, time-aligned datasets to link behavior and neural activity. These results lay the groundwork for future clinical studies aimed at validating multimodal biomarkers and testing whether such data can inform adaptive DBS programming.