Multimodal AI for Stimulation Detection in Newborn Resuscitation
Abstract
This master’s thesis investigates the development of a stimulationdetection system for neonatal resuscitation using deep neural networks.The objective was to create a reliable detection system utilizing videorecordings, acceleration data from the NeoBeat device [47], and annotatedfiles created with ELAN software. The study involved simulated datacollected under controlled conditions and real-world resuscitation videosfrom Haydom Lutheran Hospital in Tanzania.Advanced machine learning techniques were used to build models capableof real-time analysis. The models were trained on simulated resuscitationdata and validated on real-world data to ensure robustness. Transferlearning was applied to video data, while 2D-CNN and LSTM modelswere explored for acceleration data, with Optuna optimizing thehyperparameter search.Key findings include the impressive performance of the MoViNet modelintegrated with a custom-built 2D-CNN, achieving an accuracy of 0.95.The multimodal AI approach significantly improved detection accuracyand stability by combining video and acceleration data. However,challenges such as false positives during high movement periods indicatethe need for further refinement.Future work should incorporate real-world videos into the trainingdataset, expand detectable stimulation features, and optimize the modelsfor real-time deployment in clinical settings. This research demonstratesthe potential of deep learning to enhance neonatal resuscitation practices,aiming to reduce neonatal mortality and improve long-termdevelopmental outcomes.