Manual medical image segmentation is still a core task in many clinical workflows but remains tedious and cognitively demanding. Traditional 2D interfaces often fragment attention and do not match the precision or ergonomics needed for complex anatomical annotation tasks
In this project, we developed and evaluated an immersive segmentation platform using the Meta Quest 3 headset and the Logitech MX Ink stylus. The system supports 2D annotation of CT scans in a unified XR workspace, designed to mirror pen-on-paper feeling while offering 3D spatial interaction and volumetric visualization. A user study showed promising usability (SUS score of 66), with participants praising the intuitive stylus controls and reduced cognitive load during annotation.
Looking ahead, future work will focus on enhancing stylus precision, integrating AI-assisted features, and supporting collaborative use in surgical planning and medical education. These improvements aim to bridge the gap between conventional desktop tools and immersive clinical interfaces.
Publication
de Paiva, L.F., Luijten, G., Santos, A.S.F., Kim, M., Puladi, B., Kleesiek, J. and Egger, J., 2025. Beyond the Desktop: XR-Driven Segmentation with Meta Quest 3 and MX Ink. arXiv preprint arXiv:2506.04858.
Project Partners
- Institut für Künstliche Intelligenz in der Medizin (IKIM)
• Prof. Jan Egger
• Prof. Jens Kleesiek
• Dr. Moon Kim
• Lisle Faray de Paiva
• Ana Sofia Ferreira Santos
• Gijs Luijten - Logitech International S.A.
• Dr. Aidan Kehoe - Institute of Medical Informatics
• Dr. Behrus Puladi
Contact
Lisle Faray de Paiva (lisle.faraydepaiva@uk-essen.de)
