Semantic segmentation of brain structures from medical images, in particular Magnetic Resonance Imaging (MRI), plays an important role in many neuroimaging applications. Deep learning based segmentation algorithms are now achieving state-of-the-art segmentation results but currently require large amounts of annotated data under predefined segmentation protocols and data inclusion/exclusion criteria. The rigidity of such approaches forbids natural interactions by humans and thus limits the usefulness for non-routine questions.
In this project, we will develop a novel AI agent able to segment neuroanatomical structures based on a textual prompt describing the structure to be segmented. Similar to how a senior clinician would provide feedback to a trainee, we will further allow providing the AI agent with additional textual information relating to the case at hand and any previous segmentation proposal by the AI agent.
Large language models have recently enabled agile human-AI interactions trough textual prompting but their translation to medical imaging question remains limited. A key challenge in designing vision-language models for volumetric neuroimaging tasks is the lack of pre-existing suitable foundation AI models. The project will take advantage of pre-trained medical language models as well as prior neuroanatomical knowledge captured through brain atlases, publications, and medical records to constrain the problem sufficiently. We will then develop latent image representations aligned with frozen latent representations extracted from medical language.
In year 1, the student will focus on assembling a collection of datasets of brain MRIs annotated according to a wide range of established protocols. A bespoke segmentation pipeline will be developed to handle the large number of potentially overlapping classes. In year 2, the focus will move to the development of a neuroimaging vision-language model. In year 3, agile text prompting and refinement will be designed. Year 4 will be dedicated to validation studies and thesis write-up.