edi-summ: multimodal clinical summarization

Overview

Co-authored EDI-Summ, an image-guided encoder-decoder model for multimodal clinical document summarization. The work was published as a long paper at ACL 2024.

My contribution

• Implemented image cross-attention in BART’s decoder.
• Profiled ResNet, VGG, and ViT image encoders and multimodal fusion choices.
• Ran ablations and failure analysis across three clinical benchmarks.
• Contributed to evaluation using ROUGE, BLEU, METEOR, statistical analysis, and expert review.

Architecture

clinical text → modality-aware encoder → BART decoder + image cross-attention → grounded summary

Result

EDI-Summ outperformed BART, GPT-4V, and multimodal baselines across the three evaluated benchmarks. The important result was not only higher aggregate scores, but evidence that decoder-side image grounding contributed useful clinical information.

Publication

From Sights to Insights: Towards Summarization of Multimodal Clinical Documents

Proceedings of ACL 2024, Volume 1: Long Papers, pages 13117–13129.