Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM

Abstract

• we identify anatomical regions in chest X-rays to generate focused sentences that center on key visual elements
• the pre-trained LLM can generate structured chest X-ray reports tailored to prompted anatomical regions and clinical contexts

Introduction

• Automatically generating reports could reduce physician workload and diagnostic errors.
• Medical report generation issues
  • Structural deficiency
  • lack of interpretability and interactivity
• LLM guidde by anatomical regions and clinical contextual prompts to achieve high interpretability and interactivity
  • detect anatomical regions in chest X-rays to generate region-focused descriptions, establishing an anatomy foundation for structured report
  • our model incorporates clinical contextual information, including the patient’s medical history and the reason for examination, etc., typically provided by the physician
  • we utilize a large language model to integrate anatomical region descriptions, anatomical prompts, and clincial contextual prompts into a single anatomically based structured by coordinating and consolidating these data sources

Method

• We propose a structured report generation framework, guided by anatomy and clinical prompts, to simulate radiologist workflow.

  • Sentence Generator

    • First, we identify anatomical regions in chest X-rays and extract per-region features.
    • Faster R-CNN with a ResNet-50 backbone for anatomy detection and feature extraction

  • Sentence Generator

    • Then, a sentence generator produces region descriptions, forming the basis for the structured report.
    • 

  • Anatomy Prompts Generation

    • Concurrently, we generate anatomical prompts indicating sentence presence and abnormalities per region.
    • 
    • P1 and P2 are the anatomical location and abnormality prompts
    • Anatomy prompts generation
      • sentence detection, abnormal detection, and a prompts converter

  • Structured report generation

    • Finally, we integrate region descriptions, anatomical prompts, and clinical context from doctors into prompts for a large language model, it generates the final structured report.
    • clinical context P3(history, indications, reasons for examination)
    • 

Experiment

• MIMIC-CXR: chest radiograph images and reports