Qwen-VL for Angiography

Term definitions12 terms

ARCADE

Public X-ray angiography challenge dataset. This project uses the SYNTAX vessel-segment track, not stenosis.

Arterio

Coronary angiography labeling platform at Gdańsk University of Technology; this project supplies vessel masks for its labeling / doctor-assist flow.

clDice

Topology-aware Dice on vessel skeletons (Shit et al., CVPR 2021); flags thin-structure breaks area Dice can miss.

DeepStack

Last hidden state plus ViT layers 8/16/24, fused by learnable softmax weights before the decoder.

Dice

Mask overlap 2|P∩G|/(|P|+|G|); primary training/selection metric under severe foreground imbalance.

LoRA

Low-rank adapters on vision qkv/MLP layers so the 8B tower can adapt without full fine-tuning.

Qwen3-VL

Qwen/Qwen3-VL-8B-Instruct. Its vision tower is used as a feature extractor.

SegDecoder

CNN that maps 4096-ch 16×16 features through five upsample stages to 512×512 vessel logits.

SYNTAX

ARCADE track labeling coronary segments by SYNTAX numbering; merged here into one binary vessel class.

U-Net

ResNet-34 ImageNet encoder–decoder baseline (segmentation-models-pytorch); comparison only, not the Arterio serve path.

ViT

Vision Transformer – the Qwen vision tower’s patch encoder; intermediate layers feed DeepStack.

XCA

X-ray coronary angiography – 512×512 grayscale projection frames of contrast-filled coronaries (ARCADE).

Teaching a Multimodal LLM's Vision Tower to Draw Coronary Vessels

The problem

Coronary angiograms (X-ray coronary angiography, or XCA) are read constantly in cardiology, and a lot of downstream decision-support work – stenosis detection, vessel-health scoring, doctor-assist overlays – depends on having a clean pixel mask of the vessel tree for every frame. Right now that mask mostly comes from a human tracing it by hand. That does not scale, and it is the bottleneck we set out to remove: given an XCA frame, produce a binary vessel mask automatically, accurately enough to be useful as a labeling aid.

We built this inside the Arterio ecosystem, where the near-term goal is an online labeling path: an operator (or eventually a doctor-assist flow) sends a frame, gets a mask back, and moves on. That framing matters for everything downstream – it's why we cared more about not missing thin vessels than about raw pixel-perfect accuracy, and why a heavier, slower model was still an acceptable choice if it labeled well.

Authors

Four of us at Gdańsk University of Technology: myself (Jan Bancerewicz), Franciszek Borys, Julia Augustyniak, and Patryk Lewandowski, coordinated by Ph.D. Tomasz Dziubich. Graded 5.0.

Two paths, one question

We trained two very different models on the same ARCADE SYNTAX split so we could compare them:

• State-of-the-art baseline – a ResNet-34 U-Net, the kind of purpose-built segmenter you'd reach for by default. (This is not a language model. It is a traditional computer vision model.)
• The model we actually want to ship – a LoRA-adapted Qwen3-VL vision tower with a small (~6M-parameter) SegDecoder bolted on top.

We didn't pick Qwen because it segments well out of the box – it doesn't, it generates text. We picked it because we wanted to give a try to emerging technologies know how close a generalist vision-language backbone could get to a purpose-built segmenter if we adapted it properly. The U-Net's job in this project is entirely to answer that question, not to be the thing we serve.

Why the naive approach fails

Qwen3-VL was never trained to output masks. Our first real bug came from wiring the segmentation decoder to the wrong Hugging Face class (AutoModel), which left the vision MLP/merger randomly initialized – this was a block for a while,however, we managed to identify the issue and fix it.

Getting dense prediction out of a model like this means reshaping model.visual features into a fixed spatial grid, and using a loss that respects how sparse the foreground actually is – vessels cover roughly 2–5% of pixels. We settled on 0.5·BCE + 0.5·Dice with the decoder's output bias initialized to logit(0.1), so training doesn't spend its first epochs just learning "predict background."

How we solved the problem

Architecture

Canonical flow: Qwen LoRA+DeepStack, trained once, then run online-style in a single stage. U-Net is comparison-only; the hybrid refiner is a documented failed experiment, not a serving option.

Pipeline in code

• convert_mask.py merges the COCO-style SYNTAX polygons into binary vessel masks.
• dataset.py / augmentations.py feed 512×512 RGB frames.
• train_qwen_seg_new.py discards the language model, runs the vision tower (patch 16, merge 2 → 256 tokens / 16×16), optionally fuses the final hidden state with ViT layers 8/16/24 via DeepStack, then upsamples through SegDecoder.
• train_unet.py trains the ImageNet-pretrained ResNet-34 baseline under the identical loss, so the comparison is fair.
• qwen_unet_pipeline.py was our hybrid experiment: soft Qwen mask concatenated with RGB, fed into a U-Net refiner.
• evaluate.py adds clDice on top of area-based metrics; inference_qwen_unet.py and generate_plots.py produce panels and figures.

One asymmetry worth flagging: Qwen trains unfairly on compute (8B parameters, batch size 1, two 16 GB cards) but fairly on loss, its loss function remains consistent, making the ablation numbers directly comparable on the variable we controlled.

Developement map

Start: ARCADE frames and polygons, plus a classical U-Net we could already train. Qwen was already locked in as the multimodal backbone for Arterio labeling – but not yet a competitive pixel segmenter.

Along the way: U-Net's test Dice landed at 0.7955 and stayed there. Qwen started at a frozen last-hidden-state Dice of 0.619, climbed to 0.672 with DeepStack, and reached 0.758 with LoRA added on top. A hybrid refiner looked genuinely strong on validation (0.818) – and then fell apart at test time, actually hurting the teacher model it was supposed to improve (Qwen-only 0.761 → refined 0.696).

End: a reproducible Qwen LoRA+DeepStack pipeline producing masks for Arterio's online labeling flow. U-Net remains the published accuracy comparison, not the serving path. Weights are not redistributed – only metrics and plots are public. The group deliverable is complete; integration with Arterio's doctor-assist system is the next phase.

Results

ARCADE SYNTAX test set, N = 300 frames; hardware: 2× RTX 4080 SUPER 16 GB.

U-Net wins on pure accuracy (0.795 vs 0.758), but the gap is almost entirely recall (0.713 vs 0.761), with identical precision (0.835 both). Qwen misses thin distal branches more than it invents vessels that aren't there – for a labeling tool, that's the failure mode we'd rather have. DeepStack alone added +0.053 Dice; LoRA added +0.086 on top of that.

Takeaway: intermediate ViT features plus LoRA are what actually make the Qwen path viable. Cascade refinement failed because it never saw the teacher's real test-time errors – more on that below.

Crucial decisions

1. Binary SYNTAX merge. Going multi-class would have buried the real question – can this vision tower segment at all – under a much harder problem.
2. DeepStack as four fused scalars. Worth +5.3 Dice on its own; the early ViT layers preserve thin-structure detail the last layer alone loses.
3. LoRA at 0.1× the decoder's learning rate. Worth +8.6 Dice without a full 8B fine-tune on two 16 GB cards.
4. clDice in evaluation. Area-based Dice can look fine while missing topology breaks that actually matter clinically.

Issues with the Qwen-based approach

The cascade refiner was trained on the teacher's best train-set masks, so it never encountered the kind of errors Qwen actually makes at test time – it learned to thicken and hallucinate false positives instead of correcting them. Validation Dice for the hybrid was inflated by threshold and curriculum choices that didn't generalize. Errors cluster around catheters and low-contrast distal branches.

Lessons

1. Evolution: data and masks → U-Net baseline → Qwen frozen → DeepStack → LoRA → hybrid (negative result) → Integration with Arterio Platform.
2. Trade-offs made: constrained the task to binary segmentation, trained in bf16, applied PEFT to satisfy memory constraints, and adopted hybrid parameters that yielded suboptimal results.
3. What worked: a rigorous loss comparison against U-Net, monotonic ablations, standardized JSON and plot logging, and topology-aware evaluation.
4. Friction points: cascade trained on the teacher's best-case outputs, non-transferable validation threshold search, GPU pinning issues, and evolving specifications.
5. Next time: abandon the cascade approach or train it on actual teacher failure modes; freeze the evaluation protocol before tuning thresholds; run integration sanity tests; log latency and VRAM footprint prior to targeting production SLAs.
6. Core insight: a refiner must learn a teacher's failure distribution, not its optimal outputs.