MUSE: Evolving Grounded Multimodal Skills
for Cross-Episode Agent Improvement
From experience to reusable skills.
Average accuracy gain
GPT-5.4 · 72.52% → 87.26%MathVerse accuracy gain
GPT-5.4 · 62.10% → 93.40%Average accuracy
Claude-Sonnet-4.6 · four benchmarksMethod & evaluation details
Build the library
Independently solved, error-free seed trajectories with correct answers are distilled into structured profiles: applicability, visual hints, reasoning hints, verifier hints, and retrieval descriptors.
Reuse on new tasks
The main setting uses 20 labeled seeds, 20 skill profiles, and 20 offline rounds. The library is frozen before evaluation. Reuse is checked against the current image, without evaluation labels or model parameter updates.
Results
Five backbones on four benchmarks; two 8B models on M3CoT.
Accuracy (%)
Full comparison table
Mean accuracy over three independent runs. “Avg.” is the unweighted mean across four benchmarks.
Baselines follow the manuscript’s implementations. AMem retrieves evolution-stage memory snippets.
Paper-reported results · Table 1 & Appendix ↗
One skill. A new image.
A counting procedure transfers to a new chart.
What makes a skill useful?
Keep visual guidance.
Removing multimodal skill descriptions lowers accuracy from 83.1% to 71.3%, even when the solver still sees the image.
Refine beyond the first round.
Continued skill evolution raises accuracy from 79.0% to 93.4% compared with first-round profiles alone.
Evolve with the domain.
Continued evolution outperforms freezing after Commonsense on the following Science and Mathematics stages.