2025 · ICML 2025 — DataWorld: Unifying Data Curation Frameworks Across Domains workshop · Preprint
Beyond Atomic Geometry Representations in Materials Science: A Human-in-the-Loop Multimodal Framework
Can Polat , Hasan Kurban , Erchin Serpedin , Mustafa KurbanResearch highlights
MCS-Set: adding context to crystal representations
Combine atomic structures, images, and structured annotations for materials prediction and constrained generation.
Why we investigated it
Geometry-only datasets leave out context needed to evaluate how models use visual and textual information.
What the work contributes
A human-in-the-loop workflow creates MultiCrystalSpectrumSet with lattice and coordination annotations.
What the evidence shows
The reported evaluations reveal modality-specific performance gaps and show why annotation quality matters for generalization.
Build on this work
Use the dataset and framework described in the paper to compare multimodal property prediction and partially constrained crystal generation.
- Multimodal materials
- Data curation
- Crystal generation
- Human-in-the-loop learning
Abstract
Most materials science datasets are limited to atomic geometries (e.g., XYZ files), restricting their utility for multimodal learning and comprehensive data-centric analysis. These constraints have historically impeded the adoption of advanced machine learning techniques in the field. This work introduces MultiCrystalSpectrumSet (MCS-Set), a curated framework that expands materials datasets by integrating atomic structures with 2D projections and structured textual annotations, including lattice parameters and coordination metrics. MCS-Set enables two key tasks: (1) multimodal property and summary prediction, and (2) constrained crystal generation with partial cluster supervision. Leveraging a human-in-the-loop pipeline, MCS-Set combines domain expertise with standardized descriptors for high-quality annotation. Evaluations using state-of-the-art language and vision-language models reveal substantial modality-specific performance gaps and highlight the importance of annotation quality for generalization. MCS-Set offers a foundation for benchmarking multimodal models, advancing annotation practices, and promoting accessible, versatile materials science datasets. The dataset and implementations are available at https://github.com/KurbanIntelligenceLab/MultiCrystalSpectrumSet.
Cite
@misc{polat2025beyond,
title = {Beyond Atomic Geometry Representations in Materials Science: A Human-in-the-Loop Multimodal Framework},
author = {Can Polat and Hasan Kurban and Erchin Serpedin and Mustafa Kurban},
year = 2025,
journal = {ICML 2025 — DataWorld: Unifying Data Curation Frameworks Across Domains workshop},
doi = {10.48550/arxiv.2506.00302},
url = {https://arxiv.org/pdf/2506.00302},
}