2026 · Digital Discovery
SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models
Can Polat , Mustafa Kurban , Erchin Serpedin , Hasan KurbanResearch highlights
SCALAR: evaluating reliability beyond prediction error
Measure hallucination, consistency, and reasoning as materials foundation models encounter larger structures.
Why we investigated it
A model can improve numerical accuracy while becoming less consistent or producing invalid answers.
What the work contributes
SCALAR evaluates prediction, physics-grounded reasoning prompts, and inverse retrieval across controlled nanoparticle sizes.
What the evidence shows
Explicit reasoning often reduces error and hallucination but can destabilize consistency or output validity. Effects vary across the evaluated models.
Build on this work
Use the SCALAR code and data to report several reliability measures under explicit scale shifts.
- Materials foundation models
- Structural hallucination
- Scientific reasoning
- Scale generalization
Abstract
Large language models are increasingly applied to materials science reasoning, yet their behavior under physically structured distribution shifts remains poorly understood. We introduce SCALAR (Structural Consistency And Logic Across Regimes), a benchmark for evaluating geometric scale generalization and its connection to structural hallucination, consistency, and reasoning in materials foundation models. Given canonical crystal representations, models must reason about derived nanoparticle structures obtained through supercell expansion and spherical carving from validated bulk crystal structures across length scales spanning 25 to 18,123 atoms: 1,743 base structures (83 materials × 21 radii), augmented by split-aware rotations to approximately 100,000 structures across training and evaluation splits. SCALAR defines three tasks: (i) CIF→property prediction; (ii) a Chain-of-Thought variant with explicit physics-grounded reasoning; and (iii) inverse retrieval identifying crystals from candidates given target properties. Outputs are evaluated via structured metrics capturing numeric error, hallucination, cross-prompt consistency, scale-trend reasoning, output validity, and retrieval fidelity. Experiments across diverse foundation models reveal large, model-dependent shifts under explicit reasoning, often reducing hallucination and error, but frequently destabilizing consistency or validity, demonstrating that scale generalization cannot be inferred from accuracy alone.
Cite
@article{polat2026scalar,
title = {SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models},
author = {Can Polat and Mustafa Kurban and Erchin Serpedin and Hasan Kurban},
year = 2026,
journal = {Digital Discovery},
doi = {10.1039/d6dd00393a},
url = {https://pubs.rsc.org/dd/article-pdf/doi/10.1039/D6DD00393A/14823982/d6dd00393a.pdf},
}