Material Discovery Bench

All models are tasked with finding materials where $\kappa \ge$ $kappa, ${\epsilon }_0\le$ $epsilon, $Y\ge$ $youngs and $G\ge$ $shear. The pinned values are chosen to define the appropriate desired window, and the objective below is what the model is given.

  Propose dynamically stable, novel, BEOL-compatible crystalline materials that meet all of the following targets: thermal conductivity $kappa, STATIC dielectric constant $epsilon, Young's modulus $youngs, and shear modulus $shear. Every candidate must also come with a BEOL temperature and process compatible synthesis recipe that an expert review judges WOULD ATTEMPT — a candidate whose recipe is judged not worth attempting does not count. Novelty means that the material has never been deposited as a thin-film in BEOL compatible conditions in the reported literature.

We equipped the models with a set of tools to accomplish this task. These were designed to be similar to tools that would be available to a computational materials scientist. The tools we provided the model were:

• Web search (using Exa as the provider)
• A coding sandbox with python and bash capabilities and some relevant materials science packages — pymatgen¹, mp_api² and ASE³
• Machine learning based tools to compute 1) the dynamic stability, 2) the lattice thermal conductivity, 3) the static dielectric constant, and 4) the compliance tensor

The model was given no stopping condition, and proceeds until it hits an error or exhausts its total token budget of 100 million tokens. We use the AI Security Institute's open source Inspect framework⁴ to benchmark these models. In the next section we discuss in more detail the implemented tools used by the model to screen the proposed candidates. Then we discuss the synthesis scoring procedure.

Tools provided

Below, we list the machine learning based tools we used in this study to compute the various properties. We leverage machine learning interatomic potentials (MLIPs), in particular the universal point edge transformer (UPET) foundation machine learning model PET-MAD⁵. In future work direct density functional theory calculations can be incorporated in place of these MLIP calculations, or a hybrid approach can be taken.

Synthesis grading procedure

We provided some LLM generated recipes to human experts to grade independently. From these gradings we built rubrics to grade any proposed recipe. Each rubric is a penalty based format, where the recipe is deducted for incorrectly specifying or omitting any information. There are two types of penalties — critical and fixable. A critical penalty automatically guarantees the recipe would not be attempted. A judge is allowed to decide from the list of fixable penalties whether to attempt a recipe or mark it as unlikely to succeed. For our grader we use a worst of three GPT-5.6 Sol with OpenAI's web search capabilities, as it correlated best with human feedback.

Example Grading

Tool: 2.45 GHz microwave-plasma CVD (low-temperature, seeded).
- Substrate: 300 mm Si wafer with 100 nm PECVD SiO2; sputter a 3 nm AlN or 2 nm h-BN (0001)
  buffer to template hexagonal stacking.
- Seeding: spin-coat 5 nm detonation-nanodiamond colloid (0.1 g/L in DMSO), 60 s ultrasonic,
  N2 blow-dry; seed density >1e11 cm-2; O2-plasma descum 10 s before seeding.
- Gas: 0.4% CH4 in H2, 300 sccm total, 15 Torr; microwave power 600 W with pulsed duty cycle 30%
  to hold substrate at 380-400 °C (pyrometer + He-backside-cooled stage).
- Bias: -80 V DC pulsed substrate bias for the first 10 min (bias-enhanced nucleation), then float.
- Growth: 8 h → 80-150 nm continuous film; endpoint by in-situ laser reflectance interferometry.
- Post: 5 min H2 plasma at 350 °C for surface termination; optional 400 °C, 30 min N2 anneal.
- Phase ID: UV Raman (lonsdaleite 1315-1325 cm-1 vs cubic 1332 cm-1, no 1580 cm-1 G-band), GIXRD
  hexagonal (100)/(002)/(101) reflections absent in cubic diamond, cross-section TEM/SAED for
  ABAB stacking; graphitic/DLC contamination shown by D/G bands.

This is Claude Opus 5 proposing hexagonal diamond — lonsdaleite — by seeded microwave-plasma CVD, graded against the PECVD rubric. The grader returned WOULD NOT ATTEMPT: one critical penalty, which ends the judgement on its own, alongside five fixable ones.

The penalties applied, of the rubric's 17 criteria:

• There is no specific processing or choices leading to some desired phase formation. This is the critical one, and it decides the verdict on its own.
• If a plasma is used it is not well specified — source, power, gases, bias.
• Gas-flow or deposition sequence is missing or misspecified.
• Exhaust handling, including pumping and scrubbing or abatement, is not specified.
• The processing conditions can form the desired phase, but they are inadequate.
• Proposed characterization cannot validate the composition and proposed phase.

The critical penalty, in the grader's words:

and its overall summary:

That last clause is the distinction the critical class exists to draw. The recipe would deposit a film; it would not deposit this film, and tightening the parameters would not change that.

References

1. S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson and G. Ceder. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science 68, 314–319 (2013).
2. A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder and K. A. Persson. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials 1, 011002 (2013).
3. A. Hjorth Larsen et al. The atomic simulation environment — a Python library for working with atoms. Journal of Physics: Condensed Matter 29, 273002 (2017).
4. UK AI Security Institute. Inspect: An open-source framework for large language model evaluations. https://inspect.aisi.org.uk
5. A. Mazitov, F. Bigi, M. Kellner, P. Pegolo, D. Tisi, G. Fraux, S. Pozdnyakov, P. Loche and M. Ceriotti. PET-MAD, a lightweight universal interatomic potential for advanced materials modeling. Nature Communications 16, 10653 (2025). arXiv:2503.14118
6. C. Lin, J. Han, B. Xu and N. Marzari. First-principles phonon physics using the Pheasy code. arXiv:2508.01020 (2025).
7. A. Togo and I. Tanaka. First principles phonon calculations in materials science. Scripta Materialia 108, 1–5 (2015). See also A. Togo, First-principles phonon calculations with Phonopy and Phono3py, Journal of the Physical Society of Japan 92, 012001 (2023).
8. F. Zhou, W. Nielson, Y. Xia and V. Ozoliņš. Lattice anharmonicity and thermal conductivity from compressive sensing of first-principles calculations. Physical Review Letters 113, 185501 (2014). See also Compressive sensing lattice dynamics. II. Efficient phonon calculations and long-range interactions, Physical Review B 100, 184309 (2019).
9. A. Togo, L. Chaput and I. Tanaka. Distributions of phonon lifetimes in Brillouin zones. Physical Review B 91, 094306 (2015).
10. K. Yan, A. Saxton, X. Qian, X. Qian and S. Ji. A space group symmetry informed network for O(3) equivariant crystal tensor prediction. International Conference on Machine Learning (ICML) 2024. arXiv:2406.12888
11. K. Choudhary, G. Cheon, E. Reed and F. Tavazza. High-throughput density functional perturbation theory and machine learning predictions of infrared, piezoelectric and dielectric responses. npj Computational Materials 6, 64 (2020).