Ranked by Overall M̄ for the selected conditioning. Click a column header to sort. Scroll the table sideways for all metrics.
Overall M̄ by model
Mean morality metric across the four chains, conditioned vs unconditioned.
M̄ per model: conditioned vs unconditioned
Bars show the mean morality metric; whiskers show its confidence interval.
Chain-following Δ
Conditioned M̄ minus unconditioned M̄ for this chain (positive = the model moves toward the chain when told to follow it).
Table
Chain-following Δ across all chains
Per-Norm Breakdown
Conditioned vs Unconditioned
Chain-Following Delta
Implicit Chain Scores
Cross-Variant Consistency
Category Refusal Breakdown
Refusal Rate by Prompt Version
Chain-Following Δ by Prompt Version
Pick up to four models.
M̄ per chain
Chain-following Δ per chain
Headline metrics
Actions and consequences
Optimal action per chain
Cross-variant consistency on this scenario
Valid-choice T=0 agreement across the three prompt versions, pooled over chains and conditionings.
Model choices
Action Distribution (Conditioned)
Action Distribution (Unconditioned)
Modal Choice (Conditioned)
Modal Choice (Unconditioned)
Sample Justifications
Conditioned
Unconditioned
Data
All scenarios, raw model responses, aggregated metrics and paper figures are public in the Hugging Face dataset
. Each archive contains a MANIFEST.md listing its files, call counts, coverage and per-file SHA-256 checksums. Downloads open in a new tab.
Reproducing the figures
Unzip corda_v2_benchmark.zip, use its code/ folder as the working directory, copy the ICLR_2/ folder from corda_v2_results_raw.zip into it and follow code/README.md. No model calls are needed.
Citation
The paper is under double-blind review; authors are withheld.
About this benchmark
This benchmark evaluates how well large language models follow hierarchical moral reasoning chains: structured priority orderings of ethical norms (e.g. “avoid personal harm” ranks above “minimise total harm”).
Each model is tested under two conditions: conditioned (given an explicit moral chain to follow) and unconditioned (no guidance). The gap between them reveals whether a model can adapt its moral reasoning when instructed.
90 scenarios in six types (trolley variants, interpersonal, medical, resource allocation, animal, mixed robot/human), each in three prompt versions (canonical, novel, first-person advice). Every model answers each prompt once at T=0 and ten times at T=1 (11,880 calls per model).
Metrics
| Metric | Range | Meaning |
|---|---|---|
| Overall M̄ | 0 – 1 | Mean morality metric across all chains (1 = fully norm-compliant) |
| Adj. Overall M̄ | 0 – 1 | Mean adjusted morality metric, excluding norms with no active scenarios |
| Chain-following Δ | −1 – 1 | Conditioned M̄ minus unconditioned M̄ (positive = follows instructions) |
| Refusal Rate | 0 – 100% | Fraction of prompts where the model refused to choose |
| Sampling Consistency | 0 – 100% | Agreement between deterministic (T=0) and modal stochastic (T=1) choices |
| Valid-Choice Cross-Variant Consistency | 0 – 100% | Family-weighted T=0 agreement across all three variants, excluding triplets containing DEFAULT |
| Valid-Choice Triplet Coverage | 0 – 100% | Fraction of variant triplets containing three valid semantic choices |
| Canonical–Novel Agreement | 0 – 100% | Family-weighted valid-choice agreement; pairs containing DEFAULT are excluded |
| Novel–Advice Agreement | 0 – 100% | Family-weighted valid-choice agreement; pairs containing DEFAULT are excluded |
| Canonical–Advice Agreement | 0 – 100% | Family-weighted valid-choice agreement; pairs containing DEFAULT are excluded |
| Implicit Chain | chain name | Which moral chain the model naturally aligns with when unconditioned |
Family-weighted means each scenario (one scenario with its three prompt versions) contributes equally, however many chains it is scored under.
Adjusted M̄ (Adj. Overall M̄): some norms cannot be exercised in the current scenario set (e.g. avoid_personal_robot_harm never triggers). Adjusted M̄ sets those norms’ weights to zero so they do not artificially inflate scores. Use it when comparing models across chains with different coverage profiles.
Moral chains
Norms are listed from highest to lowest priority.
About this page
This is a static build of the benchmark leaderboard: every number was precomputed from the aggregated results with the benchmark’s own Python code, so no server is involved. Uploading new submissions for scoring needs the evaluation pipeline in corda_v2_benchmark.zip (see the Data tab).