November changelog: Runtimes & OIDC cloud credentials, plus imports with Claude Code & opencode

Read more

LLM Rankings for Infrastructure as Code

Ten language models each wrote nine reusable OpenTofu modules for AWS, GCP and Azure. We scored all 90 with checks against real tofu plan output and an LLM judge calibrated against human grades.

Release
2026-10
Models
10
Modules scored
90
Run date
2026-10-22

Leaderboard

The overall score is 70% mechanical checks against real plans and 30% LLM judge. The top four have overlapping rank intervals, so read them as a group. Select a column heading to sort.

iac-bench 2026-10 leaderboard: overall score with 95% confidence interval, rank interval, worst attempt, mechanical and judge scores, median score per cloud, failed plans, security failures, time and cost per attempt.
1Claude Opus 5.5AnthropicClosed88.987.5–91.51–385.086.494.787.988.690.20/90/9243 s$0.40
2Claude Fable 5.1AnthropicClosed86.983.7–90.21–579.085.091.285.685.689.50/90/9302 s$1.55
3GPT-6 AstraOpenAIClosed86.786.0–89.81–572.083.294.772.891.395.90/90/9189 s$0.81
4Claude Sonnet 5.5AnthropicClosed86.184.8–88.22–581.682.594.784.890.682.80/92/9161 s$0.17
5Kimi K3Moonshot AIOpen82.457.3–84.75–925.380.686.081.179.286.91/94/9494 s$0.44
6GPT-6 LunaOpenAIClosed81.579.8–82.15–771.379.784.274.384.585.70/90/9144 s$0.005
7Gemini 3.1 ProGoogleClosed73.069.2–76.26–959.971.971.974.279.365.40/95/9168 s$0.25
8DeepSeek V4 ProDeepSeekOpen69.264.4–79.36–960.969.567.582.562.163.20/95/9322 s$0.098
9Claude Haiku 5.5AnthropicClosed68.066.1–90.61–925.359.291.287.826.889.42/91/9107 s$0.012
10Gemini 3.6 FlashGoogleClosed54.340.0–61.510–1015.050.962.362.849.251.01/98/9279 s$0.35

Worst attempt is the lowest score of a model's 9 attempts. Mechanical and Judge are each component scored on its own, on the same 0–100 scale. Cloud columns are the median of three attempts. Failed plans and security failures count attempts out of 9. Time is the median wall-clock time of an attempt. Cost is the average per attempt at list prices on 2026-10-23, excluding judging.

Which model should I use?

The top four are statistically close, so treat these as starting points, not verdicts. Each card links to the model report.

Score vs. cost

Generation cost varies by a factor of about 280 across the field and is only loosely related to score. Marked models are on the Pareto frontier: no other model beats them on both cost and score.

Scatter plot of overall score against generation cost on a log scale. The Pareto frontier runs through GPT-6 Luna, Claude Sonnet 5.5 and Claude Opus 5.5.
Overall score against the estimated cost of one attempt at each of the three briefs (log scale, list prices on 2026-10-23). The line is the Pareto frontier.
Overall score and estimated generation cost per attempt, cheapest first.
ModelCost per attemptOverallPareto frontier
GPT-6 Luna$0.00581.5On frontier
Claude Haiku 5.5$0.01268.0
DeepSeek V4 Pro$0.09869.2
Claude Sonnet 5.5$0.1786.1On frontier
Gemini 3.1 Pro$0.2573.0
Gemini 3.6 Flash$0.3554.3
Claude Opus 5.5$0.4088.9On frontier
Kimi K3$0.4482.4
GPT-6 Astra$0.8186.7
Claude Fable 5.1$1.5586.9

Best model by cloud

Each cloud had one brief. Scores are the median of three attempts per model.

AWS: S3 bucket

  1. 1.Claude Opus 5.587.9
  2. 2.Claude Haiku 5.587.8
  3. 3.Claude Fable 5.185.6
Field median
81.8
Most missed check
Access log delivery permission (H13): missed in 30 of 30 attempts

GCP: Cloud SQL for PostgreSQL

  1. 1.GPT-6 Astra91.3
  2. 2.Claude Sonnet 5.590.6
  3. 3.Claude Opus 5.588.6
Field median
82.2
Most missed check
Databases dropped on removal (H11): missed in 24 of 28 attempts

Azure: App Service

  1. 1.GPT-6 Astra95.9
  2. 2.Claude Opus 5.590.2
  3. 3.Claude Fable 5.189.5
Field median
86.2
Most missed check
Interface budget (A7): missed in 19 of 28 attempts

Where models struggle

Some weaknesses showed up across vendors, sizes and price points. Whichever model you use, these are the places to check its infrastructure code first.

Input validation
The weakest judged area. Only 32 of 90 attempts earned full credit for variables and validation, and every model fell short at least once. Some modules had no validation at all. More often, error-prone inputs such as SKUs, time windows and retention periods went unchecked, or two inputs could be set to contradict each other.
Documentation that is wrong
Nearly every module had documentation, but 26 of 90 attempts scored zero for it because something in it was false: a description promising behavior the code does not implement, a service limit stated wrong, or a comment the next line contradicts. Longer READMEs give the judge more claims to check.
The provider's quiet defaults
Models configure the obvious feature and miss the default or permission that makes it safe or working. No S3 attempt, out of 30, produced access logs S3 could deliver: 19 offered no logging, and the other 11 never gave S3 permission to write the logs. 24 of 28 Cloud SQL modules that planned delete a database's data when it is removed from the inputs. 10 of 28 App Service modules that planned left basic-auth publishing on, several of them right next to correct HTTPS and TLS settings.
Security as a setting
In 15 attempts the secure value was the default, but any consumer could change it through an input, for example a connection mode that also accepts unencrypted clients. The module is safe as written and unsafe as used.
Version constraints
Provider constraints that allow the next major version appeared in 19 of 30 AWS modules and 22 of 28 GCP modules, but only 2 of 28 Azure modules. The habit follows the provider, not the model.
Sizing the interface to the audience
Only about half of attempts earned full credit for interface design. The usual miss is handing settings that need service expertise, such as encryption algorithms or upload cleanup windows, to developers the brief describes as new to the service.

How the ranking works

Each model got the same three briefs of a few sentences, each naming a use case and an audience: an S3 bucket for application teams, a Cloud SQL for PostgreSQL instance for a DBA team, and an App Service web app for an SRE team. The briefs never list the security, durability or hygiene settings a senior reviewer expects, so the score measures what the model adds on its own.

Each model made three attempts per brief. A module that does not plan gets no mechanical credit and counts as a failed plan.

Mechanical rubric (70%)
Fixture-specific checks evaluated against real tofu plan output.
Judge (30%)
A calibrated LLM judge grading design and craft against a published quality doctrine.
Disclosure
Massdriver wrote the briefs, rubrics and doctrine. The judge is Claude Opus 5.5, from the same vendor as four of the ranked models. Ranked by the mechanical score alone, the ten models fall in the same order.

LLM rankings FAQ

In iac-bench 2026-10, Claude Opus 5.5 has the highest overall score (88.9). Claude Fable 5.1, GPT-6 Astra and Claude Sonnet 5.5 follow within three points, and their rank intervals overlap, so the four form a top group. The benchmark uses OpenTofu, the open-source fork of Terraform, which reads the same HCL. Terraform itself was not run.

Each cloud had one brief, so these are results for one service per cloud. On AWS (an S3 bucket), Claude Opus 5.5 (87.9) and Claude Haiku 5.5 (87.8) lead. On GCP (Cloud SQL for PostgreSQL), GPT-6 Astra (91.3) leads Claude Sonnet 5.5 (90.6). On Azure (App Service), GPT-6 Astra (95.9) leads Claude Opus 5.5 (90.2). GPT-6 Astra also has the lowest S3 score in the top group (72.8).

GPT-6 Luna scored 81.5 for about half a cent per attempt, with no security failures. Claude Sonnet 5.5 is the cheapest model in the top group, at about $0.17 per attempt. Claude Opus 5.5 has the highest score, at about $0.40 per attempt.

Kimi K3 places fifth (82.4), ahead of GPT-6 Luna and Gemini 3.1 Pro. DeepSeek V4 Pro places eighth (69.2). Both trail the four leaders.

Input validation and documentation are the weakest areas across every model: only 32 of 90 attempts fully validated their inputs, and 26 of 90 had documentation with a false statement in it. Models also miss provider defaults, such as the permission S3 needs to deliver access logs (missed by all 30 attempts). 25 of 90 attempts failed a mandatory security item. The section above, Where models struggle, has the details.

Not reliably. The briefs never mention security, and 25 of 90 attempts failed at least one mandatory security item: 6 of 30 S3 modules did not guarantee the public access block, 9 of 28 Cloud SQL modules that planned did not require encrypted connections, and 10 of 28 App Service modules that planned did not turn off basic-auth publishing. Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra and GPT-6 Luna had none in nine attempts each. Gemini 3.6 Flash failed in eight of nine.

A module must first plan with its own example inputs. Fixture-specific checks then read the plan and the source for security, data protection, fit for the audience and service expertise. They check what the module does, never what it names things. That mechanical score is 70%. The judge grades design and craft in 14 sections of a published doctrine, 0 to 2 each, for the other 30%. Each model's score per cloud is the median of its three attempts, and the overall score is the mean of the three.

For each fixture, the judge and a human graded the same 8 modules section by section, 112 grades each. The judge counts only if it matches the human on at least 6 of 8 modules in every section. Claude Opus 5.5 matched 93%, 97% and 98% of the grades on the three fixtures. A second judge, GPT-5.5, failed that bar on two fixtures and was not used.

The judge is Claude Opus 5.5, and it accounts for 30% of the score. Ranked by the mechanical checks alone, where no model grades another, the ten models fall in exactly the same order. Massdriver, which builds infrastructure tooling, wrote the briefs, rubrics and doctrine.

No. Every module was planned against real AWS, GCP and Azure accounts, and nothing was applied. Problems that only appear at apply time, such as a cloud API rejecting a value, are not scored.

Each provider's defaults, with no temperature, reasoning-effort or token-budget overrides. Every model got the same prompt, the same tools and the same limits: 150 messages and 30 minutes per attempt. Most providers serve these models under undated names, so the results are tied to the run date, 2026-10-22.

Three attempts per brief is enough to separate groups but not neighbors. Claude Opus 5.5 is the only model whose 95% rank interval (1–3) stays within the top three. Gemini 3.6 Flash ranks last in at least 95% of bootstrap draws.

Yes. The benchmark code, briefs, rubrics and judge doctrine are public on GitHub, tagged 2026-10, and the links at the bottom of this page go straight to them. The data release has every transcript, all 90 modules, every score and judge grade with its evidence, and the calibration. It is licensed CC BY 4.0, and the code at that tag reproduces every number on this page.

Read the full reports