GPT-6.1 Sol on iac-bench 2026-10

Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings.

Summary

GPT-6.1 Sol places third of 16 by composite (87.2) and sits in the second group with Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra, Kimi K3.1 Pro and Claude Sonnet 5.5, with a 95% rank interval of 2–8. In the bootstrap it finishes ahead of Astra in 49% of resamples, ahead of Fable 5.1 in 54%, ahead of Sonnet in 71% and ahead of Kimi K3.1 Pro in 67%. It is the cheapest model in that group: $1.02 for all 9 attempts, against $1.52 for Sonnet and $7.26 for Astra.

All 9 attempts planned, with no security findings. App Service (G003) is its strongest fixture, third in the field by median composite. S3 (G001) is eleventh.

Scores

Rubric items not at full credit, counted over attempts that planned: G001: A5 1/3, B1 2/3, H6 2/3, H7 2/3, H9 2/3, H12 3/3, H13 3/3, H14 1/3, H15 1/3; G002: B1 3/3, H4 3/3, H5 3/3, H11 3/3; G003: B2 2/3. Definitions are in Scoring criteria.

Strengths

• Security posture fixed in code in all 9 modules. On Cloud SQL this includes connector_enforcement = "REQUIRED" and validation that rejects overrides of IAM authentication. On App Service, both basic-auth publishing flags are off on the app and the slot; Claude Sonnet 5.5 leaves them on in 2/3 (H2).
• Login grants for IAM users (3/3). Every Cloud SQL attempt grants both cloudsql.client and cloudsql.instanceUser (E6), the grant GPT-6 Luna omits.
• Managed-identity image pulls (3/3, E4). AcrPull goes to the identities the app and the staging slot pull with, where GPT-6 Luna uses registry passwords.
• Smaller interfaces than Claude Sonnet 5.5: median 6, 17 and 17 inputs against 10, 26 and 27. It stays within the App Service budget (A7), which Sonnet exceeds in 3/3.

Weaknesses

• S3 retention and logging:
  • no bucket policy admits the logging service (H13, 3/3);
  • access logging offered in one attempt only (H9 lost 2/3);
  • expiration and noncurrent-version rules in one attempt only (H6, H7 lost 2/3). The other two expire only incomplete multipart uploads.
• Databases dropped on removal (H11, 3/3). No google_sql_database sets deletion_policy. Backups and point-in-time recovery cannot be turned off (H4/H5 at 0.75 in 3/3).
• Provider constraints that admit a new major in 5 of 9 attempts (B1), including >= 6.0, < 8.0 in all three Cloud SQL modules.
• Raw CORS on S3. cors_rules mirrors the provider's cors_rule block. Interface design 1 in 6 of 9 S3 readings, each citing it.
• Duplicate registry inputs on App Service, as with GPT-6 Astra: an ACR resource ID and a registry_url that can disagree. Data sources vs inputs 1 in 8 of 9 readings.
• A latent type error in G002 e3: iam_principals passes IAM_USER-style values to google_sql_user.type, which expects CLOUD_IAM_USER. The judge caught it (Variables and validation 1). The rubric did not, because the sweep does not populate collection inputs.

Characteristic patterns

• Optional features behind object-typed inputs. Every App Service attempt turns zone balancing on through attributes of a capacity object, and the S3 e3 attempt enables access logging when an access_logging object is supplied. The rubric credits both (H7 3/3, H9 in e3).
• A TLS-deny statement that exempts AWS service principals in 3/3 S3 attempts (excerpt below).
• Slot warm-up wiring. Two App Service attempts point WEBSITE_SWAP_WARMUP_PING_PATH at the health-check path, so a swap waits for the staging slot to respond.

Representative excerpt

G001/gpt-6.1-sol-e1/main.tf lines 63–75

    Statement = [{
      Sid       = "DenyInsecureTransport"
      Effect    = "Deny"
      Principal = "*"
      Action    = "s3:*"
      Resource  = [aws_s3_bucket.assets.arn, "${aws_s3_bucket.assets.arn}/*"]
      Condition = {
        Bool = {
          "aws:SecureTransport"       = "false"
          "aws:PrincipalIsAWSService" = "false"
        }
      }
    }]

The TLS-deny statement carries a second condition that exempts AWS service principals, with the comment "Exempt direct AWS service principals as their network context can be redacted." It appears in all three S3 attempts and in no other model's modules.

Results · Methodology · Scoring criteria