Qwen3.8 2.4T-A95B on iac-bench 2026-10

Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings.

Summary

Qwen3.8 2.4T-A95B places eleventh of 16 by composite (81.6), between GLM-5.3 (82.5) and GPT-6 Luna (81.5), with a 95% rank interval of 7–13. Against its neighbours the bootstrap is close to a coin flip: it finishes ahead of Kimi K3 in 48% of resamples, GPT-6 Luna in 45% and GLM-5.3 in 43%, and ahead of Kimi K3.1 Pro in 2%. All 9 attempts cost $4.40, against $3.52 for Kimi K3.1 Pro, $3.96 for Kimi K3, $8.46 for GLM-5.3 and $0.88 for DeepSeek V4 Pro.

All 9 attempts planned. 4 of 9 carry a security finding: one on S3 and three on App Service. Cloud SQL (G002) is its strongest fixture, fifth in the field by median composite (89.0) and the highest of the open-weight models. App Service (G003) is its weakest, thirteenth (71.7).

Scores

Rubric items not at full credit, counted over attempts that planned: G001: A5 2/3, B1 3/3, H1 1/3, H9 2/3, H12 3/3, H13 3/3, H15 2/3; G002: A4 1/3, B1 2/3, H4 3/3, H5 3/3, H11 2/3; G003: A6 1/3, A7 3/3, H1 2/3, H2 2/3, H3 1/3, H4 1/3. Definitions are in Scoring criteria.

Strengths

• Cloud SQL posture fixed in code (3/3). ipv4_enabled = false and ssl_mode = "ENCRYPTED_ONLY" are literals, where Kimi K3 lets ssl_mode be weakened in 3/3. The private network is derived from environment (apps-<environment>).
• Identity-only database access. IAM database login (E3), declared users (E5) and workload identities (E7) in 3/3, with no password users. One attempt merges cloudsql.iam_authentication, log_connections and log_disconnections over the DBA's flags, rejects any input that sets them, and refuses to disable deletion protection in prod.
• S3 expertise. 6 of 7 expertise items in 3/3, including bucket-owner-enforced ownership (E1), a TLS-deny policy (E4) and ready-to-attach consumer policies (E9). S3 judge median 94.7.
• Every module plans (9/9), where Kimi K3 and GLM-5.3 each fail one App Service attempt. Autoscale rules (E1), the container port (E2) and a staging slot (H9) appear in 3/3.

Weaknesses

• Security findings: H1 on S3 (e1); on App Service H2 (e1), H1 and H2 (e2), H1 (e3). Kimi K3.1 Pro has none. The cause is under Characteristic patterns.
• Documentation that contradicts the code (Documentation 0 in 5 of 9 attempts; only DeepSeek V4 Pro has more, 6/9):
  • every App Service README describes deploying a new image to the staging slot and then swapping, but one image input feeds both the production app and the slot (3/3);
  • Cloud SQL descriptions ask for full service-account emails, which become google_sql_user names that PostgreSQL rejects until the .gserviceaccount.com suffix is trimmed (3/3). The judge caught it; the rubric did not.
• Large App Service interfaces: 27, 29 and 41 inputs against a budget of 20 (A7 partial or zero, 3/3). The 41-input attempt also exposes http2_enabled (A6).
• Gaps on S3 and Cloud SQL:
  • no bucket policy admits the logging service (H13, 3/3); logging (H9) and a provider block with a region input (H15) each appear in 1/3;
  • backups and point-in-time recovery cannot be turned off (H4/H5 at 0.75, 3/3), and backups never outlive the instance (E1, 0/3);
  • no deletion_policy in either attempt that declares databases (H11).
• Provider constraints that admit a new major in 5 of 9 attempts (B1): >= 5.0 or >= 6.0 on S3 (3/3), and >= 5.30.0, < 7.0.0 spanning two majors on Cloud SQL (2/3).

Characteristic patterns

• Wide spread within a fixture. Mechanical scores run 0.57–0.87 on S3 and 0.53–0.80 on App Service, against a spread of 0.05 on Cloud SQL, all from modules that plan. Each sample hardcodes a different subset of the security posture and exposes the rest:

  • S3 e1 adds allow_public_access, which turns off all four public-access-block settings (H1, 18 of 86 rubric weight). The other two S3 attempts keep them fixed;
  • App Service e1 hardcodes HTTPS-only, TLS 1.2 and FTP off, but never sets the basic-auth publishing flags, which default to on (H2);
  • e3 turns basic auth off and hardcodes TLS 1.2, then exposes https_only and the managed identity as inputs (H1, H4; excerpt below);
  • e2 sets neither basic-auth flag, leaves https_only off the slot, and exposes minimum_tls_version and ftps_state with validations that accept "1.0" and "AllAllowed" (H1, H2, H3).

  Kimi K3.1 Pro varies by at most 0.075 on any fixture. Kimi K3 and GLM-5.3 have wider App Service spreads, but theirs come from a failed plan.

• Long transcripts. Median 37,210 output tokens per attempt, second in the field after GLM-5.3. The longest (App Service e2: 68,729 tokens, $1.02, 41 inputs) is also its lowest-scoring attempt (59.4).

Representative excerpt

G003/qwen3.8-2.4t-a95b-e3/main.tf lines 48–67

  https_only                    = var.https_only
  public_network_access_enabled = var.public_network_access_enabled
  virtual_network_subnet_id     = var.virtual_network_subnet_id

  # HTTP APIs are stateless; session affinity only pins users to instances
  # that may be recycled, so keep it off.
  client_affinity_enabled = false

  # Images arrive from a container registry, never over password-based
  # publishing endpoints.
  ftp_publish_basic_authentication_enabled       = false
  webdeploy_publish_basic_authentication_enabled = false

  dynamic "identity" {
    for_each = var.managed_identity_enabled ? ["SystemAssigned"] : []

    content {
      type = identity.value
    }
  }

One App Service resource, two postures. Basic-auth publishing is hardcoded off with a comment explaining why, while HTTPS-only and the managed identity are left to inputs (https_only, managed_identity_enabled). Both default to the secure setting, but either can be switched off, which costs this attempt H1 and H4. Each App Service sample draws this line in a different place.

Results · Methodology · Scoring criteria