Kimi K3.1 Pro on iac-bench 2026-10

Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings.

Summary

Kimi K3.1 Pro places sixth of 16 by composite (86.5), the highest open-weight model, with a 95% rank interval of 3–9. It is third on the mechanical score alone (85.6, behind Claude Fable 5.2 and Claude Opus 5.5) and ninth on the judge score (88.6). In the bootstrap it finishes ahead of Claude Sonnet 5.5 in 54% of resamples, ahead of Claude Fable 5.1 in 35%, GPT-6.1 Sol in 33% and GPT-6 Astra in 27%, and ahead of Kimi K3 in more than 99%. All 9 attempts cost $3.52, about the same as Claude Opus 5.5 ($3.58), against $1.52 for Sonnet, $1.02 for Sol and $3.96 for Kimi K3.

All 9 attempts planned, with no security findings. Kimi K3 had four findings and one failed gate. S3 (G001) is its strongest fixture, tied fourth in the field by median composite (85.6, with Claude Fable 5.1). App Service (G003) has its highest median (87.8) but places eighth, and Cloud SQL (G002) places seventh (86.0, up from 79.2 for Kimi K3).

Scores

Rubric items not at full credit, counted over attempts that planned: G001: A5 3/3, H12 3/3, H13 3/3, H15 3/3; G002: A8 1/3, B1 3/3, H4 3/3, H5 3/3, H9 1/3, H11 2/3; G003: A7 3/3. Definitions are in Scoring criteria.

Strengths

• The Cloud SQL security gap of Kimi K3 is closed. ssl_mode = "ENCRYPTED_ONLY" is a literal in 3/3 attempts, next to ipv4_enabled = false (H2 passes 3/3; Kimi K3 failed it 3/3 by validating an input that admitted unencrypted connections). No S3 attempt offers public reads (H1, which Kimi K3 lost in 1/3).
• Connection auditing and forced IAM login. Every Cloud SQL module merges consumer flags into a baseline and then re-applies cloudsql.iam_authentication = "on" last, so it cannot be overridden. log_connections and log_disconnections are on in 3/3 (E8, which Kimi K3 misses in 3/3).
• Hardcoded App Service posture on the app and the slot. Both basic-auth publishing flags off, HTTPS-only and TLS 1.2 in 3/3, and every attempt plans: the staging slot sets health_check_eviction_time_in_min, the pairing that failed Kimi K3's gate. Autoscale, WEBSITES_PORT, metric alerts and AcrPull (E1–E4) appear in 3/3; Key Vault references (E5) in none.
• Tighter provider constraints. B1 is lost in 3 of 9 attempts, all on Cloud SQL, against 6 of 9 for Kimi K3 and Claude Fable 5.1 and 5 of 9 for GPT-6.1 Sol.

Weaknesses

• The largest App Service interfaces in the field: 50, 51 and 54 inputs against a budget of 20, so A7 scores zero in 3/3. Kimi K3 used 20–34, and the next-largest median is Gemini 3.6 Flash at 40. Many are attribute pass-throughs (ip_restrictions, vnet_route_all_enabled, container_start_time_limit). Interface design 1 in 6 of 9 readings.
• Duplicate and conflicting registry inputs. container_registry_id beside container_registry_url in 3/3, and a managed identity's client ID beside its principal ID in e2. A registry password is accepted as an alternative to AcrPull in 3/3. Data sources vs inputs 1 in 8 of 9 readings, Sensitive values 1 in 6.
• Cloud SQL regressions against Kimi K3:
  • service-account users keep the .gserviceaccount.com suffix, which Cloud SQL rejects, in 3/3 (Kimi K3 trims it in 3/3). The e1 and e3 READMEs document the full email as the username: Documentation 0 in 5 of 9 readings;
  • backups and point-in-time recovery are always on with no opt-out (H4/H5 at 0.75, 3/3), where Kimi K3 offered one;
  • a customer-managed key can be supplied in 1/3 (E2), against 3/3 for Kimi K3.
• Carried over from Kimi K3: transaction-log retention capped at 7 days in 3/3 while ENTERPRISE_PLUS (35 days) is offered; Cloud SQL provider ranges such as >= 5.20.0, < 7.0.0 that span two majors; no AWS provider block (H15, 3/3); S3 over budget with 12–13 inputs (A5, 3/3).
• Databases dropped on removal (H11 lost 2/3). e1 and e3 set no deletion_policy. e2 defaults it to ABANDON through an optional() attribute of the databases map, which earns full credit.
• S3 logging that cannot deliver (H13, 3/3). e1 and e3 write to a caller-supplied bucket. e2 creates its own log bucket, but the policy grants logging.amazonaws.com instead of logging.s3.amazonaws.com (excerpt below).

Characteristic patterns

• Fewer tokens, longer runs. 233k output tokens across 9 attempts against 316k for Kimi K3, yet a median of 31 minutes per App Service attempt, second only to GLM-5.3. It was served through OpenRouter pinned to Moonshot AI's own endpoint; Kimi K3 was routed without a pinned host, so timings are not directly comparable.
• Identity-first Cloud SQL, as in Kimi K3: private IP only, IAM database login, workload identities and Query Insights in 3/3. e2 also grants cloudsql.instanceUser and cloudsql.client to each IAM user (E6), which Kimi K3 never does. Validation stays as dense as Kimi K3's: 20–21 blocks per Cloud SQL module, 22–28 per App Service module.

Representative excerpt

G001/kimi-k3.1-pro-e2/main.tf lines 254–270

  statement {
    sid    = "AllowServerAccessLogDelivery"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["logging.amazonaws.com"]
    }

    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.logs[0].arn}/access-logs/*"]

    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.this.arn]
    }

The one S3 attempt that creates its own log bucket also writes a delivery grant for it, scoped to this bucket's ARN, but names the service principal logging.amazonaws.com. S3 server access logs are delivered by logging.s3.amazonaws.com, so the statement admits nothing and the rubric's delivery check (H13) fails.

Results · Methodology · Scoring criteria