flake: CI required checks aggregator fails when gen job is cancelled

[ibetitsmike]

CI Run Link: https://github.com/coder/coder/actions/runs/18689201542

Branch: main
Commit: 86f0f39863a27040acd17dd6bc354cc6a430df7c (Steven Masley) — coder/coder@86f0f39

Summary:

• The "required" aggregator job failed because it detected a cancelled required check: gen.
• There were no failing tests; most test jobs succeeded. The gen job shows an explicit cancellation.

Evidence:

• required job log shows cancelled check and exits 1:

Checking required checks
- changes: success
- fmt: success
- lint: success
- gen: cancelled
- test-go-pg: success
- test-go-pg-17: success
- test-go-race-pg: success
- test-js: success
- test-e2e: success
- offlinedocs: success
- check-build: skipped

One of the required checks has failed or has been cancelled
##[error]Process completed with exit code 1.

• gen job log shows cancellation (no test failure):

##[error]The operation was canceled.
Post job cleanup.
Terminate orphan process: pid (...) (make)

Classification: Infrastructure/CI pipeline

• Root cause: The required checks aggregator treats cancelled jobs as failures. The gen job was cancelled mid-run, which triggered the aggregator to fail and sent a Slack alert, despite all test suites passing.
• Not a test flake, not a data race, and no panic/OOM found.

Timing verification:

• required job failed at 2025-10-21T15:45:36Z, matching the Slack notification window for this run.

Duplicates search (last 30 days & historical keywords):

• Queries used in coder/internal:
  • "One of the required checks has failed or has been cancelled"
  • gen cancelled required checks
  • ci required checks cancelled
  • required checks aggregator
• Closest prior issue: flake: CI failure in main - gen job (pnpm setup) and build job (Java setup) #929 (different root cause: network timeouts in setup steps). No active duplicate found for cancelled gen causing required to fail.

Assignment analysis (component ownership):

• This is owned by CI workflow maintainers. Recent ownership signals for .github/workflows/ci.yaml:
  • ci: make changes required (#20131) — Ethan
  • ci: make test-go-pg-17 a required check (#19722) — Ethan
  • ci: ping slack on ci failures / prompt changes (#19835, #19435) — Ethan
  • Recent edits also by Michael for Slack agent wiring (#20379)
• Given recent substantive changes adding and tuning the required checks mechanism were by Ethan, assigning to @ethanndickson for triage.

Recommendations:

• Consider allowing specific jobs (e.g., gen) to be treated as neutral on cancellation, or gate the aggregator until all required jobs are completed/succeeded/explicitly skipped.
• Investigate why gen was cancelled mid-run within the same workflow (concurrency/cancel-in-progress or runner preemption). If expected, update the aggregator’s logic; if not, address job stability.

Repro/Next steps:

• Review .github/workflows/ci.yaml: required job logic for handling cancelled jobs.
• Check workflow concurrency settings and any conditions that could cancel gen.
• If cancellations are expected in some paths, adjust required to ignore those paths or re-run gen automatically.

Quality Checklist:

• Identified actual failing job and validated timing
• Downloaded and reviewed failing job logs (required) and cancelled job logs (gen)
• Searched coder/internal for duplicates with multiple query patterns
• Classified as Infrastructure, not a test/data race/process crash
• Assignment based on component ownership history, not PR/commit author of the failing run

[flake-investigator]

New occurrence on main: required job failed due to a cancelled required check (fmt).

CI Run Link: https://github.com/coder/coder/actions/runs/18756570682
Branch: main
Commit: a106d67c07c7dc326ccce5a4eafc2e534412acd9 (Mathias Fredriksson) — coder/coder@a106d67

Evidence from required job log:

Checking required checks
- changes: success
- fmt: cancelled
- lint: success
- gen: success
- test-go-pg: success
- test-go-pg-17: success
- test-go-race-pg: success
- test-js: success
- test-e2e: success
- offlinedocs: success
- check-build: skipped

One of the required checks has failed or has been cancelled
##[error]Process completed with exit code 1.

fmt job log shows no failures; steps complete successfully, but the job’s overall conclusion is reported as cancelled in the workflow summary, which trips the aggregator.

Timing verification: required failed at 2025-10-23T17:40:45Z; Slack alert fired at 17:40:57Z for this run.

Classification: Infrastructure/CI aggregator behavior — identical symptom to this issue but with fmt instead of gen.

[flake-investigator]

New occurrence of this flake today.

CI Run Link: https://github.com/coder/coder/actions/runs/18847720861
Branch: main
Commit: 5a3ceb38f0636e63d273cb64aef230b252b544a5 (Dean Sheather) — coder/coder@5a3ceb3

Evidence from required job:

Checking required checks
- changes: success
- fmt: success
- lint: success
- gen: cancelled
- test-go-pg: success
- test-go-pg-17: success
- test-go-race-pg: success
- test-js: success
- test-e2e: success
- offlinedocs: success
- check-build: skipped

One of the required checks has failed or has been cancelled
##[error]Process completed with exit code 1.

Notes:

• All test suites passed; only the "gen" job shows cancelled. Its steps completed successfully but the job conclusion is "cancelled" (job id 53776271436).
• Timing aligns with Slack alert: required failed at 2025-10-27T16:23:34Z; Slack at 16:23:41Z.
• This matches the issue’s prior occurrences; classifying as Infrastructure/CI pipeline.

cc: @f0ssel for visibility

[deansheather]

Seems like this a flake on GitHub or Depot's part. It looks like the GHA runner for the gen job (in the above case) didn't report it's status or crashed.

[flake-investigator]

Reoccurrence on main

CI Run: https://github.com/coder/coder/actions/runs/18865828600
Branch: main
Commit: e720afa9d0d4721753abe096ecd16621ae6c53a6 (Spike Curtis)

Timing: required job failed at 2025-10-28T06:26:22Z (Slack alert at 06:26:28Z)

Evidence from required job logs:

Checking required checks
- changes: success
- fmt: cancelled
- lint: success
- gen: cancelled
- test-go-pg: success
- test-go-pg-17: success
- test-go-race-pg: success
- test-js: success
- test-e2e: success
- offlinedocs: success
- check-build: skipped

One of the required checks has failed or has been cancelled

Notes:

• Both gen and fmt were marked cancelled by the API, but their logs show all steps completed successfully with no failures.
• All test jobs passed; this is the same aggregator failure mode as described in this issue.

Classification: Infrastructure/workflow (required checks aggregator interpreting cancelled jobs as failure).

[flake-investigator]

New occurrence: required checks aggregator failed due to fmt job marked as cancelled.

CI Run Link: https://github.com/coder/coder/actions/runs/18865991866
Branch: main
Commit: dec6d310a89431b7e14b76481352edb4c33e5116 — Dean Sheather — coder/coder@dec6d31

Evidence from the required job:

Checking required checks
- changes: success
- fmt: cancelled
- lint: success
- gen: success
- test-go-pg: success
- test-go-pg-17: success
- test-go-race-pg: success
- test-js: success
- test-e2e: success
- offlinedocs: skipped
- check-build: skipped

One of the required checks has failed or has been cancelled
##[error]Process completed with exit code 1.

Notes:

• fmt job logs show all steps finishing without errors and typical post-job cleanup; however the job conclusion is reported as cancelled in the workflow summary, which causes the aggregator to fail.
• No failing tests, no data races, and no panic/OOM indicators in this run.
• Timing verification: required job failed at 2025-10-28T06:42:30Z, and the Slack alert was emitted immediately after.

Classification: Infrastructure/CI workflow (aggregator logic treats cancelled jobs as failure).

Potential follow-ups:

• Consider relaxing aggregator logic to tolerate cancelled for specific jobs (or treat as neutral) when the job actually completed steps successfully, or ensure job result reporting can’t be cancelled in this benign case.
• Investigate why fmt was marked cancelled despite normal step completion.

[ethanndickson]

I took a look at a failing job in the Depot dashboard, and it looks like the cancelled jobs are getting stuck on the launching runner step, and this is counting towards the configured timeout-minutes:

I've sent a message in the depot shared slack channel about the runners taking as long as they are to start up. If this persists we can always bump the job-level timeout, or move the timeout onto a specific step, so the time spent waiting for the runner is not included.