)]}'
{
  "commit": "b005aa3a97bdd6512eaaeaf707d96768f236dc46",
  "tree": "d86f5c5eba518d379e9130d3acc2c31dadb6100b",
  "parents": [
    "c87b47d4a43a69a01b0f8c83636babf5ff70ccc8"
  ],
  "author": {
    "name": "David Ostrovsky",
    "email": "david@ostrovsky.org",
    "time": "Sun Aug 23 15:56:07 2026 +0200"
  },
  "committer": {
    "name": "David Ostrovsky",
    "email": "david@ostrovsky.org",
    "time": "Sun Aug 23 18:30:34 2026 +0200"
  },
  "message": "nodepool: re-enable host-key-checking to gate node SSH readiness\n\nThe main pool set host-key-checking: False, which removes nodepool\u0027s\nSSH-readiness gate: nodes can be handed to the executor before SSH/host-\nkey-scan readiness is confirmed, so ~1/3 of builds fail at the first\nAnsible connect with RETRY_LIMIT (~15-18s, no console logs). This is the\nchronic CI flakiness tracked in issue 502048297.\n\nSerial console of three consecutive failing (RETRY_LIMIT) nodes shows\nthey boot cleanly and become SSH-ready at a consistent ~31s after VM\ncreation -- not slow, not resource-starved:\n\n  node 10581: created 14:27:20Z, sshd started 14:27:51Z  (+31s)\n  node 10582: created 14:29:44Z, sshd started 14:30:13Z  (+29s)\n  node 10583: created 14:32:05Z, sshd started 14:32:38Z  (+33s)\n\nOn this Debian image the host keys are regenerated ~2s before sshd\nbinds, so a post-sshd ssh-keyscan should capture the regenerated key,\nnot the baked-in one.\n\nWithout a readiness gate nodepool marks the node ready when GCE reports\nRUNNING (~+10-15s), long before sshd is up at ~+31s. The executor then\ndoes ~19s of merge/checkout and connects at ~+30-34s -- right on the\nreadiness boundary -- so a few seconds of jitter decides each attempt:\nconnect before the node is SSH-ready -\u003e RETRY; connect after -\u003e success.\nThat is the ~1/3 pass rate. (The exact failing state -- sshd not yet\nbound vs the zuul authorized_keys not yet written -- is not pinned, as\nthe retried builds\u0027 console is discarded.)\n\nnodepool\u0027s GCE driver defaults host-key-checking to True: when true it\nssh-keyscans a node before supplying it and relaunches nodes that never\nbecome reachable, absorbing the wait inside the launch pipeline instead\nof racing the executor. It relies on the launcher reaching the node\ninternal IP, which is how this deployment is configured. It was disabled\nto dodge the GCE baked-in-host-key race; the serial evidence above\nindicates a post-sshd scan gets the regenerated key, so restoring the\ndefault should not reintroduce it -- validate the launcher log post\nmerge.\n\nUpstream confirms both mechanism and guidance: the GCE driver docs\n(zuul/nodepool: [1]) say host-key-checking defaults to True and \"helps\nensure that nodes are ready to receive SSH connections before they ar\nsupplied to the requestor\", to be disabled only for launcher/node\nnetwork separation or non-SSH platforms -- neither applies here. The\ngating is code-confirmed by zuul/nodepool: [2] (host-key-checking\ncontrols the statemachine/GCE driver\u0027s keyscan).\n\nThis is a hypothesis-driven change; nodepool config is not speculatively\ntestable. Validate post-merge: watch the RETRY_LIMIT rate and the\nlauncher log for ssh-keyscan / host-key failures and relaunches. If\nhost-key failures appear, roll back.\n\n[1] https://review.opendev.org/c/zuul/nodepool/+/841908\n[2] https://review.opendev.org/c/zuul/nodepool/+/821481\n\nIssue: 502048297\nChange-Id: I673366b351c74482e7c11f7b4e0c235fd6e330fa\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "73199f1343c6a5d6aaccc06e5e618c92262ea11b",
      "old_mode": 33188,
      "old_path": "nodepool/nodepool.yaml",
      "new_id": "e751f0bf4fcf2a7c5f689dd90d1e6a8560b8d5d1",
      "new_mode": 33188,
      "new_path": "nodepool/nodepool.yaml"
    }
  ]
}