Pass@k vs. Pass^k

Two readings of repeated success that answer opposite risk questions. Pass@k measures whether at least one of k attempts solves the case and is common in code generation and candidate search. Pass^k, a less standardized notation, requires all k executions to pass and exposes reliability under repetition. A system can have high Pass@k while remaining inconsistent.

Under independent attempts with constant success probability p, the approximations are 1 - (1 - p)^k for Pass@k and p^k for Pass^k. Real executions share prompts, models, data, and failure modes, so independence must be tested. The first metric fits cases where finding one solution is enough; the second shows whether an operation can be repeated without relying on luck.