QCircuitEval

Two checks, one verdict

Generated code is run in the task's framework, then lowered to a shared intermediate representation. From that representation, two graders operate the same way for Qiskit, Cirq, PennyLane, and CUDA-Q. The functional grader simulates the circuit and compares its behavior with an independently derived target. That alone is not enough: code can reproduce the expected output without performing the requested computation. The structural grader therefore inspects registers, gates, measurements, and source for evidence of the required construction and for banned shortcuts. Both graders must pass, and construction requirements are run before the functional checks.

No LLM takes part in grading. Every verdict comes from a deterministic verifier, and each contract has one verifier route. Only a verified pass counts toward a score. A functional or construction mismatch returns as a semantic failure. When the grader cannot justify a decision, it returns an execution error rather than a guessed pass, and the same applies when a result falls inside the uncertainty band. Exceeding a resource, time, memory, or dimension limit returns as a resource limitation.

Hellinger fidelity

Quantum programs can produce measurement statistics, prepared states, or unitaries. Because measurements are stochastic, many tasks compare distributions rather than a single return value. For a target distribution $p$ and a candidate $q$, the grader uses the Hellinger fidelity, $F_H$:

The index $z$ ranges over every possible measured bitstring. The Hellinger fidelity is symmetric, bounded in $[0,1]$, and more stable than KL divergence when probability is spread across many small outcomes, the regime where a single-reference KL threshold is most brittle.

The acceptance test is on Hellinger infidelity, $1-F_H(p,q)$, against the contract's pinned target in the declared bit order. Gate and unitary tasks instead compare operators up to global phase:

Other semantic objects use checks suited to what they are. States are compared by trace distance under the contract's phase and observation policy. Unitaries and isometries use an operator norm, restricted to the logical subspace or with ancilla restore where the contract declares it. Channels are compared through the normalized Choi representation under Frobenius distance. Instruments are checked on branch probabilities together with the normalized post-measurement states. Classical input-output tasks run the full finite truth table rather than a sample of examples, and distribution tasks compare the whole probability vector. Shor and Steane encoders use phase-sensitive exact state checks, so a program that matches the measurement support with the wrong codeword signs fails. Arithmetic and Boolean tasks are evaluated across their full declared input domain. Parameterized quantum error correction (QEC) tasks run every declared diagnostic case, including no error and each permitted single-error location. Every case must pass.

What counts as the same answer

The target is derived independently of any one canonical solution, so a circuit passes when it realizes the contracted behavior, not when it matches a particular reference listing. Different constructions can still overlap on that contracted distribution. Readout order is not open: prompts and contracts pin little-endian bit order, and a reversed measurement map fails because 011 and 110 do not overlap. A circuit that hardcodes the expected bit string also fails the structural checks, even if its simulated histogram matches.

Gate and unitary tasks still treat global phase as unobservable. Multiplying a state by $e^{i\phi}$ does not change any measurement probabilities, so two circuits that differ only by that overall factor realize the same physical operation. $S$ and $RZ(\pi/2)$ look like different listings, but $RZ(\pi/2)=e^{-i\pi/4}S$, so both are treated as equivalent:

Why simulation is not enough

A functional grader can be fooled by code that writes the expected answer directly. This supposed adder returns the demonstration result 0110 without allocating operands or performing any addition:

The functional grader sees the expected output; the structural grader sees that no adder was built. Requiring both checks accepts genuinely different valid circuits without turning output matching into a loophole.

The adder is the crudest version of the problem, and the structural grader is built to catch the subtler ones too. It rejects precomputed counts and probabilities, a unitary handed back in place of a program, and a nontrivial dense unitary on the full register. It also rejects amplitude injection, where a program writes the target state straight into the register. Each of these reproduces the expected output while skipping the computation the task asked for.

Beyond rejecting fabricated answers, contracts state what a valid program has to contain. A contract can set minimum qubits, operations, measurements, and entangling gates, require that particular wires are measured while forbidding others, and name the interactions that must occur between data, ancilla, control, and target registers. Where a task calls for an explicit construction, the grader enforces the gate families and decompositions the contract specifies and bans the obvious substitutes, among them library calls, optimizers, eigensolvers, simulators, samplers, decoders, and dense evolution.

The harder evasions need checks of their own. A net-unitary nonlocality check finds entangling gates that cancel one another out, and restricting that check to the measured register catches an entangler parked on unused ancillas, where it satisfies a gate count without touching the result. Error correction tasks strip canceling padding and require physical errors that depend on the arguments they were given, and a contract can insist on multi-controlled corrections. Cross-case invariance catches a function that branches on public arguments and hardcodes each row of the truth table rather than computing it. Every one of these routes has an adversarial test behind it.