pdtbench — an agentic trading benchmark with a control group

the question this answers

Give a language model a historical market window and it will often turn a profit. That number on its own is close to meaningless: the model may have read the chart during pretraining. Standard backtests cannot separate a model that trades well from a model that remembers what happened next.

pdtbench is built around a control group, the way a trial is. It was built at the PDT Partners hackathon, where it took first place.

the synthetic twin

Every real window is paired with a synthetic twin — matched on total return (±3pp), volatility (±15%) and regime, so the trade to make is identical and only the identity differs. The twin is the same problem stripped of anything the model could have memorised.

Beating the real window by more than its twin is the edge. That gap, not the raw return, is the score that means something.

memorisation as a measurable slope

Regressing that gap against how often the model can name the window turns “is this memorisation?” into a slope you can measure, rather than a suspicion you argue about.

no look-ahead, by construction

The engine fills at the next unseen open, and a read-audit test makes look-ahead impossible by construction rather than by convention.

Baselines and the agent enter through the same in-process MCP server, so fee parity holds by construction rather than by assertion. Neither side can be quietly advantaged by a different code path.

a scoring rule that cannot be gamed by sitting out

Sharpe is floored at 0.25 × buy-and-hold daily volatility, because plain Sharpe on a mostly-cash account rewards deploying 1% of capital for two lucky days. An agent that never trades scores exactly zero.

state of the work

218 tests, none touching the network. Environment, baselines and analysis are complete; the live model batch is the next step.