You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$5,000 left to reach the funding goal.
Open, reproducible audits that test whether published AI-trading and forecasting backtests use information that was actually available at the time—and measure how data-timing errors, survivorship bias, and incorrect price handling affect reported results.
Minimum to start: $2,500. Funding goal: $5,000.
Proposed use of funds:
60% — Historical data: Data needed to investigate gaps that free sources cannot reliably fill, particularly delisted-company prices and first-published versus subsequently revised data.
25% — Compute and storage: Infrastructure for the nightly collection and preservation of dated evidence.
15% — Research time: Designing audits, specifying controls, reviewing results, and publishing reproducible reports.
The minimum amount would support a narrower implementation of the research plan; the full goal would provide more room to address historical-data gaps and carry out the planned audits with better evidence coverage. The exact scope will depend on the availability and cost of the required data.
This funding would support public research, not a commercial service or a claim of profitable trading. No paying audit clients are currently in place.
Many published AI-trading and forecasting results are evaluated on data that would not have been available to a real decision-maker at the time.
Examples include joining financial filings by date without accounting for the time they became public, testing historical performance with today's index constituents, and calculating transaction costs using adjusted prices that incorporate later corporate actions.
These problems can make a backtest appear more successful than it would have been under historically available information.
Statistical methods for detecting overfitting address an important part of backtest reliability. They cannot, by themselves, establish that the underlying data was historically available and correctly represented. If a model has access to information from the future, even a statistically careful evaluation may be measuring the wrong thing.
This project focuses on that upstream problem: auditing the data, timestamps, and assumptions beneath published backtests, then measuring the effect of identifiable defects wherever the evidence permits.
The intended users include researchers comparing trading models, developers maintaining public benchmarks, and investors or analysts evaluating claims about historical performance. The work will provide open evidence and tools; it will not assume that any particular strategy is profitable.
The public repository market-data-audit contains reproducible investigations of published repositories and research claims. Each audit distinguishes measured effects from defects identified in code whose effects could not be established.
Key findings include:
46% of SEC filing facts in a tested date-only join were treated as available before EDGAR accepted the corresponding filing. This is a result of the audited data and join method, not a claim about every SEC dataset or pipeline.
Using FinRL's Dow 30 ticker list produced an annualised advantage of 4.48 percentage points over the real index in the specific comparison audited. Testing today's S&P 500 constituents backwards produced an apparent annual advantage of approximately 5.1–5.6 percentage points in the studies conducted.
Commission estimates based on adjusted closing prices were approximately five times the correct estimate in the audited case.
Across 18 FRED macroeconomic series, the direction of the monthly change differed from the first published value in approximately one month out of seven in the comparison conducted.
Three of eight public cases were confirmed and fixed upstream by project maintainers:
stefan-jansen/machine-learning-for-trading: issue #1158 and PR #1161; the author independently reproduced the finding.
HKUDS/Vibe-Trading: the issue was fixed in under a day, with a known-answer regression test.
TauricResearch/TradingAgents: the reported issue was fixed on 16 October.
Two additional cases identify mechanisms in code but do not have a measured effect; they are not counted as confirmed, measured defects.
Related studies include the historical-universe comparison described above and an investigation of favourite–longshot bias in prediction markets, finding an effect of approximately one percentage point on both Polymarket and Manifold in the cases studied.
The research uses a preregistration discipline: the rule and expected result are committed before measurement, known-answer controls accompany the relevant calculations, and failed expectations—including my own—remain part of the record.
A nightly collection has also been running unattended, with a hash-chained record, covering approximately 500 companies across prices, options, analyst data, news, SEC filings, and Japanese and Korean filings. Its purpose is to preserve dated evidence and make later checks against the information available at a given time more credible.
These results are available for independent inspection in the public repository and linked study materials.
Over six months, the project will aim to deliver:
1. Twelve public audits
Each audit will examine a published AI-trading or forecasting repository or benchmark and produce a report in a consistent format covering:
the decision or research claim the audit addresses;
the verdict and supporting evidence;
findings classified as Measured, Read in code, or Not measured;
sample sizes, denominators, assumptions, and controls;
the method needed to reproduce the finding and verify a proposed fix.
All audits will be published regardless of outcome, including cases where no defect is found, the expected effect is absent, or the effect cannot be measured.
2. An open checklist and reproducible scripts
The project will publish reusable checks for:
look-ahead caused by missing time-of-day information;
survivorship bias in historical asset universes;
price types, corporate actions, and transaction-cost calculations;
one-bar execution delays;
overlap between training and test data.
The aim is to make these checks usable by other researchers, not to keep the audit methods proprietary.
3. A forward-dated research record
The project will maintain a dated log of forecasts and their subsequent outcomes. This will provide evidence that can be checked later, rather than relying solely on a retrospective assertion that the process was point-in-time.
The funding would support the public research and evidence infrastructure. It would not fund a promise to develop a profitable trading strategy.
The evaluation criteria will be stated in advance:
Audit delivery: Aim to complete 12 public audits within six months, with at least 8 showing a measurable effect on a defined headline result.
Transparent denominators: Report the number and proportion of audits where no defect is found or where an effect cannot be measured. These outcomes will not be hidden or excluded from the record.
Independent engagement: Seek at least four additional maintainer responses beyond the three cases already confirmed as fixed. Responses and fixes will be reported whether positive, negative, or absent.
Prediction accuracy: Publish cases where my preregistered prediction of an effect was wrong, including the direction or magnitude of the error where it can be established.
Reproducibility: Provide the evidence, scripts, assumptions, and controls needed for readers to inspect the measurements.
The target of eight audits with measured effects is a goal, not a condition for suppressing or omitting less conclusive results. The project will publish all completed audits and report the final denominator.
I spent five years working with pre-match and live sports betting markets, where prices represent probabilities and decisions depend on what was known at the time. That experience made the distinction between information available at a decision point and information learned later central to how I evaluate data.
I now apply that perspective to AI-trading and forecasting research. My work combines data-integrity investigation, point-in-time validation, experimental design, and reproducible controls.
I work independently from Ukraine. The research is remote, the main outputs are public, and the project does not require privileged access to proprietary trading systems.
How I work: I define the research questions, measurement rules, specifications, and acceptance criteria. I use AI coding agents—including Claude Code, Gemini, and Codex—to assist with implementation, then verify the resulting code and measurements through tests, known-answer controls, and reproducible checks. My responsibility is to establish what should be tested, evaluate whether the implementation meets those requirements, and report what the evidence supports.
The public repository documents this approach and provides examples of the resulting work.
No paying clients yet. This application funds public research. A possible future paid-audit service is a separate plan and is not presented as existing revenue.
Uncertain fundraising outcome. A comparable project focused on look-ahead audits of AI-trading benchmarks reportedly raised $0 over two months. I therefore do not treat this grant as guaranteed income or assume that technical merit alone will secure funding.
Historical-data limitations. Coverage of delisted companies and other historically unavailable data is a major constraint. Funding may reduce this limitation, but it cannot guarantee complete coverage.
Measurement is not always possible. A defect visible in code does not automatically establish a measurable change in performance. Such cases will be reported separately rather than assigned an unsupported numerical effect.
No promise of investment returns. Finding data defects can change how a backtest should be interpreted. It does not establish that a corrected strategy will make money.
Main repository: OleksanderSS/market-data-audit
Sample audit report: audits/sample-report.md in the repository.
Vibe-Trading: issue #1790 and fix commit d3c7681 in HKUDS/Vibe-Trading.
Machine Learning for Trading: issue #1158 and PR #1161 in stefan-jansen/machine-learning-for-trading.
TradingAgents: issue #1518 in TauricResearch/TradingAgents.
These links provide examples of the reported issues and the resulting fixes or evidence. The repository and linked reports contain the detailed methods and measurements.
There are no bids on this project.