You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$20,000 left to reach the funding goal.
My own experience as a disabled immigrant and 15+ year expertise in Fintech have shown me evidence that financial products -even at the basic customer service level- are inherently exclusionary by design. I have worked in financial inclusion -advised the US Federal Reserve and W3C- but we are entering a world where the gap is exponentially growing.
As financial institutions rapidly shift from basic chatbots to autonomous AI agents for onboarding, credit navigation, dispute resolution, and customer support, current AI safety benchmarks are failing to keep up. Most existing evaluations are static, measuring surface-level policy compliance or generic capability rather than how an agent behaves when interacting with a real human who has complex, non-standard needs.
In Kindlee's preliminary 2025 limited benchmark study "The Kindlee Cost of Trust" across live customer-facing banking bots, we observed an 82.7% critical failure rate in edge-case journeys (compared to 18.2% in standard flows). When older adults, disabled individuals, the neurodiverse, or people facing language barriers interact with these systems, agents frequently collapse into dead-end loops, demand inaccessible authentication steps, or issue improper restrictions. Worse, standard "LLM-as-a-judge" evaluators routinely grant these broken runs a false pass because the agent sounded polite while completely failing the user.
This project scales The Kindlee Benchmark 2026 into an empirical, open-access research study and synthetic dataset. We will evaluate frontier agentic configurations across 1,000+ multi-turn financial scenarios to quantify context degradation, failure disparities, and evaluator false-pass rates under the EU AI Act and UK FCA Consumer Duty frameworks, as well as initial comparisons with US institutions
We are expanding our initial study to test our methodology at scale, quantify the impact and measure the effectiveness of our contextual feedback. More specifically, we aim to scale Multi-Turn Red-Teaming for Atypical Journeys, Execute 1,000+ multi-turn evaluation runs subjecting frontier models (e.g., GPT-4o, Claude 3.5, Llama 3.1) and in-production deployed financial agents to synthetic user personas representing disabled, elderly, low-digital-literacy, and neurodivergent populations in financial gatekeeping scenarios. We will:
Quantify Disparate Impact & Interaction Friction: Measure failure rate gaps between standard user journeys and vulnerable user profiles
Expose Evaluator False-Pass Rates: Compare and re-evaluate automated judge scores to detect harmful agent behavior.
Publish Open Science Artifacts: Release a public empirical research report (The Kindlee Benchmark 2026) and a de-identified synthetic evaluation dataset (CC BY 4.0) to give researchers, regulators, and builders a ground-truth baseline for fair agentic AI.
Right now, now very many companies have a trust answer to the question "How do we know if our AI agent violates FCA Consumer Duty or EU AI Act accessibility rules?", and our last study showed most organizations do not meet compliance standards. This project expands on our previous empirical evidence, and delivers:
Publication of The Kindlee Benchmark 2026 Report: An empirical paper detailing failure taxonomy, context collapse rates, and evaluator reliability metrics across major frontier models in financial flows.
Release of a Static Synthetic Benchmark Slice: A open-access dataset (CC BY 4.0) containing 1,000 static synthetic edge-case interaction traces and human-annotated evaluations, giving researchers and regulators a baseline for safe agent behavior.
Evaluator False-Pass Audit: A statistically validated audit mapping the false-pass rate gap between generic automated judges (LLM-as-a-judge) and human expert assessments in edge-case journeys.
This creates a stronger signal for the state of AI fairness and provides the tools for in-house evaluators to reference to.
Specific amounts vary based on the final award -Total Requested: $17,500 USD (Minimum: $7,500 USD / Maximum: $20,000 USD) - but we expect:
Model Compute & API Execution ($7,500): Token costs for running multi-turn synthetic interaction sweeps across commercial APIs (GPT-4o, Claude 3.5) and open-weights baselines.
Synthetic Scenario Structuring & Analysis ($6,000): Designing representative edge-case conversation flows, data curation, and statistical analysis of interaction friction.
Domain Expert Annotation Protocol ($2,500): Blind human expert verification to evaluate model outputs and calibrate false-pass detection thresholds.
Open Science Publishing & Distribution ($1,500): Report formatting, dataset hosting infrastructure, and public open-access dissemination.
Carla (Charlie) Canino (Principal Investigator): Founder & CEO of Kindlee Inc. (Techstars Miami '24 / J.P. Morgan Portfolio). 15+ years leading payment risk, fraud prevention, and enterprise product strategy (ex-Activision Blizzard, ex-Grover, ex-Worldpay). Advisory background with the U.S. Federal Reserve Faster & Secure Payments Task Force and contributor to W3C accessible payment standards. Lived experience as a disabled, Venezuelan-American, and LGBTQ+ founder.
Advisory & Governance: Supported by senior advisors including Ezekiel Canestrari (former COO of ClearBank Europe), Florent Stroppa (Lead Product Advisor) alongside experts in machine learning safety, enterprise risk, and regulatory alignment.
Capicua Development Partner: Capicua is a fintech specialized development firm based in NYC that came to us through the Founder Pledge network -Carla is a pledger- and has helped us with integration, design and execution of the previously architected platform, integration to Vanta, and provides insight on product direction and feasibility.
Political externalities delayed our product and market alignment in 2024/2025, however we pivoted and still found a wedge. This time around the risks might be:
Cause 1 (Token Compute & Rate Limits): High-volume multi-turn interaction testing across commercial APIs can encounter rate limits or token cost inflation.
Mitigation: We run baseline synthetic sweeps on open-weights models locally to refine evaluation scripts before executing targeted sweeps on commercial APIs.
Cause 2 (Automated Evaluator Variance): Generic automated judges may show inconsistent scoring across different model families.
Mitigation: We use a double-blind human expert annotation subset to ground-truth calibrate evaluator scoring logic.
Outcome if Project Fails: Even if multi-model sweeps run at a reduced scale, the published synthetic scenario dataset and empirical failure taxonomy will remain fully accessible to the research community as an auditable baseline.
We have raised $120,000 USD in equity capital through Techstars Miami / J.P. Morgan Portfolio in 2024, and in 2025 approximately 35K EUR supported by angel backing, and Capicua's in-kind development investment.
(IP Boundary Note: All pre-existing commercial software, Kindlee’s core scoring algorithms, KCTI, proprietary scoring weights, taxonomy, commercial quantification and contextual remediation logic, private client traces, and commercial persona libraries constitute Background IP and are permanently excluded from grant-funded open deliverables.)