You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$60,000 left to reach the funding goal.
As per the Sijal method, a "Case File" will be compiled across the various sessions and their associated outputs. It will be built off:
Toulmin Model (Stephen Toulmin - 1958)
ODNI ICD 203
Through extensive investigation of the 'missing piece' identified during Session #19, some key insights have become apparent regarding this design.
Having considered prior applications of CoT, I concluded that the most suitable course of action would be to consider the problem computationally. By starting from a point of Dimensional Analysis and iterating via a programmed form of thinking, we can define a series of discrete actions to follow:
Any failure to gather relevant information will of course limit how well downstream tasks can link that information together.
Some indicators:
Liu et al. demonstrate degraded ability to use information towards the middle of the input, compared to very early or very late, for some very long-context models ("Lost in the middle", TACL 2024).
On average, models were only able to leverage 10-20% of the length of context provided (BABILong, NeurIPS 2024).
In particular, GPT-4 would use around 10% of its maximum 128,000 tokens.
The best system on MultiHiertt (ACL 2022) had approximately 64% of failures from failing to combine one or more required values.
Bisociation (Arthur Koestler - 1964) → Klein's Paths of Insight (Gary Klein - 2013) → Cognitive graphs and predictive linking (TransE - 2013) → Abduction (Charles S. Peirce - 1903)
Failure and Guarantee:
The Collapse of Spurious Connections.
A guarantee lies in the completeness of the connection. The connection is valuable only if it leads to an outcome that meaningfully impacts the decision.
As described above, this is the crux of the method:
There are a number of desirable features of PoT that I would like to take advantage of:
On average about 12% higher performance than CoT; 8% higher for verbal mathematics questions, 15% higher for all finance-related question groups. Adding self-consistency gave further improvements. (TMLR 2023)
How I would implement this:
Identify all relevant decision units in the conclusion, all "hold unless" criteria and associated thresholds.
Perform reverse Buckingham's Pi on combinations of length 2-4 over all quantities, to generate all candidate equations with the correct units as one of the decision units.
Clearly this would be too laborious to leave up to the models.
Perform calculations with an appropriate package for handling units (such as pint).
For all other quantities with matching units, calculate the difference, the ratio, whether above or below any thresholds, and whether any bounds were crossed.
Where applicable, that is, if any are upper and lower bounds, repeat for each bound and keep only comparisons satisfied across all bounds.
Record 19 of Sijal contained the following:
Why ? , if all 6 members voted "yes", did this issue fail with a decision of "no"?
This was explained by reference to the "usual" practice of counting "yes" votes as dissent on the first reading. On this basis, 11 out of 18 members agreed (61%) and the result was "supported with dissent".
However, as noted above, it made more sense to read the question the other way around: would this process filter OUT desirable items? Hence all the "yes" votes would be against the procedure, giving agreement of 17 out of 18 (94%) (The old story).
So this raises the novel issue of whether the angle should be read normally or inverted. As ever, a good example of the value of anomalies to suggest an improved hypothesis, as advocated by Klein (The best story)
This would manifest as numbers: any difference of more than 1 between two apparently comparable numbers; or inconsistency between any calculated figure and a given one. Or as words: apply NLI to all pertinent combinations, check with at least 1 other program, followed by ACH, asking what missing variable would reconcile the difference. (The outcome is a brand)
Given a combination lock, one would of course attempt all possibilities, 000-999. No guarantee of being any quicker, but obviously much more certain.
Therefore:
Unit search followed by applying the codes
Deterministic exact match, plus NLI using 2 separate programs, plus ACH score
Each case in the form of Bastiat-Merton, satisfying anchorage, to produce 2nd degree scores
After Ronald Howard, only useful to the extent that it may change a decision.
As above, conduct With and Without analysis on a mini-court. This would also test the "holds unless..." point and to what extent conclusions turn on particular numbers.
Any useful information only discovered towards the end of discussions, for example, by Sijal, would therefore be pointless if it did not impact on the outcome, or the case as to why. No point, however insightful, if it will not have this effect. Indeed, this is the very purpose of conducting a discovery process.
Note: Filters (1), (2), and (5) are all from the patent system, and that wasn't by any coincidence. It's because the patent system's logic is by far the most mature in human history when it comes to testing "Is this a real discovery?"
Prior Disclosure Filter (Patent Law)
Non-Obviousness Test (Graham v. John Deere, 1966)
Materiality Threshold (Financial Auditing Standards (PCAOB))
Specificity / Barnum Filter (Forer Effect, 1949)
Prior Art Search (Patent System (USPTO/EPO and PRISMA 2020)
Operation: Defining a Sijal based on the strongest lexical element already:
Impossible Lexical Order: Degree of change (overturning the overall judgment> turning the lens judgment> changing "what to do" > achieving "holds unless"), then number of sources, then number of lenses, then number of pieces, then margin of sensitivity.
For a tie only: paired comparisons with three models, order of presentation varied and combined via the Bradley-Terry rule.
A scheme that takes one discovery or null: it's structurally impossible to produce 4.
Always follow this structure:
BLUF, that is, the bottom line up front, with relevant figures
Because... Therefore... sequence leading to this bottom line
Why it is important, that is, relevant to the overall judgment, in one line
Something new, that is, a novel statement, as far as I can tell
Once complete, run relevant code to verify:
Make discovery certain, rather than leave it to luck
By generating every quantity x unit that can combine with the relevant decision unit, we ensure that there is no reliance on what the LLM happens to focus on.
Prevent important facts being overlooked
The whole exercise is conducted in code (Python) via a package for handling units, not by reference to the LLM output. We will also verify that we have picked up every figure appearing in the input.
Effectively no Type I errors
Candidates must pass 7 tests in turn: Previously disclosed, Not obvious, Material, Sufficiently specific, Prior art, Would change decision and Blindly obvious. One fail means they are dropped.
It's okay to report "No Discovery"
All cut-offs and models used will be logged before any work begins. Once a set of candidates has been reviewed, no further lowering of these criteria is permitted. The model should frequently conclude that there is nothing to discover.
Traceable and reproducible
A cryptographic digest will accompany each discovery, incorporating the finding, associated query terms, data extracts, models and parameters applied. This can then be double-checked by others.
Distinguish between the function of the LLM vs our algorithm.
The model is only responsible for the proposal and formulation. All the calculations, comparisons, and final decision-making are done in code, not in the model.
Planted discovery cases
40% of the eval set has one planted discovery in it (half computational, a quarter contradiction, a quarter second order). The system has to find that.
Empty cases
30% of the evaluation set contains no valid discovery. Any output on these is a false positive.
Deceptive cases
20% contain near-valid candidates which have been planted there deliberately (a judge has already said that it's too obvious, doesn't alter the decision, too general, already published). The right answer is to abstain.
Real cases
10% are genuine files from actual institutions where the finding is first spotted by independent experts.
Success metrics:
The release rule:
If the number of false positives goes above the target then the gates get tightened irrespective of what happens to the hit rate afterwards (a false positive in a sealed file is worse than honest abstention).
This funding provides a 12 month runway over two phases: first to build out Sijal's discovery pipeline and then to turn that into an open benchmark for the research community.
Phase 1 - Months 1–6: Building the discovery pipeline
Infrastructure ($4,000)
Replacing a laptop that's only half broken is the way to go with this job. In the Gaza market the cost of a working machine is currently around $4,000.
Living costs ($12,000)
$2k per month for six months is the absolute minimum you'd need to work full time on this project if you were to do it as a sole occupation (i.e. without any other employment which could in turn delay or stop the project).
Daily development costs ($7,200)
Internet and electricity in Gaza are both extremely unreliable and extremely expensive. You'll need around $40 per day to get the consistent access you need to do the work.
API costs ($1,500)
The pipeline uses a certain number of models in every session (dual extraction, two-model NLI, three-model mini-court, prior art search, etc.). That budget covers around 400 dev/eval sessions during the build.
Independent evaluation ($4,000) requires at least two blind annotators per output, with a third in place to resolve any disagreements. The builder of the outputs is not permitted to carry out the evaluation.
Phase 2 - Months 7–12: Sijal Benchmark
Living and daily costs ($19,200): $2,000 per month for living costs and $40 per day for infrastructure during the second six months.
Benchmark construction and publication ($8k) - this is the cost of taking the evaluation set and turning it into an open, published benchmark for models to test against. That's the annotation cost, the verification cost and the publication cost.
API costs ($1,500)
Funds further sessions for benchmarking validation and stress testing against a range of models.
Reserve ($2,600)
Allocated for unforeseen expenditure in an unpredictable environment.
Total: $60,000
Phase 1 delivers a validated discovery pipeline with a quantified false positive rate. Phase 2 provides an open benchmark that forms the infrastructure for the wider research community.
Just me for now. I'm a Software developer (Python) based in Palestine, Gaza, with 8 years' experience, working full-time on Sijal and Saql since April 2026. Here's the chain of problems that this proposal is based on. Each step in the chain is something I've either built or measured.
Every output has to be traceable back to an input. I came from data systems (building validation, duplicate detection, audit logs etc). When I started incorporating language models into those pipelines, the model-generated numbers were still being checked against the source in the code.
Don't let the model reason freely. Constrain it to a known good method. That's the principle underpinning Sijal, an institutional reasoning engine in production. There's a designated devil's advocate who argues the other sideargues the opposing case at full strength; a number of judges, each running on different models, all make their own judgments, and every fact they cite carries a verbatim quote matched to its source. Saql uses the same principle in its debate evaluation.
Treat the outputs as measurements. Re-running Sijal’s vote on 30+ contested questions (some of which were in both Arabic and English) gave the same results. Where a counting error had caused one of the sealed files to show a near-unanimous verdict as a 61% split,I didn’t edit it. The correction was sealed as a new record, and the original was left in place as a verifiable record, marked as superseded. The error was corrected and a new record created, the original being left in place as a verifiable record, marked as superseded. In Saql, the human and the model score separately on the same behavioral anchors; agreement is measured via quadratic weighted kappa and the criteria are never amalgamated into a single score (that's the point of the position paper I’m writing for the ArgMining workshop).
Find where it breaks. A test case for Sijal’s discovery step had a finding planted in it (a three-step calculation spread across three sources). A general-purpose agent failed on that in four ways: it didn't do the arithmetic, it repeated the judges, it produced lines that would work for any organization, and it returned four findings rather than one. A procedure can be used to get the model to argue the other side; it can’t guarantee the model will multiply three numbers from three sources.” three numbers from three sources. So this proposal moves that step out of the model and into code.
The thread that runs through all four of them is that AI judgment becomes auditable when you choose which of the steps the model isn't allowed to do.
Sijal: fasil.quotefather.com · GitHub: github.com/abdyazouri · LinkedIn: linkedin.com/in/abd-iv7l
Most likely causes, roughly in order:
Extraction is the big bottleneck. The pipeline can only link numbers that it actually extracts correctly (with units and denominators). If "1.2 kg per family meal" gets reduced to "1.2 kg" then the discovery is lost. The actual session text in both Arabic and English is vastly more complex than the test cases so the computational hit rate may fall well below the 85% target.
The gates are in place for safety. The release rule is such that it makes the gates even more stringent if the false positive rate exceeds 5% (even if that means the hit rate subsequently falls). So the system is more likely to fail by saying "no discovery" rather than reporting a false positive.
The only reliable path is the computational one. Contradiction detection and second-order effects both depend on NLI and open-ended reasoning, where the targets are already modest (50% and 30% respectively).
If the majority of real session discoveries end up in those categories, then the most robust method will only be applicable to the smallest subset.
The evaluation may be less rigorous than it seems. Planted cases authored by the same individual who constructed the pipeline may reward the pipeline's design rather than its actual capability. Real cases would require institutional approval, and Sijal has no customers at the moment. The annotators may also fall short of the target agreement (κ ≥ 0.7), as FEVER's annotators only achieved 0.68.
(Main cause) Operational constraints. I'm working solo out of Gaza where both power and internet are spotty and work can be cut off at any time. API costs could also go up if every evaluation case requires a full Sijal session. Code and data are stored off site so it's time lost rather than work done.
Outcomes if it fails:
Sijal will keep on going, as discovery is a separate research layer and the decision files are generated independently. Abstention is the default, so a failure will just result in silence rather than false claims in the sealed files. The deterministic parts can still be used to check the numbers in Sijal's decision files via unit-aware extraction, coverage checks, and arithmetic checks. I'll publish the results in any event (regardless of outcome), including the hit rate, the false positive rate, a breakdown of the misses by stage and the evaluation set. A well-documented negative result on the limits of LLM discovery will be of benefit to others in the field and will help to define the scope for Phase 2.
Little so far:
About $1,130 (800 JOD) in donations after I posted the project in a Reddit community.
$1k in API credits (in kind) via Anthropic's startup program.
No grants, no investment, nothing beyond that.
I have two applications currently pending for Sijal: a BlueDot rapid grant and an application to the Epistemics Fund via Lightcone Commons, with a decision on the latter due mid-December 2026.
I'll update this page if either of those comes through.