Declaration of use of AI: I have used AI to keep this application precise and to the point.

Project summary

Today, every organisation choosing to deploy open-weight model typically has to perform similar safety and security evaluations independently and repeatedly. Individual developers and researchers also repeat the same process as well. This creates massive duplication: hundreds and thousands of organisations execute highly similar safety tests, if not same. This consuming compute, energy, researcher time, and engineering resources redundantly.

The duplication is not just wasteful but that's exactly why open-weight safety goes unmeasured. When every org repeats private evaluations, nobody has a public, comparable, longitudinal picture. Hence nobody can tell whether open-weight safety is improving or degrading across releases. And, no regulator or releaser has a shared factual baseline to act on. This matters the most because safety measured at release is not safety in deployment. Anyone can fine-tune safeguards away. Existing evaluation work from METR, Apollo, AI Lab Watch, SaferAI, the Midas project - targets frontier closed models, where the gap doesn't exist in the same form. No one is systematically tracking safety degradation under fine-tuning across open-weight releases. Asking $13,400 for three-four months to take it from one model to twenty+.

What are this project's goals? How will you achieve them?

OSMSI will provide open-weight models a Safety Cards, and a public Safety Index using six dimensions (Jailbreak resistance, Harmful capability, Prompt injection, Uncertainty calibration, Agentic safety, Interpretability aspects [exploration / v2]. The method needs further work). The goal is to create a shared repository of reproducible safety evaluations that developers and organizations can use as a common baseline. Organizations can then focus their resources on the evaluations that are genuinely specific to their deployment context.

OSMSI treats model safety evaluation as shared infrastructure rather than a problem that every organization should independently solve from scratch.

Months 1 - freeze framework v1, publish the Safety Card schema, harden pipelines.

Months 2-3 - evaluate 10–15 open-weight models across the five scored dimensions. Establish a held-out private test set alongside the public one.

Months 3-4 - publish the leaderboard, report the public/private performance gap as a first-class metric, and run a fine-tuning degradation pilot on a subset of models.

How will this funding be used?

Total requested: $13,400 - three-four months, compute only, no salary. Built alongside full-time employment.

Compute, GPUs - $8,000. Dedicated GPU infrastructure for evaluation runs.

Compute, grader - $2,000. Model APIs for grading.

Storage, hosting, CI/CD - $1,500. Artifact and log storage, leaderboard hosting, pipeline CI.

Software and subscriptions - $500. Tooling and licences.

Contingency - $1,400. Model pricing changes, re-runs after methodology revisions.

Coffee & Red Bull - $900 ($8 a day)

Total - $13,400.

Who is on your team? What's your track record on similar projects?

Netan Mangal | AI Research Engineer @ Secure Agentics (UK) | LinkedIn

• Research Engineer at Secure Agentics, working on AI security and evaluation for enterprise deployments.

• 5+ YoE in large scale data-tech, applied AI, ex-crypto exchange.

OSMSI is an open-source project that builds directly on that practical experience evaluating foundation models for safety and deployment.

What are the most likely causes and outcomes if this project fails?

Contamination. As benchmarks become public, they leak into training data and scores drift upward without safety improving. Outcome: the index degrades into a memorisation test.

How much money have you raised in the last 12 months, and from where?

$2,000 from BlueDot Impact Rapid Grant, July 2026.