@atilgandev Good question!

This is actual a central problem with any system that links a claim with prior experiments that tested "the same claim". No two experiments are precisely the same.

We started by focusing on direct replications, where authors try to reproduce a previous experiment as closely as they can. These are rather rare however outside of the psychology.

Next we broadened our search to "close replications" and we distinguished two different types - close experiments and close extensions. Close experiments are looking for the same claim/effect as a prior work but using slightly different methods. Close extensions, which are very common in fields like genetics and biomedicine, are testing if an effect generalizes (usually to a different human population).

Finally, we have been collecting "conceptual replications". These are the most loose form of replication, where two researchers are looking for a similar "general effect" but in different ways and different contexts.

In our database viewer, you can filter to a subset of the four types of replication (direct, close extension, close experiment, conceptual).

I absolutely agree that surfacing changes in experimental context could be valuable. For instance, a drug that worked in younger group may not work in an older population. A psychological effect found in Americans may not work in Japanese. A failure to replicate often means that an effect does not generalize. However, we choose not to do this since it adds an additional level of complexity. A judgement call has to be made somewhere whether the change in context is actually relevant. An experiment done in a room with windows may get a different result than one done in a room with no windows. Do the windows matter? Someone has to make a call using scientific theory and we feel it's not our business to be doing that.

Another issue is the relevant differences are often unknown or not something you can tell from what is said in the paper (for instance the experimental finding may have to do with how a flask was shaken or an animal was handled. It seems scientists regularly discover new ways that mouse cage design effect the biology of mice).

Additionally, from a metascience perspective, what we are interested in is the claims scientists make, which tend to be of a more general character anyway. Scientists usually don't say "we found that X happens in people tested at 1 pm on a Monday afternoon in a room with windows who had just had lunch and were being paid and looked at a 20 inch computer monitor and were all educated Americans in a room with humidity level Y and temperature Z ...". They usually make some more generalizable claim like "a computer monitor that has a lower frame rate degrades performance on a visual task". There is a strong bias towards overclaiming the generalizability and importance of an effect, this is called importance hacking (https://www.clearerthinking.org/post/importance-hacking-a-major-yet-rarely-discussed-problem-in-science)

All of this is in line with this paper by Nosek and Errington, two major names in metascience: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691
See also https://metascienceobservatory.org/docs/defining-replication

Does that make sense? I'm very open to suggestions on how to deal with replications that aren't direct / have important changes, so I'd be curious to hear if you have any thoughts.

Brief update (8/6/2026) - we recently published an article about what we've learned so far: https://moreisdifferent.blog/p/what-weve-learned-so-far-at-the-metascience

We also have put some information on our website about two other initiatives we've been working on - our forensic metascience agent and Bird's Eye Reviews: https://metascienceobservatory.org/articles

Brief update (8/6/2026) - we recently published an article about what we've learned so far: https://moreisdifferent.blog/p/what-weve-learned-so-far-at-the-metascience

We also have put some information on our website about two other initiatives we've been working on - our forensic metascience agent and Bird's Eye Reviews: https://metascienceobservatory.org/articles

Just a note (since this post has pretty high ranking in Google searches) - we've been building this at the Metascience Observatory! Gavin is on our advisory board and has supported the initiative financially through a Manifund regrant! A similar project, but focused on the social sciences, is FORRT's FReD project.

I'm very excited about this and it seems like a great value for 20k!