Executive Summary

validate.science analyzes scientific papers to identify claims that may not be fully supported by the evidence presented. Our system:

1. Extracts claims — Identifies testable statements from the paper
2. Finds evidence — Locates statistical results (sample sizes, p-values, study design)
3. Checks the math — Verifies reported statistics are calculated correctly
4. Flags mismatches — Highlights where claims may exceed what the evidence supports

How It Works

Our analysis pipeline processes documents through seven stages, each designed to extract specific information and apply targeted validation checks.

Stage 1: Document Processing

PDF text is extracted with section boundaries preserved (Abstract, Methods, Results, Discussion). Section identification enables context-aware analysis—claims in the Discussion section are treated differently than those in Results.

Stage 2: Claim Extraction

A large language model identifies up to 15 atomic, testable claims from the document. We focus on empirical assertions rather than background statements, definitions, or methodological descriptions.

Criteria for extraction:

• Must be a testable empirical claim (not a definition or background)
• Must be specific enough to evaluate against evidence
• Prioritizes claims from Results and Conclusions sections

Stage 3: Claim Classification

Each claim is classified along three dimensions:

Stage 4: Evidence Extraction

Statistical evidence is extracted from the document, including:

• Sample sizes (N, n, participants)
• Test statistics (t, F, χ², r, z)
• P-values and significance levels
• Confidence intervals
• Effect sizes (Cohen's d, η², etc.)
• Study design indicators (RCT, observational, cross-sectional)

Stage 5: Claim-Evidence Matching

Claims are matched to relevant evidence using semantic embedding similarity. Each claim is compared to each evidence item using cosine similarity, and matches above a threshold are retained. This allows claims to be evaluated against the specific evidence that supports (or fails to support) them.

Stage 6: Burden-of-Proof Check

Three deterministic rules are applied to detect epistemic overreach:

1. Causal-from-Correlation: Causal claim + correlational/observational design → flag
2. Overgeneralization: Broad population claim + small/narrow sample → flag
3. Underpowered: Strong claim + inadequate sample size → flag

Stage 7: Risk Scoring

An epistemic risk score (0–100%) is computed based on the number and severity of failure modes detected. Claims with scores above the threshold (default: 50%) are flagged for review.

Detection Methods

We use a two-tier detection system that separates high-confidence statistical errors from potential methodological issues. This honest approach ensures users know exactly how much to trust each finding.

Tier 2: Potential Issues (Review Suggested)

These detections identify claims that may exceed what the evidence can support. They are presented as suggestions for author review, not definitive errors. Currently one detection method is enabled based on validated precision.

Tier 1: Statistical Errors (High Confidence)

These detections identify mathematical inconsistencies in reported statistics, using the same techniques as Statcheck and GRIM. 79% precision, 95% recall validated on 154,961 statistical tests from the Hartgerink 2016 dataset.

Evidence Quality

Validation Evidence

Our detection methods are validated against established benchmarks and real-world papers with known issues. All results are reproducible.

Statcheck Benchmark

We validated our statistical error detection against the Hartgerink 2016 Statcheck dataset, containing 155,000 statistical results from psychology papers.

Confusion Matrix

Famous Failed Papers

We tested against papers with known replication failures or author disavowals to validate our detection of real-world issues.

Key Finding: Facial Feedback Main Result

The 1988 Strack et al. "pen in teeth" study—a foundational paper in embodied cognition—contains a gross statistical error in its main result:

Reported:  t(89) = 1.85, p = .03 (significant)
Computed:  p = 0.068 (NOT significant)

The main result claiming that holding a pen in teeth improves humor ratings is based on an incorrect p-value. The effect is not statistically significant at conventional thresholds.

Internal Test Suite

We maintain an internal test suite of 6 test cases including synthetic papers with planted errors and real papers with known issues.

Academic Foundations

Our detection methods are grounded in peer-reviewed research on statistical error detection and scientific methodology. Each technique is based on established academic work.

Statcheck: P-Value Verification

Our p-value recalculation method is based on Statcheck, developed by Nuijten et al. (2016). Their analysis of 250,000+ p-values from psychology articles found:

• 49.6% of papers contained at least one statistical inconsistency
• 12.9% had "gross errors" where significance status was affected
• Errors were equally distributed across top and bottom journals

We implement the same recalculation logic for t-tests, F-tests, χ² tests, correlations, and z-tests, achieving 100% agreement on p-value calculations.

GRIM Test: Impossible Means

The GRIM (Granularity-Related Inconsistency of Means) test was developed by Brown & Heathers (2017). For integer-scale data (e.g., Likert scales 1-7), they showed that:

• Mean × N must equal an integer (or close to one with rounding)
• Applied to 260 papers: 36% contained at least one impossible mean
• Simple arithmetic check catches fabricated or misreported data

GRIMMER Test: Impossible SDs

The GRIMMER test (Anaya 2016) extends GRIM logic to standard deviations. SD must be mathematically possible given N and scale constraints, providing an additional check for data integrity.

Power Analysis and Sample Size

Our underpowered detection is informed by extensive research on statistical power in science:

• Ioannidis (2005): "Why most published research findings are false" demonstrated how low power leads to unreliable findings
• Button et al. (2013): Found median power in neuroscience was ~21%, leading to inflated effect sizes and low replication rates

Replication Crisis Ground Truth

Papers that failed major replication attempts provide ground truth for validating our detection methods:

• Many Labs (Klein et al. 2014): Large-scale replications of classic effects
• Open Science Collaboration (2015): Found only 36% of psychology findings replicated, with effect sizes typically half of originals

We use these papers to test whether our system identifies real issues without producing false positives.

Claim-Evidence-Burden Analysis

Our semantic analysis of claims exceeding their evidence is novel but grounded in established scientific principles:

• Causal inference: Only randomized controlled trials can establish causation; observational studies establish association
• External validity: Small, narrow samples cannot support claims about broad populations (the "WEIRD" problem)
• Statistical power: Small samples yield unreliable estimates regardless of p-value significance

Limitations & Honest Assessment

We believe in transparency about what our system can and cannot do. No automated tool can replace expert human review.

What We Detect Well

What We Miss

Known Limitations

What This Tool Is NOT

• Not peer review — Cannot evaluate theoretical contributions, novelty, or importance
• Not fraud detection — Finding statistical errors ≠ finding misconduct
• Not a quality stamp — Absence of flags does not mean a paper is good
• Not definitive — All flags are potential issues for human review

Appropriate Uses

• Pre-submission check for authors to catch errors before publication
• Quick screening during peer review to prioritize manual checking
• Teaching tool to illustrate common statistical issues
• Research tool for studying error prevalence in literature

Reproducibility

All benchmark results are reproducible. Our code is open source and benchmark data is publicly available.

Running the Benchmarks

# Clone the repository
git clone https://github.com/validate-science/validate-science.git
cd validate-science

# Install dependencies
npm install

# Run Statcheck benchmark (requires dataset download)
npm run benchmark

# Run full benchmark suite and freeze results
npm run benchmark:freeze

Statcheck Dataset

The Statcheck benchmark uses the Hartgerink 2016 dataset:

• Source: OSF Repository (osf.io/gdr4q)
• Citation: Nuijten, M. B., et al. (2016). Behavior Research Methods, 48(4), 1205-1226.
• Contents: 258,103 statistical results from 30,717 psychology articles

Version Information

Benchmark Workflow

To create a new benchmark version:

1. Update PIPELINE_VERSION in src/services/version.ts
2. Run npm run benchmark:freeze to save results
3. Run npm run benchmark:publish <version> to publish
4. Results are saved to data/benchmarks/

See docs/BENCHMARKING.md for full documentation.

Code References

Version History

Each methodology version represents a frozen snapshot of benchmark results at a point in time. Older versions remain available for reference.

Version Naming

Versions follow the format v{semver}-{YYYY-MM-DD}:

• semver: Semantic version of the pipeline (MAJOR.MINOR.PATCH)
• date: Date the benchmark was frozen

Multiple benchmarks may exist for the same pipeline version if run on different dates. Only one version is published as "current" at any time.

References