Non-Reductionist Philosophy

Perplexity - ERSA-steelman-critique

22 min read

ERSA Framework: Steelman and Red Team Analysis

Executive Summary

This document provides a critical examination of the ERSA (Explanatory Robustness & Strength Assessment) framework by presenting the strongest possible criticisms (Steelman) and by playing the role of adversary (Red Team). The goal is to identify genuine weaknesses, unexamined assumptions, and places where ERSA could fail or be misused.

PART 1: STEELMAN CRITIQUE (Strongest Possible Objections)

Steelman Argument 1: ERSA is Fundamentally Reductionist About Complex Systems

The Argument:

ERSA reduces scientific maturity to a single numerical score across wildly different domains. This assumes that:

  1. Complexity in physics can be compared to complexity in psychology
  2. Causal mechanisms in chemistry are analogous to causal mechanisms in economics or sociology
  3. A single framework can assess what are fundamentally different types of knowledge claims

The Problem:

A system’s behavior might follow rules that are fundamentally incomparable across domains:

  • Physics: Describes law-like regularities that hold everywhere, always
  • Chemistry: Describes regularities dependent on precise conditions but still law-like
  • Biology: Describes systems with multiple evolutionary pathways that can produce identical outcomes
  • Psychology: Describes systems influenced by consciousness, interpretation, and meaning-making
  • Economics: Describes human behavior that itself responds to predictions about that behavior
  • Sociology: Describes emergent properties that vanish when analyzed at individual level

The ERSA framework assumes these are all on the same maturity spectrum, but they might not be. A theory might be “mature” in physics (predictive to 10 decimal places) but fundamentally immature in economics (predictive only at aggregate level, breaks down under reflexivity).

Historical Example:

When physics frameworks were applied to psychology (behaviorism), it created a century of wasted research because the framework didn’t fit the domain. The reductionist approach assumed psychological laws would be as universal as physical laws. They weren’t.

Consequence for ERSA:

  • Economics theories might permanently plateau at ERSA 5-6 even with perfect research methodology
  • Psychology theories might never reach ERSA 8-9 despite centuries of evidence
  • The scale might be inappropriately comparing apples and oranges, creating illusion of commensurability where none exists

Counter-consideration: Maybe domain-specific weightings solve this? But then you’re not really using ERSA as a universal scale—you’re using multiple scales pretending to be one.

Steelman Argument 2: ERSA Enshrines the Tyranny of the Majority (Consensus Bias)

The Argument:

ERSA heavily weights scientific consensus (95%+ agreement = ERSA 8+). But:

  1. Scientific consensus has repeatedly been wrong:

    • Steady-state cosmology (consensus, wrong)
    • Continental drift initially (against consensus, right)
    • Ulcers from stress (consensus, wrong; actually bacteria)
    • Stomach acid from acid foods (consensus, wrong; acid produced endogenously)

  2. Major paradigm shifts often start with minority positions:

    • Einstein vs. established Newtonian physics
    • Plate tectonics vs. geology establishment
    • Evolution vs. religious establishment (not science, but illustrates consensus problem)

  3. ERSA rewards conformity: A fringe researcher with novel ideas starts at ERSA 1-2 even if their ideas are brilliant. They face a massive uphill battle to overcome ERSA 4-5, the “Valley of Death,” which ERSA now codifies as expected.

The Problem:

By making ERSA dependent on consensus, ERSA essentially says “scientific maturity = how many people agree.” But this:

  • Disadvantages unconventional thinkers
  • Advantages those who work on fashionable topics with institutional funding
  • Could systematically underrate theories that challenge current paradigms
  • Might actually slow scientific progress by rewarding conformity

The Historical Test:

If you calculated ERSA scores retroactively:

  • Quantum mechanics when Einstein was skeptical: ERSA 4 (minority position)
  • Plate tectonics 1950s: ERSA 2 (nearly rejected by establishment)
  • Heliocentrism 1600s: ERSA 2 (Galileo punished)
  • Evolution 1900s: ERSA 3 (many biologists skeptical)

All now ERSA 9-10, but ERSA framework would have PENALIZED those who believed them when true believers were minorities.

Consequence: ERSA might be systematically biased toward established orthodoxy and against revolutionary ideas that later prove correct.

Steelman Argument 3: ERSA Conflates “Evidence Quality” with “Theoretical Maturity”

The Argument:

ERSA assumes that:

  • More evidence → Higher ERSA
  • Better-quality evidence → Higher ERSA
  • Lots of replicated studies → Foundational theory

But this assumes a particular relationship between evidence and truth that isn’t guaranteed.

Problem Case 1: Efficient Markets Hypothesis (EMH)

  • Strong theoretical framework
  • Backed by 50+ years of economic research
  • High-quality empirical studies across multiple countries
  • If judged by ERSA standards: Should be ERSA 6-7 by 2000

Reality: EMH was fundamentally wrong. Market anomalies, behavioral finance, and empirical evidence contradicted it. But the evidence quality and theoretical coherence made it appear mature.

The problem: Lots of evidence supporting a wrong framework ≠ mature framework.

Problem Case 2: Newtonian Mechanics in Quantum Regime

  • Centuries of evidence
  • Perfect predictions at macroscopic scale
  • Foundational to all engineering
  • By ERSA standards: ERSA 9

But: Fundamentally breaks down at quantum and relativistic scales. The theory wasn’t “wrong”—but it was incomplete in ways not detectable by evidence at macro scale.

Consequence: ERSA might rate theories as “mature” that are actually incomplete, broken, or wrong in domains not yet tested.

Steelman Argument 4: ERSA Assumes Falsification is Possible (Popper Problem)

The Argument:

ERSA incorporates Popperian falsifiability as a fundamental principle. But Popper’s criterion itself has been heavily criticized:

Why falsification is problematic:

  1. No single falsification is definitive: Any anomaly can be explained by adjusting auxiliary hypotheses rather than rejecting core theory. (Lakatos recognized this—the protective belt absorbs anomalies.)

  2. Measurement error is always present: How do you know a measurement contradicting your theory reflects reality vs. measurement error?

  3. Complex systems can’t be falsified cleanly: In social sciences, any prediction can be undermined by confounding variables. You can never say “this proves the theory false” because unmeasured confounders might explain the discrepancy.

  4. Falsification doesn’t work for probabilistic theories: A theory predicting “70% of people will do X” isn’t falsified by 30% not doing X—that’s the predicted outcome! Yet ERSA treats this as testable/falsifiable.

  5. Historical scientists didn’t follow falsifiability: Einstein didn’t abandon Newtonian mechanics because of a few anomalies. He built a better framework that explained everything Newton explained PLUS more. This is not pure falsification.

Consequence: By enshrining falsifiability as core principle, ERSA might systematically underrate theories that can’t be cleanly falsified (most social sciences, complex systems, probabilistic domains).

Steelman Argument 5: ERSA is Culturally Contingent, Not Universal

The Argument:

ERSA assumes there’s a universal metric for “scientific maturity.” But:

  1. Different cultures define evidence differently:

    • Western science privileges RCTs and quantification
    • Traditional medicine privileges long-term observation and holistic patterns
    • Indigenous knowledge privileges intergenerational testing
    • These aren’t inferior—they’re different epistemologies

  2. ERSA privileges Western scientific methodology: The Bradford Hill criteria, GRADE, RCTs—all emerged from Western academic medicine and epidemiology. This framework might not apply to:

    • Ayurvedic medicine (different theory of causation)
    • Traditional ecological knowledge (different testing methods)
    • Theological claims (non-empirical domain)
    • Philosophical theories (not empirically testable)

  3. ERSA assumes empiricism is the highest form of knowledge: But many domains aren’t primarily empirical:

    • Mathematics (proven through logic, not observation)
    • Ethics (not subject to empirical testing)
    • Aesthetics (subjective by definition)

Consequence: ERSA might be parochially Western and empiricist, unable to assess non-Western or non-empirical knowledge systems fairly.

Steelman Argument 6: ERSA Disguises Subjective Judgments as Objective Measurement

The Argument:

ERSA appears objective—scoring things 0-4 on each Bradford Hill criterion, calculating composite scores. But underneath, it’s intensely subjective:

  1. Who decides what counts as “strong” evidence?

    • Effect size of 1.2 vs. 1.5—is that 2/4 or 3/4?
    • No universal answer; depends on domain expert judgment

  2. How do you weight conflicting criteria?

    • What if Consistency is high but Dose-Response is low?
    • Theory scores ERSA 4.5 or 5.5?
    • Different reasonable people would score differently

  3. Publication bias is unmeasurable:

    • You estimate how many studies are in “file drawers”
    • But you’re guessing; this profoundly affects ERSA rating
    • Two researchers could reasonably differ by 1-2 ERSA levels

  4. Bradford Hill criteria are themselves subjective:

    • “Plausibility” depends on domain knowledge and creativity
    • “Coherence” with existing knowledge—but which existing knowledge?
    • “Analogy”—analogous in what way? To whom?

The Illusion of Precision:

ERSA appears to give you ERSA 5.7 vs. ERSA 5.2, suggesting precision and objectivity. But underlying this are dozens of subjective calls that could easily shift the score by ±1.0 or more.

Consequence: ERSA might create false confidence in precision where disagreement is inevitable. Two equally expert researchers might give ERSA scores differing by 2-3 levels based on different judgments.

Steelman Argument 7: ERSA Privileges Quantity Over Quality in Perverse Ways

The Argument:

ERSA rewards:

  • More studies (consistency score based on number of replications)
  • More disciplines (cross-domain integration bonus)
  • Longer history (temporal stability bonus)

But this creates perverse incentives:

  1. Quantity over novelty: A theory with 100 mediocre studies beats a theory with 3 brilliant revolutionary studies. ERSA punishes new insights.

  2. Fashionable topics over important topics: Popular research areas (cancer, Alzheimer’s) accumulate 50+ studies. Rare diseases can’t reach ERSA 5+ no matter how solid the evidence, just due to small sample population.

  3. RCT-able topics over non-RCT-able: Surgery innovations can’t be randomized (can’t randomly assign people to “good surgery” vs. “bad surgery” control group). But non-RCT-able topics get perpetually downgraded by ERSA despite solid evidence.

  4. Institutional research over grassroots research: Small research programs studying important questions might accumulate only 3-4 high-quality studies but ERSA rates them low. Huge pharmaceutical-funded research programs might have 30 studies but many designed to confirm marketability rather than truth.

Consequence: ERSA might systematically disadvantage rare diseases, non-RCT-able topics, disruptive innovations, and underfunded research areas, creating a framework that reflects institutional research power rather than actual knowledge.

Steelman Argument 8: ERSA Doesn’t Account for Paradigm Incommensurability

The Argument:

Kuhn argued (and many philosophers agree) that competing paradigms are incommensurable—you can’t compare them using universal metrics because they operate on different assumptions.

Example: Homeopathy vs. Modern Medicine

Modern medicine asks: “Does this treatment cause biochemical changes detectable by measurement?”

Homeopathy asks: “Does this treatment resonate with the body’s vital energy?”

These aren’t answering the same question. If you use ERSA (modern scientific framework), homeopathy scores ERSA 0. But that’s not because homeopathy is wrong—it’s because homeopathy operates in a different paradigm that rejects the empirical testing assumption.

The Problem:

  • Comparing ERSA scores for theories from different paradigms is comparing apples to oranges
  • ERSA assumes there’s a “true” universal framework (empiricist, quantitative, Popperian)
  • But incommensurable paradigms might have equal validity within their own logic

Consequence: ERSA might systematically disadvantage theories from non-empiricist paradigms while falsely claiming universal applicability.

Steelman Argument 9: ERSA Incentivizes Gaming Through Protective Belts

The Argument:

ERSA incorporates Lakatos’s concept of research programs with “protective belts” that absorb anomalies. But this creates a perverse incentive:

Every time evidence contradicts a theory, researchers can:

  1. Add auxiliary hypotheses
  2. Claim the protective belt was adjusted, not the core theory threatened
  3. Maintain the same ERSA level despite evidence

Example: Psychoanalytic theory

  • Predicts “repressed trauma causes neurosis”
  • Evidence: Many people with trauma don’t have neurosis
  • Adjustment: “They have ‘defensive splitting’ or ‘dissociation’”
  • Result: Theory unfalsifiable; protective belt expands to explain anything
  • ERSA: Stays low because unfalsifiable, OR stays medium because it’s “progressive” by generating new protective hypotheses?

Real Example: Efficient Markets Hypothesis (EMH)

  • Predicts no predictable patterns in stock prices
  • Evidence: Consistent “momentum effects,” “value anomalies”
  • Adjustment: “These require including behavioral factors” OR “Market microstructure effects”
  • Result: Theory expands but core claim remains unfalsified

The Problem:

ERSA rewards “progressive” research programs that generate new hypotheses to explain anomalies. But this means you can have an ERSA 6 theory that’s actually un-falsifiable and just constantly adding explanations.

Consequence: ERSA might reward epicyclical theories (adding explanations to defend core idea) while appearing to reward genuinely progressive science.

Steelman Argument 10: ERSA Assumes Linear Progress (Kuhn’s Problem)

The Argument:

ERSA assumes scientific progress is roughly linear: ERSA 1 → 2 → 3 → … → 9 → 10.

But Kuhn and others argued scientific progress is non-linear:

  • Long periods of normal science (refining existing theory)
  • Sudden paradigm shifts (crisis, new theory, revolution)
  • Regression (in some senses, losing knowledge of old framework)

Problem Cases:

  1. Pre-paradigm science: Before a dominant framework emerges, science might involve competing schools with no obvious linear progression. Particle physics in the 1920s had many competing theories—which was “progressing”?

  2. Paradigm transitions: When physics shifted from Newtonian to relativistic, some knowledge was “lost” (Euclidean space as fundamental reality). By what metric did physics “progress”?

  3. Revolutions aren’t incremental: The shift from Ptolemaic to Heliocentric astronomy wasn’t moving up an ERSA scale—it was a complete restructuring of how to think about the problem.

Consequence: ERSA might misapply a linear scale to non-linear progress, especially problematic for revolutionary theories that require paradigm shifts.

PART 2: RED TEAM ATTACK (Strategic Vulnerabilities)

Red Team Attack 1: ERSA Can Be Used to Suppress Dissent

The Attack:

Once a theory reaches ERSA 8-9, ERSA framework can be weaponized to suppress alternative theories:

  • Researcher proposes alternative to ERSA 9 theory
  • Dismissal: “That theory is only ERSA 2; your idea is 4 levels below consensus”
  • Result: Authority-crushing of dissent

Historical: Semmelweis proposed hand-washing prevented childbed fever (later ERSA 4-5 evidence). Medical establishment had ERSA 8 consensus that miasma caused disease and humors needed to be balanced. Semmelweis was institutionalized and died. ERSA at the time would have suppressed his eventually-correct view.

Modern Risk:

ERSA could become a tool for defending orthodoxy. “The consensus is ERSA 8.5; your fringe idea is ERSA 1.2.”

This looks scientific but actually privileges those with institutional power to define consensus.

Red Team Attack 2: ERSA Scores are Unfalsifiable

The Attack:

How would you FALSIFY an ERSA rating?

You can’t. If ERSA rates Theory X at 5.0, and new evidence contradicts it:

  • If you lower the ERSA to 4.5: “As expected, evidence evolved the rating”
  • If you keep it at 5.0: “Core findings are robust; this is just auxiliary hypothesis refinement”
  • No matter what, ERSA can be defended

This violates ERSA’s own Popperian principle: ERSA requires theories to be falsifiable, but ERSA itself isn’t falsifiable.

Red Team Attack 3: The Impossibility of Domain Weighting

The Attack:

ERSA proposes different domains weight Bradford Hill criteria differently (physics emphasizes coherence; psychology emphasizes consistency). But:

  1. Who decides the weights? Scientists in each domain? (They disagree)
  2. Weights become politicized: Economics researchers might downweight “Dose-Response” because many economic theories don’t have it. This looks like domain-appropriate adjustment; it’s actually downweighting a criterion that contradicts their research.
  3. Weighting flexibility destroys comparability: If you can weight criteria however you want, two theories can have identical ERSA scores despite fundamentally different evidence profiles. ERSA loses meaning.

Red Team Attack 4: ERSA Creates Illusory Precision

The Attack:

ERSA 5.7 sounds more precise than ERSA 5-6. But underlying this:

  • Multiple subjective judgments (±0.5-1.0 each)
  • Disagreement between raters (±1-2)
  • Arbitrary scoring thresholds (why is “strong” 3/4, not 2.5/4?)

Result: False precision.

You think ERSA 5.7 vs. ERSA 5.2 is meaningful difference. It isn’t—both are fundamentally “mid-range evidence” and would be rated differently by different experts.

Red Team Attack 5: ERSA Becomes a Proxy War

The Attack:

Researchers will game ERSA:

  1. Publish many small studies (rather than one large one) to increase “Consistency” score
  2. Find adjacent domains to apply theory to to boost “Cross-domain integration”
  3. Fund studies to generate large samples to reduce imprecision, boost ERSA
  4. Dismiss contradicting evidence by claiming it’s lower-quality evidence or publication bias without evidence

Result: ERSA becomes not a measure of truth, but a measure of research productivity and funding.

Red Team Attack 6: ERSA Contradicts Its Own Foundations

The Attack:

ERSA incorporates Popper (falsifiability) + Lakatos (protective belts protect unfalsifiable cores) + GRADE (different evidence types weighted) + Bloom’s (knowledge is multidimensional).

These frameworks are in tension:

  • Popper: Everything must be falsifiable
  • Lakatos: Cores are unfalsifiable by design
  • GRADE: Quality assessment is subjective and domain-dependent
  • Bloom’s: Knowledge has multiple dimensions; can’t reduce to single score

Trying to incorporate all of them creates an internally contradictory framework.

Red Team Attack 7: The Consensus-Minority Problem

The Attack:

ERSA both:

  1. Relies heavily on consensus (ERSA 8+ requires 95%+ agreement)
  2. Tries to accommodate paradigm shifts (ERSA 10+ for revolutionary theories)

But these are contradictory:

Revolutionary theories, by definition, start as minority positions. ERSA would rate them ERSA 1-2. By the time they reach consensus (ERSA 8), they’re no longer revolutionary—they’re mainstream.

So ERSA either:

  • Underrates emerging revolutionary theories (ERSA 1-2 when they might be right)
  • OR abandons consensus-weighting (loses one of its key features)

Result: ERSA can’t coherently handle theory transitions from minority to consensus.

Red Team Attack 8: ERSA Assumes “More Evidence” Means “Better Theory”

The Attack:

Large, well-funded research areas accumulate more evidence than small, underfunded ones. This creates systematic bias toward theories that are:

  • Funded by institutions
  • Popular with researchers
  • Easy to test with available methods
  • In wealthy countries

Underfunded theories (rare diseases, neglected regions, unconventional ideas) will perpetually score lower on ERSA simply because fewer people study them.

This isn’t a measure of truth—it’s a measure of funding and fashion.

Red Team Attack 9: ERSA Cannot Handle Under-Determined Theories

The Attack:

Some theories are “under-determined”—multiple incompatible theories fit all available evidence equally well.

Example: Wave-particle duality in quantum mechanics. Both interpretations (wave picture, particle picture) fit identical evidence. Neither is “more mature” because evidence literally cannot distinguish them.

ERSA assumes there’s a fact-of-the-matter (theory is either true or false). Under-determined theories violate this assumption.

Consequence: ERSA can’t coherently rate under-determined theories. You’d have to assign both ERSA 8 (both fit evidence) while admitting they’re incompatible. This breaks ERSA’s logic.

Red Team Attack 10: ERSA Weaponizes Evidence Against Emerging Fields

The Attack:

When a new field (e.g., psychiatric epigenetics, immunotherapy for cancer) emerges with promising early findings:

ERSA assessment: “Early studies, small sample sizes, inconsistent results, limited replication—ERSA 2.0”

True assessment: “Revolutionary new field with promising preliminary findings; could transform medicine”

ERSA penalizes newness. This is actually a feature (don’t overstate uncertainty into confidence). But it’s also a weapon against disruptive innovation.

Consequence: ERSA might systematically underrate emerging fields that could be revolutionary.

PART 3: Legitimate Criticisms (Not Steelman, Not Red Team—Actually Valid)

Legitimate Criticism 1: Complexity of Application

Issue: ERSA requires expertise in:

  • Epistemology
  • Bradford Hill criteria
  • GRADE methodology
  • Lakatos’s philosophy
  • Domain-specific knowledge
  • Statistics
  • Psychology (to avoid cognitive biases in assessment)

No single person has all this. Applying ERSA well might require a team of 5-10 specialists.

Result: ERSA is impractical for routine use. It’s too complex.

Fair counter: This is true of any comprehensive framework. Worth the complexity if it’s accurate.

Legitimate Criticism 2: It’s a Framework, Not a Formula

Issue: ERSA doesn’t tell you what score a theory gets. It tells you how to think about scoring a theory. Two experts might honestly disagree by 2-3 ERSA levels.

Result: ERSA isn’t as objective as it appears.

Fair counter: Honesty about subjectivity is a strength—better than pretending objectivity you don’t have.

Legitimate Criticism 3: Missing Dimensions

Missing:

  • Predictive failure rates (how often theory makes wrong predictions?)
  • Paradigm flexibility (how hard is it to overthrow?)
  • External validity (does it work outside laboratory?)
  • Ethical implications (does theory have harmful applications?)
  • Tractability (can we actually study this?)

ERSA might need additional dimensions.

PART 4: How ERSA Could Fail (Failure Modes)

Failure Mode 1: Consensus Enforcement

ERSA becomes tool for suppressing dissent and locking in orthodoxy. Minority researchers working on revolutionary ideas get dismissed as “ERSA 2” and thus unworthy of attention.

Failure Mode 2: Bureaucratization

ERSA becomes bureaucratic compliance tool. Funding agencies demand “ERSA 5 minimum” for funding. Researchers design studies to maximize ERSA score rather than find truth.

Failure Mode 3: Domain Collapse

Different domains’ theories get compared inappropriately. Social science theory stays ERSA 4-5 forever due to inherent complexity. Physical science theory reaches ERSA 9 within decades. But this reflects domain differences, not theory quality, leading to inappropriate dismissal of social science.

Failure Mode 4: Protective Belt Gaming

Researchers use “progressive research program” language to defend unfalsifiable theories. “We’re protecting the hard core while adjusting the protective belt”—which is just epic cycling dressed in Lakatosian language.

Failure Mode 5: False Precision

Researchers cite “ERSA 5.7 suggests this theory is moderately robust” when actually it means “I scored three subjective criteria, got different values depending on how I interpreted them, and averaged them.” False confidence.

PART 5: Responses to Criticisms

Response to Steelman 1 (Reductionism):

Defense: ERSA isn’t claiming all domains are identical. It’s claiming they share some common features (testability, evidence quality, replicability) while acknowledging domain-specific constraints through weighting and qualifiers.

But: This is weak. Fundamentally different domains might not be comparable.

Response to Steelman 2 (Consensus Bias):

Defense: ERSA doesn’t say “consensus = truth.” It says “consensus + longevity + multiple confirmations = high confidence.” These are proxies for truth, not definitions of it. And ERSA explicitly allows for ERSA 10 (paradigm-shifting theories with extraordinary evidence) even against consensus.

Stronger defense needed: ERSA should explicitly penalize “consensus suppression” and reward “productive dissent” more than it currently does.

Response to Steelman 3 (Evidence Quality ≠ Theoretical Maturity):

Defense: ERSA incorporates mechanisms (like novel prediction requirement, mechanism understanding requirement) that go beyond just evidence accumulation. Efficient Markets Hypothesis would score low on “novel prediction” and “mechanism complexity” dimensions.

But: Unclear whether ERSA would have caught EMH as wrong.

Response to Steelman 4 (Falsification Problems):

Defense: ERSA doesn’t require single falsifications. It requires convergent evidence across multiple Bradford Hill criteria. One anomaly doesn’t falsify; multiple anomalies across different methods do. This is more sophisticated than pure Popperianism.

Fair: This is actually ERSA’s strength.

Response to Steelman 5 (Cultural Contingency):

Defense: ERSA explicitly recognizes this. The framework includes “domain-specific weighting” and acknowledges that non-empirical domains (philosophy, mathematics) might not fit.

Weakness: ERSA still assumes empiricism as default. Non-empirical domains would have multiple criteria scored 0/4 (Experiment, Dose-Response), making them permanently low-ERSA despite being internally rigorous.

Response to Steelman 6 (Subjective Judgments Disguised as Objective):

Defense: ERSA is transparent about this. It’s explicitly a framework requiring expert judgment, not an algorithm. It makes assumptions visible for criticism.

Fair: This is actually an advantage of ERSA over simpler metrics that hide subjectivity.

Response to Steelman 7 (Quantity Over Quality):

Defense: ERSA distinguishes evidence quality through GRADE assessment. A single high-quality RCT beats 100 low-quality observational studies.

But: ERSA still weights “number of studies” in Consistency scoring. More replicating studies = higher consistency, even if previous studies were biased.

Response to Steelman 8 (Paradigm Incommensurability):

Defense: ERSA can’t solve the fundamental Kuhnian problem, but neither can any other framework. At least ERSA acknowledges it through paradigm-level scoring and qualifiers.

Fair point: Valid weakness without clear solution.

Response to Steelman 9 (Gaming via Protective Belts):

Defense: ERSA includes “Research Program Health” qualifier (Progressive vs. Degenerating) specifically to catch this. If a theory is just adding protective belt hypotheses without novel predictions, it scores “Degenerating.”

Weakness: Determining “novel” is subjective. Researcher claims their new protective hypothesis is “novel prediction”; skeptic says it’s “epicyclical defense.”

Response to Steelman 10 (Non-Linear Progress):

Defense: ERSA includes ERSA 10-11 for “paradigm-shifting” theories, acknowledging non-linear jumps. It’s not purely linear.

But: ERSA still primarily operates as ordinal scale (1, 2, 3…) suggesting linearity even if paradigm shifts exist.

PART 6: Recommendations for Strengthening ERSA

Recommendation 1: Explicit Dissent Scoring

Add a dimension: “Productive Scientific Dissent.”

  • 0/4: Theory universally accepted with no serious doubters
  • 1/4: Minority of credible scientists disagree
  • 2/4: Significant minority (20-30%) disagree, some with credible arguments
  • 3/4: Substantial disagreement (40-50%)
  • 4/4: Fundamental debate ongoing

This captures that productive science has dissent, and too-perfect consensus might be suspicious.

Recommendation 2: Predictive Failure Rate Tracking

Add: “Prediction Accuracy Score” measuring percentage of novel predictions that were confirmed:

  • High accuracy = many novel predictions confirmed
  • Low accuracy = many novel predictions disconfirmed
  • This catches theories with evidence but poor predictive performance

Recommendation 3: Paradigm Flexibility Assessment

Add: “Paradigm Dependence Score”:

  • 1/4: Theory strongly dependent on specific paradigm; would fail in alternative frameworks
  • 2/4: Theory mostly paradigm-independent but some assumptions depend on current framework
  • 3/4: Theory is paradigm-independent; works across frameworks
  • 4/4: Theory would likely survive paradigm shift

Recommendation 4: Tractability Index

Add separate dimension: “Research Tractability” (0-10 scale):

  • Separate from ERSA core score
  • Rates how easy theory is to study empirically
  • Rare diseases might score ERSA 4 (limited evidence) but Tractability 2 (inherently hard to study)
  • Prevents unfair comparison of inherently tractable vs. intractable topics

Recommendation 5: Funding-Independence Score

Add: “Funding Independence Ratio”

  • 0/4: Theory studied only by heavily funded institutions
  • 1/4: Mostly well-funded research
  • 2/4: Mix of funded and unfunded research
  • 3/4: Significant research by under-resourced groups
  • 4/4: Distributed research independent of funding concentration

This prevents ERSA from just measuring research fashion and funding.

Recommendation 6: Explicit Uncertainty Ranges

Instead of ERSA 5.7, require: “ERSA 5.7 ± 1.2”

The uncertainty range reflects genuine disagreement between experts. Forces honest acknowledgment that scoring is imprecise.

Recommendation 7: Dissent Documentation

For any theory with ERSA 6+, require documentation of:

  • Best arguments against the theory
  • Alternatives that fit some evidence but not all
  • Known limitations and unsolved problems

This prevents ERSA from becoming tool for suppressing legitimate criticism.

Recommendation 8: Paradigm Shift Prediction

For theories at ERSA 7-9, make explicit prediction:

  • Is this theory likely to be overthrown in next 50 years?
  • What evidence would overturn it?
  • What would a successor theory look like?

This forces acknowledgment that mature theories might be incomplete.

Recommendation 9: Multi-Discipline Assessment Teams

Require ERSA assessments to be done by teams including:

  • Domain expert (knows theory well)
  • Skeptic/critic (argues against theory)
  • Philosopher of science (understands epistemology)
  • Statistician (understands methods)
  • External reviewer (from adjacent field)

This reduces bias from single perspective.

Recommendation 10: Sunset Clause

ERSA scores shouldn’t be permanent. Require re-assessment:

  • ERSA -1 to 2: Re-assess every 5 years
  • ERSA 3-5: Re-assess every 10 years
  • ERSA 6-8: Re-assess every 20 years
  • ERSA 9+: Re-assess every 50 years

This prevents ERSA from lock-in where initial rating sticks forever.

Summary: ERSA’s Fundamental Tension

ERSA tries to be:

  1. Universal (applicable across all domains) but domains are incommensurable
  2. Objective (systematic and reproducible) but based on expert judgment
  3. Predictive (showing where theories are progressing) but science is non-linear
  4. Progressive (allowing paradigm shifts) but also consensus-dependent
  5. Falsifiable (following Popper) but incorporating unfalsifiable hard cores (Lakatos)

The fundamental tension: You cannot simultaneously maximize all five of these. ERSA tries to do so, creating internal contradictions.

What ERSA does well:

  • Makes assessment framework visible and transparent
  • Incorporates multiple epistemological perspectives
  • Acknowledges subjectivity rather than hiding it
  • Provides systematic way to think about theory maturity
  • Allows domain-specific adaptation

What ERSA does poorly:

  • Doesn’t resolve fundamental philosophical problems (Kuhn, Popper, paradigm incommensurability)
  • Might suppress dissent through consensus weighting
  • Can be gamed through protective belt expansion or study proliferation
  • Might reflect research funding patterns more than truth
  • Creates false precision through numerical scoring

Verdict: ERSA is a valuable framework for organizing thinking about scientific maturity, but dangerous if treated as objective measurement rather than transparent systematic judgment. The framework is best used with full acknowledgment of its limitations and active attempts to counter its biases.

Graph View

Backlinks