Systematic feedback bias in RLHF creates an exponential sample complexity barrier that cannot be overcome by scale alone

Gaikwad proves that when feedback is systematically biased on a fraction α of contexts with bias strength ε, distinguishing between two true reward functions that differ only on problematic contexts requires exp(n·α·ε²) samples. This is super-exponential in the fraction of problematic contexts. The intuition: a broken compass that points wrong in specific regions creates a learning problem that compounds exponentially with the size of those regions. You cannot 'learn around' systematic bias without first identifying where the feedback is unreliable. This explains empirical puzzles like preference collapse (RLHF converges to narrow value subspace), sycophancy (models satisfy annotator bias not underlying preferences), and bias amplification (systematic annotation biases compound through training). The MAPS framework (Misspecification, Annotation, Pressure, Shift) can reduce the slope and intercept of the gap curve but cannot eliminate it. The gap between what you optimize and what you want always wins unless you actively route around misspecification—and routing requires knowing where misspecification lives.