Sure So Far · No. 04

Why Do Scientists' Error Bars Keep Coming Out Too Small?

Notes and sources · checked 2 November 2026 ·

In 1913 Robert Millikan gave the charge of the electron with a small plus-or-minus, and today's value lies well outside it; the episode asks whether misses like that are routine, and why. Across physics, medicine, nuclear data and lab comparisons, big misses turn up far more often than the error bars allow, though the typical gap is about the size the bars say. In the video we put "too small?" at about 90 % and "why?" at 50 %; both are our judgment, not a computed figure. The first is measured; the second is partly settled, since hidden errors and expectation both play a part and nobody can yet say how much of each.

Millikan's number

R. A. Millikan, "On the Elementary Electrical Charge and the Avogadro Constant", Physical Review 2, 109–143 (1913), DOI 10.1103/PhysRev.2.109, is in the public domain (Internet Archive scan). He timed oil drops falling between two plates and rising with a voltage on (pp. 109, 122); every charge was a whole multiple of one smallest step. On p. 140: e = 4.774 ± .009 × 10⁻¹⁰ electrostatic units (1.5924 ± 0.0030 × 10⁻¹⁹ C). D. Goodstein, "In Defense of Robert Andrews Millikan", Engineering and Science 63(4) (2000) (PDF), covers the apparatus and the 1923 Nobel Prize.

Today e is exactly 1.602 176 634 × 10⁻¹⁹ C, fixed in the SI since 20 May 2019 (NIST value; NIST on the redefinition; BIPM). With NIST's statcoulomb factor, 3.335 641 × 10⁻¹⁰ C (Guide to the SI, App. B.9), that is 4.803 205 in his units: Millikan was short by 3.24 times his ±, or 0.61 %. In fairness, he called his ± estimated limits rather than probable errors, which would be much smaller (p. 143): four factors, each allowed at most 0.1 %, combined as 2 parts in 1,000 (p. 140).

The drops he left out

The paper says its 58 drops cover 60 consecutive days (p. 133), and on p. 138:

this is not a selected group of drops

D. Allchin, "Flirting with Fraud" (1992, SHiPS), counts 175 drops in the notebooks, the first 68 before 13 February 1912. That leaves 107 for the period the paper covers, A. Franklin's 1981 count as cited by Henrion and Fischhoff (below): 58 published, 49 not. Goodstein, a defender, says about 100, with about 25 abandoned partway, and counts 63 days rather than 60. He transcribes a page of 14 March 1912 marked in red:

Beauty Publish

That drop made the paper. APS News (2006) confirms margin notes like it. Allchin has the same words crossed out on 10 April 1912, because Brownian motion, the random jiggling of a tiny drop, had spoiled the run, and notes beside dropped runs that the answer was too high (15 March) or too high by 1½ % (16 April 1912).

Critics call it cooking the data; M. Jeng, Am. J. Phys. 74, 578 (2006) (arXiv:physics/0508199), says a few drops went simply because their charges were too far off. Goodstein replies that drops too small jiggled and drops too large fell too fast, and you couldn't know until you'd measured. Both sides cite Franklin's reanalysis: put the usable drops back and the value and its limits barely move. The selection made the data look tidier; it didn't make the number wrong.

The air, and the hopes

What made it wrong was the air. Millikan's formula uses its viscosity, which he allowed to shift e by 0.1 % at most (p. 140); the value he used was off by more (Henrion and Fischhoff, p. 791; Feynman). In "Cargo Cult Science", his 1974 Caltech commencement address (Engineering and Science 37(7), text), Richard Feynman said later values crept up step by step, and why:

When they got a number closer to Millikan's value they didn't look so hard.

A speech, not data. Harder evidence comes from J. A. Bearden of Johns Hopkins, Physical Review 37, 1210–1229 (1931) (scan), who measured e with X-rays and crystals: 4.806 ± 0.003 × 10⁻¹⁰ esu, 0.06 % from today's value. He called his values "entirely too high" (p. 1229) and blamed the crystals, because they clashed with other constants that rested on Millikan's e.

A habit, not a fluke

M. Henrion and B. Fischhoff, "Assessing uncertainty in physical constants", Am. J. Phys. 54, 791–798 (1986) (copy), stretched old error bars into 98 % ranges, which the later value should escape 2 % of the time. It escaped 11 % of the time for 27 measurements of the speed of light (1875–1958), 29 % for 14 of G (1798–1983, leaving out three that missed by more than ten standard deviations) and 57 % for 40 recommended values (eight reviews, 1929–1969, judged against 1973). When light-speed results drifted, deBray proposed that light was slowing by about 4 km/s a year. Raymond Birge declared the speed of light settled in 1941; by 1950 the recommended value had moved 2.4 of his standard deviations, and that value was soon replaced too. Ordinary people in lab studies do the same: their 98 % ranges miss 20–40 % of the time. (The paper counts Millikan's miss as three standard deviations, uncovered 15 years on; that reads his ± as one standard deviation, though he meant a limit.)

Forty-one thousand measurements

D. C. Bailey of the University of Toronto, "Not Normal: the uncertainties of scientific measurements", Royal Society Open Science 4, 160600 (2017) (doi.org/10.1098/rsos.160600), took 41,000 measurements of 3,200 quantities, from medicine and physics to lab comparisons in chemistry and toxicology, and asked how many combined error bars apart each pair was. On a bell curve, more than five happens about 6 times in 10 million (5.7 × 10⁻⁷). His Table 1: 2.0 % of medical pairs, 2.4 % in particle physics, 3.3 % nuclear, 6.2 % in key comparisons between national standards labs, 7.8 % for constants, 12 % across lab comparisons. That is up to five orders of magnitude too often.

The twist: the typical bar is about the right size, in physics even a little big (§5). The excess sits in fat tails (Student's t, ν ≈ 2–3); even two-bar gaps occur in 16–35 % of pairs, not 4.6 %. So doubling every bar doesn't fix it (§4.3). Publication bias can't make tails this heavy, and old data aren't to blame: the newest data have the heaviest tails. In physics, unlike medicine, results published close together agree better (§3.6), which Bailey reads as expectation or shared methods.

When misses get traced, the culprit is a hidden error, not a forgotten fact: at NIST a sample ring slipped for 35 years, throwing off several half-lives (§4.2). Bailey's model, plausible but unproven, is that such errors come the way complex systems fail, software included: mostly small, now and then huge. Even the six national labs with the most key comparisons only reach ν ≈ 10 on physical standards, and for the best work he expects about 0.1 % of results beyond five bars, about a thousand times the bell.

G, the stubborn one

G, the number you need to weigh the Earth, is notoriously variable (Bailey §3.4); it is why the Earth's mass in our first episode (5.97200–5.97431 × 10²⁴ kg) is solid only to three digits. In the mid-1990s Germany's PTB found G = 6.715 40 ± 0.000 56 (83 ppm; Matters of Gravity No. 3, 1994, arXiv:gr-qc/9402002). The accepted value was 6.672 59(85), 128 ppm (Cohen & Taylor, CODATA 1986), so PTB was 0.64 % high, 50.4 of the accepted value's error bars. Finding the cause, a wrong assumption about a torque transmitter, took 8 years (Bailey §4.2).

Today's G = 6.674 30(15) × 10⁻¹¹ m³ kg⁻¹ s⁻² (22 ppm; NIST) rests on the 16 measurements in Table XXX of the CODATA 2022 report (P. J. Mohr et al., Rev. Mod. Phys. 97, 025002 (2025), arXiv PDF). By our count only 5 reach it with their error bars as published, where a bell curve expects 10.9. CODATA multiplies every bar by 3.9 (since 2018), after which 11 reach: agreement on paper, with no word on which is wrong.

NIST then repeated the Paris (BIPM) measurement on the same torsion balance over about ten years, blind: S. Schlamminger et al., Metrologia 63 (online 16 April 2026), DOI 10.1088/1681-7575/ae570f, CC BY 4.0 (abstract via Crossref). A colleague, Patrick Abbott, scrambled the data, and the reveal came on 11 July 2024 (NIST news). Result: 6.673 87(38) × 10⁻¹¹ (5.7 × 10⁻⁵), 250 ppm below BIPM, though within one standard deviation of the world average. No single cause turned up; the likeliest explanation is overlooked errors, probably several (Physics 19, 64). The next official values take results available by 31 December 2026 and are due early in 2027 (CODATA); on 2 November 2026 NIST still listed the 2022 set. We'll add the new G here when it appears.

A few details

Try it

For each question write a low and a high number, far enough apart that you're 98 % sure the answer lies between: the Moon's average distance in km; the seconds sunlight takes to reach us; how many Earths would fill the Sun; how many Earth masses make Jupiter; the length of a Mars day in hours. The video shows a drawn example from one made-up reader, not a measurement. Expect one or two misses; honest 98 % ranges would miss at least once only about one time in ten (1 − 0.98⁵ = 9.6 %), and twice or more about four times in a thousand. Five trivia questions aren't a lab, but a range that feels safe usually isn't.

Answers: 384,400 km, though the distance swings by about 40,000 km in a typical orbit (mean perigee 363,300, mean apogee 405,500 km; NASA). About 499 s, or 8 min 19 s: 149 597 870 700 m (IAU 2012, B2) over 299 792 458 m/s (NIST). About 1.3 million by volume (1,304,000; NASA). About 318 (317.83; NASA). 24.6597 h, about 24 h 40 min (NASA).

Image credits