Can You Repeat That? Replication Bias and Caffeine/Health Connections

National and International coffee days (yes there are two) passed in a flash last weekend. Also in a flash went the last three years since Hearing Economics updated research findings on the effects of caffeine and coffee on bodily systems. Today’s ridiculously long post aims to correct that glaring gap.

No worries — the caffeine obsession is intact at Hearing Economics which serves as a kind of blogger coffee house for coffee obsessives. For example, we received this communique from fellow blogger, coffee enthusiast, marketing entrepreneur, and self-styled “bio-hacker” Gary Kirwan:

I came across an old blog post of yours talking about international coffee day here and wanted to let you know I wrote a big, referenced article on the difference between National Coffee Day in the US and International Coffee Day in the rest of the world. I found the Wikipedia page didn’t go into enough detail and might confuse people about the difference so I went and found the details and linked it all in one place. Here is a direct link explaining International Coffee Day which is on 1 Oct each year and here is another direct link explaining National Coffee Day, which is on 29 September each year. 

I hope you might find it useful for your blog post.
Thank you Gary, very useful. Hearing Economics is happy to promote your sites and confirm official dates on our calendar for observant coffee drinking and writing.

Again, What Does Coffee Have to Do with Health, Hearing or Economics?


More than in the past” is the short answer to that header. The economics of coffee lies in increased productivity of those who drink, as discussed in a 2014 coffee post and presumed  to continue in 2017. As far as health studies go, coffee is a natural because it is consumed by people of all ages, irrespective of gender, throughout the world. Its effects, if they exist, are manifest in the data and only await study designs that pull them out.

On the whole-body level, the 2015 revised USDA nutritional guidelines  dietary guideline advisory committee reviewed the effects of coffee on health and pronounced it good for us, which marks a first. Underscoring this pronouncement is the finding from two studies (row 4 and end note 4 of the table below) that coffee affects mortality in a good way. Coffee may not make us immortal, but at least it may give us a few more years to drink it, which is certainly cause for joy and perhaps increased productivity. Readers can peruse the ever-lengthening table to see coffee/caffeine effects on specific body systems and some diseases; also to note that the “benefits” column is full to the brim while the “adverse” column is down to dregs.

We just do not know how coffee affects hearing, balance, or tinnitus…. but we’d sure like to know. (2012 post). 

And if they ever find a connection between hearing loss and coffee, I’m doomed. (2012 comment, Anon)

The absence of research and meta-analyses on effects of caffeine and coffee in the auditory and vestibular systems in changing, albiet slowly. Fortunately, change is generally in the right direction for coffee drinkers and rodents:

  • Good. Inverse relationships between tinnitus and coffee consumption were found in two 2014 studies of epidemiological data. In the US, Glickman and colleagues reported it for incident tinnitus in women; in the UK, McCormack et al. reported it for prevalence of transient and persistent tinnitus in middle age subjects.
  • Good. In two earlier animals studies (2008 and 2009), coffee protected and facilitated recovery from auditory neuropathy in mice. 
  • Bad. A 2016 animal study by Zawawi et al found recovery from noise induced hearing loss in guinea pigs was impaired by caffeine.

And that’s it so far for coffee/caffeine effects: Robust studies on tinnitus, absence of studies on hearing loss in humans, a few animals models in the works, essentially zilch for vestibular effects.

Despite slim pickings, the media had a field day with the Zawawi study, blaring out headlines such as:

Your Daily Coffee Consumption Could Be Making You Deaf.”

Probably not, unless you are one of the 8 female albino guinea pigs selected at random to receive daily doses of noise and intraperitoneally-injected caffeine.  Before we believe that headline, let’s get the study replicated with more guinea pigs and then move on to see if the probabilities in the animal model have application to humans who drink it. 


Got Research?


Which brings this interminable post to the point of replication as an explanation of its title. As mentioned, coffee is the perfect variable in epidemiological research because it’s practically ubiquitous and it can be counted in cups. Hence, the extremely long table below, which contains a multitude of studies that run the gamut from case studies to meta-analyses, from animal models to human subjects. 

A quick glance at the table is enough to realize that not all findings have the same significance, despite their P values. Study after study shows coffee significantly affecting one or another bodily system,  which begs two questions about research in general and this table in particular. The following paragraphs are reprinted from a prior post as a reminder that research findings reflect probabilities, not certainties, and that anything can happen by chance, especially when your sample size is small. 

Q1:  How many of the significant findings in the table are statistical flukes?

A1: Probably more than you’d think.  Most studies use a P value, usually .05, as a statistical line in the sand to distinguish “significant” findings ([p<.05) from normal sampling variation (p ≥ .05).  But these are probabilities, meaning that your finding has a 5% chance (1 in 20) of being a chance event.  Worse, if you replicate your study to ensure you’ve really got something, you have a 9.75% chance1 that one of your significant findings is a random event, another way of saying it’s wrong.  Even worse is what the popular media has dubbed science’s “replication crisis” in which one-time significant findings are accepted as the real deal without repeating to check for chance.

Even if there is no real effect in your data, you have over a 40% chance of finding a fake effect if you do the analysis 10 times.  Apparently, some researchers have caught on to that and keep at it till they get the (random) result that yields (spurious) significance.  So says Christopher Labos, MD, in a delightful article that tells you why you need to take the table below with a  grain of salt and not put all your money into Starbucks stock.

Q2:  What does statistical significance imply about clinical significance?

A2:  Nothing if the study is biased.   Dr. Labos explains a variety of bias types that render findings meaningless, including selection bias, which he illustrates perfectly with coffee:

a 1981 NEJM study showed an association between coffee consumption and pancreatic cancer. The selection bias occurred when the controls were recruited for the study. The control group had a high incidence of peptic ulcer disease, and so as not to worsen their symptoms, they drank little coffee. Thus, the association between coffee and cancer was artificially created because the control group was fundamentally different from the general population in terms of their coffee consumption. When the study was repeated with proper controls, no effect was seen.


Are You Feeling Lucky?


Dr. Labos makes one more helpful coffee-driven statistical point about pitfalls of risk descriptions, such as in the NHSII-tinnitus study above.  His main point is that non-case studies need to use relative risk ratios (RRR) and not odds ratios because the latter overstate risk and scare people.  Audiologists, don’t worry — the tinnitus study used hazard ratios, which are a special form of relative risk ratio that works to minimize selection bias.

His secondary point is that relative risks are ratios that say nothing about absolute risks, which are rarely reported in studies even though they are the ones that are most useful in making health decisions.  He uses another excellent coffee example (also worth reading in full) to show why the relative risk of having a heart attack within an hour of consuming a cup of coffee (1.5, or 50% increased risk) translates in real life to an absolute risk of one heart attack per 2 million cups of coffee.

Now that we’re all primed with statistical knowledge and recognize that Hearing Economics suffers from confirmation bias in its selection process, we can risk reading through the table while sipping cups of coffee and listening to our tinnitus diminish and our hearts beat reassuringly.  Happy National and International Coffee Days for Audiologists and their patients!



Adverse Effects




Driving Safety

Reduced risk of accidents for long-haul truckers (63%)[i]


Physical capacity

Exercise performance

Activity level/motivation

  • Increased benefit (used to be considered “doping”)
  • Ergogenic aid for perceived effort & neural substrate availability[ii]
  • >2 c/day increases physical activity, may depend on genetic variation (ADORA2A gene)[iii]

Mortality (all causes)

Risk of dying over 13 year period reduced 10%-15% with 2+ cups/day[iv]



  • Dependence abuse


  • Tolerance

“little evidence”

  • Withdrawal Syndrome

Benign, 50% who stop have headache; tiredness; &/or decreased attentiveness and contentedness

Cardiovascular Disease

  • Heart failure

Progressive risk reduction up to 4-5 cups/day[v]

> 5 cups/day increases risk progressively

  • Myocardial infarction

Protective Effect

  • Heart arrhythmia

Habitual consumption reduces AF risk[vi]

May trigger in very susceptible individuals


  • Cardiovascular events (mainly heart attack)


Increased in 18-45 year olds with hypertension if ≥4 cups/day[vii]
  • Blood Pressure

“small but significant” reduction in systolic and diastolic blood pressure[viii]

BP spikes w/ caffeine x tranylcypromine  interaction (case study)[ix]

  • Ischemic Stroke

Reduced risk of stroke for 1-3 cups/day[x]

  • Intracerebral  hemorrhage

17% risk reduction with 1+ cups/day[xi]


Possible trigger factor for intracranial aneurysm rupture (subarachnoid hemorrhage)[xii]

  • Women:  CV disease

No deleterious effects[xiii]

No helpful effects

Endocrine Disorders

  • Insulin sensitivity




  • Type II diabetes

inverse risk reduction: 33% lower with 6 cups/day (caffeinated or decaf)[xiv];

caffeinated increased intake (1.5 c/day)  reduces risk; decreased intake increases risks[xv]


>4 cups/day reduces risk of onset  of Type II diabetes in post-menopausal women[xvi]


Digestive System                       


Reduced death from oral cancers[xvii]


Liver cancer

3 cups/day reduces risk of hepatocellular carcioma (HCC) by as much as 50%[xviii]


Liver cirrhosis (alcohol related)

Inverse risk, 22% – 65% reduction with 1 to 4 cups/day[xix]


Liver fibrosis

Lowered in some patients[xx], [xxi]



Colorectal cancer

15% lower risk, 4+ cups/day[xxii]

52% lower odds of colon cancer recurrence, 4 cups/day[xxiii]


Bowel surgery

Post surgical bowel movement 14 hours sooner with caffeine[xxiv]


Mental Processes & Disease

Anxiety Symptoms


Possible link

  • Alzheimer’s disease

Reduction of risk

  • Memory

Improves memory consolidation[xxv]

  • Depression

Reduced (relative to sweet drinks), 4 or more cups/day[xxvi]

  • Depression in Women

15-20% decreased risk with consumption of 2-4 cups/day[xxvii]

  • Cognitive Decline


  • Mild cognitive decline (MCD)


reduced rate of decline[xxviii]


lower risk if routine consumption of1-2 cups/day [xxix]

  • Cognitive Decline in Women

Protective effect, increasing with age, for > 3 cups/day[xxx]


risk decreases as consumption increases to 7 cups/day

Risk increases above 8 cups/day

Nervous System Disorders

  • Parkinson’s disease

Dose-related protective effect


“modest” improvement in motor function[xxxii]



  • Gout

Reduced incidence, dose-dependent


Reproductive Systems

  • Uterine/Endometrial cancer

>3 cups/day lowers risk[xxxiii]

  • Prostate Cancer

6 cups/day: “striking decrease” in fatal, metastatic disease[xxxiv]

  • Pregnancy & Postpartum

Coffee intake in pregnancy does not cause behavioral problems in offspring[xxxv]


No sleep consequences for infants at 3 months[xxxvi]

  • Erectile Dysfunction (ED)

Inverse relationship, 2-3 cups/day reduces odds of prevalent ED except in diabetic men[xxxvii]


Auditory/Vestibular System

  • Auditory neuropathy

Facilitates recovery from diabetic & experimentally-induced AN (auditory neuropathy) in mice[xxxviii], [xxxix]

  • Noise induced threshold shifts

Impairs recovery from noise induced hearing loss in guinea pigs[xl]

  • Tinnitus

inverse relationship of caffeine intake and:

  • Tinnitus incidence in women[xli]
  • Prevalence modestly reduced for transient and persistent tinnitus[xlii]


  • Vestibular system

Does not affect vestibular & balance test results[xliii], [xliv]


Visual System

  • Dry Eye

May reduce by boosting tear production[xlv]

  • Glaucoma

Increased risk for >500 mg/day caffeine[xlvi]


  • Aging


“powerful anti-aging botanical”  purported to reduce signs of skin aging[xlvii]

  • Skin cancer
  • 17% reduced relative risk of basal cell carcinoma (BCC) with 3 cups per day[xlviii]
  • 20% risk reduction of malignant melanoma with ≥4 cups caffeinated coffee/day[xlix]


This is the 9th post in the coffee series. Click links for previous posts published on April 25 2012, May 2 2012, Sept 25 2012, Oct 2 2012, Sept 24 2013, Oct 2 2013, Sept 23 2014, Sept 30 2014




[i] Sharwood LN et al.  Use of caffeinated substances and risk of crashes in long distance drivers of commercial vehicles: case-control study.  BMJ 2013;346:f1140. 

[ii] Cole KJ et al.  Effect of caffeine ingestion on perception of effort and subsequent work production. Int J Sport Nutr. 1996 Mar;6(1):14-23.

[iii] American College of Sports Medicine (ACSM) 61st Annual Meeting. Abstracts 423 and 316. Presented May 28, 2014.  Two presentations reported in Medscape.

[iv] Freedman ND et al. Association of Coffee Drinking with Total and Cause-Specific Mortality. N  Engl J Med 2012; 366:1891-1904, May 17, 2012; Ding M et al. Association of Coffee Consumption with Total and Cause-Specific Mortality in Three Large Prospective Cohorts. Circulation; 136(13); Nov 16, 2015.

[v] Mostofsky E et al. Habitual Coffee Consumption and Risk of Heart Failure: A Dose–Response Meta-Analysis. Circulation Heart Fail. 2012; 5(4):401-405.

[vi] Cheng M, Hu Z, Lu X, et al. Caffeine intake and atrial fibrillation incidence: Dose response meta-analysis of prospective cohort studies. Canadian J Cardiol 2014: DOI:10.1016/j.cjca.2013.12.026. Abstract

[vii] Palatini P et al. Coffee consumption and risk of cardiovascular events in hypertensive patients. Results from the HARVEST. ScienceDirect 2016;212(1 June): 131-137.

[viii] Thomas-Jean F, et al. Are you tea or coffee addict? The long term beneficial effect on blood pressure.  Parallel Oral Session 7A, 5/17/13, 9:40am.  23rd European Meeting on Hypertension & Cardiovascular Protection.  Milan, June 14 −17, 2013.

[ix] van der Hoeven N et al.  Severe hypertension related to caffeinated coffee and tranylcypromine: A case studyAnn Intern Med. 2014;160(9):657-658. doi:10.7326/L14-5009-8

[x] D’Elia L et al. no title given. European Society of Hypertension (ESH) 2012: 22nd European Meeting on Hypertension and Cardiovascular Protection. April 26 – 29, 2012; London, United Kingdom.

[xi] Kokubo Y et al.  The impact of green tea and coffee consumption on the reduced risk of stroke incidence in Japanese population:  The Japan public health center-based study cohort.  Stroke, 3/14/2013.

[xii] Vlak MHM et al. Trigger Factors and Their Attributable Risk for Rupture of Intracranial Aneurysms.  Stroke, 5/11/2011.

[xiii] Garcia-Lopez E. no title given.  American J of Clin Nutrition; May 13, 2011.

[xiv] Ding M et al.  Caffeinated and Decaffeinated Coffee Consumption and Risk of Type 2 Diabetes: A Systematic Review and a Dose-Response Meta-analysis.  doi: 10.2337/dc13-1203. Diabetes Care February 2014 vol. 37 no. 2 569-586.

[xv] Bhupathiraju S N et al. Changes in coffee intake and subsequent risk of type 2 diabetes: three large cohorts of US men and women.  Diabetologia. Published online April 24, 2014. 

[xvi] Goto A et al. Coffee and caffeine consumption in relation to sex hormone-binding globulin and risk of type 2 diabetes in postmenopausal women. Diabetes. 2011 Jan;60(1):269-75. Epub 2010 Oct 28.

[xvii] Bakalar N. Risks: Coffee Linked to Fewer Oral Cancer Deaths.

[xviii] Bravi F, La Vecchia C, Gallus S, et al. Coffee Reduces Risk for Hepatocellular Carcinoma: An Updated Meta-analysis. Clinical Gastroenterology and Hepatology. 2013; Loomis, D, et al. Carcinogenicity of drinking coffee, mate, and very hot beverages. The Lancet Oncology , Volume 17 , Issue 7 , 877 – 878.

[xix] Kennedy OJ et al. Systematic review with meta-analysis: coffee consumption and the risk of cirrhosis. Alimentary Pharmacology and Therapeutics 2016;43:562-574.  doi:10.1111/apt.13523

[xx] Molloy JW et al. (2012), Association of coffee and caffeine consumption with fatty liver disease, nonalcoholic steatohepatitis, and degree of hepatic fibrosis. Hepatology, 55: 429–436. doi: 10.1002/hep.24731

[xxi] Bambha K.  no title given.  Presented at The Liver Meeting 2012: American Assn for the Study of Liver Diseases (AASLD) 63rd Annual Meeting.  Abstract 99. November 11, 2012.

[xxii] Rashmi S et al. Caffeinated and decaffeinated coffee and tea intakes and risk of colorectal cancer in a large prospective study. Am J Clin Nutr, June 13, 2012, doi: 10.3945/​ajcn.111.031328.

[xxiii] Brendan J et al. Coffee Intake, Recurrence, and Mortality in Stage III Colon Cancer: Results From CALGB 89803 (Alliance). Journal of Clinical Oncology, August 2015

[xxiv]  Müller SA et al. Randomized clinical trial on the effect of coffee on postoperative ileus following elective colectomy. Brit J Surg. 2012:99:1530-1538.

[xxv] Borota d et al.  Post-study caffeine administration enhances memory consolidation in humans. Nature Neuroscience (2014) doi:10.1038/nn.3623; Published online 12 January 2014 Corrected online 17 January 2014.

[xxvi] Chen, American Academy of Neurology’s 65th Annual Meeting. Abstract 2257. Released January 8, 2013.  Sweetened Drinks May Boost Depression, Coffee Reduce It.  

[xxvii] Lucas, M et al.  Coffee, Caffeine, and Risk of Depression Among Women.  & Brauser D.  Coffee May Keep Depression Away

[xxviii] Arab L et al.  Epidemiologic evidence of a relationship between tea, coffee, or caffeine consumption and cognitive decline.  Adv Nutr January 2013  vol. 4: 115-122, 2013

[xxix] Solfrizzi V et al. Coffee Consumption Habits and the Risk of Mild Cognitive Impairment: The Italian Longitudinal Study on Aging. Journal of Alzheimer’s Disease, Volume 47, Issue 4 (September 2015).

[xxx] Richie K et al. The neuroprotective effects of caffeine. A prospective population study (the Three City Study).  Neurology, August 7, 2007 vol. 69( 6), pp 536-54. doi: 10.1212/01.wnl.0000266670.35219.0c

[xxxi] Tanskanen A. et al.  Heavy coffee drinking and the risk of suicide. European J of Epidemiology, 16(9),  (2000), 789-791, DOI: 10.1023/A:1007614714579

[xxxii][xxxii] Postuma RB. no title given. Movement Disorder Society’s 16th International Congress of Parkinson’s Disease and Movement Disorders. Abstract 421. Presented June 18, 2012

[xxxiii] Gunter MR et al. A prospective investigation of coffee drinking and endometrial cancer incidenceInt J Cancer. 2012 Aug 15;131(4):E530-6. doi: 10.1002/ijc.26482. Epub 2011 Nov 17; Loomis, D, et al. Carcinogenicity of drinking coffee, mate, and very hot beverages. The Lancet Oncology , Volume 17 , Issue 7 , 877 – 878.

[xxxiv] Wilson KM et al. Coffee Consumption and Prostate Cancer Risk and Progression in the Health Professionals Follow-up Study. JNCI J Natl Cancer Inst (2011). doi: 10.1093/jnci/djr151. First published online: May 17, 2011

[xxxv] Loomans E et al. Caffeine Intake During Pregnancy and Risk of Problem Behavior in 5- to 6-Year-Old Children.  Pediatrics .2011-3361; published ahead of print July 9, 2012, doi:10.1542/peds.2011-3361.

[xxxvi] Santos I, et al.  Maternal Caffeine Consumption and Infant Nighttime Waking. : Prospective Cohort Study.  Pediatrics, 2, 2012.

[xxxvii] Lopez DS et al. Role of Caffeine Intake on Erectile Dysfunction in US Men: Results from NHANES 2001-2004. PLOS One;10(4); 2015.

[xxxviii] Hong BN et al. High-dosage pyridoxine-induced auditory neuropathy and protection with coffee in miceBiol Phar Bull. 2009 Apr; 32(4); 597-603.

[xxxix] Hong BN et al.  Coffee improves auditory neuropathy in diabetic mice.  Neurosci Lett. 2008 Aug 29; 441(3): 302-6.

[xl] Zawawi F et al. Association of Caffeine and Hearing Recovery After Acoustic Overstimulation Events in a Guinea Pig Model. JAMA Otolaryngol Head Neck Surg. 2016 Apr;142(4):383-8. doi: 10.1001/jamaoto.2015.3938.

[xli] Glicksman JT, et al.  A prospective study of caffeine intake and risk of incident tinnitus. Am J Med, 127(8), 3/6/2014.

[xlii] McCormack, A et al.  Association of Dietary Factors with Presence and Severity of Tinnitus in a Middle-Aged UK Population. PLOSOne. Dec 12 2014.

[xliii] FELIPE, Lilian et al . Evaluation of the caffeine effect in the vestibular testRev. Bras. Otorrinolaringol.,  São Paulo ,  v. 71, n. 6, Dec.  2005 .   

[xliv] Enriquez A, Sklaar J, Viirre E, Chase B. Effects of Caffeine on Postural Stability. Int Tinnitus J. 2009;15(2):163-163

[xlv] Arita R.  Caffeine Increases Tear Volume Depending on Polymorphisms within the Adenosine A2a Receptor Gene and Cytochrome P450 1A2. Ophthalmology, Volume 119, Issue 5 , Pages 972-978, May 2012.

[xlvi] Pasquale LR et al (2012), The Relationship between Caffeine and Coffee Consumption and Exfoliation Glaucoma or Glaucoma Suspect: A Prospective Study in Two Cohorts.  Invest. Ophthalmol. Vis. Sci. September 21, 2012 vol. 53 no. 10 6427-6433. 

[xlvii]Revaleskin Coffee Berry treatments.

[xlviii] Turati F et al. Coffee and cancers of the upper digestive and respiratory tracts: meta-analyses of observational studies. Ann Oncol. 2011 Mar;22(3):536-44.  Epub 2010 Oct 13.

[xlix] Loftfield E et al.  Coffee Drinking and Cutaneous Melanoma Risk in the NIH-AARP Diet and Health Study,.JNCI: Journal of the National Cancer Institute, Volume 107, Issue 2, 1 February 2015.




1If significance level (alpha) is set to a probability of .05 (1-0.95) and the experiment is repeated, then probability is 1-0.95^2 = 0.0975. Further replications continue to raise the probability stakes by the same formula.

About Holly Hosford-Dunn

Holly Hosford-Dunn, PhD, graduated with a BA and MA in Communication Disorders from New Mexico State, completed a PhD in Hearing Sciences at Stanford, and did post-docs at Max Planck Institute (Germany) and Eaton-Peabody Auditory Physiology Lab (Boston). Post-education, she directed the Stanford University Audiology Clinic; developed multi-office private practices in Arizona; authored/edited numerous text books, chapters, journals, and articles; and taught Marketing, Practice Management, Hearing Science, Auditory Electrophysiology, and Amplification in a variety of academic settings.