Imagine if President Joe Biden, under advisement from the CQA, made a series of foreign policy blunders so grave they almost lead to nuclear war. Imagine if Biden, due to his reservations about the CQA’s activities, withheld full air support for a botched CQA military operation to overthrow the socialist Maduro regime in Venezuela, fostering anger among CQA agents.

The CQA’s anger only deepened after learning Biden was considering cutting their agency’s budget by 20%.

Imagine if the mission’s failure and Biden’s subsequent diplomatic overtures to the Maduro government (aimed at easing tensions) created resentment among anti-Maduro Venezuelan refugees, who longed to topple the Maduro government, install a US-backed liberal democracy, and return to their home country. In fact, some refugees even took part in the CQA’s failed mission, and continued to work closely with the agency.

Imagine if Maduro, following his rise to power, banned gambling and shut down Mob-run casinos, angering many an East Cost wise guy. Imagine if the CQA had long relied on the Mob for its failed assassination attempts on Maduro.

Now, imagine if President Biden was assassinated. The Shooter acted alone and was motivated by pro-Maduro and anti-American sentiments (or so you are told).

Imagine if, two days later, The Shooter was killed. The Second Shooter claims to have been motivated by his love for Biden and anger at The Shooter.

The Shooter’s sympathies for Maduro are evident in his defection to Venezuela years earlier, founding of a pro-Maduro organization, visits to Cuban and Venezuelan embassies, and socialist beliefs (or so you are told).

You later discover The Shooter returned to the U.S. following his supposed defection to Venezuela and even received a government loan upon his return. You later discover that the address listed on the socialist pamphlets The Shooter distributed on behalf of his pro-Maduro organization was that of a restaurant that served as a front for CQA and anti-Maduro Venezuelan refugee operations. You later discover The Shooter was the only member of this organization. You later come across a mysterious photo depicting The Shooter standing aside a top CQA official.

You later discover The Shooter never visited the Cuban and Venezuelan embassies at all: rather, someone impersonating The Shooter made these visits. The security cameras were mysteriously offline as well.

You later discover The Second Shooter had ties to the Mob and anti-Maduro Venezuelan militants, and that his financial troubles mysteriously lifted in the weeks preceding Biden’s assassination. You learn The Second Shooter personally witnessed the assassination (and, that morning, asked an undercover government agent if he “wanted to see the fireworks”), and left immediately afterwards to go to the hospital where Biden died.

You later discover one of the doctors who treated The Second Shooter in jail and diagnosed his psychosis (as The Second Shooter’s defense rested upon a “temporary insanity” argument) was a central participant in the CQA’s mind-control program and even used some of that program’s techniques on The Second Shooter. The Second Shooter was never the same after these visits.

You later discover that in the months before Biden’s assassination, another plot to murder the president was uncovered, one organized by anti-Maduro militant groups. You later discover the Secret Service agent who discovered this plan was arrested for mysterious “misconduct” charges and tied up in court during the entirety of the Pelosi Commission’s investigation into Biden’s murder.

You later discover the CQA handler of its anti-Maduro and Mafia partners was in charge of the flow of information between the CQA and the Pelosi Commission. But this was concealed by the handler’s use of an assumed name and only disclosed years later. You later encounter declassified documents confirming the CQA withheld information from the Pelosi Commission.

You watch in horror as several high-ranking Mob bosses are mysteriously gunned down in their homes days before each was slated to testify to the Pelosi Commission.

You hear news of the Pelosi Commission’s finding that The Shooter acted alone. You later discover several members of the Committee voiced opposition to this conclusion, and that the Committee’s meager funding limited the scope of its inquiry.

A subsequent Congressional investigation concludes Biden was murdered as the result of a conspiracy. You hear of forensic audio evidence on which this conclusion rested indicating the presence of a second shooter.

Now, read the above again, but replace “Biden” with “John F. Kennedy,” “Central Q Agency” with “Central Intelligence Agency,” “Maduro” with “Castro,” “Venezuela” with “Cuba,” “Lee Harvey Oswald” with “The Shooter,” “Jack Ruby” with “The Second Shooter,” and “Pelosi Commission” with “Warren Commission.” And imagine if the above, concerning JFK and Oswald and co., was all factually true - because it is.

Still believe the official narrative that Lee Harvey Oswald acted alone when he shot JFK?

“If you torture the data long enough, it will confess to anything”

- Ronald Coase

In my last post, I briefly mentioned p-hacking in my discussion of how many social science papers are unreliable, and how terrible research practices partially underlie the sorry state of the literature. In this post, I want to describe “p-hacking” in greater detail and illustrate the concept by doing some p-hacking myself.

p-values, explained

If you haven’t taken a statistics course or it’s been a while, you might have forgotten what a p-value is in the first place (or never learned, which is also fine!). Here is a quick refresher.

The p-value of a statistical test is the probability of obtaining a false positive result (i.e., a result that seems significant, but isn’t). To illustrate the concept, imagine you are a researcher analyzing this dataset of ~60 thousand OkCupid profiles.1 Let’s assume this is a representative sample of the population of all OkCupid users:

Suppose you want to determine whether the average age of male and female OkCupid users is meaningfully different.

Your first instinct might be to simply… find the average age by gender and compare. And huzzah! The numbers are different:

  sex   mean_age
  <chr>    <dbl>
1 f         32.8
2 m         32.0

Unfortunately, it’s not that simple. Top-line statistics can be numerically different without indicating a statistically significant difference due to random variation in the specifics of a given sample. To illustrate, suppose I create 1,000 sub-samples from the initial 60 thousand-large dataset (each consisting of 1,000 male profiles selected at random). Here is the distribution of average male age across the samples:

As you can see, each particular sample has a slightly different sample mean (i.e., the average (of age in this case) for a sample drawn from a larger population).

The theoretical distribution of potential sample means is called the sampling distribution, and in the case of the mean for a normally-distributed population, it looks an awful lot like what I created above (i.e. bell-shaped).2

Before returning to p-values in particular, I want to explain another topic that will frame the p-value discussion: hypothesis testing.

Hypothesis testing

Let’s return to the initial research question: whether male OkCupid users are (on average) older or younger than female users (i.e., whether there is a statistically meaningful difference in the average age of male and female users).

For a hypothesis test in statistics, one establishes a “null hypothesis” that says there is not a significant difference/effect. With hypothesis testing, you assume that there is no such difference between the means (or, in the case of a regression coefficient, that the coefficient is not meaningfully different from 0). In this example, the null hypothesis H_0 would be:

H_0: mean_age_male = mean_age_female

With hypothesis testing, you start by assuming the null hypothesis is true. So in this case, you would assume that the average age of male OkCupid users comes from the same sampling distribution as the mean age of female users.

We then “test” this hypothesis by seeing how likely it is that both means do in fact come from the same sampling distribution. If this seems unlikely, we reject the null hypothesis and accept the alternative (which is the logical complement of the null). In this case, the alternative hypothesis H_1 would be that the mean age of female OkCupid users is not equal to the mean age of male users):

H_1: mean_age_male ≠ mean_age_female

To recap: we can think of the sample mean as falling somewhere on a theoretical distribution of all potential sample means called the sampling distribution. In case it helps, here’s a diagram I made (with the m-looking greek letter mu the sample mean for a given sample i (so, mu_1 corresponds to potential sample 1, mu_2 corresponds to potential sample 2, etc.).

To determine whether to reject the null hypothesis, first imagine the average age of females is situated on the male sampling distribution (which is what we are saying with the null hypothesis). We then find the number of standard deviations the average age of females is from the center of the distribution.3 Here is where the mean age of females falls on the male sampling distribution:

Here is the information we need to calculate the number of standard deviations the mean age of females is from the mean age of men (which is our test statistic, t):

  sex   mean_age sd_age     n
  <chr>    <dbl>  <dbl> <int>
1 f         32.8  10.0  24117
2 m         32.0   9.03 35829

We find that t is approximately 10. Ten standard deviations out on the distribution! As a result, the p-value (the probability we obtained a false positive result, i.e. the probability our test statistic actually does come from the null distribution) is minute (<2.2e-16), and well below the 0.05 threshold typically used to define “statistical significance.” So we reject the null hypothesis in favor of the alternative hypothesis, that the mean age of female OkCupid users is meaningfully different from the mean age of male OkCupid users.

p-hacking, explained

P-values provide valuable information about the statistical significance of a difference in means (such as in the example above) or the effect of an independent variable on a dependent variable in the case of regression analysis. Unfortunately, p-values are easily-abused because they are calculated with the assumption that statistical tests are conducted one by one rather than repeatedly until the researcher obtains a “significant” result.

Many researchers compare numerous variables simultaneously in the hopes that one or more relationships are significant. And they do so without correcting for multiple comparisons (which is required due to the single-test assumption). Researchers also sometimes try to expand their sample until they reach significance. From this overview of p-hacking:

A lot of pressure rests on researchers to produce statistically significant results. For many researchers, statistical significance is the cornerstone of their academic career…

Now, what does a researcher do confronted with messy, non-significant results? According to several much-cited studies (for example John et al., 2012; Simmons et al., 2011), a common reaction is to start sampling again (and again, and again, …) in the hope that a somewhat larger sample size can boost significance. Another reaction is to wildly conduct hypothesis tests on the existing sample until at least one of them becomes significant (see for example: Simmons et al., 2011; Kerr, 1998 ). These practices, along with some others, are commonly known as p-hacking, because they are designed to drag the famous p-value right below the mark of .05 which usually indicates statistical significance. Undisputedly, p-hacking works (for a demonstration try out the p-hacker app).

…

As a showcase, we want to introduce two researchers: The HARKer takes existing data and conducts multiple independent hypothesis tests (based on multiple uncorrelated variables in the data set) with the goal to publish the ones that become significant. For example, the HARKer tests for each possible correlation in a large data set whether it differs significantly from zero. On the other hand, the Accumulator uses optional stopping. This means that he collects data for a single research question test in a sequential manner until either statistical significance or a maximum sample size is reached.

…

We can see that HARKing produces higher false positive rates than optional stopping with the same number of tests. This can be explained through the dependency on the first sample in the case of optional stopping: Given that the null hypothesis is true, this sample is not very likely to show extreme effects in any direction (however, there is a small probability that it does). Every extension of this sample has to “overcome” this property not only by being extreme in itself but also by being extreme enough to shift the test on the overall sample from non-significance to significance. In contrast, every sample in the multiple testing case only needs to be extreme in itself.

To illustrate the ease with which one can obtain a fake result, I will randomly generate data and find significant relationship between my made-up variables. If you want to try this for yourself, I recommend checking out this p-hacking demo.

For this example, suppose we are studying the predictors of cognitive ability, and imagine our dependent variables y1, y2, and y3 (which are meant to measure cognitive ability, the outcome of interest) are IQ, SAT score, and GRE score. Suppose we have a sample size of 100 and randomly generate data for y1, y2, and y3 as follows:

# y (IQ)
# normally distributed with a mean of 100 and a standard deviation of # 15
y1 = rnorm(100,mean=100,sd=15)

# SAT
# normally distributed with a mean of 1051 and a standard deviation 
# of 211
y2 = rnorm(100,mean=1051,sd=211)

# GRE
# normally distributed with a mean of 150 and a standard deviation of # 9
y3 = rnorm(100,mean=150,sd=9)

And suppose we randomly generate data for ten regressors, x1, x2, …, x10:

# Regressors:
x1 <- rbinom(100,size=1,prob=0.5)
x2 <- rbinom(100,size=1,prob=0.2)
x3 <- rbinom(100,size=5,prob=0.3)
x4 <- rnorm(100,mean=0,sd=1)
x5 <- rnorm(100,mean=10,sd=2)
x6 <-rnorm(100,mean=100,sd=10)
x7 <- rnorm(100,mean=0,sd=1)
x8 <- rpois(100,lambda=150)
x9 <- rlnorm(100,meanlog=0,sd=1)
x10 <- rbernoulli(100,p=0.1)

Lets first regress y1 (IQ) against all 10 regressors:

> m1 <- lm(y1~.,data=df)
> summary(m1)

Call:
lm(formula = y1 ~ ., data = df)

Residuals:
    Min      1Q  Median      3Q     Max 
-30.812  -9.426  -2.442   9.792  41.549 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)  
(Intercept) 127.854450  49.017901   2.608   0.0107 *
y2            0.005603   0.007924   0.707   0.4814  
y3            0.110255   0.238585   0.462   0.6451  
x1           -3.540796   3.404144  -1.040   0.3012  
x2           -2.959543   4.639410  -0.638   0.5252  
x3           -4.319908   1.646945  -2.623   0.0103 *
x4            0.623863   2.026799   0.308   0.7590  
x5            1.345545   0.912235   1.475   0.1438  
x6           -0.101083   0.170585  -0.593   0.5550  
x7           -0.754545   1.735537  -0.435   0.6648  
x8           -0.277310   0.140113  -1.979   0.0510 .
x9           -0.776470   0.842022  -0.922   0.3590  
x10          -2.239609   5.221288  -0.429   0.6690  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 15.79 on 87 degrees of freedom
Multiple R-squared:  0.151,	Adjusted R-squared:  0.03395 
F-statistic:  1.29 on 12 and 87 DF,  p-value: 0.239

And voila! We have found a statistically significant relationship (under p<0.05 criteria) between x3 and y1 (IQ), with a respectable p-value4 of 0.0103. And we also found a just barely-insignificant relationship between x8 and y1.

This isn’t surprising: under p < 0.05 significance criteria, each time an independent variable is regressed against the dependent variable there is a 5% chance of obtaining a spurious relationship (1-0.95=0.05). So in the case of ten dependent variables, the probability of obtaining at least one false positive is about 40% (1-(0.95)^10 = 0.401).

Yet many seemingly-upstanding researchers would take the above results and write about the relationship between x3 and y1 without mentioning the insignificant p-values for the other regressors. Indeed, the studies cited in the quote above (John et al. (2012) and Simmons et al. (2011)) testify to the stunning prevalence of p-hacking techniques and other shady research methods. In their survey of nearly 6,000 academic psychologists, John et al. (2012) find that, for the “Bayesian-truth-serum (BTS) condition” (one portion of respondents had their answers run through a scoring algorithm “used to provide incentives for truth telling”):5

[N]early 1 in 10 research psychologists has introduced false data into the scientific record and…the majority of research psychologists have engaged in practices such as selective reporting of studies, not reporting all dependent measures, collecting more data after determining whether the results were significant, reporting unexpected findings as having been predicted, and excluding data post hoc.

This article is getting to be a bit long, so I’ll save further discussion of p-hacking and other questionably research practices for another post. I hope this article gave you a better understanding of p-values, why p-hacking is so disturbingly easy, and the frightening ubiquity of p-hacking techniques (and thus unreliable research!).

Leave a comment

Share Irrational Actor

2

The sampling distribution is bell-shaped but not normally-distributed. Rather it is t-distributed. The t-distribution is similar to the normal distribution but has “fatter” tails:

My focus on social science isn’t just because it’s difficult, since STEM fields are plenty challenging in their own right. There is also my personal interest in social science (especially economics, which was one of my majors in college), and the fact that findings in social science are highly relevant to public policy debates (many of which I am personally invested in).

But my emphasis on social science and desire to interrogate its theories in these posts is also due to a recognition that social science is actually harder than other kinds of academic inquiry. Unlike researchers in the natural sciences, those in social science are studying largely invisible social phenomena that are imprecisely defined and measured. They usually resort to using observational data (data gathered passively rather than for the express purpose of academic research) and can rarely run the tightly-controlled experiments commonplace in some natural science fields.

Here is how Nature editors described some of the challenges facing social scientists in a piece responding to Congressional Republicans’ 2012 push to end NSF funding for political science research (emphasis mine):

The social sciences are an easy target for this type of attack because they are less cluttered with technical terminology and so seem easier for the layperson to assess. …

Part of the blame must lie with the practice of labelling the social sciences as soft, which too readily translates as meaning woolly or soft-headed. Because they deal with systems that are highly complex, adaptive and not rigorously rule-bound, the social sciences are among the most difficult of disciplines, both methodologically and intellectually. They suffer because their findings do sometimes seem obvious. Yet, equally, the common-sense answer can prove to be false when subjected to scrutiny. There are countless examples of this, from economics to traffic planning.

And here is Sociologist Rene Bekkers listing the reasons he thinks social science is especially challenging:

1. No Laws

All we have is probabilities.

2. All Experts

The knowledge we have is continuously contested. The objects of study think they know why they do what they do.

3. Zillions of Variables

Everything is connected, and potentially a cause – like a bowl of well-tossed spaghetti.

4. Many Levels of Action

Nations, organizations, networks, individuals, time all have different dynamics.

5. Imprecise Measures

Few instruments have near perfect validity and reliability.

Conclusion

Social science is not as easy as rocket science. It is way more complicated.

Now, that social science is difficult is not the only reason it’s worth expending considerable effort scrutinizing its findings and theories (rather than taking them at face value). There is also the specter of shoddy methodology and, far too often, something akin to outright fraud.

In other words, large swaths of social science research are unreliable due to avoidable failings on the part of the researchers. FantasticAnachronism describes the situation in grim detail:

There's a popular belief that weak studies are the result of unconscious biases leading researchers down a "garden of forking paths". Given enough "researcher degrees of freedom" even the most punctilious investigator can be misled.

I find this belief impossible to accept. The brain is a credulous piece of meat but there are limits to self-delusion. Most of them have to know. It's understandable to be led down the garden of forking paths while producing the research, but when the paper is done and you give it a final read-over you will surely notice that all you have is a n=23, p=0.049 three-way interaction effect (one of dozens you tested, and with no multiple testing adjustments of course). At that point it takes more than a subtle unconscious bias to believe you have found something real. And even if the authors really are misled by the forking paths, what are the editors and reviewers doing? Are we supposed to believe they are all gullible rubes?

…

Even when they do accuse someone of wrongdoing they use terms like "Questionable Research Practices" (QRP). How about Questionable Euphemism Practices?

• When they measure a dozen things and only pick their outcome variable at the end, that's not the garden of forking paths but the greenhouse of fraud.

• When they do a correlational analysis but give "policy implications" as if they were doing a causal one, they're not walking around the garden, they're doing the landscaping of forking paths.

• When they take a continuous variable and arbitrarily bin it to do subgroup analysis or when they add an ad hoc quadratic term to their regression, they're...fertilizing the garden of forking paths? (Look, there's only so many horticultural metaphors, ok?)

If you’re not familiar with the so-called “Replication Crisis” and the way researchers routinely produce spurious findings, I encourage you to read this piece by Scott Alexander (part of a larger series on the reliability of scientific research). Here's an enlightening excerpt about a darkly revealing psychology paper, beginning with four tricks to make almost any research yield “statistically significant” results (practices that are employed by a shamefully large number of actual researchers):

1. Measure multiple dependent variables, then report the ones that are significant. For example, if you’re measuring whether treatment for a certain psychiatric disorder improves life outcomes, you can collect five different measures of life outcomes – let’s say educational attainment, income, self-reported happiness, whether or not ever arrested, whether or not in romantic relationship – and have a 25%-ish probability one of them will come out at significance by chance. Then you can publish a paper called “Psychiatric Treatment Found To Increase Educational Attainment” without ever mentioning the four negative tests.

2. Artificially choose when to end your experiment. Suppose you want to prove that yelling at a coin makes it more likely to come up tails. You yell at a coin and flip it. It comes up heads. You try again. It comes up tails. You try again. It comes up heads. You try again. It comes up tails. You try again. It comes up tails again. You try again. It comes up tails again. You note that it came up tails four out of six times – a 66% success rate compared to expected 50% – and declare victory. Of course, this result wouldn’t be significant, and it seems as if this should be a general rule – that almost by the definition of significance, you shouldn’t be able to obtain it just be stopping the experiment at the right point. But the authors of the study perform several simulations to prove that this trick is more successful than you’d think:

3. Control for “confounders” (in practice, most often gender). I sometimes call this the “Elderly Hispanic Woman Effect” after drug trials that find that their drug doesn’t have significant effects in the general population, but it does significantly help elderly Hispanic women. The trick is you split the population into twenty subgroups (young white men, young white women, elderly white men, elderly white women, young black men, etc), in one of those subgroups it will achieve significance by pure chance, and so you declare that your drug must just somehow be a perfect fit for elderly Hispanic women’s unique body chemistry. This is not always wrong (some antihypertensives have notably different efficacy in white versus black populations) but it is usually suspicious.

4. Test different conditions and report the ones you like. For example, suppose you are testing whether vegetable consumption affects depression. You conduct the trial with three arms: low veggie diet, medium veggie diet, and high veggie diet. You now have four possible comparisons – low-medium, low-high, medium-high, low-medium-high trend). One of them will be significant 20% of the time, so you can just report that one: “People who eat a moderate amount of vegetables are less likely to get depression than people who eat excess vegetables” sounds like a perfectly reasonable result.

Then they run simulations to show exactly how much more likely you are to get a significant result in random data by employing each trick:

The image demonstrates that by using all four tricks, you can squeeze random data into a result significant at the p < 0.05 level about 61% of the time.

Try this p-hacker demo to see for yourself.

Due to these research practices, as well as other distortions like publication bias, the existing social science literature is far less reliable than we would expect even if we concede that the research itself is incredibly challenging.

However, there is a real question as to whether a non-expert blogger with only an undergraduate education (in one social science, economics) is really equipped to identify credible research. But professional researchers themselves don’t seem to be very good at this either. The graph below shows how papers that *failed* to replicate are cited just as often as papers that passed replication.

One workaround here is to use broadly applicable rules of thumb to assess a theory/finding. Regardless of the specific field, if a paper has a small sample size, a p-value close to 0.05, uses regression analysis and attempts to “control for confounders,” has small effect sizes, and/or conducts extensive subgroup analysis/turns continuous variables into categorical ones, it should probably be regarded as low-quality.

Likewise, papers that have large sample sizes, very small p-values, large effect sizes, etc.; use a randomized controlled trial methodology or another method capable of establishing causality; correct for multiple comparisons; pre-register their study designs so they can’t adjust their methods as they go (as described by Scott Alexander above); etc. are more likely to have reliable findings. Again, you can draw these conclusions based on just a few metrics (sample size (n), significance level (p), etc.) even if you know little about the subject matter under study.

Share Irrational Actor

Blogging as a (promising) genre of writing

So I definitely want to use this as a space to examine elements of social science (especially economics), and use theories and evidence I deem credible to understand the institutions and forces most relevant in the world of 2021. I also want to try my hand at making predictions, and hope to weigh in on public policy debates and current events based on my understanding of the relevant theory, literature, and (sometimes) my own analysis. But I have another motivation for starting this blog that I want to touch on briefly.

Blogs as a genre of writing are situated somewhere between longer-form essays/articles/op-eds (which do a good job articulating things carefully and persuasively at the expense of greater time & effort by the author and fewer opportunities for reader engagement), and social media posts (which basically do the opposite). They strike a nice balance between quantity and quality of posts.

They also allow for iterative improvement of one’s ideas (blog post idea —> writing the post itself and clarifying/refining my thoughts —> after posting, engaging with reader feedback/criticism —> a new, “better” idea that incorporates what I learned through writing and reading responses to the first post —> rinse and repeat). The Substack platform is especially good at facilitating this process thanks to its user-friendly social features (leave a comment on this post if you want to see for yourself!). And, because blog posts are usually edited more lightly than essays/news articles and churned out more quickly, this iterative process occurs at a reasonable clip.

So welcome to Irrational Actor, and I hope you will engage critically and thoughtfully with the ideas I express here. And please share your own ideas as well in the comments section. The more people that read and comment on these posts, the higher the quality of comment-section discourse, so please share this blog with friends and family if you think they would find it interesting!

Leave a comment