P-values, p-hacking and critical values from software
The p-value is the smallest significance level at which you would reject the null: the probability, if H₀ were true, of a test statistic at least as extreme as yours. Reject when p < α. P-hacking, re-running an analysis until p dips below α, manufactures false positives.
Watch the lesson 13 min
The video follows this page: the same examples, then the checkpoint questions.
Before this: How a hypothesis test works: four steps from claim to decision
On the exam
- Explain hypothesis testing and its components, including statistical significance, Type I and Type II errors, and the power of a test; construct appropriate hypothesis tests; and interpret the results
Interpret a p-value and explain how p-hacking distorts statistical significance. Expect 'reject at 5% but not at 1%?', turning a one-tailed p-value into a two-tailed one, a scenario to label as p-hacking (and its effect on Type I errors), and which spreadsheet function gives a critical value.
How deeply to learn each part
- Know the idea
- Understand it in general. It frames other topics, but you will not calculate with it.
- Use it
- Understand it well enough to solve a question with it. Recognize when it applies.
- Memorize
- Learn it by heart. The exam will not give it to you.
You know this one. The full lesson is folded away. The takeaways, formulas and checkpoint below are your quick refresher, and the checkpoint is a fair test of whether you still do.
You tested strong here. The full lesson is folded away. The takeaways, formulas and checkpoint below are your quick refresher, and the checkpoint is a fair test of whether you still know it.
What is a p-value?
Memorize
The p-value is the probability of getting a test statistic at least as extreme as the one observed, if were true. It is also the smallest significance level at which would be rejected.
Which tail counts depends on the alternative hypothesis.
- Right-tail test Memorize P(T ≥ t) under H₀
- Left-tail test Memorize P(T ≤ t) under H₀
- Two-tailed test Memorize 2 × P(T ≥ |t|) under H₀
The decision rule: reject if p < α; fail to reject if p ≥ α. It gives the same answer as the critical-value method, plus a measure of strength of evidence : the smaller the p-value, the stronger the evidence against .
What a p-value is not: it is not the probability that is true, and 1 − p is not the probability that the strategy works.
Exam tip Two-sided alternative with a one-tailed p-value from software? Double it before you compare with α.
What is p-hacking?
Use it
P-hacking is reworking an analysis (another period, another benchmark, dropping awkward months) until a result crosses the significance line, then reporting only that version. It inflates the rate of Type I errors: results look significant when nothing is there.
- Selective performance reporting Use it choosing the period, benchmark or ratio that flatters
- Data mining for strategies Use it testing many rules until one wins
- Manipulating metrics Use it smoothing data or dropping outliers before computing value at risk (VaR) or Sharpe ratios
- Defenses Use it how to keep false positives out
- Confirmatory testing Know the idea fix the hypothesis and the data first, then repeat the test
- Out-of-sample validation Use it test on data the strategy has never seen
- Transparent methods Use it report every test you ran
- Ethical standards Know the idea fair, complete performance presentation
-
- Selective performance reporting picks the period, benchmark or ratio that flatters the result. It is a performance-presentation problem as much as a statistical one: see Standard III(D), Performance Presentation for the duty to present performance fairly and completely.
- Data mining tests many rules and keeps the winner.
- Metric manipulation smooths the data or drops outliers before computing value at risk (VaR) or Sharpe ratios.
The defenses: fix the hypothesis and the data before testing (confirmatory testing ), confirm with repeat tests, validate on data the strategy has never seen (out-of-sample validation ), report every test you ran, and follow ethical reporting standards.
Try this 0 of 3
- Find a statistic with p just under 5% (two-tailed).
- On the Hacking tab, run tries until a false positive appears.
- Switch on "Report only the best" after 20 tries.
t = 2.15 with 59 degrees of freedom: right-tail p-value 0.0178, so reject at 5% and fail to reject at 1%. Reject whenever p is below α.
Drag the test statistic, tap the chart, or focus the handle and use the arrow keys.
- p-value area
- 5% critical value
Show working
p = P(T ≥ 2.15) with 59 degrees of freedom = 0.0178.
At 5%: 0.0178 < 0.05 → reject H₀. At 1%: 0.0178 ≥ 0.01 → fail to reject H₀.
The critical values say the same: 1.671 (5%) and 2.391 (1%).
In a spreadsheet: T.DIST.RT(2.15, 59) = 0.0178.
The p-value is the smallest significance level at which H₀ would be rejected. It is not the probability that H₀ is true.
Neither calculator gives t-distribution areas: the p-value comes from software or a table.
On the first tab, drag the statistic and watch the shaded tail and the decisions at 5% and 1%. On the second, run variations of a strategy with no real edge and count the false positives.
Where do critical values and p-values come from in practice?
Know the idea
In practice they come from software rather than printed tables. These are the spreadsheet critical values functions in Excel and Google Sheets (df stands for degrees of freedom):
| Function | Returns | Example |
|---|---|---|
| T.INV(prob, df) | t-value with prob to its left (one-tailed critical values) | T.INV(0.95, 59) = 1.671 |
| T.INV.2T(prob, df) | two-tailed t critical value | T.INV.2T(0.05, 59) = 2.001 |
| T.DIST.RT(x, df) | right-tail probability beyond a t-statistic x (a one-tailed p-value) | T.DIST.RT(2.15, 59) = 0.018 |
| NORM.S.INV(prob) | standard normal value with prob to its left | NORM.S.INV(0.975) = 1.960 |
| NORM.INV(prob, mean, sd) | same, for any normal distribution | NORM.INV(0.99, 0, 4.8) = 11.17 (in % units) |
| CHISQ.INV.RT(prob, df) | right-tail chi-square critical value (Excel’s older CHIINV does the same) | CHISQ.INV.RT(0.05, 19) = 30.144 |
| F.INV(prob, df1, df2) | F-value with prob to its left (lower critical value) | F.INV(0.025, 25, 40) = 0.472 |
| F.INV.RT(prob, df1, df2) | right-tail F critical value | F.INV.RT(0.05, 25, 40) = 1.783 |
You will not type formulas in the exam, but a question can show a function and ask what it returns.
Memorize this
Memorizep-values
| Item | Rule |
|---|---|
| Definition | smallest α at which is rejected |
| Decision | reject if |
| Two-tailed | one-tail area beyond |
| Strength | smaller , stronger evidence against |
| Not | the probability that is true |
Memory hook "p below α, the null must go."
Know it cold Same decision as the critical-value method, every time.
Memorizep-hacking
| Form | Example |
|---|---|
| Selective reporting | best period, benchmark or ratio only |
| Data mining | test many rules, keep the winner |
| Metric manipulation | smooth data, drop outliers before VaR or Sharpe |
| Effect | more Type I errors (false positives) |
| Defense | out-of-sample tests, confirmatory tests, full disclosure |
Memory hook Hacking manufactures false positives: Type I.
- Right-tail test Memorize P(T ≥ t) under H₀
- Left-tail test Memorize P(T ≤ t) under H₀
- Two-tailed test Memorize 2 × P(T ≥ |t|) under H₀
- Selective performance reporting Use it choosing the period, benchmark or ratio that flatters
- Data mining for strategies Use it testing many rules until one wins
- Manipulating metrics Use it smoothing data or dropping outliers before computing value at risk (VaR) or Sharpe ratios
- Defenses Use it how to keep false positives out
- Confirmatory testing Know the idea fix the hypothesis and the data first, then repeat the test
- Out-of-sample validation Use it test on data the strategy has never seen
- Transparent methods Use it report every test you ran
- Ethical standards Know the idea fair, complete performance presentation
-
Quick example
A one-tailed test of a mean gives t = 2.25 with 35 degrees of freedom. The area to its right is p = 0.015: reject at 5%, not at 1%.
Quick example
A left-tail test gives t = −1.80 with 24 degrees of freedom. The area to its left is p = 0.042: reject at 5%, not at 1%.
Quick example
Software gives a right-tail p-value of 0.018 for t = 2.15. For a two-sided alternative, p = 2 × 0.018 = 0.036.
- 0.018×2= 0.036
- 0.018ENTER2× 0.036
ENTER enter (on the HP, puts the number on the stack)
How it is asked
-
Reject at which levels?
p = 0.018. Reject at 5%? At 1%?
5% yes, 1% no. Trap: "1.8% chance H₀ is true".
-
One-tailed to two-tailed
One-tailed p 0.035, Hₐ is "not equal", α 5%.
Two-tailed p = 0.07: fail to reject. Trap: comparing 0.035 with 0.05.
-
Name the practice
An analyst keeps changing the sample period until p < 0.05 and reports that version.
P-hacking; it raises the Type I error rate.
-
Which function?
Two-tailed t critical value, 5%, 25 degrees of freedom.
T.INV.2T(0.05, 25) = 2.060. T.INV(0.05, 25) gives −1.708.
Exam traps
- Reading the p-value as the probability that the null is true.
- Comparing a one-tailed p-value with α in a two-tailed test. Double it first.
- Thinking p-hacking raises Type II errors. It manufactures false positives, which are Type I errors.
- Treating a p-value of 0.049 as strong evidence and 0.051 as none: the evidence barely differs.
Checkpoint
3 exam-style questions. About 90 seconds each.
1Software reports a one-tailed p-value of 0.04 for a test statistic. The analyst's alternative hypothesis is that the mean differs from zero, and the significance level is 5%. The analyst should:
2An analyst tests one trading rule over 30 different look-back windows and reports only the window with a p-value of 0.03, describing the rule as significant at 5%. This practice most likely:
3Which spreadsheet function returns the critical value for a two-tailed t-test at the 5% significance level with 40 degrees of freedom?
Key takeaways
- p-value = the smallest α at which H₀ is rejected; reject when p < α.
- Two-tailed p-value = 2 × the one-tail area beyond |t|.
- P-hacking (selective reporting, data mining, massaged metrics) inflates Type I errors.
- Defenses: fix the test in advance, confirm it, and validate out of sample.
Quick example
A one-tailed test of a mean gives t = 2.25 with 35 degrees of freedom. The area to its right is p = 0.015: reject at 5%, not at 1%.
Quick example
A left-tail test gives t = −1.80 with 24 degrees of freedom. The area to its left is p = 0.042: reject at 5%, not at 1%.
Quick example
Software gives a right-tail p-value of 0.018 for t = 2.15. For a two-sided alternative, p = 2 × 0.018 = 0.036.
- 0.018×2= 0.036
- 0.018ENTER2× 0.036
ENTER enter (on the HP, puts the number on the stack)
All Quantitative Methods formulas, every lesson
Practice exam-style questions on Estimation and Hypothesis Testing
Questions about this lesson?
Where this comes back
- Parametric vs non-parametric tests Estimation and Hypothesis Testing
- Guidance for Standard III: Duties to Clients Ethical and Professional Standards
Questions candidates ask
What does a p-value of 0.03 mean?
If the null hypothesis were true, a test statistic at least this extreme would turn up only 3% of the time. You would reject the null at the 5% level but not at the 1% level. It is not the probability that the null is true.
What is p-hacking in investment management?
Reworking an analysis until it crosses the significance line and reporting only that version: picking flattering periods or benchmarks, mining many strategies for a winner, or cleaning data before computing VaR or Sharpe ratios. It produces false positives; out-of-sample testing and full disclosure guard against it.
How do you convert a one-tailed p-value to a two-tailed one?
For a symmetric statistic such as t or Z, double it: a one-tailed p-value of 0.018 becomes 0.036 for a two-sided alternative. Then compare the doubled value with α.
Spotted an error? Tell us and we will fix it.