Type I and Type II errors and the power of a test
A Type I error (a false positive) rejects a true null; its probability is α. A Type II error (a false negative) fails to reject a false null; its probability is β. Power is 1 − β. Raising α raises power and Type I risk; a larger sample or a larger effect lowers both errors.
Before this: How a hypothesis test works: four steps from claim to decision
On the exam
- Explain hypothesis testing and its components, including statistical significance, Type I and Type II errors, and the power of a test; construct appropriate hypothesis tests; and interpret the results
Explain Type I and Type II errors and the power of a test. Expect scenario questions that name the error, and what happens to α, β and power when the significance level, the sample size or the size of the effect changes.
How deeply to learn each part
- Know the idea
- Understand it in general. It frames other topics, but you will not calculate with it.
- Use it
- Understand it well enough to solve a question with it. Recognize when it applies.
- Memorize
- Learn it by heart. The exam will not give it to you.
You know this one. The full lesson is folded away. The takeaways, formulas and checkpoint below are your quick refresher, and the checkpoint is a fair test of whether you still do.
You tested strong here. The full lesson is folded away. The takeaways, formulas and checkpoint below are your quick refresher, and the checkpoint is a fair test of whether you still know it.
What can go wrong with a decision?
A test uses a sample, so it can be wrong in two different ways. Either it rejects a null that is actually true, or it fails to reject a null that is actually false. The size of each risk is something you can state and, to a degree, control.
The four outcomes
Memorize
- Reject H₀ Memorize
- H₀ true: Type I error Memorize false positive (false alarm); probability α
- H₀ false: correct Memorize power; probability 1 − β
-
- Fail to reject H₀ Memorize
- H₀ true: correct Memorize probability 1 − α
- H₀ false: Type II error Memorize false negative (missed signal); probability β
-
A Type I error is rejecting a true null hypothesis: a false positive , or false alarm. Its probability is the significance level, α. A Type II error is failing to reject a false null hypothesis: a false negative , or missed signal. Its probability is β. The confidence level, 1 − α, is the probability of not making a Type I error.
The power of a test is the probability of correctly rejecting a false null.
Exam tip Power is 1 − β, never 1 − α. The quantity 1 − α is the confidence level.
Go deeper Putting numbers on power (beyond the exam)
Power depends on the cut-off for rejection and on where the true mean really is. With a known σ, the cut-off for a one-tailed test of : μ ≤ μ₀ is a fixed number of standard errors above μ₀. This calculation is an illustration, not something the exam asks you to do.
Try this 0 of 4
- Keep n = 25 and μ₁ = 1.0%, and tighten α to 1%. Read the power.
- Back at α = 5% and μ₁ = 1.0%, raise n until power is at least 0.900.
- Keep n at 25 and α at 5%, and raise power above 0.95 another way.
- Slide the true mean down to 0 and read the power.
Sample size 25, standard error 0.4%. Cut-off 0.658%, so α is 0.050. If the true mean is 1.0%, β is 0.196 and power is 0.804. Move the cut-off and you trade false alarms for missed signals; more data or a bigger effect shrinks both.
Drag the Cut-off for rejecting H₀, tap the chart, or focus the handle and use the arrow keys.
- If H₀ is true (μ = 0)
- If μ = 1.0%
- α 0.050
- β 0.196
- Power 0.804
Red: α, rejecting a true H₀. Amber: β, missing a real effect. Green hatch: power.
Show working
Standard error = σ ÷ √n = 2 ÷ √25 = 0.4%.
Cut-off = z(α) × SE = 1.645 × 0.4 = 0.658%. Reject H₀ when the sample mean is above it. α = P(sample mean > cut-off | μ = 0) = 0.050.
If the true mean is 1.0%: z = (0.658 − 1.0) ÷ 0.4 = −0.855.
Power = P(Z > −0.855) = 0.804; β = 1 − power = 0.196.
Neither calculator has N(z): look the z up in the table. The keystrokes give the cut-off and z.
Lower α pushes the cut-off right: fewer false alarms, more missed signals. A bigger n narrows both curves and cuts both errors. So does a true mean farther from H₀.
- 2÷25√x=STO1SE 0.4
- ×1.645=cut-off 0.658
- −1=÷RCL1=z = −0.855
√x square root · STO store · RCL recall a stored number
- 2ENTER25g√x÷STO1SE 0.4
- 1.645×cut-off 0.658
- 1−RCL1÷z = −0.855
ENTER enter (on the HP, puts the number on the stack) · g blue shift key · √x square root · STO store · RCL recall a stored number
Display: −0.855
Drag the cut-off line, or change α, n and the true mean, and watch the shaded areas for α, β and power change.
What moves the risks?
- Lower α (same n) Memorize fewer false alarms; β rises, power falls
- Higher α (same n) Memorize β falls, power rises; more false alarms
- Larger sample, n Memorize standard error shrinks; β falls and power rises at the same α
- Larger effect size Memorize true value farther from H₀; power rises, β falls
Moving the cut-off only shifts risk from one error to the other: it is a trade. The significance level is a choice made before looking at the data, so it does not depend on n. The sample size changes the standard error, which improves both β and power at once. So does a larger effect size , a bigger gap between the true value and the value in . To lower both errors at once, increase the sample size or test for a larger effect.
Go deeper Why 5% is the usual choice
The 5% level is a convention, not a law of nature. A researcher who fears false alarms more than missed signals (for example, before launching an expensive fund) may pick 1%. One who fears missing a real effect (for example, a safety screen) may accept 10%. State the level before the test; picking it afterwards to fit the result defeats the purpose.
Go deeper Many tests, many false alarms
Run 20 tests at 5% on data with nothing in it and you should expect about one false positive. Searching until something looks significant is p-hacking, covered in the p-value lesson.
Memorize this
MemorizeErrors and power
| true | false | |
|---|---|---|
| Fail to reject | correct () | Type II error () |
| Reject | Type I error () | correct: power () |
Memory hook Type I: you acted when you should not have (the alarm rings, no fire). Type II: you did nothing when you should have (fire, no alarm). The order matches the letters: α pairs with I, β with II.
Know it cold Significance level . Power . Not .
MemorizeTrade-offs
| Change | Type I | Type II | Power |
|---|---|---|---|
| Lower , same | down | up | down |
| Larger , same | same | down | up |
| Larger effect, same and | same | down | up |
Memory hook Moving the cut-off only trades one error for the other. A bigger sample or a bigger effect cuts both.
Know it cold To lower both errors, increase or the effect size.
- Reject H₀ Memorize
- H₀ true: Type I error Memorize false positive (false alarm); probability α
- H₀ false: correct Memorize power; probability 1 − β
-
- Fail to reject H₀ Memorize
- H₀ true: correct Memorize probability 1 − α
- H₀ false: Type II error Memorize false negative (missed signal); probability β
-
- Lower α (same n) Memorize fewer false alarms; β rises, power falls
- Higher α (same n) Memorize β falls, power rises; more false alarms
- Larger sample, n Memorize standard error shrinks; β falls and power rises at the same α
- Larger effect size Memorize true value farther from H₀; power rises, β falls
Quick example
A test misses a false null 20% of the time (β = 0.20). Power = 1 − 0.20 = 0.80.
How it is asked
-
Name the error from a scenario
The analyst rejects the null that alpha is zero, but alpha is zero. Error?
Rejected a true null: Type I. Trap: Type II.
-
Effect of a lower significance level
Sample size fixed; α goes from 5% to 1%. What happens?
Type II probability rises, power falls. Trap: "power increases".
-
Define power
Power is the probability of…
Rejecting a false null (1 − β). Traps: rejecting a true null (that is α), failing to reject a false null (that is β).
-
Lower both errors
How can both α and β be reduced?
Larger sample (or a larger effect). Changing α alone trades one error for the other.
Exam traps
- Writing power = 1 − α. Power is 1 − β; 1 − α is the probability of correctly not rejecting a true null.
- Saying a lower significance level raises power. With the same sample size it lowers power, because β rises.
- Mixing up which error is 'rejecting a true null'. That is Type I. Failing to reject a false null is Type II.
Checkpoint
5 exam-style questions. About 90 seconds each.
1An analyst rejects the null hypothesis that a manager's alpha is zero, when in truth the alpha is zero. This is a:
2Holding sample size constant, reducing the significance level from 5% to 1% will most likely:
3The power of a test is best described as the probability of:
4An analyst wants to lower the probability of both a Type I and a Type II error in a test of a fund's mean return. The analyst should most appropriately:
5Holding the sample size constant, raising the significance level from 1% to 5% will most likely:
Key takeaways
- Type I: reject a true H₀ (probability α). Type II: fail to reject a false H₀ (probability β).
- Power = 1 − β: the chance of correctly rejecting a false null.
- Lower α with the same n means higher β. A larger sample or a larger effect lowers both errors.
- Type I is a false alarm; Type II is a missed signal.
Quick example
A test misses a false null 20% of the time (β = 0.20). Power = 1 − 0.20 = 0.80.
All Quantitative Methods formulas, every lesson
Practice exam-style questions on Estimation and Hypothesis Testing
Questions about this lesson?
Where this comes back
- t-tests of a mean, a difference in means and paired differences Estimation and Hypothesis Testing
- P-values, p-hacking and critical values from software Estimation and Hypothesis Testing
Questions candidates ask
What is the difference between Type I and Type II errors?
A Type I error rejects a null hypothesis that is actually true: a false alarm, with probability α. A Type II error fails to reject a null that is actually false: a missed signal, with probability β. Reducing one usually raises the other unless the sample size or the effect size is larger.
What is the power of a test?
Power is the probability that the test correctly rejects a false null hypothesis, equal to 1 − β. A powerful test rarely misses a real effect. Power rises with a larger sample, a higher significance level, and a larger effect size (a true value farther from the null).
How does sample size affect Type I and Type II errors?
The Type I error probability is whatever significance level you choose, so sample size does not change it. A larger sample shrinks the standard error, which lowers the Type II error probability and raises power at the same α. A larger effect size does the same.
Spotted an error? Tell us and we will fix it.