Designing studies
Covered last time:
✅ Key idea from last lecture
Power is a property of a design, not of a test. Last time we measured it. Today we build the thing being measured — and we do it before any data exist.
One idea organizes the whole lecture:
A design has exactly two jobs — get rid of bias, and shrink sampling error. Every technique today does one or the other.
📖 Reading
Whitlock & Schluter, Ch. 14 — Designing Experiments (§14.4 is bias, §14.5 is sampling error) and Ch. 1 — Statistics and Samples. Gotelli & Ellison, Ch. 6 — Designing Successful Field Studies, and Ch. 7 — A Bestiary of Experimental and Sampling Designs.
Lamberti and Resh 1983
🗳️ Five times today, you make the call
Five slides are marked 🗳️ DECIDE. Each one asks a multiple-choice question and I will not tell you the answer on that slide.
How it works: write your letter down — actually write it — then hold up fingers 1–4. If the room splits, turn to the person next to you, argue for ninety seconds, and we vote again. Nothing here is graded. The point is that you commit before you find out.
Your study produces an estimate. It can miss the truth two different ways:
This distinction is the whole lecture, because the two problems have completely different cures:
⚠️ Why this matters more than any test
A p-value describes sampling error only. No statistical test can detect bias in your own design. That is why design comes first.
A confounder is a bias machine: it moves the estimate off the truth, and it does not care how many samples you take.
| Problem | Gets worse with | Cured by |
|---|---|---|
| Bias | nothing — more data does not help | controls, randomization, blinding |
| Sampling error | small n, high variance | replication, balance, blocking |
Before either job, one question: what is your research question?
Gotelli & Ellison’s four canonical forms:
Each one implies a different design. “I will go measure some things and see what turns up” implies none of them.
📖 Reference
Gotelli & Ellison, Ch. 6 — Designing Successful Field Studies.
Statistics has no test for causality. The design carries the causal claim; the analysis just reports numbers. Strength of inference runs on a gradient, and what moves you up it is control of confounding:
Our running example, all day: on small islands, does lizard predation control spider density?
⚠️ The same regression, three different claims
lm(spiders ~ lizards) returns the identical output at all three levels. What changes is what you are entitled to say about it.
Not experiments at all — you use natural variation in the predictor. Survey plots across an existing gradient of lizard density.
What you gain: real scale, real systems, long time spans, questions you could never manipulate.
What you give up:
📖 Reference
Whitlock & Schluter §14.7 — “What if you can’t do experiments?” — is exactly this problem, and covers matching and adjusting for covariates as the two partial fixes.
A manuscript lands on your desk. The authors surveyed 40 island plots and report that spider density falls as lizard density rises. They did not measure moisture.
You know that moisture is related to both lizards and spiders on these islands.
Your review says the authors must:
(a) Control for moisture — it is a confounder, and failing to control for it invalidates the result
(b) Not control for moisture — doing so could erase the very effect they are measuring
(c) Nothing yet — you cannot tell which advice is right from the data alone
(d) Control for moisture and report the uncontrolled result, letting readers choose
✍️ Write your letter down, then vote.
✅ The answer is (c) — and this is the uncomfortable part of observational work.
If moisture drives lizards and spiders independently, it is a confounder and (a) is right. If moisture drives lizards, which then eat spiders, it is a mediator on the causal path — and controlling for it deletes the effect, so (b) is right.
The data cannot distinguish these two. The arrows come from the biology, not from the numbers. Anyone who tells you “always control for everything you measured” is wrong.
You cannot randomize, so you attack confounding directly:
Know which one you have — the diagrams on the right are the whole distinction:
You set the predictor yourself and measure the response. Randomized and controlled, this is the strongest design available.
What it costs you — Gotelli & Ellison’s trade-offs:
Gotelli & Ellison’s two axes for manipulations — and they answer different questions:
And for sampling the response:
Mesocosms let you press a treatment and follow a trajectory — at the cost of scale.
⚠️ A snapshot can hide a complete reversal
Many treatment effects change sign as a system responds. Measure once, at one time, and you get whichever answer that moment happened to hold.
A control is a unit treated exactly like the manipulated units, except for the one thing under investigation.
That word exactly is where designs fail. Your manipulation almost always does something besides the thing you care about:
Vocabulary
The second kind is a procedural or sham control. Next slide: real data where forgetting it would have produced a completely wrong paper.
Ellison, Farnsworth & Twilley (1996) asked whether epibiont sponges promote mangrove root growth. Four treatments, 14 roots each:
sponge_df <- read_csv("data/sponge_root_growth.csv")
sponge_df %>%
group_by(treatment) %>%
reframe(summary_stats(growth_mm_d))# A tibble: 4 × 7
treatment n mean sd se ci_lower ci_upper
<chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Control 14 0.329 0.432 0.115 0.0801 0.578
2 Foam 14 0.712 0.393 0.105 0.485 0.939
3 Haliclona 14 0.765 0.348 0.093 0.564 0.966
4 Tedania 14 0.914 0.442 0.118 0.659 1.17 Control — bare rootFoam — inert foam, the sham controlHaliclona, Tedania — the two spongesRead the means before you read any test.
Four numbers. Nothing else is on this slide on purpose.
Those four means are all the evidence you have. You now have to write the one sentence of the abstract that states what you found.
Which claim do these data support?
(a) Both sponge species increase mangrove root growth
(b) Only Tedania increases root growth; Haliclona does not
(c) We cannot tell whether the sponges do anything
(d) The sponges have no effect on root growth
✍️ Write your letter down before you look at the next slide.
Then argue it with your neighbour. Everyone in this room has enough information to get this right.
| Treatment | Growth (mm/d) |
|---|---|
Control |
0.33 |
Foam |
0.71 |
Haliclona |
0.77 |
Tedania |
0.91 |
Remember what Foam is: a piece of inert plastic. It is not alive and it is not a sponge.
# Is there an artifact of having ANYTHING attached?
sponge_df %>%
filter(treatment %in% c("Control", "Foam")) %>%
t.test(growth_mm_d ~ treatment, data = .)
Welch Two Sample t-test
data: growth_mm_d by treatment
t = -2.4537, df = 25.775, p-value = 0.02122
alternative hypothesis: true difference in means between group Control and group Foam is not equal to 0
95 percent confidence interval:
-0.70372014 -0.06199415
sample estimates:
mean in group Control mean in group Foam
0.3292857 0.7121429 # Do the living sponges beat the inert foam?
sponge_df %>%
filter(treatment != "Control") %>%
mutate(kind = if_else(treatment == "Foam", "sham", "sponge")) %>%
t.test(growth_mm_d ~ kind, data = .)
Welch Two Sample t-test
data: growth_mm_d by kind
t = -0.98461, df = 26.362, p-value = 0.3338
alternative hypothesis: true difference in means between group sham and group sponge is not equal to 0
95 percent confidence interval:
-0.3923957 0.1381100
sample estimates:
mean in group sham mean in group sponge
0.7121429 0.8392857 ✅ The answer is (c). Most rooms vote (a).
Control vs Foam: p = 0.021 — a piece of inert plastic significantly increases root growthFoam vs sponges: p = 0.33 — the living sponges add nothing detectable on top of thatThe sponges sit higher than the bare control, so (a) feels obvious. But the right comparison is not against nothing — it is against an object that is not a sponge, and against that, the sponges do not measurably win.
⚠️ The paper you would have written without the foam
“Sponges promote mangrove root growth” — real p-value, clean figure, wrong conclusion. The effect is mostly having something attached to your root.
A sham control is what stands between a correct result and a confident wrong one.
Randomization is how you deconfound the variables you never thought of. It works at two levels, and both matter:
Haphazard is not random. “Whichever tree I reached first” correlates with trail access, which correlates with light, soil and disturbance.
# Selection: 10 trees from the 100 in the stand
selected <- sample(1:100, 10)
# Assignment: 5 to treatment, the rest to control
treated <- sample(selected, 5)
tibble(tree = selected,
group = if_else(tree %in% treated, "treatment", "control")) %>%
arrange(group)# A tibble: 10 × 2
tree group
<int> <chr>
1 49 control
2 100 control
3 47 control
4 71 control
5 89 control
6 65 treatment
7 25 treatment
8 74 treatment
9 18 treatment
10 24 treatment🖐 set.seed() makes it reproducible, not less random
Record the seed and anyone can reconstruct exactly which trees you assigned where.
The measurer’s expectations leak into the measurements. This is a bias, so no amount of data fixes it.
W&S spend most of Ch. 14’s opening on this because medicine learned it the hard way. Ecology mostly has not.
⚠️ This lands on our own study
You measured needles knowing which side of the tree they came from, and knowing the hypothesis. Nobody cheated. But “is this needle bent or straight, include it or not” is a judgement call, and judgement calls drift toward the expected answer.
The fix is cheap: someone relabels the envelopes, you measure blind to side.
Expectation is a powerful thing.
Replication does two jobs: it shrinks the standard error, and it is the only way to estimate variability at all.
The hard part is never how many. It is of what.
⚠️ The rule
Replicates must vary at the scale your hypothesis applies to. Anything finer is pseudoreplication (Hurlbert 1984): it shrinks your SE without adding information — a bias dressed as precision.
The next slide gives you three studies. Every one of them has been published in roughly this form.
Write down the true number of replicates per group for each study. Three numbers.
1. Fire and soil invertebrates. 20 soil cores from the burnt part of a forest, 20 from the unburnt part of the same forest. (a) 20 (b) 2 (c) 1
2. Copper and barnacle settlement. 2 aquaria — one with copper, one without. 5 settling plates in each. (a) 10 (b) 5 (c) 2 (d) 1
3. Sewage and water quality. 10 water samples collected above the outfall, 10 below it. (a) 20 (b) 10 (c) 1
✍️ Three letters, written down, before you vote.
Number 3 is the one to argue about.
The question to ask each time
“What is the smallest thing I could have independently assigned or independently chosen?”
Count those. Not the measurements.
1. Fire → (c) n = 1 per group. One burnt area, one unburnt area. The 20 cores describe that forest and support no claim about fire in general. To study fire you need several burnt and several unburnt forests.
2. Copper → (d) n = 1 per group. One aquarium per treatment. The 5 plates share water, temperature, and whatever went wrong in that tank. You need several aquaria per treatment.
3. Sewage → (c) n = 1 per group. This is the one that catches everybody. There is one upstream location and one downstream location. Ten samples from one place is ten measurements of one place.
In all three, the published version reported the bigger number — and a satisfyingly small p-value.
⚠️ What pseudoreplication does and does not do
It barely moves your means. It shrinks your standard error. So the estimate stays put while the uncertainty around it quietly collapses.
That is why it is so hard to catch by eye, and why no diagnostic plot will flag it. Only the design will.
You will do exactly this to a real dataset for homework — and get the opposite answer from the obvious one.
Our own pine data has 64 rows and 8 trees’ worth of sides — 4 trees, two sides each.
Every analysis we have run since Week 4 starts by averaging the 8 needles per side away:
p_df# A tibble: 8 × 3
team side needle_length_mm
<chr> <chr> <dbl>
1 botany shady 16.1
2 botany sunny 17.3
3 grace_jace shady 15.4
4 grace_jace sunny 15.5
5 one shady 13.2
6 one sunny 14.2
7 waterbugs shady 11.7
8 waterbugs sunny 13.8That group_by() %>% summarise() is not data cleaning. It is the line where we choose the tree as the experimental unit, because the tree is what the hypothesis is about.
One tank, many measurements, n = 1.
For a fixed total effort, equal group sizes give the smallest standard error on the difference.
🖐 When imbalance is not your fault
Mortality, lost plots and weather happen. Balance is a design goal, not a moral requirement — but plan for balance, and report what you actually ended up with.
📖 Reference
Whitlock & Schluter §14.5 — reducing the influence of sampling error: replication, balance, blocking, and choosing extreme treatment levels.
Extreme treatments
If you want to detect a dose response with few replicates, spread your doses wide. Four doses clustered near the middle waste your replicates on a range where nothing happens.
Most tests assume observations are independent — knowing one tells you nothing about another. Autocorrelation is the standard violation: things near each other in space or time resemble each other. Samples 1 m apart are not two independent samples; consecutive days are not independent days; offspring of one mother are not independent of each other.
You have two options, and only two: space your samples beyond the correlation range, or model it (Week 10 onward).
Or — you stop fighting it and use it.
Blocking: group units into sets that are alike, then randomize treatments within each set. Variation among blocks leaves the comparison instead of contaminating it.
✅ A paired design is a block of size two
Two sides of the same tree share genotype, soil, age and history. Pairing removes all of it from the comparison. That is why 7 trees do the work of 88.
Non-independence you ignore costs you. Non-independence you plan for pays you. Blocking is how you get paid.
Blocked designs need blocked analyses — Week 9 and Week 10.
Every tree slopes the same way. The trees sit at different heights. Blocking keeps the slope and throws away the heights.
Simple random — every unit has an equal chance of selection. Number the units, draw with a random number generator.
Stratified — if the population has distinct strata, sample each one separately, with n proportional to stratum size.
Simple random
Stratified
Cluster — sample subunits nested inside larger units. Used when travel or access makes anything else impossible.
Systematic — units evenly spaced; the transect is the ecological classic.
Cluster
Systematic
A published study compares a heavily visited tidepool beach with a restricted, no-access beach. Invertebrate densities are lower, and 78.2% of boulders are disturbed at the visited beach versus 1.8% at the restricted one. Sampled once, this year.
You inherit this project with funding for exactly one more sampling campaign. You want to claim that visitors reduce invertebrate density.
Where do you spend it?
(a) More quadrats at both beaches — tighten the estimates you already have
(b) Add a third, intermediate-use beach — get a gradient
(c) Re-sample both beaches next year — get a time series
(d) No campaign can rescue this claim
✍️ One letter. Write it down.
✅ The answer is (d), and it is meant to be unsatisfying.
The two beaches may always have differed — in exposure, substrate, or wave energy — which is very likely why one became popular and one became restricted. You cannot add a “before” retroactively.
Before–After–Control–Impact is the design you needed:
Why this matters beyond tidepools
Nobody randomizes an oil spill, a dam, a clearcut or a hurricane. BACI is how environmental impact assessment is supposed to work — and the reason baseline monitoring has value long before anyone knows what will hit.
BACI is a factorial design with an interaction → Week 9.
We did the mechanics last lecture. The design point is the one worth keeping:
# Our pine pilot, unpaired vs paired — same trees, same needles
pwr.t.test(d = d_paired, power = 0.80,
sig.level = 0.05, type = "paired")$n[1] 6.40683844 trees per side (88 total) unpaired, versus 7 trees paired. Nothing changed but the design.
Plan for precision, not just for power
W&S §14.6 frames this as planning the width of your confidence interval. “How precise will my estimate be?” is often the more honest question than “will I get a star?”
⚠️ Retrospective power, one more time
Computing power from the effect size you observed re-expresses your p-value and tells you nothing new. Use it to plan the next study. To argue for no effect, use an equivalence test or a CI that excludes anything meaningful.
Same two levers, two views
Power climbs with the effect size you are chasing, and the detectable effect size falls as you add replicates. You choose one of them before the study; the other is then decided for you.
With your neighbour. Five designs. For each, name the analysis — from the list on the right — that it forces you into. You have met all five.
1. Needle length measured on 40 trees, along with the light reaching each tree, in lux.
2. Sea urchin grazing measured on 4 patches inside each of 16 reefs.
3. Barnacle settlement at 2 copper levels × 2 temperatures, fully crossed.
4. Number of ant species per quadrat — counts, many of them zero.
5. Fish mass compared between two lakes, with body length also measured and known to matter.
🖐 The tell
For each one, ask: how many predictors, what kind, and is anything nested inside anything else?
Pick from these
✍️ Five answers, written down. Two minutes.
Answers: 1 regression · 2 nested ANOVA (or a mixed model) · 3 two-way ANOVA with interaction · 4 GLM · 5 ANCOVA.
This is Gotelli & Ellison Ch. 7’s whole point, and it is also the rest of this course. Write the design down and the analysis is already decided:
| Your design | The analysis | When |
|---|---|---|
| One continuous predictor | Regression | Week 6 |
| Several continuous predictors | Multiple regression | Week 7 |
| One categorical factor | One-way ANOVA | Week 8 |
| Two crossed factors, or BACI | Two-way ANOVA + interaction | Week 9 |
| Subsamples inside units; clusters | Nested ANOVA | Week 9 |
| Blocks, repeated measures, split-plot | Mixed model | Week 10 |
| Counts, or presence/absence | GLM | Week 11 |
| Factor plus a continuous covariate | ANCOVA | Week 12 |
You cannot choose your analysis at the end
You chose it the day you decided what to measure and how to lay it out. Everything after that is bookkeeping.
The sponge slide, if you remember nothing else
The inert foam grew roots almost as well as the living sponges. Without that sham control, the paper reports a sponge effect — correctly analysed, properly significant, and wrong.
Design is not the paperwork before the science. It is the science.
🗳️ One last decision — on your way out
On a scrap of paper, unsigned:
What is the single biggest threat to the validity of our own pine needle study — and what one change would you make to next year’s collection?
One sentence each. Hand it to me at the door.