# A tibble: 4 × 7
treatment n mean sd se ci_lower ci_upper
<chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Control 14 0.329 0.432 0.115 0.0801 0.578
2 Foam 14 0.712 0.393 0.105 0.485 0.939
3 Haliclona 14 0.765 0.348 0.093 0.564 0.966
4 Tedania 14 0.914 0.442 0.118 0.659 1.17 Lecture: Study Design & Sampling
Designing studies
Where We Left Off
Covered last time:
- Type I and Type II error, and the decision table
- Power — the chance of detecting an effect that is really there
- A priori power to plan a study, retrospective power to plan the next one
- Pairing our four trees took the study from needing 88 trees to needing 7
Note
✅ Key idea from last lecture
Power is a property of a design, not of a test. Last time we measured it. Today we build the thing being measured — and we do it before any data exist.
Today’s Overview
One idea organizes the whole lecture:
A design has exactly two jobs — get rid of bias, and shrink sampling error. Every technique today does one or the other.
- Bias → controls, randomization, blinding
- Sampling error → replication, balance, blocking
- Then: sampling layouts in the field, and which analysis each design leads to
Note
📖 Reading
Whitlock & Schluter, Ch. 14 — Designing Experiments (§14.4 is bias, §14.5 is sampling error) and Ch. 1 — Statistics and Samples. Gotelli & Ellison, Ch. 6 — Designing Successful Field Studies, and Ch. 7 — A Bestiary of Experimental and Sampling Designs.
Lamberti and Resh 1983
The Two Ways an Estimate Goes Wrong
Your study produces an estimate. It can miss the truth two different ways:
- Bias — your estimate is centred on the wrong value. Collect more data and you converge, confidently, on the wrong answer.
- Sampling error — your estimate is centred on the right value but bounces around it. Collect more data and it tightens up.
This distinction is the whole lecture, because the two problems have completely different cures:
Important
⚠️ Why this matters more than any test
A p-value describes sampling error only. No statistical test can detect bias in your own design. That is why design comes first.
A confounder is a bias machine: it moves the estimate off the truth, and it does not care how many samples you take.
| Problem | Gets worse with | Cured by |
|---|---|---|
| Bias | nothing — more data does not help | controls, randomization, blinding |
| Sampling error | small n, high variance | replication, balance, blocking |
Start From the Question
Before either job, one question: what is your research question?
Gotelli & Ellison’s four canonical forms:
- Spatial or temporal pattern in Y?
- Effect of factor X on Y?
- Are values of Y consistent with hypothesis H?
- What is the best estimate of parameter θ?
Each one implies a different design. “I will go measure some things and see what turns up” implies none of them.
Note
📖 Reference
Gotelli & Ellison, Ch. 6 — Designing Successful Field Studies.
The Design Carries the Causal Claim
Statistics has no test for causality. The design carries the causal claim; the analysis just reports numbers. Strength of inference runs on a gradient, and what moves you up it is control of confounding:
- Observational field study — weak
- Field study with measured covariates
- Randomized, controlled experiment — strong
Our running example, all day: on small islands, does lizard predation control spider density?
Warning
⚠️ The same regression, three different claims
lm(spiders ~ lizards) returns the identical output at all three levels. What changes is what you are entitled to say about it.
Natural Experiments
Not experiments at all — you use natural variation in the predictor. Survey plots across an existing gradient of lizard density.
What you gain: real scale, real systems, long time spans, questions you could never manipulate.
What you give up:
- Direction of causation is not identified — lizards may eat spiders, or spiders may feed lizards
- Uncontrolled variables ride along with the predictor
Note
📖 Reference
Whitlock & Schluter §14.7 — “What if you can’t do experiments?” — is exactly this problem, and covers matching and adjusting for covariates as the two partial fixes.
🗳️ DECIDE · You Are the Reviewer
A manuscript lands on your desk. The authors surveyed 40 island plots and report that spider density falls as lizard density rises. They did not measure moisture.
You know that moisture is related to both lizards and spiders on these islands.
Your review says the authors must:
(a) Control for moisture — it is a confounder, and failing to control for it invalidates the result
(b) Not control for moisture — doing so could erase the very effect they are measuring
(c) Nothing yet — you cannot tell which advice is right from the data alone
(d) Control for moisture and report the uncontrolled result, letting readers choose
Warning
✍️ Write your letter down, then vote.
The Answer · Confounder or Mediator?
Important
✅ The answer is (c) — and this is the uncomfortable part of observational work.
If moisture drives lizards and spiders independently, it is a confounder and (a) is right. If moisture drives lizards, which then eat spiders, it is a mediator on the causal path — and controlling for it deletes the effect, so (b) is right.
The data cannot distinguish these two. The arrows come from the biology, not from the numbers. Anyone who tells you “always control for everything you measured” is wrong.
Strengthening a Natural Experiment
You cannot randomize, so you attack confounding directly:
- Reduce it — choose plots that are alike in everything except lizard density (this is matching)
- Adjust for it — measure the covariates and put them in the model
- Name it — you cannot adjust for a confounder you never wrote down
Know which one you have — the diagrams on the right are the whole distinction:
- A confounder drives both X and Y — adjust for it
- A mediator sits on the path X → M → Y — adjusting for it removes the effect you came to measure
Manipulative Experiments
You set the predictor yourself and measure the response. Randomized and controlled, this is the strongest design available.
What it costs you — Gotelli & Ellison’s trade-offs:
- Scale for replication — replicable plots are small plots
- Organism for tractability — small, short-lived species
- Treatments for replication — every extra treatment level costs replicates
- Confounding does not disappear; you now own it
Press, Pulse, Snapshot, Trajectory
Gotelli & Ellison’s two axes for manipulations — and they answer different questions:
- Pulse — apply the treatment once, then walk away. Measures recovery: how fast does the system bounce back?
- Press — maintain the treatment continuously. Measures the new equilibrium the system settles into.
And for sampling the response:
- Snapshot — measure once at the end. Cheap, and blind to everything in between.
- Trajectory — measure repeatedly through time. Expensive, and the only way to see dynamics.
Mesocosms let you press a treatment and follow a trajectory — at the cost of scale.
Important
⚠️ A snapshot can hide a complete reversal
Many treatment effects change sign as a system responds. Measure once, at one time, and you get whichever answer that moment happened to hold.
Controls — and Why One Is Usually Not Enough
A control is a unit treated exactly like the manipulated units, except for the one thing under investigation.
That word exactly is where designs fail. Your manipulation almost always does something besides the thing you care about:
- Toxicology: controls get injected too — same needle, same handling, no toxin
- Predator exclosures: a cage changes flow, light and sediment. You need two controls — an open plot and a cage-with-holes
- Anything you attach, bury, or visit does something merely by being attached, buried or visited
Note
Vocabulary
The second kind is a procedural or sham control. Next slide: real data where forgetting it would have produced a completely wrong paper.
Randomization
Randomization is how you deconfound the variables you never thought of. It works at two levels, and both matter:
- Selection — which units enter the study. Makes the sample represent the population.
- Assignment — which unit gets which treatment. Makes the groups exchangeable before treatment.
Haphazard is not random. “Whichever tree I reached first” correlates with trail access, which correlates with light, soil and disturbance.
Blinding
The measurer’s expectations leak into the measurements. This is a bias, so no amount of data fixes it.
- Single-blind — the subject does not know the treatment
- Double-blind — the person measuring does not know either
- A placebo is what makes subject-blinding possible at all
W&S spend most of Ch. 14’s opening on this because medicine learned it the hard way. Ecology mostly has not.
Important
⚠️ This lands on our own study
You measured needles knowing which side of the tree they came from, and knowing the hypothesis. Nobody cheated. But “is this needle bent or straight, include it or not” is a judgement call, and judgement calls drift toward the expected answer.
The fix is cheap: someone relabels the envelopes, you measure blind to side.
Expectation is a powerful thing.
Replication — and at What Scale
Replication does two jobs: it shrinks the standard error, and it is the only way to estimate variability at all.
The hard part is never how many. It is of what.
Warning
⚠️ The rule
Replicates must vary at the scale your hypothesis applies to. Anything finer is pseudoreplication (Hurlbert 1984): it shrinks your SE without adding information — a bias dressed as precision.
The next slide gives you three studies. Every one of them has been published in roughly this form.
We Already Fixed This Once
Our own pine data has 64 rows and 8 trees’ worth of sides — 4 trees, two sides each.
Every analysis we have run since Week 4 starts by averaging the 8 needles per side away:
# A tibble: 8 × 3
team side needle_length_mm
<chr> <chr> <dbl>
1 botany shady 16.1
2 botany sunny 17.3
3 grace_jace shady 15.4
4 grace_jace sunny 15.5
5 one shady 13.2
6 one sunny 14.2
7 waterbugs shady 11.7
8 waterbugs sunny 13.8That group_by() %>% summarise() is not data cleaning. It is the line where we choose the tree as the experimental unit, because the tree is what the hypothesis is about.
One tank, many measurements, n = 1.
Blocking, and Our Paired Design
Blocking: group units into sets that are alike, then randomize treatments within each set. Variation among blocks leaves the comparison instead of contaminating it.
- Chambers differ? Run every temperature in every chamber — block by chamber
- Days differ? Run all treatments each day — block by time
- Ridiculously variable sites? Put all treatments at each site
Important
✅ A paired design is a block of size two
Two sides of the same tree share genotype, soil, age and history. Pairing removes all of it from the comparison. That is why 7 trees do the work of 88.
Non-independence you ignore costs you. Non-independence you plan for pays you. Blocking is how you get paid.
Blocked designs need blocked analyses — Week 9 and Week 10.
Every tree slopes the same way. The trees sit at different heights. Blocking keeps the slope and throws away the heights.
Simple Random and Stratified
Simple random — every unit has an equal chance of selection. Number the units, draw with a random number generator.
- What nearly every test assumes you did
- Genuinely awkward in the field; haphazard is the common substitute, and it is not the same thing
Stratified — if the population has distinct strata, sample each one separately, with n proportional to stratum size.
- More precise than simple random when strata differ, because within-stratum variance is all that is left
- Strata must go into the model, not be ignored
Simple random
Stratified
Cluster and Systematic
Cluster — sample subunits nested inside larger units. Used when travel or access makes anything else impossible.
- Cheap, and less precise than simple random for the same total n
- The clusters are your replicates → nested ANOVA (Week 9) or a mixed model (Week 10)
Systematic — units evenly spaced; the transect is the ecological classic.
- Ideal for describing change along a gradient
- One real risk: your spacing lines up with a periodicity in the system
Cluster
Systematic
🗳️ DECIDE · Where Do You Spend the Money?
A published study compares a heavily visited tidepool beach with a restricted, no-access beach. Invertebrate densities are lower, and 78.2% of boulders are disturbed at the visited beach versus 1.8% at the restricted one. Sampled once, this year.
You inherit this project with funding for exactly one more sampling campaign. You want to claim that visitors reduce invertebrate density.
Where do you spend it?
(a) More quadrats at both beaches — tighten the estimates you already have
(b) Add a third, intermediate-use beach — get a gradient
(c) Re-sample both beaches next year — get a time series
(d) No campaign can rescue this claim
Warning
✍️ One letter. Write it down.
The Two Ways to Buy Power
Note
Same two levers, two views
Power climbs with the effect size you are chasing, and the detectable effect size falls as you add replicates. You choose one of them before the study; the other is then decided for you.