# install.packages("readxl")
# install.packages("janitor")
library(tidyverse)
library(readxl) # reading Excel files
library(janitor) # cleans variable names, excel poop, rounding issues
pine_csv_df <- read_csv("data/pine_needles.csv")
pine_excel_df <- read_excel("data/pine_needles.xlsx")Lecture: Wrangling Your Data
Importing more than one way, reshaping columns, and saving what you make
Where We Left Off
Last time you:
- Turned a field observation into a testable H₀ / Hₐ
- Built a project folder (
data/,scripts/,figures/,outout/) - Loaded
pine_needles.csvand made your firstggplot
✅ Key idea from Getting Started
You can get data into R and make a plot. Today we learn to reshape what you have and save what you make — the two skills every later module depends on.
Today’s roadmap
- Importing data more than one way
- The five wrangling verbs
filter()in depth — every boolean operator,%in%, AND vs. ORmutate()— unit conversions and log transformscase_when()— recoding and binning- Catching duplicate rows
- Saving your work with
write_csv()
Part 1 · Importing Data More Than One Way
The same data, two file formats
Field data doesn’t always arrive as a .csv. Excel workbooks are everywhere in ecology — you need both tools.
✅ Key idea
read_csv() comes from readr (loaded with tidyverse).
Excel files need the separate readxl package
— install once, library() every session.
📖 New word
Package vs. bundle — tidyverse is a bundle of packages (readr, dplyr, ggplot2, …). readxl isn’t included; it always needs its own library() line.
Look before you trust it — every time
glimpse(pine_csv_df)Rows: 48
Columns: 6
$ date <chr> "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20…
$ group <chr> "cephalopods", "cephalopods", "cephalopods", "cephalopods", …
$ n_s <chr> "n", "n", "n", "n", "n", "n", "s", "s", "s", "s", "s", "s", …
$ sun <chr> "shady", "shady", "shady", "shady", "shady", "shady", "sunny…
$ tree_no <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, …
$ length_mm <dbl> 20, 21, 23, 25, 21, 16, 15, 16, 14, 17, 13, 15, 19, 18, 20, …
dim(pine_csv_df)[1] 48 6
🖐 Notice
Six columns: date, group, n_s, sun, tree_no, length_mm or what you call them.
Same structure whether it came from the .csv or the .xlsx.
⚠️ Watch out!
Excel loves to turn a plain number column into a date, or a tree_no like 1 into 1.0.
Always glimpse() an Excel import before doing anything else.
Part 2 · The Five Wrangling Verbs
The Five Wrangling Verbs
Almost everything you do to a data frame is one of these five functions, chained with the pipe |>:
| Function | What it does |
|---|---|
filter() |
keep rows matching a condition |
select() |
keep columns by name |
mutate() |
add or change a column |
arrange() |
sort rows |
summarize() |
collapse rows to a summary |
All take a data frame as input and return a data frame — that’s what makes them chainable.
✅ Key idea
summarize() is Describing Your Data’s whole topic.
Today: the other four, plus saving what you build.
Read a pipeline out loud
pine_csv_df |>
filter(...) |> # then
select(...) |> # then
mutate(...) |> # then
arrange(...) # then
Each |> reads as “then.”
filter() — Picking Rows
# Keep only the shady (sheltered) side
pine_csv_df |> filter(n_s == "n") |> head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
# Keep only needles longer than 20 mm
pine_csv_df |> filter(length_mm > 20) |> head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 21
2 3/20/25 cephalopods n shady 1 23
3 3/20/25 cephalopods n shady 1 25
4 3/20/25 cephalopods n shady 1 21
5 3/20/25 salmon n shady 2 23
6 3/20/25 salmon n shady 2 21
⚠️ Watch out!
=assigns a value==tests equality.filter(n_s = "n")is an error- you want
filter(n_s == "n").
Common comparison operators:
| Symbol | Meaning |
|---|---|
== |
exactly equal |
!= |
not equal |
> < |
greater / less than |
>= <= |
greater / less / or equal |
is.na() |
is the value missing? |
!is.na() |
is the value not missing? |
filter() — Combining Conditions
# Combine two conditions with a comma (AND)
pine_csv_df |> filter(n_s == "s", length_mm > 15) |> head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods s sunny 1 16
2 3/20/25 cephalopods s sunny 1 17
3 3/20/25 crayfish s sunny 3 16
4 3/20/25 crayfish s sunny 3 17
5 3/20/25 snail s sunny 4 17
6 3/20/25 snail s sunny 4 16
AND vs. OR
- A comma between conditions means AND
- —every condition must be true for a row to be kept.
- Use
|between conditions for OR instead
filter() — OR, With a Real Example
🔮 Predict first: n_s == "n" | length_mm > 22 keeps a row if the side is shady or the needle is longer than 22mm.
Will this ever keep a sunny needle?
Under what condition?
# Shady-side needles, OR any exceptionally long needle
pine_csv_df |>
filter(n_s == "n" | length_mm > 22) |>
head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
⚠️ Watch out!
|is the pipe character, not the pipe operator|> or %>%- Look related but do completely different jobs
- one is logical OR, the other chains steps together.
filter() — Matching One of Several Values with %in%
# Keep rows where group is one of two named teams
pine_csv_df |>
filter(group %in% c("cephalopods", "salmon")) |>
head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
# The long way, for comparison — %in% replaces a chain of |
pine_csv_df |>
filter(group == "cephalopods" | group == "salmon") |>
head()# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
✅ Key idea
x %in% c(a, b, c)asks “isxequal to any of these?”- it’s the clean way to write an OR across many possible values without a long chain of
|.
- it’s the clean way to write an OR across many possible values without a long chain of
filter() — Missing Values with is.na()
# A tiny illustrative dataset with missing values
survey_df <- tibble(
site = c("A", "B", "C", "D"),
count = c(12, NA, 8, NA)
)
survey_df |> filter(is.na(count)) # rows that ARE missing# A tibble: 2 × 2
site count
<chr> <dbl>
1 B NA
2 D NA
survey_df |> filter(!is.na(count)) # rows that are NOT missing# A tibble: 2 × 2
site count
<chr> <dbl>
1 A 12
2 C 8
✅ Key idea
is.na(x)is itself a boolean testTRUE/FALSE— so it plugs intofilter()- exactly like
==or>does.
📖 Coming up
pine_csv_df has no missing values, so is.na() has nothing to catch here.
Describing Your Data digs into why missing values matter so much once you’re computing a sample size or a mean.
select() — Picking Columns
# Keep only side and length
pine_csv_df |> select(n_s, length_mm) |> head()# A tibble: 6 × 2
n_s length_mm
<chr> <dbl>
1 n 20
2 n 21
3 n 23
4 n 25
5 n 21
6 n 16
# Drop the date column (minus sign)
pine_csv_df |> select(-date) |> head()# A tibble: 6 × 5
group n_s sun tree_no length_mm
<chr> <chr> <chr> <dbl> <dbl>
1 cephalopods n shady 1 20
2 cephalopods n shady 1 21
3 cephalopods n shady 1 23
4 cephalopods n shady 1 25
5 cephalopods n shady 1 21
6 cephalopods n shady 1 16
✅ Key idea
select() doesn’t touch rows — it only trims the columns you’re carrying forward. Useful the moment a dataset has 30+ columns and you need 4 of them.
- Name the columns you want, in any order
- A leading
-drops a column instead of keeping it
select() — Picking Columns by Pattern
# Keep columns that contain a word
pine_csv_df |> select(contains("n")) |> head()# A tibble: 6 × 4
n_s sun tree_no length_mm
<chr> <chr> <dbl> <dbl>
1 n shady 1 20
2 n shady 1 21
3 n shady 1 23
4 n shady 1 25
5 n shady 1 21
6 n shady 1 16
Useful helpers:
starts_with("x") - super useful for similar units or types of data in variableends_with("_mm")contains("sun")
📖 R4DS §3.3 — select()
mutate() — Creating and Changing Columns
🔮 Predict first: length_mm / 10 converts millimeters to centimeters.
Will mutate() replace length_mm, or add a new column?
How many columns will the result have?
# Add a converted-units column
pine_csv_df |>
mutate(length_cm = length_mm / 10) |>
head()# A tibble: 6 × 7
date group n_s sun tree_no length_mm length_cm
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20 2
2 3/20/25 cephalopods n shady 1 21 2.1
3 3/20/25 cephalopods n shady 1 23 2.3
4 3/20/25 cephalopods n shady 1 25 2.5
5 3/20/25 cephalopods n shady 1 21 2.1
6 3/20/25 cephalopods n shady 1 16 1.6
✅ Key idea
mutate() keeps every existing column and adds new ones to the right.
To overwrite a column, reuse its name:
mutate(length_mm = round(length_mm, 1)).
mutate() — More Examples
# Add a log-transformed column
pine_csv_df |>
mutate(log_length = log(length_mm)) |>
head()# A tibble: 6 × 7
date group n_s sun tree_no length_mm log_length
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20 3.00
2 3/20/25 cephalopods n shady 1 21 3.04
3 3/20/25 cephalopods n shady 1 23 3.14
4 3/20/25 cephalopods n shady 1 25 3.22
5 3/20/25 cephalopods n shady 1 21 3.04
6 3/20/25 cephalopods n shady 1 16 2.77
# mutate() can build on a column it just created
pine_csv_df |>
mutate(
length_cm = length_mm / 10,
size_class = if_else(length_mm > 18, "long", "short")
) |>
head()# A tibble: 6 × 8
date group n_s sun tree_no length_mm length_cm size_class
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> <chr>
1 3/20/25 cephalopods n shady 1 20 2 long
2 3/20/25 cephalopods n shady 1 21 2.1 long
3 3/20/25 cephalopods n shady 1 23 2.3 long
4 3/20/25 cephalopods n shady 1 25 2.5 long
5 3/20/25 cephalopods n shady 1 21 2.1 long
6 3/20/25 cephalopods n shady 1 16 1.6 short
📖 Why log-transform?
- Many biological measurements are right-skewed
- a log transform pulls extreme values in and can make a distribution more symmetric before you summarize or test it.
- We’ll check whether it matters for our data in a later module
- Know what log you are doing log2 log10 or ln?????
Part 2b · case_when() — Recoding and Binning
case_when() — Turning a Number into a Category
🔮 Predict first: We’re about to bin length_mm into "short", "medium", and "long". Needle 1 in the data is 20mm. Which bin does it land in — and does that depend on whether the cutoff is < or <=?
pine_csv_df |>
mutate(
size_class = case_when(
length_mm < 16 ~ "short",
length_mm < 20 ~ "medium",
TRUE ~ "long"
)
) |>
count(size_class)# A tibble: 3 × 2
size_class n
<chr> <int>
1 long 16
2 medium 17
3 short 15
✅ Key idea
case_when() checks conditions top to bottom and uses the first one that’s TRUE.
Once a row matches length_mm < 16, R never even looks at the later conditions for that row.
case_when() — Always End With TRUE ~
# What happens if we forget the catch-all?
pine_csv_df |>
mutate(
size_class = case_when(
length_mm < 16 ~ "short",
length_mm < 20 ~ "medium"
# no TRUE ~ ... here
)
) |>
count(size_class)# A tibble: 3 × 2
size_class n
<chr> <int>
1 medium 17
2 short 15
3 <NA> 16
⚠️ Watch out!
Drop the TRUE ~ "long" line and every “long” needle silently becomes NA instead of erroring. case_when() never warns you — always finish with a TRUE ~ ... catch-all, even when you’re sure you’ve covered every case.
case_when() — Recoding Categories, Not Just Numbers
# Give each field team a short code
pine_csv_df |>
mutate(
team_code = case_when(
group == "cephalopods" ~ "CEPH",
group == "salmon" ~ "SALM",
group == "crayfish" ~ "CRAY",
group == "snail" ~ "SNAI",
TRUE ~ "UNK"
)
) |>
distinct(group, team_code)# A tibble: 4 × 2
group team_code
<chr> <chr>
1 cephalopods CEPH
2 salmon SALM
3 crayfish CRAY
4 snail SNAI
🖐 Notice
case_when() isn’t just for numeric cutoffs — every condition on the left of a ~ can be any boolean test: ==, %in%, >, even combinations with & and |.
The "UNK" catch-all should never actually appear here — if it does, that’s your signal a team name was misspelled somewhere upstream.
arrange() — Sorting Rows
# Shortest needles first
pine_csv_df |> arrange(length_mm)# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 salmon s sunny 2 12
2 3/20/25 salmon s sunny 2 12
3 3/20/25 salmon s sunny 2 12
4 3/20/25 cephalopods s sunny 1 13
5 3/20/25 salmon s sunny 2 13
6 3/20/25 crayfish s sunny 3 13
7 3/20/25 cephalopods s sunny 1 14
8 3/20/25 salmon s sunny 2 14
9 3/20/25 salmon s sunny 2 14
10 3/20/25 crayfish s sunny 3 14
# ℹ 38 more rows
# Longest first
pine_csv_df |> arrange(desc(length_mm))# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 25
2 3/20/25 crayfish n shady 3 25
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 salmon n shady 2 23
5 3/20/25 crayfish n shady 3 23
6 3/20/25 snail n shady 4 23
7 3/20/25 cephalopods n shady 1 21
8 3/20/25 cephalopods n shady 1 21
9 3/20/25 salmon n shady 2 21
10 3/20/25 crayfish n shady 3 21
# ℹ 38 more rows
- Default is ascending (smallest first)
desc()reverses it
arrange() — Sorting by Multiple Columns
# Sort by side, then by length within each side
pine_csv_df |> arrange(n_s, desc(length_mm))# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 25
2 3/20/25 crayfish n shady 3 25
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 salmon n shady 2 23
5 3/20/25 crayfish n shady 3 23
6 3/20/25 snail n shady 4 23
7 3/20/25 cephalopods n shady 1 21
8 3/20/25 cephalopods n shady 1 21
9 3/20/25 salmon n shady 2 21
10 3/20/25 crayfish n shady 3 21
# ℹ 38 more rows
- Most useful at the end of a pipeline, to inspect results in a sensible order
arrange() never changes which rows you have — only the order they print in.
The Full Pipeline
pine_wrangled_df <- pine_csv_df |>
filter(length_mm > 0) |> # drop impossible values
select(group, n_s, sun, length_mm) |> # keep what we need
mutate(
length_cm = length_mm / 10,
log_length = log(length_mm),
size_class = case_when(
length_mm < 16 ~ "short",
length_mm < 20 ~ "medium",
TRUE ~ "long"
)
) |>
arrange(n_s, desc(length_mm)) # sort for inspection
head(pine_wrangled_df)# A tibble: 6 × 7
group n_s sun length_mm length_cm log_length size_class
<chr> <chr> <chr> <dbl> <dbl> <dbl> <chr>
1 cephalopods n shady 25 2.5 3.22 long
2 crayfish n shady 25 2.5 3.22 long
3 cephalopods n shady 23 2.3 3.14 long
4 salmon n shady 23 2.3 3.14 long
5 crayfish n shady 23 2.3 3.14 long
6 snail n shady 23 2.3 3.14 long
✅ Key idea
The pipe |> is what makes this readable top to bottom — a recipe, not a nest of parentheses.
Read it out loud:
- Take
pine_csv_df,
- then filter to real values,
- then select four columns,
- then add three new columns,
- then sort by side and length.
Part 3 · A Duplicate-Row Problem
A Duplicate-Row Problem
Remember pseudoreplication from Getting Started — one tree standing in for many? Duplicate rows are the data-entry version of the same mistake: the same measurement counted twice inflates your sample size without adding real information.
needle_check_df <- read_csv("data/pine_needles_error.csv")
nrow(needle_check_df)[1] 36
n_distinct(needle_check_df)[1] 18
Something’s wrong: nrow() and n_distinct() don’t match. Some rows are exact repeats.
📖 New words
n_distinct()— counts how many unique rows existdistinct()— returns the data with duplicate rows removed - be careful!!!\- unique() - looks for the unique rows or names in rows for mispellinbg
Finding the duplicates
# Which rows are exact duplicates of an earlier row?
needle_check_df |> filter(duplicated(needle_check_df))# A tibble: 18 × 5
name species needle_id needle_name length_mm
<chr> <chr> <dbl> <chr> <dbl>
1 John a 1 one 21.1
2 John a 2 two 21.2
3 John a 3 three 21.5
4 Mary a 1 one 21
5 Mary a 2 two 21.1
6 Mary a 3 three 21.3
7 Steve a 1 one 21.4
8 Steve a 2 two 21.3
9 Steve a 3 three 21.9
10 Harry a 1 one 24.1
11 Harry a 2 two 24.2
12 Harry a 3 three 24.3
13 Stan a 1 one 22.1
14 Stan a 2 two 22.2
15 Stan a 3 three 22.4
16 Jake a 1 one 20.3
17 Jake a 2 two 20.4
18 Jake a 3 three 20.1
🖐 Field habit
Before you trust any dataset — yours or a collaborator’s — check nrow() against n_distinct(). It takes five seconds and catches a real, common mistake.
Removing duplicates with distinct()
# Keep only one copy of each unique row
needle_clean_df <- needle_check_df |> distinct()
nrow(needle_check_df)[1] 36
nrow(needle_clean_df)[1] 18
⚠️ Watch out!
distinct() only catches exact duplicate rows. A typo’d duplicate (John vs john) slips right through — always eyeball your data too.
Part 4 · Saving Your Work
write_csv() — the mirror image of read_csv()
write_csv(pine_wrangled_df, "data/pine_needles_wrangled.csv")Raw data is read-only. pine_needles.csv never gets overwritten — write_csv() always creates a new, separate file. That’s why we named it pine_needles_wrangled.csv, not pine_needles.csv.
✅ Key idea
If you can’t reproduce a result from raw data + a script, something’s wrong with the script — never with the raw file. Keeping raw data untouched is what makes that guarantee possible.
Naming your saved files
- Never overwrite
data/pine_needles.csv— that file is raw - Save wrangled output with a name that says what happened:
pine_needles_wrangled.csv, notpine_needles2.csvorfinal.csv - Anyone re-running your script from scratch should get the exact same output file
⚠️ Watch out!
final.csv, final_v2.csv, final_FINAL.csv — this is exactly the mess a project folder with relative paths and clear names is meant to prevent.
🖐 Try it yourself
Predict what write_csv(pine_wrangled_df, "data/pine_needles_wrangled.csv") does if that file doesn’t exist yet. What if it already does?
Wrap-up
Today you:
- Imported the same data from a
.csvand an.xlsx - Used
filter(),select(),mutate(), andarrange()— and chained them into a pipeline - Filtered with the full boolean toolkit:
==,!=,>/<,%in%,is.na(), AND (,) vs. OR (|) - Recoded and binned columns with
case_when()— always ending in aTRUE ~catch-all - Converted units and log-transformed a column with
mutate() - Caught and removed duplicate rows with
distinct() - Saved a new file with
write_csv()without touching the raw data
🖐 Before next class
Finish the worksheet: wrangle pine_needles.csv into a new data frame with at least one unit conversion, one log transform, and one case_when() category, then save it with write_csv().
Up next — Describing Your Data
- Mean, median, variance, SD, SE
- The
length()trap and thesum(!is.na())fix group_by()+summarize()for tidy group comparisons
Getting unstuck
When code breaks — and it will, that is normal:
- Read the error message out loud; it usually names the line
- Check the usual suspects:
library(tidyverse)andlibrary(readxl)loaded? Spelling? A missing)or|>at the start of a line? ?function_nameopens the help page- Bring the exact error (copy-paste it) to class or office hours
✅ Key idea
Every working scientist googles error messages daily. Getting stuck is not failing — it is the job.