Importing more than one way, reshaping columns, and saving what you make
Last time you:
data/, scripts/, figures/ , outout/)pine_needles.csv and made your first ggplotNote
✅ Key idea from Getting Started
You can get data into R and make a plot. Today we learn to reshape what you have and save what you make — the two skills every later module depends on.
Note
Today’s roadmap
filter() in depth — every boolean operator, %in%, AND vs. ORmutate() — unit conversions and log transformscase_when() — recoding and binningwrite_csv()Field data doesn’t always arrive as a .csv. Excel workbooks are everywhere in ecology — you need both tools.
Note
✅ Key idea
read_csv() comes from readr (loaded with tidyverse).
Excel files need the separate readxl package
— install once, library() every session.
Note
📖 New word
Package vs. bundle — tidyverse is a bundle of packages (readr, dplyr, ggplot2, …). readxl isn’t included; it always needs its own library() line.
Rows: 48
Columns: 6
$ date <chr> "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20…
$ group <chr> "cephalopods", "cephalopods", "cephalopods", "cephalopods", …
$ n_s <chr> "n", "n", "n", "n", "n", "n", "s", "s", "s", "s", "s", "s", …
$ sun <chr> "shady", "shady", "shady", "shady", "shady", "shady", "sunny…
$ tree_no <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, …
$ length_mm <dbl> 20, 21, 23, 25, 21, 16, 15, 16, 14, 17, 13, 15, 19, 18, 20, …
Tip
🖐 Notice
Six columns: date, group, n_s, sun, tree_no, length_mm or what you call them.
Same structure whether it came from the .csv or the .xlsx.
Warning
⚠️ Watch out!
Excel loves to turn a plain number column into a date, or a tree_no like 1 into 1.0.
Always glimpse() an Excel import before doing anything else.
Almost everything you do to a data frame is one of these five functions, chained with the pipe |>:
| Function | What it does |
|---|---|
filter() |
keep rows matching a condition |
select() |
keep columns by name |
mutate() |
add or change a column |
arrange() |
sort rows |
summarize() |
collapse rows to a summary |
All take a data frame as input and return a data frame — that’s what makes them chainable.
Note
✅ Key idea
summarize() is Describing Your Data’s whole topic.
Today: the other four, plus saving what you build.
Tip
Read a pipeline out loud
pine_csv_df |>
filter(...) |> # then
select(...) |> # then
mutate(...) |> # then
arrange(...) # then
Each |> reads as “then.”
filter() — Picking Rows# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 21
2 3/20/25 cephalopods n shady 1 23
3 3/20/25 cephalopods n shady 1 25
4 3/20/25 cephalopods n shady 1 21
5 3/20/25 salmon n shady 2 23
6 3/20/25 salmon n shady 2 21
Warning
⚠️ Watch out!
= assigns a value== tests equality. filter(n_s = "n") is an errorfilter(n_s == "n").Common comparison operators:
| Symbol | Meaning |
|---|---|
== |
exactly equal |
!= |
not equal |
> < |
greater / less than |
>= <= |
greater / less / or equal |
is.na() |
is the value missing? |
!is.na() |
is the value not missing? |
filter() — Combining Conditions# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods s sunny 1 16
2 3/20/25 cephalopods s sunny 1 17
3 3/20/25 crayfish s sunny 3 16
4 3/20/25 crayfish s sunny 3 17
5 3/20/25 snail s sunny 4 17
6 3/20/25 snail s sunny 4 16
Tip
AND vs. OR
| between conditions for OR insteadfilter() — OR, With a Real ExampleNote
🔮 Predict first: n_s == "n" | length_mm > 22 keeps a row if the side is shady or the needle is longer than 22mm.
Will this ever keep a sunny needle?
Under what condition?
# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
Warning
⚠️ Watch out!
| is the pipe character, not the pipe operator |> or %>%filter() — Matching One of Several Values with %in%# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
# A tibble: 6 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20
2 3/20/25 cephalopods n shady 1 21
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 cephalopods n shady 1 25
5 3/20/25 cephalopods n shady 1 21
6 3/20/25 cephalopods n shady 1 16
Note
✅ Key idea
x %in% c(a, b, c) asks “is x equal to any of these?”
|.filter() — Missing Values with is.na()# A tibble: 2 × 2
site count
<chr> <dbl>
1 B NA
2 D NA
# A tibble: 2 × 2
site count
<chr> <dbl>
1 A 12
2 C 8
Note
✅ Key idea
is.na(x) is itself a boolean test
TRUE/FALSE — so it plugs into filter()== or > does.Tip
📖 Coming up
pine_csv_df has no missing values, so is.na() has nothing to catch here.
Describing Your Data digs into why missing values matter so much once you’re computing a sample size or a mean.
select() — Picking Columns# A tibble: 6 × 2
n_s length_mm
<chr> <dbl>
1 n 20
2 n 21
3 n 23
4 n 25
5 n 21
6 n 16
# A tibble: 6 × 5
group n_s sun tree_no length_mm
<chr> <chr> <chr> <dbl> <dbl>
1 cephalopods n shady 1 20
2 cephalopods n shady 1 21
3 cephalopods n shady 1 23
4 cephalopods n shady 1 25
5 cephalopods n shady 1 21
6 cephalopods n shady 1 16
Note
✅ Key idea
select() doesn’t touch rows — it only trims the columns you’re carrying forward. Useful the moment a dataset has 30+ columns and you need 4 of them.
- drops a column instead of keeping itselect() — Picking Columns by PatternUseful helpers:
starts_with("x") - super useful for similar units or types of data in variableends_with("_mm")contains("sun")📖 R4DS §3.3 — select()
mutate() — Creating and Changing ColumnsNote
🔮 Predict first: length_mm / 10 converts millimeters to centimeters.
Will mutate() replace length_mm, or add a new column?
How many columns will the result have?
# A tibble: 6 × 7
date group n_s sun tree_no length_mm length_cm
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20 2
2 3/20/25 cephalopods n shady 1 21 2.1
3 3/20/25 cephalopods n shady 1 23 2.3
4 3/20/25 cephalopods n shady 1 25 2.5
5 3/20/25 cephalopods n shady 1 21 2.1
6 3/20/25 cephalopods n shady 1 16 1.6
Note
✅ Key idea
mutate() keeps every existing column and adds new ones to the right.
To overwrite a column, reuse its name:
mutate(length_mm = round(length_mm, 1)).
mutate() — More Examples# A tibble: 6 × 7
date group n_s sun tree_no length_mm log_length
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 20 3.00
2 3/20/25 cephalopods n shady 1 21 3.04
3 3/20/25 cephalopods n shady 1 23 3.14
4 3/20/25 cephalopods n shady 1 25 3.22
5 3/20/25 cephalopods n shady 1 21 3.04
6 3/20/25 cephalopods n shady 1 16 2.77
# A tibble: 6 × 8
date group n_s sun tree_no length_mm length_cm size_class
<chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> <chr>
1 3/20/25 cephalopods n shady 1 20 2 long
2 3/20/25 cephalopods n shady 1 21 2.1 long
3 3/20/25 cephalopods n shady 1 23 2.3 long
4 3/20/25 cephalopods n shady 1 25 2.5 long
5 3/20/25 cephalopods n shady 1 21 2.1 long
6 3/20/25 cephalopods n shady 1 16 1.6 short
Tip
📖 Why log-transform?
case_when() — Recoding and Binningcase_when() — Turning a Number into a CategoryNote
🔮 Predict first: We’re about to bin length_mm into "short", "medium", and "long". Needle 1 in the data is 20mm. Which bin does it land in — and does that depend on whether the cutoff is < or <=?
Note
✅ Key idea
case_when() checks conditions top to bottom and uses the first one that’s TRUE.
Once a row matches length_mm < 16, R never even looks at the later conditions for that row.
case_when() — Always End With TRUE ~Warning
⚠️ Watch out!
Drop the TRUE ~ "long" line and every “long” needle silently becomes NA instead of erroring. case_when() never warns you — always finish with a TRUE ~ ... catch-all, even when you’re sure you’ve covered every case.
case_when() — Recoding Categories, Not Just Numbers# A tibble: 4 × 2
group team_code
<chr> <chr>
1 cephalopods CEPH
2 salmon SALM
3 crayfish CRAY
4 snail SNAI
Tip
🖐 Notice
case_when() isn’t just for numeric cutoffs — every condition on the left of a ~ can be any boolean test: ==, %in%, >, even combinations with & and |.
The "UNK" catch-all should never actually appear here — if it does, that’s your signal a team name was misspelled somewhere upstream.
arrange() — Sorting Rows# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 salmon s sunny 2 12
2 3/20/25 salmon s sunny 2 12
3 3/20/25 salmon s sunny 2 12
4 3/20/25 cephalopods s sunny 1 13
5 3/20/25 salmon s sunny 2 13
6 3/20/25 crayfish s sunny 3 13
7 3/20/25 cephalopods s sunny 1 14
8 3/20/25 salmon s sunny 2 14
9 3/20/25 salmon s sunny 2 14
10 3/20/25 crayfish s sunny 3 14
# ℹ 38 more rows
# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 25
2 3/20/25 crayfish n shady 3 25
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 salmon n shady 2 23
5 3/20/25 crayfish n shady 3 23
6 3/20/25 snail n shady 4 23
7 3/20/25 cephalopods n shady 1 21
8 3/20/25 cephalopods n shady 1 21
9 3/20/25 salmon n shady 2 21
10 3/20/25 crayfish n shady 3 21
# ℹ 38 more rows
desc() reverses itarrange() — Sorting by Multiple Columns# A tibble: 48 × 6
date group n_s sun tree_no length_mm
<chr> <chr> <chr> <chr> <dbl> <dbl>
1 3/20/25 cephalopods n shady 1 25
2 3/20/25 crayfish n shady 3 25
3 3/20/25 cephalopods n shady 1 23
4 3/20/25 salmon n shady 2 23
5 3/20/25 crayfish n shady 3 23
6 3/20/25 snail n shady 4 23
7 3/20/25 cephalopods n shady 1 21
8 3/20/25 cephalopods n shady 1 21
9 3/20/25 salmon n shady 2 21
10 3/20/25 crayfish n shady 3 21
# ℹ 38 more rows
Tip
arrange() never changes which rows you have — only the order they print in.
pine_wrangled_df <- pine_csv_df |>
filter(length_mm > 0) |> # drop impossible values
select(group, n_s, sun, length_mm) |> # keep what we need
mutate(
length_cm = length_mm / 10,
log_length = log(length_mm),
size_class = case_when(
length_mm < 16 ~ "short",
length_mm < 20 ~ "medium",
TRUE ~ "long"
)
) |>
arrange(n_s, desc(length_mm)) # sort for inspection
head(pine_wrangled_df)# A tibble: 6 × 7
group n_s sun length_mm length_cm log_length size_class
<chr> <chr> <chr> <dbl> <dbl> <dbl> <chr>
1 cephalopods n shady 25 2.5 3.22 long
2 crayfish n shady 25 2.5 3.22 long
3 cephalopods n shady 23 2.3 3.14 long
4 salmon n shady 23 2.3 3.14 long
5 crayfish n shady 23 2.3 3.14 long
6 snail n shady 23 2.3 3.14 long
Note
✅ Key idea
The pipe |> is what makes this readable top to bottom — a recipe, not a nest of parentheses.
Read it out loud:
- Take
pine_csv_df,
- then filter to real values,
- then select four columns,
- then add three new columns,
- then sort by side and length.
Remember pseudoreplication from Getting Started — one tree standing in for many? Duplicate rows are the data-entry version of the same mistake: the same measurement counted twice inflates your sample size without adding real information.
[1] 36
[1] 18
Important
Something’s wrong: nrow() and n_distinct() don’t match. Some rows are exact repeats.
Note
📖 New words
n_distinct() — counts how many unique rows existdistinct() — returns the data with duplicate rows removed - be careful!!!\# A tibble: 18 × 5
name species needle_id needle_name length_mm
<chr> <chr> <dbl> <chr> <dbl>
1 John a 1 one 21.1
2 John a 2 two 21.2
3 John a 3 three 21.5
4 Mary a 1 one 21
5 Mary a 2 two 21.1
6 Mary a 3 three 21.3
7 Steve a 1 one 21.4
8 Steve a 2 two 21.3
9 Steve a 3 three 21.9
10 Harry a 1 one 24.1
11 Harry a 2 two 24.2
12 Harry a 3 three 24.3
13 Stan a 1 one 22.1
14 Stan a 2 two 22.2
15 Stan a 3 three 22.4
16 Jake a 1 one 20.3
17 Jake a 2 two 20.4
18 Jake a 3 three 20.1
Tip
🖐 Field habit
Before you trust any dataset — yours or a collaborator’s — check nrow() against n_distinct(). It takes five seconds and catches a real, common mistake.
distinct()Warning
⚠️ Watch out!
distinct() only catches exact duplicate rows. A typo’d duplicate (John vs john) slips right through — always eyeball your data too.
write_csv() — the mirror image of read_csv()Important
Raw data is read-only. pine_needles.csv never gets overwritten — write_csv() always creates a new, separate file. That’s why we named it pine_needles_wrangled.csv, not pine_needles.csv.
Note
✅ Key idea
If you can’t reproduce a result from raw data + a script, something’s wrong with the script — never with the raw file. Keeping raw data untouched is what makes that guarantee possible.
data/pine_needles.csv — that file is rawpine_needles_wrangled.csv, not pine_needles2.csv or final.csvWarning
⚠️ Watch out!
final.csv, final_v2.csv, final_FINAL.csv — this is exactly the mess a project folder with relative paths and clear names is meant to prevent.
Tip
🖐 Try it yourself
Predict what write_csv(pine_wrangled_df, "data/pine_needles_wrangled.csv") does if that file doesn’t exist yet. What if it already does?
Today you:
.csv and an .xlsxfilter(), select(), mutate(), and arrange() — and chained them into a pipeline==, !=, >/<, %in%, is.na(), AND (,) vs. OR (|)case_when() — always ending in a TRUE ~ catch-allmutate()distinct()write_csv() without touching the raw dataTip
🖐 Before next class
Finish the worksheet: wrangle pine_needles.csv into a new data frame with at least one unit conversion, one log transform, and one case_when() category, then save it with write_csv().
Note
Up next — Describing Your Data
length() trap and the sum(!is.na()) fixgroup_by() + summarize() for tidy group comparisonsWhen code breaks — and it will, that is normal:
library(tidyverse) and library(readxl) loaded? Spelling? A missing ) or |> at the start of a line??function_name opens the help pageNote
✅ Key idea
Every working scientist googles error messages daily. Getting stuck is not failing — it is the job.