Lecture: Wrangling Your Data

Importing more than one way, reshaping columns, and saving what you make

Wrangling
R
tidyverse
Reading CSV and Excel files, the filter/select/mutate/arrange verbs, the full boolean-operator toolkit (==, !=, >, <, is.na(), %in%, & vs. |), recoding and binning with case_when(), log and unit-conversion transforms, catching duplicate rows, and writing your cleaned data back out — all on the pine needle dataset.
Author

Bill Perry

Where We Left Off

Last time you:

  • Turned a field observation into a testable H₀ / Hₐ
  • Built a project folder (data/, scripts/, figures/ , outout/)
  • Loaded pine_needles.csv and made your first ggplot
Note

✅ Key idea from Getting Started

You can get data into R and make a plot. Today we learn to reshape what you have and save what you make — the two skills every later module depends on.

Note

Today’s roadmap

  1. Importing data more than one way
  2. The five wrangling verbs
  3. filter() in depth — every boolean operator, %in%, AND vs. OR
  4. mutate() — unit conversions and log transforms
  5. case_when() — recoding and binning
  6. Catching duplicate rows
  7. Saving your work with write_csv()

Part 1 · Importing Data More Than One Way

The same data, two file formats

Field data doesn’t always arrive as a .csv. Excel workbooks are everywhere in ecology — you need both tools.

# install.packages("readxl")
# install.packages("janitor")
library(tidyverse)
library(readxl)   # reading Excel files
library(janitor)  # cleans variable names, excel poop, rounding issues

pine_csv_df   <- read_csv("data/pine_needles.csv")
pine_excel_df <- read_excel("data/pine_needles.xlsx")
Note

✅ Key idea

read_csv() comes from readr (loaded with tidyverse).
Excel files need the separate readxl package
— install once, library() every session.

Note

📖 New word

Package vs. bundletidyverse is a bundle of packages (readr, dplyr, ggplot2, …). readxl isn’t included; it always needs its own library() line.

Look before you trust it — every time

glimpse(pine_csv_df)
Rows: 48
Columns: 6
$ date      <chr> "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20/25", "3/20…
$ group     <chr> "cephalopods", "cephalopods", "cephalopods", "cephalopods", …
$ n_s       <chr> "n", "n", "n", "n", "n", "n", "s", "s", "s", "s", "s", "s", …
$ sun       <chr> "shady", "shady", "shady", "shady", "shady", "shady", "sunny…
$ tree_no   <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, …
$ length_mm <dbl> 20, 21, 23, 25, 21, 16, 15, 16, 14, 17, 13, 15, 19, 18, 20, …
dim(pine_csv_df)
[1] 48  6
Tip

🖐 Notice

Six columns: date, group, n_s, sun, tree_no, length_mm or what you call them.
Same structure whether it came from the .csv or the .xlsx.

Warning

⚠️ Watch out!

Excel loves to turn a plain number column into a date, or a tree_no like 1 into 1.0.
Always glimpse() an Excel import before doing anything else.

Part 2 · The Five Wrangling Verbs

The Five Wrangling Verbs

Almost everything you do to a data frame is one of these five functions, chained with the pipe |>:

Function What it does
filter() keep rows matching a condition
select() keep columns by name
mutate() add or change a column
arrange() sort rows
summarize() collapse rows to a summary

All take a data frame as input and return a data frame — that’s what makes them chainable.

Note

✅ Key idea

summarize() is Describing Your Data’s whole topic.
Today: the other four, plus saving what you build.

Tip

Read a pipeline out loud

pine_csv_df |>
  filter(...) |>   # then
  select(...) |>   # then
  mutate(...) |>   # then
  arrange(...)     # then

Each |> reads as “then.”

filter() — Picking Rows

# Keep only the shady (sheltered) side
pine_csv_df |> filter(n_s == "n") |> head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        20
2 3/20/25 cephalopods n     shady       1        21
3 3/20/25 cephalopods n     shady       1        23
4 3/20/25 cephalopods n     shady       1        25
5 3/20/25 cephalopods n     shady       1        21
6 3/20/25 cephalopods n     shady       1        16
# Keep only needles longer than 20 mm
pine_csv_df |> filter(length_mm > 20) |> head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        21
2 3/20/25 cephalopods n     shady       1        23
3 3/20/25 cephalopods n     shady       1        25
4 3/20/25 cephalopods n     shady       1        21
5 3/20/25 salmon      n     shady       2        23
6 3/20/25 salmon      n     shady       2        21
Warning

⚠️ Watch out!

  • = assigns a value
  • == tests equality. filter(n_s = "n") is an error
  • you want filter(n_s == "n").

Common comparison operators:

Symbol Meaning
== exactly equal
!= not equal
> < greater / less than
>= <= greater / less / or equal
is.na() is the value missing?
!is.na() is the value not missing?

filter() — Combining Conditions

# Combine two conditions with a comma (AND)
pine_csv_df |> filter(n_s == "s", length_mm > 15) |> head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods s     sunny       1        16
2 3/20/25 cephalopods s     sunny       1        17
3 3/20/25 crayfish    s     sunny       3        16
4 3/20/25 crayfish    s     sunny       3        17
5 3/20/25 snail       s     sunny       4        17
6 3/20/25 snail       s     sunny       4        16
Tip

AND vs. OR

  • A comma between conditions means AND
    • —every condition must be true for a row to be kept.
  • Use | between conditions for OR instead

filter() — OR, With a Real Example

Note

🔮 Predict first: n_s == "n" | length_mm > 22 keeps a row if the side is shady or the needle is longer than 22mm.
Will this ever keep a sunny needle?
Under what condition?

# Shady-side needles, OR any exceptionally long needle
pine_csv_df |>
  filter(n_s == "n" | length_mm > 22) |>
  head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        20
2 3/20/25 cephalopods n     shady       1        21
3 3/20/25 cephalopods n     shady       1        23
4 3/20/25 cephalopods n     shady       1        25
5 3/20/25 cephalopods n     shady       1        21
6 3/20/25 cephalopods n     shady       1        16
Warning

⚠️ Watch out!

  • | is the pipe character, not the pipe operator |> or %>%
  • Look related but do completely different jobs
  • one is logical OR, the other chains steps together.

filter() — Matching One of Several Values with %in%

# Keep rows where group is one of two named teams
pine_csv_df |>
  filter(group %in% c("cephalopods", "salmon")) |>
  head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        20
2 3/20/25 cephalopods n     shady       1        21
3 3/20/25 cephalopods n     shady       1        23
4 3/20/25 cephalopods n     shady       1        25
5 3/20/25 cephalopods n     shady       1        21
6 3/20/25 cephalopods n     shady       1        16
# The long way, for comparison — %in% replaces a chain of |
pine_csv_df |>
  filter(group == "cephalopods" | group == "salmon") |>
  head()
# A tibble: 6 × 6
  date    group       n_s   sun   tree_no length_mm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        20
2 3/20/25 cephalopods n     shady       1        21
3 3/20/25 cephalopods n     shady       1        23
4 3/20/25 cephalopods n     shady       1        25
5 3/20/25 cephalopods n     shady       1        21
6 3/20/25 cephalopods n     shady       1        16
Note

✅ Key idea

  • x %in% c(a, b, c) asks “is x equal to any of these?”
    • it’s the clean way to write an OR across many possible values without a long chain of |.

filter() — Missing Values with is.na()

# A tiny illustrative dataset with missing values
survey_df <- tibble(
  site  = c("A", "B", "C", "D"),
  count = c(12, NA, 8, NA)
)

survey_df |> filter(is.na(count))    # rows that ARE missing
# A tibble: 2 × 2
  site  count
  <chr> <dbl>
1 B        NA
2 D        NA
survey_df |> filter(!is.na(count))   # rows that are NOT missing
# A tibble: 2 × 2
  site  count
  <chr> <dbl>
1 A        12
2 C         8
Note

✅ Key idea

  • is.na(x) is itself a boolean test
    • TRUE/FALSE — so it plugs into filter()
    • exactly like == or > does.
Tip

📖 Coming up

pine_csv_df has no missing values, so is.na() has nothing to catch here.
Describing Your Data digs into why missing values matter so much once you’re computing a sample size or a mean.

select() — Picking Columns

# Keep only side and length
pine_csv_df |> select(n_s, length_mm) |> head()
# A tibble: 6 × 2
  n_s   length_mm
  <chr>     <dbl>
1 n            20
2 n            21
3 n            23
4 n            25
5 n            21
6 n            16
# Drop the date column (minus sign)
pine_csv_df |> select(-date) |> head()
# A tibble: 6 × 5
  group       n_s   sun   tree_no length_mm
  <chr>       <chr> <chr>   <dbl>     <dbl>
1 cephalopods n     shady       1        20
2 cephalopods n     shady       1        21
3 cephalopods n     shady       1        23
4 cephalopods n     shady       1        25
5 cephalopods n     shady       1        21
6 cephalopods n     shady       1        16
Note

✅ Key idea

select() doesn’t touch rows — it only trims the columns you’re carrying forward. Useful the moment a dataset has 30+ columns and you need 4 of them.

  • Name the columns you want, in any order
  • A leading - drops a column instead of keeping it

select() — Picking Columns by Pattern

# Keep columns that contain a word
pine_csv_df |> select(contains("n")) |> head()
# A tibble: 6 × 4
  n_s   sun   tree_no length_mm
  <chr> <chr>   <dbl>     <dbl>
1 n     shady       1        20
2 n     shady       1        21
3 n     shady       1        23
4 n     shady       1        25
5 n     shady       1        21
6 n     shady       1        16

Useful helpers:

  • starts_with("x") - super useful for similar units or types of data in variable
  • ends_with("_mm")
  • contains("sun")

📖 R4DS §3.3 — select()

mutate() — Creating and Changing Columns

Note

🔮 Predict first: length_mm / 10 converts millimeters to centimeters.
Will mutate() replace length_mm, or add a new column?
How many columns will the result have?

# Add a converted-units column
pine_csv_df |>
  mutate(length_cm = length_mm / 10) |>
  head()
# A tibble: 6 × 7
  date    group       n_s   sun   tree_no length_mm length_cm
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>     <dbl>
1 3/20/25 cephalopods n     shady       1        20       2  
2 3/20/25 cephalopods n     shady       1        21       2.1
3 3/20/25 cephalopods n     shady       1        23       2.3
4 3/20/25 cephalopods n     shady       1        25       2.5
5 3/20/25 cephalopods n     shady       1        21       2.1
6 3/20/25 cephalopods n     shady       1        16       1.6
Note

✅ Key idea

mutate() keeps every existing column and adds new ones to the right.
To overwrite a column, reuse its name:
mutate(length_mm = round(length_mm, 1)).

mutate() — More Examples

# Add a log-transformed column
pine_csv_df |>
  mutate(log_length = log(length_mm)) |>
  head()
# A tibble: 6 × 7
  date    group       n_s   sun   tree_no length_mm log_length
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>      <dbl>
1 3/20/25 cephalopods n     shady       1        20       3.00
2 3/20/25 cephalopods n     shady       1        21       3.04
3 3/20/25 cephalopods n     shady       1        23       3.14
4 3/20/25 cephalopods n     shady       1        25       3.22
5 3/20/25 cephalopods n     shady       1        21       3.04
6 3/20/25 cephalopods n     shady       1        16       2.77
# mutate() can build on a column it just created
pine_csv_df |>
  mutate(
    length_cm  = length_mm / 10,
    size_class = if_else(length_mm > 18, "long", "short")
  ) |>
  head()
# A tibble: 6 × 8
  date    group       n_s   sun   tree_no length_mm length_cm size_class
  <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>     <dbl> <chr>     
1 3/20/25 cephalopods n     shady       1        20       2   long      
2 3/20/25 cephalopods n     shady       1        21       2.1 long      
3 3/20/25 cephalopods n     shady       1        23       2.3 long      
4 3/20/25 cephalopods n     shady       1        25       2.5 long      
5 3/20/25 cephalopods n     shady       1        21       2.1 long      
6 3/20/25 cephalopods n     shady       1        16       1.6 short     
Tip

📖 Why log-transform?

  • Many biological measurements are right-skewed
    • a log transform pulls extreme values in and can make a distribution more symmetric before you summarize or test it.
  • We’ll check whether it matters for our data in a later module
  • Know what log you are doing log2 log10 or ln?????

Part 2b · case_when() — Recoding and Binning

case_when() — Turning a Number into a Category

Note

🔮 Predict first: We’re about to bin length_mm into "short", "medium", and "long". Needle 1 in the data is 20mm. Which bin does it land in — and does that depend on whether the cutoff is < or <=?

pine_csv_df |>
  mutate(
    size_class = case_when(
      length_mm < 16 ~ "short",
      length_mm < 20 ~ "medium",
      TRUE           ~ "long"
    )
  ) |>
  count(size_class)
# A tibble: 3 × 2
  size_class     n
  <chr>      <int>
1 long          16
2 medium        17
3 short         15
Note

✅ Key idea

case_when() checks conditions top to bottom and uses the first one that’s TRUE.
Once a row matches length_mm < 16, R never even looks at the later conditions for that row.

case_when() — Always End With TRUE ~

# What happens if we forget the catch-all?
pine_csv_df |>
  mutate(
    size_class = case_when(
      length_mm < 16 ~ "short",
      length_mm < 20 ~ "medium"
      # no TRUE ~ ... here
    )
  ) |>
  count(size_class)
# A tibble: 3 × 2
  size_class     n
  <chr>      <int>
1 medium        17
2 short         15
3 <NA>          16
Warning

⚠️ Watch out!

Drop the TRUE ~ "long" line and every “long” needle silently becomes NA instead of erroring. case_when() never warns you — always finish with a TRUE ~ ... catch-all, even when you’re sure you’ve covered every case.

case_when() — Recoding Categories, Not Just Numbers

# Give each field team a short code
pine_csv_df |>
  mutate(
    team_code = case_when(
      group == "cephalopods" ~ "CEPH",
      group == "salmon"      ~ "SALM",
      group == "crayfish"    ~ "CRAY",
      group == "snail"       ~ "SNAI",
      TRUE                   ~ "UNK"
    )
  ) |>
  distinct(group, team_code)
# A tibble: 4 × 2
  group       team_code
  <chr>       <chr>    
1 cephalopods CEPH     
2 salmon      SALM     
3 crayfish    CRAY     
4 snail       SNAI     
Tip

🖐 Notice

case_when() isn’t just for numeric cutoffs — every condition on the left of a ~ can be any boolean test: ==, %in%, >, even combinations with & and |.

The "UNK" catch-all should never actually appear here — if it does, that’s your signal a team name was misspelled somewhere upstream.

arrange() — Sorting Rows

# Shortest needles first
pine_csv_df |> arrange(length_mm)
# A tibble: 48 × 6
   date    group       n_s   sun   tree_no length_mm
   <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
 1 3/20/25 salmon      s     sunny       2        12
 2 3/20/25 salmon      s     sunny       2        12
 3 3/20/25 salmon      s     sunny       2        12
 4 3/20/25 cephalopods s     sunny       1        13
 5 3/20/25 salmon      s     sunny       2        13
 6 3/20/25 crayfish    s     sunny       3        13
 7 3/20/25 cephalopods s     sunny       1        14
 8 3/20/25 salmon      s     sunny       2        14
 9 3/20/25 salmon      s     sunny       2        14
10 3/20/25 crayfish    s     sunny       3        14
# ℹ 38 more rows
# Longest first
pine_csv_df |> arrange(desc(length_mm))
# A tibble: 48 × 6
   date    group       n_s   sun   tree_no length_mm
   <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
 1 3/20/25 cephalopods n     shady       1        25
 2 3/20/25 crayfish    n     shady       3        25
 3 3/20/25 cephalopods n     shady       1        23
 4 3/20/25 salmon      n     shady       2        23
 5 3/20/25 crayfish    n     shady       3        23
 6 3/20/25 snail       n     shady       4        23
 7 3/20/25 cephalopods n     shady       1        21
 8 3/20/25 cephalopods n     shady       1        21
 9 3/20/25 salmon      n     shady       2        21
10 3/20/25 crayfish    n     shady       3        21
# ℹ 38 more rows
  • Default is ascending (smallest first)
  • desc() reverses it

arrange() — Sorting by Multiple Columns

# Sort by side, then by length within each side
pine_csv_df |> arrange(n_s, desc(length_mm))
# A tibble: 48 × 6
   date    group       n_s   sun   tree_no length_mm
   <chr>   <chr>       <chr> <chr>   <dbl>     <dbl>
 1 3/20/25 cephalopods n     shady       1        25
 2 3/20/25 crayfish    n     shady       3        25
 3 3/20/25 cephalopods n     shady       1        23
 4 3/20/25 salmon      n     shady       2        23
 5 3/20/25 crayfish    n     shady       3        23
 6 3/20/25 snail       n     shady       4        23
 7 3/20/25 cephalopods n     shady       1        21
 8 3/20/25 cephalopods n     shady       1        21
 9 3/20/25 salmon      n     shady       2        21
10 3/20/25 crayfish    n     shady       3        21
# ℹ 38 more rows
  • Most useful at the end of a pipeline, to inspect results in a sensible order
Tip

arrange() never changes which rows you have — only the order they print in.

The Full Pipeline

pine_wrangled_df <- pine_csv_df |>
  filter(length_mm > 0) |>                 # drop impossible values
  select(group, n_s, sun, length_mm) |>    # keep what we need
  mutate(
    length_cm  = length_mm / 10,
    log_length = log(length_mm),
    size_class = case_when(
      length_mm < 16 ~ "short",
      length_mm < 20 ~ "medium",
      TRUE           ~ "long"
    )
  ) |>
  arrange(n_s, desc(length_mm))            # sort for inspection

head(pine_wrangled_df)
# A tibble: 6 × 7
  group       n_s   sun   length_mm length_cm log_length size_class
  <chr>       <chr> <chr>     <dbl>     <dbl>      <dbl> <chr>     
1 cephalopods n     shady        25       2.5       3.22 long      
2 crayfish    n     shady        25       2.5       3.22 long      
3 cephalopods n     shady        23       2.3       3.14 long      
4 salmon      n     shady        23       2.3       3.14 long      
5 crayfish    n     shady        23       2.3       3.14 long      
6 snail       n     shady        23       2.3       3.14 long      
Note

✅ Key idea

The pipe |> is what makes this readable top to bottom — a recipe, not a nest of parentheses.

Read it out loud:

  • Take pine_csv_df,
    • then filter to real values,
      • then select four columns,
        • then add three new columns,
          • then sort by side and length.

Part 3 · A Duplicate-Row Problem

A Duplicate-Row Problem

Remember pseudoreplication from Getting Started — one tree standing in for many? Duplicate rows are the data-entry version of the same mistake: the same measurement counted twice inflates your sample size without adding real information.

needle_check_df <- read_csv("data/pine_needles_error.csv")

nrow(needle_check_df)
[1] 36
n_distinct(needle_check_df)
[1] 18
Important

Something’s wrong: nrow() and n_distinct() don’t match. Some rows are exact repeats.

Note

📖 New words

  • n_distinct() — counts how many unique rows exist
  • distinct() — returns the data with duplicate rows removed - be careful!!!\
  • unique() - looks for the unique rows or names in rows for mispellinbg

Finding the duplicates

# Which rows are exact duplicates of an earlier row?
needle_check_df |> filter(duplicated(needle_check_df))
# A tibble: 18 × 5
   name  species needle_id needle_name length_mm
   <chr> <chr>       <dbl> <chr>           <dbl>
 1 John  a               1 one              21.1
 2 John  a               2 two              21.2
 3 John  a               3 three            21.5
 4 Mary  a               1 one              21  
 5 Mary  a               2 two              21.1
 6 Mary  a               3 three            21.3
 7 Steve a               1 one              21.4
 8 Steve a               2 two              21.3
 9 Steve a               3 three            21.9
10 Harry a               1 one              24.1
11 Harry a               2 two              24.2
12 Harry a               3 three            24.3
13 Stan  a               1 one              22.1
14 Stan  a               2 two              22.2
15 Stan  a               3 three            22.4
16 Jake  a               1 one              20.3
17 Jake  a               2 two              20.4
18 Jake  a               3 three            20.1
Tip

🖐 Field habit

Before you trust any dataset — yours or a collaborator’s — check nrow() against n_distinct(). It takes five seconds and catches a real, common mistake.

Removing duplicates with distinct()

# Keep only one copy of each unique row
needle_clean_df <- needle_check_df |> distinct()

nrow(needle_check_df)
[1] 36
nrow(needle_clean_df)
[1] 18
Warning

⚠️ Watch out!

distinct() only catches exact duplicate rows. A typo’d duplicate (John vs john) slips right through — always eyeball your data too.

Part 4 · Saving Your Work

write_csv() — the mirror image of read_csv()

write_csv(pine_wrangled_df, "data/pine_needles_wrangled.csv")
Important

Raw data is read-only. pine_needles.csv never gets overwritten — write_csv() always creates a new, separate file. That’s why we named it pine_needles_wrangled.csv, not pine_needles.csv.

Note

✅ Key idea

If you can’t reproduce a result from raw data + a script, something’s wrong with the script — never with the raw file. Keeping raw data untouched is what makes that guarantee possible.

Naming your saved files

  • Never overwrite data/pine_needles.csv — that file is raw
  • Save wrangled output with a name that says what happened: pine_needles_wrangled.csv, not pine_needles2.csv or final.csv
  • Anyone re-running your script from scratch should get the exact same output file
Warning

⚠️ Watch out!

final.csv, final_v2.csv, final_FINAL.csv — this is exactly the mess a project folder with relative paths and clear names is meant to prevent.

Tip

🖐 Try it yourself

Predict what write_csv(pine_wrangled_df, "data/pine_needles_wrangled.csv") does if that file doesn’t exist yet. What if it already does?

Wrap-up

Today you:

  • Imported the same data from a .csv and an .xlsx
  • Used filter(), select(), mutate(), and arrange() — and chained them into a pipeline
  • Filtered with the full boolean toolkit: ==, !=, >/<, %in%, is.na(), AND (,) vs. OR (|)
  • Recoded and binned columns with case_when() — always ending in a TRUE ~ catch-all
  • Converted units and log-transformed a column with mutate()
  • Caught and removed duplicate rows with distinct()
  • Saved a new file with write_csv() without touching the raw data
Tip

🖐 Before next class

Finish the worksheet: wrangle pine_needles.csv into a new data frame with at least one unit conversion, one log transform, and one case_when() category, then save it with write_csv().

Note

Up next — Describing Your Data

  • Mean, median, variance, SD, SE
  • The length() trap and the sum(!is.na()) fix
  • group_by() + summarize() for tidy group comparisons

Getting unstuck

When code breaks — and it will, that is normal:

  1. Read the error message out loud; it usually names the line
  2. Check the usual suspects: library(tidyverse) and library(readxl) loaded? Spelling? A missing ) or |> at the start of a line?
  3. ?function_name opens the help page
  4. Bring the exact error (copy-paste it) to class or office hours
Note

✅ Key idea

Every working scientist googles error messages daily. Getting stuck is not failing — it is the job.