Lecture: Wrangling Your Data

Importing more than one way, reshaping columns, and saving what you make

Wrangling

tidyverse

Reading CSV and Excel files, the filter/select/mutate/arrange verbs, the full boolean-operator toolkit (==, !=, >, <, is.na(), %in%, & vs. |), recoding and binning with case_when(), log and unit-conversion transforms, catching duplicate rows, and writing your cleaned data back out — all on the pine needle dataset.

Author

Bill Perry

Where We Left Off

Last time you:

Turned a field observation into a testable H₀ / Hₐ
Built a project folder (data/, scripts/, figures/, output/)
Loaded pine_data.csv and made your first ggplot
Saw why 64 rows is not 64 independent observations

Note

✅ Key idea from Getting Started

You can get data into R and make a plot. Today we learn to reshape what you have and save what you make — the two skills every later module depends on.

Note

Today’s roadmap

Importing data more than one way
The five wrangling verbs
filter() in depth — every boolean operator, %in%, AND vs. OR
mutate() — unit conversions and log transforms
case_when() — recoding and binning
Catching duplicate rows
Saving your work with write_csv()

Part 1 · Importing Data More Than One Way

The same data, two file formats

Field data doesn’t always arrive as a .csv. Excel workbooks are everywhere in ecology — you need both tools.

📥 Download the data 📥 The messy version

# install.packages("readxl")
# install.packages("janitor")
library(tidyverse)
library(readxl)   # reading Excel files
library(janitor)  # cleans messy column names, Excel artifacts, rounding

pine_csv_df   <- read_csv("data/pine_data.csv")
pine_excel_df <- read_excel("data/pine_data.xlsx")

Note

✅ Key idea

read_csv() comes from readr (loaded with tidyverse).
Excel files need the separate readxl package
— install once, library() every session.

Note

📖 New word

Package vs. bundle — tidyverse is a bundle of packages (readr, dplyr, ggplot2, …). readxl isn’t included; it always needs its own library() line.

📖 R4DS §7 — Data import

Look before you trust it — every time

glimpse(pine_csv_df)

Rows: 64
Columns: 6
$ source_file      <chr> "Bartell_Shingledecker_tree_needle_data_2026.csv", "B…
$ team             <chr> "grace_jace", "grace_jace", "grace_jace", "grace_jace…
$ replicate        <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2,…
$ side             <chr> "sunny", "sunny", "sunny", "sunny", "sunny", "sunny",…
$ needle_length_mm <dbl> 18.44, 12.48, 17.73, 13.98, 14.18, 19.17, 10.92, 17.3…
$ needle_width_mm  <dbl> 0.79, 0.99, 1.12, 0.94, 1.00, 0.99, 0.81, 1.29, 0.75,…

dim(pine_csv_df)

[1] 64  6

Tip

🖐 Notice

Six columns: source_file, team, replicate, side, needle_length_mm, needle_width_mm.
Same structure whether it came from the .csv or the .xlsx.

Warning

⚠️ Watch out!

Excel loves to turn a plain number column into a date, or a replicate like 1 into 1.0.
Always glimpse() an Excel import before doing anything else.

📖 R4DS §20 — Spreadsheets is entirely about importing from Excel and Sheets.

Part 2 · The Five Wrangling Verbs

The Five Wrangling Verbs

Almost everything you do to a data frame is one of these five functions, chained with the pipe |>:

Function	What it does
`filter()`	keep rows matching a condition
`select()`	keep columns by name
`mutate()`	add or change a column
`arrange()`	sort rows
`summarize()`	collapse rows to a summary

All take a data frame as input and return a data frame — that’s what makes them chainable.

Note

✅ Key idea

summarize() is Describing Your Data’s whole topic.
Today: the other four, plus saving what you build.

Tip

Read a pipeline out loud

pine_csv_df |>
  filter(...) |>   # then
  select(...) |>   # then
  mutate(...) |>   # then
  arrange(...)     # then

Each |> reads as “then.”

Note

📖 Reference

📖 R4DS §3 — Data transformation is the chapter for all five verbs and the pipe.

`filter()` — Picking Rows

# Keep only the shady side
pine_csv_df |> filter(side == "shady") |> head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 shady             16.6            0.75
2 Bartell_Shingledecker_… grac…         2 shady             17.0            0.84
3 Bartell_Shingledecker_… grac…         3 shady             14.3            0.86
4 Bartell_Shingledecker_… grac…         4 shady             15.6            0.98
5 Bartell_Shingledecker_… grac…         5 shady             15.1            1.35
6 Bartell_Shingledecker_… grac…         6 shady             13.1            0.77

# Keep only needles longer than 17 mm
pine_csv_df |> filter(needle_length_mm > 17) |> head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 sunny             18.4            0.79
2 Bartell_Shingledecker_… grac…         3 sunny             17.7            1.12
3 Bartell_Shingledecker_… grac…         6 sunny             19.2            0.99
4 Bartell_Shingledecker_… grac…         8 sunny             17.3            1.29
5 Bartell_Shingledecker_… grac…         7 shady             17.9            0.69
6 heal_thome_pineneedles… bota…         2 shady             20.7            0.77

Warning

⚠️ Watch out!

= assigns a value
== tests equality — filter(side = "shady") is an error
- you want filter(side == "shady").

Common comparison operators:

Symbol	Meaning
`==`	exactly equal
`!=`	not equal
`>` `<`	greater / less than
`>=` `<=`	greater / less / or equal
`is.na()`	is the value missing?
`!is.na()`	is the value not missing?

`filter()` — Combining Conditions

# Combine two conditions with a comma (AND)
pine_csv_df |> filter(side == "sunny", needle_length_mm > 15) |> head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 sunny             18.4            0.79
2 Bartell_Shingledecker_… grac…         3 sunny             17.7            1.12
3 Bartell_Shingledecker_… grac…         6 sunny             19.2            0.99
4 Bartell_Shingledecker_… grac…         8 sunny             17.3            1.29
5 heal_thome_pineneedles… bota…         1 sunny             16.8            0.98
6 heal_thome_pineneedles… bota…         2 sunny             18.7            1.07

Tip

AND vs. OR

A comma between conditions means AND
- —every condition must be true for a row to be kept.
Use | between conditions for OR instead

`filter()` — OR, With a Real Example

Note

🔮 Predict first: side == "shady" | needle_length_mm > 18 keeps a row if the side is shady or the needle is longer than 18 mm.
Will this ever keep a sunny needle?
Under what condition?

# Shady-side needles, OR any exceptionally long needle
pine_csv_df |>
  filter(side == "shady" | needle_length_mm > 18) |>
  head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 sunny             18.4            0.79
2 Bartell_Shingledecker_… grac…         6 sunny             19.2            0.99
3 Bartell_Shingledecker_… grac…         1 shady             16.6            0.75
4 Bartell_Shingledecker_… grac…         2 shady             17.0            0.84
5 Bartell_Shingledecker_… grac…         3 shady             14.3            0.86
6 Bartell_Shingledecker_… grac…         4 shady             15.6            0.98

Warning

⚠️ Watch out!

| is the pipe character (logical OR), not the pipe operator |> (or %>%)
They look alike but do completely different jobs — one is logical OR, the other chains steps together.

`filter()` — Matching One of Several Values with `%in%`

# Keep rows where team is one of two named teams
pine_csv_df |>
  filter(team %in% c("botany", "waterbugs")) |>
  head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 heal_thome_pineneedles… bota…         1 shady             16.5            0.65
2 heal_thome_pineneedles… bota…         2 shady             20.7            0.77
3 heal_thome_pineneedles… bota…         3 shady             17.6            0.94
4 heal_thome_pineneedles… bota…         4 shady             15.1            0.72
5 heal_thome_pineneedles… bota…         5 shady             17.6            0.79
6 heal_thome_pineneedles… bota…         6 shady             15.3            0.9

# The long way, for comparison — %in% replaces a chain of |
pine_csv_df |>
  filter(team == "botany" | team == "waterbugs") |>
  head()

# A tibble: 6 × 6
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 heal_thome_pineneedles… bota…         1 shady             16.5            0.65
2 heal_thome_pineneedles… bota…         2 shady             20.7            0.77
3 heal_thome_pineneedles… bota…         3 shady             17.6            0.94
4 heal_thome_pineneedles… bota…         4 shady             15.1            0.72
5 heal_thome_pineneedles… bota…         5 shady             17.6            0.79
6 heal_thome_pineneedles… bota…         6 shady             15.3            0.9

Note

✅ Key idea

x %in% c(a, b, c) asks “is x equal to any of these?”
- it’s the clean way to write an OR across many possible values without a long chain of |.

`filter()` — Missing Values with `is.na()`

# A tiny illustrative dataset with missing values
survey_df <- tibble(
  site  = c("A", "B", "C", "D"),
  count = c(12, NA, 8, NA)
)

survey_df |> filter(is.na(count))    # rows that ARE missing

# A tibble: 2 × 2
  site  count
  <chr> <dbl>
1 B        NA
2 D        NA

survey_df |> filter(!is.na(count))   # rows that are NOT missing

# A tibble: 2 × 2
  site  count
  <chr> <dbl>
1 A        12
2 C         8

Note

✅ Key idea

is.na(x) is itself a boolean test
- TRUE/FALSE — so it plugs into filter()
- exactly like == or > does.

Tip

📖 Coming up

pine_csv_df has no missing values, so is.na() has nothing to catch here.
Describing Your Data digs into why missing values matter so much once you’re computing a sample size or a mean.

Note

📖 Reference

📖 R4DS §12 — Logical vectors covers ==, &, |, %in% and is.na().
📖 R4DS §18 — Missing values

`select()` — Picking Columns

# Keep only side and length
pine_csv_df |> select(side, needle_length_mm) |> head()

# A tibble: 6 × 2
  side  needle_length_mm
  <chr>            <dbl>
1 sunny             18.4
2 sunny             12.5
3 sunny             17.7
4 sunny             14.0
5 sunny             14.2
6 sunny             19.2

# Drop the provenance column we do not need (minus sign)
pine_csv_df |> select(-source_file) |> head()

# A tibble: 6 × 5
  team       replicate side  needle_length_mm needle_width_mm
  <chr>          <dbl> <chr>            <dbl>           <dbl>
1 grace_jace         1 sunny             18.4            0.79
2 grace_jace         2 sunny             12.5            0.99
3 grace_jace         3 sunny             17.7            1.12
4 grace_jace         4 sunny             14.0            0.94
5 grace_jace         5 sunny             14.2            1   
6 grace_jace         6 sunny             19.2            0.99

Note

✅ Key idea

select() doesn’t touch rows — it only trims the columns you’re carrying forward. Useful the moment a dataset has 30+ columns and you need 4 of them.

Name the columns you want, in any order
A leading - drops a column instead of keeping it

`select()` — Picking Columns by Pattern

# Keep columns that contain a word
pine_csv_df |> select(contains("n")) |> head()

# A tibble: 6 × 2
  needle_length_mm needle_width_mm
             <dbl>           <dbl>
1             18.4            0.79
2             12.5            0.99
3             17.7            1.12
4             14.0            0.94
5             14.2            1   
6             19.2            0.99

Useful helpers:

starts_with("needle") — super useful when related variables share a prefix
ends_with("_mm")
contains("needle")

📖 R4DS §3.3 — select()

`mutate()` — Creating and Changing Columns

Note

🔮 Predict first: needle_length_mm / 10 converts millimeters to centimeters.
Will mutate() replace needle_length_mm, or add a new column?
How many columns will the result have?

# Add a converted-units column
pine_csv_df |>
  mutate(needle_length_cm = needle_length_mm / 10) |>
  head()

# A tibble: 6 × 7
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 sunny             18.4            0.79
2 Bartell_Shingledecker_… grac…         2 sunny             12.5            0.99
3 Bartell_Shingledecker_… grac…         3 sunny             17.7            1.12
4 Bartell_Shingledecker_… grac…         4 sunny             14.0            0.94
5 Bartell_Shingledecker_… grac…         5 sunny             14.2            1   
6 Bartell_Shingledecker_… grac…         6 sunny             19.2            0.99
# ℹ 1 more variable: needle_length_cm <dbl>

Note

✅ Key idea

mutate() keeps every existing column and adds new ones to the right.
To overwrite a column, reuse its name:
mutate(needle_length_mm = round(needle_length_mm, 1)).

`mutate()` — More Examples

# Add a log-transformed column
pine_csv_df |>
  mutate(log_length = log(needle_length_mm)) |>
  head()

# A tibble: 6 × 7
  source_file  team  replicate side  needle_length_mm needle_width_mm log_length
  <chr>        <chr>     <dbl> <chr>            <dbl>           <dbl>      <dbl>
1 Bartell_Shi… grac…         1 sunny             18.4            0.79       2.91
2 Bartell_Shi… grac…         2 sunny             12.5            0.99       2.52
3 Bartell_Shi… grac…         3 sunny             17.7            1.12       2.88
4 Bartell_Shi… grac…         4 sunny             14.0            0.94       2.64
5 Bartell_Shi… grac…         5 sunny             14.2            1          2.65
6 Bartell_Shi… grac…         6 sunny             19.2            0.99       2.95

# mutate() can build on a column it just created
pine_csv_df |>
  mutate(
    needle_length_cm  = needle_length_mm / 10,
    size_class = if_else(needle_length_mm > 15, "long", "short")
  ) |>
  head()

# A tibble: 6 × 8
  source_file             team  replicate side  needle_length_mm needle_width_mm
  <chr>                   <chr>     <dbl> <chr>            <dbl>           <dbl>
1 Bartell_Shingledecker_… grac…         1 sunny             18.4            0.79
2 Bartell_Shingledecker_… grac…         2 sunny             12.5            0.99
3 Bartell_Shingledecker_… grac…         3 sunny             17.7            1.12
4 Bartell_Shingledecker_… grac…         4 sunny             14.0            0.94
5 Bartell_Shingledecker_… grac…         5 sunny             14.2            1   
6 Bartell_Shingledecker_… grac…         6 sunny             19.2            0.99
# ℹ 2 more variables: needle_length_cm <dbl>, size_class <chr>

Tip

📖 Why log-transform?

Many biological measurements are right-skewed
- a log transform pulls extreme values in and can make a distribution more symmetric before you summarize or test it.
We’ll check whether it matters for our data in a later module
Know which log you are using. In R, log() is the natural log (ln, base e); log2() and log10() are the other two. Always say which one in your methods.

📖 R4DS §13 — Numbers

Part 2b · `case_when()` — Recoding and Binning

`case_when()` — Turning a Number into a Category

Note

🔮 Predict first: We’re about to bin needle_length_mm into "short" (< 13), "medium" (< 16), and "long". The first needle in the data is 18.44 mm. Which bin does it land in — and would that change if the cutoffs used <= instead of <?

pine_csv_df |>
  mutate(
    size_class = case_when(
      needle_length_mm < 13 ~ "short",
      needle_length_mm < 16 ~ "medium",
      TRUE           ~ "long"
    )
  ) |>
  count(size_class)

# A tibble: 3 × 2
  size_class     n
  <chr>      <int>
1 long          21
2 medium        25
3 short         18

Note

✅ Key idea

case_when() checks conditions top to bottom and uses the first one that’s TRUE.
Once a row matches needle_length_mm < 13, R never even looks at the later conditions for that row.

📖 R4DS §12.5 — if_else() and case_when()

`case_when()` — Always End With `TRUE ~`

# What happens if we forget the catch-all?
pine_csv_df |>
  mutate(
    size_class = case_when(
      needle_length_mm < 13 ~ "short",
      needle_length_mm < 16 ~ "medium"
      # no TRUE ~ ... here
    )
  ) |>
  count(size_class)

# A tibble: 3 × 2
  size_class     n
  <chr>      <int>
1 medium        25
2 short         18
3 <NA>          21

Warning

⚠️ Watch out!

Drop the TRUE ~ "long" line and all 21 “long” needles silently become NA instead of erroring. case_when() never warns you — always finish with a TRUE ~ ... catch-all, even when you’re sure you’ve covered every case.

`case_when()` — Recoding Categories, Not Just Numbers

# Give each field team a short code
pine_csv_df |>
  mutate(
    team_code = case_when(
      team == "botany"     ~ "BOT",
      team == "grace_jace" ~ "GRJ",
      team == "one"        ~ "ONE",
      team == "waterbugs"  ~ "WBG",
      TRUE                 ~ "UNK"
    )
  ) |>
  distinct(team, team_code)

# A tibble: 4 × 2
  team       team_code
  <chr>      <chr>    
1 grace_jace GRJ      
2 botany     BOT      
3 one        ONE      
4 waterbugs  WBG

Tip

🖐 Notice

case_when() isn’t just for numeric cutoffs — every condition on the left of a ~ can be any boolean test: ==, %in%, >, even combinations with & and |.

The "UNK" catch-all should never actually appear here — if it does, that’s your signal a team name was misspelled somewhere upstream.

`arrange()` — Sorting Rows

# Shortest needles first
pine_csv_df |> arrange(needle_length_mm)

# A tibble: 64 × 6
   source_file            team  replicate side  needle_length_mm needle_width_mm
   <chr>                  <chr>     <dbl> <chr>            <dbl>           <dbl>
 1 Needle_Length_Suhr:Bu… wate…         2 shady             7.23            1.16
 2 Needle_Length_Suhr:Bu… wate…         1 shady            10.5             1.22
 3 Kehr_and_Baalke.csv    one           8 shady            10.6             0.98
 4 Needle_Length_Suhr:Bu… wate…         2 sunny            10.7             1.08
 5 Bartell_Shingledecker… grac…         7 sunny            10.9             0.81
 6 Needle_Length_Suhr:Bu… wate…         7 shady            11.3             1.13
 7 Kehr_and_Baalke.csv    one           4 sunny            11.4             1.17
 8 Kehr_and_Baalke.csv    one           1 shady            11.5             1.11
 9 heal_thome_pineneedle… bota…         8 shady            11.6             0.66
10 Needle_Length_Suhr:Bu… wate…         3 shady            11.6             1.27
# ℹ 54 more rows

# Longest first
pine_csv_df |> arrange(desc(needle_length_mm))

# A tibble: 64 × 6
   source_file            team  replicate side  needle_length_mm needle_width_mm
   <chr>                  <chr>     <dbl> <chr>            <dbl>           <dbl>
 1 heal_thome_pineneedle… bota…         2 shady             20.7            0.77
 2 Needle_Length_Suhr:Bu… wate…         3 sunny             19.5            1.58
 3 Bartell_Shingledecker… grac…         6 sunny             19.2            0.99
 4 heal_thome_pineneedle… bota…         8 sunny             18.8            1.12
 5 heal_thome_pineneedle… bota…         2 sunny             18.7            1.07
 6 heal_thome_pineneedle… bota…         6 sunny             18.5            1.18
 7 Bartell_Shingledecker… grac…         1 sunny             18.4            0.79
 8 Bartell_Shingledecker… grac…         7 shady             17.9            0.69
 9 Bartell_Shingledecker… grac…         3 sunny             17.7            1.12
10 Kehr_and_Baalke.csv    one           5 shady             17.7            1   
# ℹ 54 more rows

Default is ascending (smallest first)
desc() reverses it

`arrange()` — Sorting by Multiple Columns

# Sort by side, then by length within each side
pine_csv_df |> arrange(side, desc(needle_length_mm))

# A tibble: 64 × 6
   source_file            team  replicate side  needle_length_mm needle_width_mm
   <chr>                  <chr>     <dbl> <chr>            <dbl>           <dbl>
 1 heal_thome_pineneedle… bota…         2 shady             20.7            0.77
 2 Bartell_Shingledecker… grac…         7 shady             17.9            0.69
 3 Kehr_and_Baalke.csv    one           5 shady             17.7            1   
 4 heal_thome_pineneedle… bota…         5 shady             17.6            0.79
 5 heal_thome_pineneedle… bota…         3 shady             17.6            0.94
 6 Bartell_Shingledecker… grac…         2 shady             17.0            0.84
 7 Bartell_Shingledecker… grac…         1 shady             16.6            0.75
 8 heal_thome_pineneedle… bota…         1 shady             16.5            0.65
 9 Bartell_Shingledecker… grac…         4 shady             15.6            0.98
10 heal_thome_pineneedle… bota…         6 shady             15.3            0.9 
# ℹ 54 more rows

Most useful at the end of a pipeline, to inspect results in a sensible order

Tip

arrange() never changes which rows you have — only the order they print in.

The Full Pipeline

pine_wrangled_df <- pine_csv_df |>
  filter(needle_length_mm > 0) |>                 # drop impossible values
  select(team, side, needle_length_mm, needle_width_mm) |>  # keep what we need
  mutate(
    needle_length_cm  = needle_length_mm / 10,
    log_length = log(needle_length_mm),
    size_class = case_when(
      needle_length_mm < 13 ~ "short",
      needle_length_mm < 16 ~ "medium",
      TRUE                  ~ "long"
    )
  ) |>
  arrange(side, desc(needle_length_mm))            # sort for inspection

head(pine_wrangled_df)

# A tibble: 6 × 7
  team       side  needle_length_mm needle_width_mm needle_length_cm log_length
  <chr>      <chr>            <dbl>           <dbl>            <dbl>      <dbl>
1 botany     shady             20.7            0.77             2.07       3.03
2 grace_jace shady             17.9            0.69             1.79       2.89
3 one        shady             17.7            1                1.77       2.87
4 botany     shady             17.6            0.79             1.76       2.87
5 botany     shady             17.6            0.94             1.76       2.87
6 grace_jace shady             17.0            0.84             1.69       2.83
# ℹ 1 more variable: size_class <chr>

Note

✅ Key idea

The pipe |> is what makes this readable top to bottom — a recipe, not a nest of parentheses.

Read it out loud:

Take pine_csv_df,

then filter to real values,

then select four columns,

then add three new columns,

then sort by side and length.

Part 3 · A Duplicate-Row Problem

A Duplicate-Row Problem

Remember pseudoreplication from Getting Started — one tree standing in for many? Duplicate rows are the data-entry version of the same mistake: the same measurement counted twice inflates your sample size without adding real information.

needle_check_df <- read_csv("data/pine_needles_error.csv")

nrow(needle_check_df)

[1] 69

n_distinct(needle_check_df)

[1] 64

Important

Something’s wrong: nrow() and n_distinct() don’t match. Some rows are exact repeats.

Note

📖 New words

n_distinct() — counts how many unique rows exist
distinct() — returns the data with duplicate rows removed — use it deliberately, not reflexively
unique() — lists the distinct values in a single column; the fastest way to spot misspellings

Finding the duplicates

# Which rows are exact duplicates of an earlier row?
needle_check_df |> filter(duplicated(needle_check_df))

# A tibble: 5 × 6
  `Source File`    `Team Name` `Replicate #` `Side of Tree` `Needle Length (mm)`
  <chr>            <chr>               <dbl> <chr>          <chr>               
1 heal_thome_pine… botany                  1 shady          16.46               
2 Bartell_Shingle… grace_jace              2 sunny          12.48               
3 Needle_Length_S… waterbugs               5 sunny          11.7                
4 Needle_Length_S… water bugs              2 Shady          7.23                
5 Kehr_and_Baalke… one                     1 sunny          16.18               
# ℹ 1 more variable: `Needle Width (mm)` <chr>

Tip

🖐 Field habit

Before you trust any dataset — yours or a collaborator’s — check nrow() against n_distinct(). It takes five seconds and catches a real, common mistake.

Removing duplicates with `distinct()`

# Keep only one copy of each unique row
needle_clean_df <- needle_check_df |> distinct()

nrow(needle_check_df)

[1] 69

nrow(needle_clean_df)

[1] 64

Warning

⚠️ Watch out!

distinct() only catches exact duplicate rows. A typo’d duplicate (waterbugs vs Waterbugs) slips right through — always eyeball your data too.

Part 3b · The Rest of the Mess

`unique()` — what else is wrong in here?

Duplicates were only the first problem in this file. Look at the columns themselves:

names(needle_check_df)

[1] "Source File"        "Team Name"          "Replicate #"       
[4] "Side of Tree"       "Needle Length (mm)" "Needle Width (mm)"

unique(needle_clean_df$`Side of Tree`)

[1] "sunny" "shady" "Shady" "SUNNY" "Sunny" "SHADY"

Important

Two values became six. shady, Shady, SHADY — R treats every one as a separate category. Any group_by() or count() you run right now is wrong.

Note

📖 New words

clean_names() (janitor) — turns Needle Length (mm) into needle_length_mm
str_to_lower() — force text to lowercase
str_squish() — strip leading/trailing spaces

📖 R4DS §14 — Strings

Tip

🖐 It was almost worse

This file also has stray spaces ("sunny ", " shady"). You don’t see them because read_csv() trims whitespace for you by default. Prove it to yourself:

read_csv("data/pine_needles_error.csv", trim_ws = FALSE)

Now there are ten values, not six. Not every import tool is that kind — which is why we still run str_squish() below.

Fixing names and text in one pipeline

needle_fixed_df <- needle_check_df |>
  clean_names() |>                 # Needle Length (mm) -> needle_length_mm
  distinct() |>                    # drop the repeated rows
  mutate(
    side = str_squish(str_to_lower(side_of_tree)),
    team = str_squish(str_to_lower(team_name))
  )

unique(needle_fixed_df$side)

[1] "sunny" "shady"

unique(needle_fixed_df$team)

[1] "grace_jace" "botany"     "water bugs" "waterbugs"  "one"

Warning

⚠️ Watch out!

str_to_lower() collapsed Waterbugs into waterbugs — but water bugs is still its own value. Lowercasing fixes case, not spelling. That last one needs case_when().

Tip

🖐 Field habit

Run unique() on every text column the first time you open a dataset. It takes seconds and it is how you find the problems that silently split your groups.

Numbers that aren’t numbers

class(needle_fixed_df$needle_length_mm)

[1] "character"

class(needle_fixed_df$needle_width_mm)

[1] "character"

Both imported as text, not numbers — somebody typed 17.73 mm in a cell, somebody used a comma decimal (13,98), and the missing values were written four different ways.

needle_final_df <- needle_fixed_df |>
  mutate(
    team = case_when(team == "water bugs" ~ "waterbugs",
                     TRUE                 ~ team),
    needle_length_mm = needle_length_mm |>
      str_remove(" mm") |>                    # drop the typed unit
      str_replace(",", ".") |>                # comma decimal -> point
      na_if("") |> na_if("NA") |>             # blank and literal "NA"
      as.numeric(),
    needle_width_mm = needle_width_mm |>
      na_if("") |> na_if("na") |> na_if(".") |>   # three more ways to write "missing"
      as.numeric(),
    needle_length_mm = if_else(needle_length_mm == -999,
                               NA_real_, needle_length_mm)   # sentinel value
  )

class(needle_final_df$needle_length_mm)

[1] "numeric"

sum(is.na(needle_final_df$needle_length_mm))

[1] 3

sum(is.na(needle_final_df$needle_width_mm))

[1] 5

Important

⚠️ The trap: parse_number("13,98") returns 1398 — it reads the comma as a thousands separator and never warns you.

A 1398 mm pine needle is obvious if you look. Always summarize or plot a column right after you clean it.

Note

📖 New words

na_if(x, "") — turn a specific value into NA
-999 — a “sentinel” code some field sheets use for missing. R has no idea it means missing until you tell it.

📖 R4DS §13 — Numbers
📖 R4DS §18 — Missing values

Part 4 · Saving Your Work

The one-way street

Everything we did today runs in one direction:

data/     ← raw, read-only, never written to
   │  read_csv()
   ▼
scripts/  ← the cleaning code you just wrote
   │  write_csv()
   ▼
output/   ← everything R produces

The raw file is evidence. The script is the method. The output is a result you can always rebuild by re-running the method on the evidence.

Important

Never write into data/. If a script can overwrite the raw file, then one bad run destroys the only copy of what you actually measured — and no one can check your work.

Note

✅ Key idea

If you can’t reproduce a result from raw data + script, the problem is in the script — never in the raw file. Keeping data/ read-only is what makes that guarantee possible.

Tidy up before you save

needle_final_df still carries the wreckage of cleaning — the original messy columns and our fixed ones:

names(needle_final_df)

[1] "source_file"      "team_name"        "replicate_number" "side_of_tree"    
[5] "needle_length_mm" "needle_width_mm"  "side"             "team"

Nobody downstream needs team_name and team. Keep the columns you mean to keep, in the order you want them:

needle_out_df <- needle_final_df |>
  select(team, side, replicate = replicate_number,
         needle_length_mm, needle_width_mm) |>
  arrange(team, side, replicate)

glimpse(needle_out_df)

Rows: 64
Columns: 5
$ team             <chr> "botany", "botany", "botany", "botany", "botany", "bo…
$ side             <chr> "shady", "shady", "shady", "shady", "shady", "shady",…
$ replicate        <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2,…
$ needle_length_mm <dbl> 16.46, 20.71, 17.59, 15.11, 17.62, 15.34, 14.79, 11.5…
$ needle_width_mm  <dbl> 0.65, 0.77, 0.94, 0.72, 0.79, 0.90, 1.03, 0.66, 0.98,…

Tip

🖐 Field habit

select() right before write_csv() is the last checkpoint. A saved file with side and side_of_tree in it will confuse someone in three months — and that someone is usually you.

Note

select(new_name = old_name) renames while it selects — handy for dropping janitor’s replicate_number back to replicate.

`write_csv()` — the mirror image of `read_csv()`

write_csv(needle_out_df, "output/pine_needles_cleaned.csv")

Read from data/, write to output/. Same shape of call, opposite direction:

Reading	Writing
`read_csv("data/…")`	`write_csv(df, "output/…")`
file → data frame	data frame → file

Warning

⚠️ Watch out!

write_csv() overwrites without asking — no warning, no undo. That is exactly why the target is output/ and never data/: re-running your script should be free, and it is only free if the thing it clobbers is something the script can rebuild.

Note

📖 Note the argument order

write_csv(df, path) — data frame first, then where to put it. read_csv(path) takes only the path. Getting these backwards is one of the most common errors of the term.

📖 R4DS §7.5 — Writing to a file

Naming your saved files

Name the file for what happened to it: pine_needles_cleaned.csv, not pine2.csv or final.csv
If a script produces several stages, number them: 01_cleaned.csv, 02_summarised.csv
Anyone re-running your script from scratch should get a byte-identical output file

Warning

⚠️ Watch out!

final.csv, final_v2.csv, final_FINAL.csv — version control by filename. This is exactly the mess a project folder with clear names and a re-runnable script is meant to prevent.

Tip

🖐 Try it yourself

Delete your output/ folder entirely, then re-run your script top to bottom.

Everything should come back. If it doesn’t, something important only exists in your console history — and that is a bug in your script.

Wrap-up

Today you:

Imported the same data from a .csv and an .xlsx
Used filter(), select(), mutate(), and arrange() — and chained them into a pipeline
Filtered with the full boolean toolkit: ==, !=, >/<, %in%, is.na(), AND (,) vs. OR (|)
Recoded and binned columns with case_when() — always ending in a TRUE ~ catch-all
Converted units and log-transformed a column with mutate()
Caught and removed duplicate rows with distinct()
Cleaned messy headers with clean_names(), messy text with str_to_lower(), and numbers-stored-as-text with as.numeric()
Trimmed to the columns worth keeping and wrote the result to output/ — leaving data/ untouched

Tip

🖐 Before next class

Finish the worksheet: take pine_needles_error.csv from data/, clean it end to end, select() the columns worth keeping, and write the result to output/. Then delete output/, re-run your script, and check that it all comes back.

Note

Up next — Describing Your Data

Mean, median, variance, SD, SE
The length() trap and the sum(!is.na()) fix
group_by() + summarize() for tidy group comparisons

Readings

Before next class — chapter PDFs are posted on Canvas:

Gotelli & Ellison, A Primer of Ecological Statistics — Ch. 8, Managing and Curating Data (raw vs. processed data, the audit trail from field sheet to analysis)
Whitlock & Schluter, The Analysis of Biological Data — Ch. 1, Statistics and Samples, §1.4 on types of variables — the categorical/numerical distinction behind case_when() and every group_by() to come

Note

✅ Key idea

Neither statistics text teaches dplyr — that is what R4DS is for. Read the texts for why a clean, documented dataset matters, and R4DS for how.

Note

📖 R for Data Science (2e)

Free online at r4ds.hadley.nz:

§3 — Data transformation — the five verbs
§5 — Data tidying
§7 — Data import
§12 — Logical vectors — case_when()
§13 — Numbers
§14 — Strings
§18 — Missing values
§20 — Spreadsheets

Getting unstuck

When code breaks — and it will, that is normal:

Read the error message out loud; it usually names the line
Check the usual suspects: library(tidyverse) and library(readxl) loaded? Spelling? A missing ) or |> at the start of a line?
?function_name opens the help page
Bring the exact error (copy-paste it) to class or office hours

📖 R4DS §8 — Workflow: getting help

Note

✅ Key idea

Every working scientist googles error messages daily. Getting stuck is not failing — it is the job.

Other Formats

Where We Left Off

Part 1 · Importing Data More Than One Way

The same data, two file formats

Look before you trust it — every time

Part 2 · The Five Wrangling Verbs

The Five Wrangling Verbs

filter() — Picking Rows

filter() — Combining Conditions

filter() — OR, With a Real Example

filter() — Matching One of Several Values with %in%

filter() — Missing Values with is.na()

select() — Picking Columns

select() — Picking Columns by Pattern

mutate() — Creating and Changing Columns

mutate() — More Examples

Part 2b · case_when() — Recoding and Binning

case_when() — Turning a Number into a Category

case_when() — Always End With TRUE ~

case_when() — Recoding Categories, Not Just Numbers

arrange() — Sorting Rows

arrange() — Sorting by Multiple Columns

The Full Pipeline

Part 3 · A Duplicate-Row Problem

A Duplicate-Row Problem

Finding the duplicates

Removing duplicates with distinct()

Part 3b · The Rest of the Mess

unique() — what else is wrong in here?

Fixing names and text in one pipeline

Numbers that aren’t numbers

Part 4 · Saving Your Work

The one-way street

Tidy up before you save

write_csv() — the mirror image of read_csv()

Naming your saved files

Wrap-up

Readings

Getting unstuck

`filter()` — Picking Rows

`filter()` — Combining Conditions

`filter()` — OR, With a Real Example

`filter()` — Matching One of Several Values with `%in%`

`filter()` — Missing Values with `is.na()`

`select()` — Picking Columns

`select()` — Picking Columns by Pattern

`mutate()` — Creating and Changing Columns

`mutate()` — More Examples

Part 2b · `case_when()` — Recoding and Binning

`case_when()` — Turning a Number into a Category

`case_when()` — Always End With `TRUE ~`

`case_when()` — Recoding Categories, Not Just Numbers

`arrange()` — Sorting Rows

`arrange()` — Sorting by Multiple Columns

Removing duplicates with `distinct()`

`unique()` — what else is wrong in here?

`write_csv()` — the mirror image of `read_csv()`