Lecture 05 — Wrangling Data

filter, select, mutate, arrange — and chaining them into a pipeline

Bill Perry

2026-09-10

Where we left off (Lecture 04 — GGPlot II)

Setup — R and Positron installed; project folders created (data/, figures/, scripts/)
Loading data — read_excel() and read_csv(); a first look with glimpse()
Pipe — %>% reads as “then”; chains steps together
GGPlot I & II — geoms, facets, stat_summary(), scale_color_manual(), coordinates, and themes

Note

✅ Key idea from Lecture 04

You can already load data and plot it beautifully. Today we learn to reshape and clean the data itself, before it ever reaches ggplot().

Goals for Today

The core tidyverse verbs:

filter() — pick rows by a condition
select() — pick columns by name
mutate() — create new columns
arrange() — sort rows
Chain them all into a pipeline

Tip

🖐 Try it yourself

By the end you will wrangle our leaf data into exactly the columns and rows you need, in one connected pipeline.

Our tools today:

readxl
tidyverse

References:

📖 R4DS Ch 3 — Data Transformation
📖 R4DS Ch 5 — Data Tidying
🌐 Data Carpentry

Naming conventions:

data frames → _df
plots → _plot
models → _model

How to Use These Slides — Predict · Type · Run

This is a shorter lecture than usual — one focused idea, wrangling, done properly. You will switch to the activity once, at the end, and type the code yourself into your R script there.

For every code block, do three things:

Predict — before it runs, say what you think the output will be
Type it out by hand — do not copy-paste
Run it and compare to your prediction

Note

✅ Why bother?

filter(), select(), mutate(), and arrange() look interchangeable on a slide — guessing the output first is what actually forces you to tell them apart.
You will type %>% dozens of times today; typing it now is what makes it automatic later, when you’re chaining five verbs under a deadline instead of reading one line calmly.

Load Libraries and Data

# Load all packages at the top of every script ----------
library(readxl) # reading Excel files
library(tidyverse) # data wrangling + ggplot2
library(janitor) # clean_names()

# Read the leaf data from the data folder ---------------
leaf_df <- read_excel("data/2026_09_03_data_sci_leaf_area.xlsx") %>%
  clean_names()

Take a First Look

# glimpse(): one row per column — name, type, first values
glimpse(leaf_df)

Rows: 53
Columns: 8
$ twig_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "twig_1",…
$ leaf_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "l07", "l…
$ teams        <chr> "12345", "12345", "12345", "12345", "12345", "12345", "12…
$ shade        <chr> "sunny", "sunny", "sunny", "sunny", "sunny", "sunny", "sh…
$ mass_g       <dbl> 0.4000, 0.4800, 0.3400, 0.6500, 0.2700, 0.4300, 0.3900, 0…
$ petiole_mm   <dbl> 79.0000, 63.0000, 68.0000, 35.0000, 34.0000, 40.0000, 40.…
$ thickness_mm <dbl> 0.15, 0.14, 0.14, 0.15, 0.11, 0.16, 0.14, 0.15, 0.12, 0.1…
$ paper_mass_g <dbl> 0.2100, 0.2100, 0.2100, 0.2400, 0.1500, 0.3400, 0.2900, 0…

<chr> = character (shade, teams)
<dbl> = numeric (mass_g, petiole_mm, thickness_mm, paper_mass_g)

The Core Tidyverse Verbs

Five functions do most of the work in data wrangling:

Function	What it does
`filter()`	keep rows matching a condition
`select()`	keep columns by name
`mutate()`	add or change a column
`arrange()`	sort rows
`summarize()`	collapse rows to a summary

All take a data frame as first input (via %>%) and return a data frame. We’ll wrangle with the first four today — summarize() is next lecture’s topic.

📖 R4DS §3 — Data Transformation

Note

✅ Why these five?

Almost every wrangling step you’ll write this semester decomposes into some combination of picking rows, picking columns, adding a column, sorting, or collapsing to a summary — that’s filter, select, mutate, arrange, and summarize.

Think of them as building blocks:

filter()   → rows you want
select()   → columns you want
mutate()   → new columns you need
arrange()  → order the rows
summarize()→ collapse to summaries (next lecture)

`filter()` — Picking Rows

Note

🔮 Predict first: We have 53 leaves. Before you run it, guess how many rows filter(shade == "sunny") returns. Write your number down, then run it and check.

# filter() keeps rows where the condition is TRUE -------

# Keep only sunny leaves
leaf_df %>% filter(shade == "sunny") %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21
4 <NA>    <NA>    12345 sunny   0.65         35         0.15         0.24
5 <NA>    <NA>    12345 sunny   0.27         34         0.11         0.15
6 <NA>    <NA>    12345 sunny   0.43         40         0.16         0.34

# Keep only leaves heavier than 0.5 g
leaf_df %>% filter(mass_g > 0.5) %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345        sunny  0.65      35             0.15        0.24 
2 <NA>    <NA>    12345        shady  0.6       68             0.12        0.23 
3 <NA>    <NA>    12345        shady  0.57      73             0.14        0.28 
4 twig_1  l07     fighting_mo… shady  0.589     66             0.32        0.257
5 twig_1  l08     fighting_mo… shady  0.511     51             0.25        0.249
6 twig_2  l09     fighting_mo… shady  0.920      0.383        NA          NA

# Combine two conditions with , (AND)
leaf_df %>% filter(shade == "shady", mass_g > 0.5) %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345        shady  0.6       68             0.12        0.23 
2 <NA>    <NA>    12345        shady  0.57      73             0.14        0.28 
3 twig_1  l07     fighting_mo… shady  0.589     66             0.32        0.257
4 twig_1  l08     fighting_mo… shady  0.511     51             0.25        0.249
5 twig_2  l09     fighting_mo… shady  0.920      0.383        NA          NA    
6 twig_2  l10     fighting_mo… shady  0.782     67             0.27        0.342

Common comparison operators:

Symbol	Meaning
`==`	exactly equal
`!=`	not equal
`>` `<`	greater / less than
`>=` `<=`	greater / less than or equal
is.na	return all NA rows
!is.na	return all rows that are NOT NA

Warning

⚠️ Watch out!

= assigns a value.
== tests equality.
filter(shade = "sunny") → error.
filter(shade == "sunny") → correct.

Live Demo — Watch It Break (on purpose)

I will type this wrong on purpose:

# One equals sign — a very common mistake
leaf_df %>% filter(shade = "sunny")

R stops and tells us:

Error in `filter()`:
! We detected a named input.
i This usually means that you've used `=` instead of `==`.

Now the fix — two equals signs:

leaf_df %>% filter(shade == "sunny")

Tip

✅ Why show a broken filter()?

filter(shade = "sunny") is the single most common typo in this whole unit, because = and == look almost identical and only one of them tests equality. dplyr’s error message actually names the mistake — “you’ve used = instead of ==” — so once you’ve seen it fixed here, the next time R prints it at you it’s a fix, not a mystery.

`select()` — Picking Columns

# select() keeps the columns you name ------------------

# Keep only shade and mass
leaf_df %>% select(shade, mass_g) %>% head()

# A tibble: 6 × 2
  shade mass_g
  <chr>  <dbl>
1 sunny   0.4 
2 sunny   0.48
3 sunny   0.34
4 sunny   0.65
5 sunny   0.27
6 sunny   0.43

# Drop the paper_mass_g column (use minus sign)
leaf_df %>% select(-paper_mass_g) %>% head()

# A tibble: 6 × 7
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16

# Keep columns that start with a word
leaf_df %>% select(starts_with("mass")) %>% head()

# A tibble: 6 × 1
  mass_g
   <dbl>
1   0.4 
2   0.48
3   0.34
4   0.65
5   0.27
6   0.43

Why select() matters:
- Real datasets often have 50–100+ columns
- You rarely need all of them
- select() keeps your workspace clean
Useful helpers:
- starts_with("x") — columns starting with “x”
- ends_with("_mm") — columns ending with “_mm”
- contains("mass") — columns containing “mass”

📖 R4DS §3.3

`mutate()` — Creating New Columns

Note

🔮 Predict first: mass_g * 1000 converts grams to milligrams. Before running: will mutate() replace mass_g or add a new column? How many columns will the result have?

# mutate() adds a new column to the data frame ---------

# Convert grams to milligrams
leaf_df %>%
  mutate(mass_mg = mass_g * 1000) %>%
  head()

# A tibble: 6 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21
4 <NA>    <NA>    12345 sunny   0.65         35         0.15         0.24
5 <NA>    <NA>    12345 sunny   0.27         34         0.11         0.15
6 <NA>    <NA>    12345 sunny   0.43         40         0.16         0.34
# ℹ 1 more variable: mass_mg <dbl>

# Add a size category based on mass
leaf_df %>%
  mutate(size_class = if_else(mass_g > 0.5, "large", "small")) %>%
  head()

# A tibble: 6 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21
4 <NA>    <NA>    12345 sunny   0.65         35         0.15         0.24
5 <NA>    <NA>    12345 sunny   0.27         34         0.11         0.15
6 <NA>    <NA>    12345 sunny   0.43         40         0.16         0.34
# ℹ 1 more variable: size_class <chr>

# mutate() can create several new columns at once
leaf_df %>%
  mutate(
    mass_mg    = mass_g * 1000,
    petiole_cm = petiole_mm / 10 # petiole length in cm
  ) %>%
  head()

# A tibble: 6 × 10
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21
4 <NA>    <NA>    12345 sunny   0.65         35         0.15         0.24
5 <NA>    <NA>    12345 sunny   0.27         34         0.11         0.15
6 <NA>    <NA>    12345 sunny   0.43         40         0.16         0.34
# ℹ 2 more variables: mass_mg <dbl>, petiole_cm <dbl>

mutate() keeps all existing columns and adds new ones
Use if_else(condition, value_if_true, value_if_false) to create categories

Note

✅ Watch the column count

mutate() never removes columns — it adds to the right of your data frame, so leaf_df gains one column per new variable you create.

To overwrite a column instead of adding one, reuse the same name on the left: mutate(mass_g = round(mass_g, 2))

📖 R4DS §3.3

`arrange()` — Sorting Rows

# arrange() sorts rows by one or more columns ----------

# Lightest leaves first
leaf_df %>% arrange(mass_g)

# A tibble: 53 × 8
   twig_id leaf_id teams       shade mass_g petiole_mm thickness_mm paper_mass_g
   <chr>   <chr>   <chr>       <chr>  <dbl>      <dbl>        <dbl>        <dbl>
 1 twig_6  l06     fighting_m… sunny  0.193       29           0.13        0.113
 2 <NA>    4       Oscar_Karl  sunny  0.230       35.7         0.36       NA    
 3 <NA>    <NA>    12345       sunny  0.27        34           0.11        0.15 
 4 <NA>    6       Oscar_Karl  sunny  0.282       46.9         0.4        NA    
 5 <NA>    5       Oscar_Karl  sunny  0.304       51.3         0.43       NA    
 6 <NA>    <NA>    12345       sunny  0.34        68           0.14        0.21 
 7 <NA>    <NA>    12345       shady  0.34        58           0.15        0.23 
 8 <NA>    <NA>    12345       shady  0.34        45           0.13        0.27 
 9 <NA>    <NA>    leaf_oglers shady  0.344       48.3         0.67        0.196
10 <NA>    5       Oscar_Karl  shady  0.344       37.0         0.25       NA    
# ℹ 43 more rows

# Heaviest leaves first (use desc() to reverse)
leaf_df %>% arrange(desc(mass_g))

# A tibble: 53 × 8
   twig_id leaf_id teams       shade mass_g petiole_mm thickness_mm paper_mass_g
   <chr>   <chr>   <chr>       <chr>  <dbl>      <dbl>        <dbl>        <dbl>
 1 <NA>    <NA>    still_load… sunny  1.08      84             0.68        0.420
 2 twig_2  l09     fighting_m… shady  0.920      0.383        NA          NA    
 3 <NA>    <NA>    leaf_oglers shady  0.8       41.2           0.5         0.3  
 4 <NA>    <NA>    leaf_oglers sunny  0.791     53.9           0.23        0.332
 5 twig_2  l10     fighting_m… shady  0.782     67             0.27        0.342
 6 twig_5  l04     fighting_m… sunny  0.76      79             0.16        0.312
 7 twig_4  l01     fighting_m… sunny  0.737     75             0.24        0.303
 8 <NA>    <NA>    leaf_oglers sunny  0.688     45.3           0.19        0.372
 9 <NA>    <NA>    leaf_oglers shady  0.68      61.2           0.47        0.32 
10 <NA>    <NA>    still_load… sunny  0.665     64             0.69        0.427
# ℹ 43 more rows

# Sort by shade, then by mass within each shade
leaf_df %>% arrange(shade, desc(mass_g))

# A tibble: 53 × 8
   twig_id leaf_id teams       shade mass_g petiole_mm thickness_mm paper_mass_g
   <chr>   <chr>   <chr>       <chr>  <dbl>      <dbl>        <dbl>        <dbl>
 1 twig_2  l09     fighting_m… shady  0.920      0.383        NA          NA    
 2 <NA>    <NA>    leaf_oglers shady  0.8       41.2           0.5         0.3  
 3 twig_2  l10     fighting_m… shady  0.782     67             0.27        0.342
 4 <NA>    <NA>    leaf_oglers shady  0.68      61.2           0.47        0.32 
 5 <NA>    <NA>    leaf_oglers shady  0.61      59.4           0.23        0.25 
 6 <NA>    <NA>    still_load… shady  0.607     82             0.32        0.301
 7 <NA>    <NA>    12345       shady  0.6       68             0.12        0.23 
 8 twig_1  l07     fighting_m… shady  0.589     66             0.32        0.257
 9 twig_3  l11     fighting_m… shady  0.585     68             0.41        0.26 
10 <NA>    <NA>    12345       shady  0.57      73             0.14        0.28 
# ℹ 43 more rows

Default: ascending order (smallest first)
desc(column) → descending order (largest first)
Very useful before printing a table or checking outliers

Tip

arrange() is rarely used in the middle of an analysis pipeline — it is most useful at the end when you want to inspect your results in a specific order.

The Full Pipeline

# Chain all the verbs together -------------------------
# Read it top to bottom like a recipe

wrangled_df <- leaf_df %>%
  filter(mass_g > 0) %>% # remove any zeros
  select(shade, mass_g, thickness_mm) %>% # keep only needed cols
  mutate(
    mass_mg = mass_g * 1000,
    size_class = if_else(mass_g > 0.5, "large", "small")
  ) %>%
  arrange(shade, desc(mass_g)) # sort by shade, then mass

head(wrangled_df)

# A tibble: 6 × 5
  shade mass_g thickness_mm mass_mg size_class
  <chr>  <dbl>        <dbl>   <dbl> <chr>     
1 shady  0.920        NA       920. large     
2 shady  0.8           0.5     800  large     
3 shady  0.782         0.27    782. large     
4 shady  0.68          0.47    680  large     
5 shady  0.61          0.23    610  large     
6 shady  0.607         0.32    607. large

Read the pipeline out loud:

Take leaf_df,

then filter to non-zero masses,

then select three columns,

then add two new columns,

then sort by shade and mass.

Note

✅ Compare it to nested calls

Write the same pipeline as nested functions — arrange(mutate(select(filter(leaf_df, mass_g > 0), shade, mass_g, thickness_mm), mass_mg = mass_g * 1000, size_class = if_else(mass_g > 0.5, "large", "small")), shade, desc(mass_g)) — and the first thing that runs is buried in the middle. The pipe lets you read top to bottom in the order R actually executes.

→ ACTIVITY 5 starts now

🛑 Go to Activity 5 — Wrangling

Close the slides. Open the Activity 5 and work through filter, select, mutate, arrange, and the full pipeline. Type every line yourself into your script; predict before you run.

What We Learned Today

filter() — keep rows by condition
select() — keep columns by name
mutate() — add new calculated columns
arrange() — sort rows
Full pipeline: chain all verbs with %>%

References:

📖 R4DS §3 — Data Transformation
📖 R4DS §5 — Data Tidying
🌐 Data Carpentry

Up next — Lecture 06, Summary Statistics:

Mean, median, variance, SD, SE
Counting correctly with sum(!is.na())
group_by() + summarize(), and skimr

Lecture 05 — Wrangling Data

Where we left off (Lecture 04 — GGPlot II)

Goals for Today

How to Use These Slides — Predict · Type · Run

Load Libraries and Data

Take a First Look

The Core Tidyverse Verbs

filter() — Picking Rows

Live Demo — Watch It Break (on purpose)

select() — Picking Columns

mutate() — Creating New Columns

arrange() — Sorting Rows

The Full Pipeline

→ ACTIVITY 5 starts now

What We Learned Today

`filter()` — Picking Rows

`select()` — Picking Columns

`mutate()` — Creating New Columns

`arrange()` — Sorting Rows