Lecture 05 — Wrangling Data

filter, select, mutate, arrange — and chaining them into a pipeline

Bill Perry

2026-10-01

Where we left off (Lecture 04 — GGPlot II)

Setup — R and Positron installed; project folders created (data/, figures/, scripts/)
Loading data — read_excel() and read_csv(); a first look with glimpse()
Pipe — %>% reads as “then”; chains steps together
GGPlot I & II — geoms, facets, stat_summary(), scale_color_manual(), coordinates, and themes

Note

✅ Key idea from Lecture 04

You can already load data and plot it beautifully. Today we learn to reshape and clean the data itself, before it ever reaches ggplot().

Goals for Today

The core tidyverse verbs:

filter() — pick rows by a condition
select() — pick columns by name
mutate() — create new columns
arrange() — sort rows
Chain them all into a pipeline

Tip

🖐 Try it yourself

By the end you will wrangle our leaf data into exactly the columns and rows you need, in one connected pipeline.

Our tools today:

readxl
tidyverse

References:

📖 R4DS Ch 3 — Data Transformation
📖 R4DS Ch 5 — Data Tidying
🌐 Data Carpentry

Naming conventions:

data frames → _df
plots → _plot
models → _model

How to Use These Slides — Predict · Type · Run

This is a shorter lecture than usual — one focused idea, wrangling, done properly. You will switch to the activity once, at the end, and type the code yourself into your R script there.

For every code block, do three things:

Predict — before it runs, say what you think the output will be
Type it out by hand — do not copy-paste
Run it and compare to your prediction

Note

✅ Why bother?

filter(), select(), mutate(), and arrange() look interchangeable on a slide — guessing the output first is what actually forces you to tell them apart.
You will type %>% dozens of times today; typing it now is what makes it automatic later, when you’re chaining five verbs under a deadline instead of reading one line calmly.

Load Libraries and Data

# Load all packages at the top of every script ----------
library(readxl) # reading Excel files
library(tidyverse) # data wrangling + ggplot2
library(janitor) # clean_names()

# Read the leaf data from the data folder ---------------
leaf_df <-
  read_excel("data/2026_09_03_data_sci_leaf_area.xlsx") %>%
  clean_names()

Take a First Look

# glimpse(): one row per column — name, type, first values
glimpse(leaf_df)

Rows: 53
Columns: 8
$ twig_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ leaf_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ teams        <chr> "12345", "12345", "12345", "12345", "12345…
$ shade        <chr> "sunny", "sunny", "sunny", "sunny", "sunny…
$ mass_g       <dbl> 0.4000, 0.4800, 0.3400, 0.6500, 0.2700, 0.…
$ petiole_mm   <dbl> 79.0000, 63.0000, 68.0000, 35.0000, 34.000…
$ thickness_mm <dbl> 0.15, 0.14, 0.14, 0.15, 0.11, 0.16, 0.14, …
$ paper_mass_g <dbl> 0.2100, 0.2100, 0.2100, 0.2400, 0.1500, 0.…

<chr> = character (shade, teams)
<dbl> = numeric (mass_g, petiole_mm, thickness_mm, paper_mass_g)

The Core Tidyverse Verbs

Five functions do most of the work in data wrangling:

Function	What it does
`filter()`	keep rows matching a condition
`select()`	keep columns by name
`mutate()`	add or change a column
`arrange()`	sort rows
`summarize()`	collapse rows to a summary

All take a data frame as first input (via %>%) and return a data frame. We’ll wrangle with the first four today — summarize() is next lecture’s topic.

📖 R4DS §3 — Data Transformation

Note

✅ Why these five?

Almost every wrangling step you’ll write this semester decomposes into some combination of picking rows, picking columns, adding a column, sorting, or collapsing to a summary — that’s filter, select, mutate, arrange, and summarize.

Think of them as building blocks:

filter()   → rows you want
select()   → columns you want
mutate()   → new columns you need
arrange()  → order the rows
summarize()→ collapse to summaries (next lecture)

`filter()` — Picking Rows

Note

🔮 Predict first: We have 53 leaves. Before you run it, guess how many rows filter(shade == "sunny") returns. Write your number down, then run it and check.

# filter() keeps rows where the condition is TRUE -------

# Keep only sunny leaves
leaf_df %>% filter(shade == "sunny") %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16
# ℹ 1 more variable: paper_mass_g <dbl>

# Keep only leaves heavier than 0.5 g
leaf_df %>% filter(mass_g > 0.5) %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345      sunny  0.65      35             0.15
2 <NA>    <NA>    12345      shady  0.6       68             0.12
3 <NA>    <NA>    12345      shady  0.57      73             0.14
4 twig_1  l07     fighting_… shady  0.589     66             0.32
5 twig_1  l08     fighting_… shady  0.511     51             0.25
6 twig_2  l09     fighting_… shady  0.920      0.383        NA   
# ℹ 1 more variable: paper_mass_g <dbl>

# Combine two conditions with , (AND)
leaf_df %>% filter(shade == "shady", mass_g > 0.5) %>% head()

# A tibble: 6 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345      shady  0.6       68             0.12
2 <NA>    <NA>    12345      shady  0.57      73             0.14
3 twig_1  l07     fighting_… shady  0.589     66             0.32
4 twig_1  l08     fighting_… shady  0.511     51             0.25
5 twig_2  l09     fighting_… shady  0.920      0.383        NA   
6 twig_2  l10     fighting_… shady  0.782     67             0.27
# ℹ 1 more variable: paper_mass_g <dbl>

Common comparison operators:

Symbol	Meaning
`==`	exactly equal
`!=`	not equal
`>` `<`	greater / less than
`>=` `<=`	greater / less than or equal
is.na	return all NA rows
!is.na	return all rows that are NOT NA

Warning

⚠️ Watch out!

= assigns a value.
== tests equality.
filter(shade = "sunny") → error.
filter(shade == "sunny") → correct.

Live Demo — Watch It Break (on purpose)

I will type this wrong on purpose:

# One equals sign — a very common mistake
leaf_df %>% filter(shade = "sunny")

R stops and tells us:

Error in `filter()`:
! We detected a named input.
i This usually means that you've used `=` instead of `==`.

Now the fix — two equals signs:

leaf_df %>% filter(shade == "sunny")

Tip

✅ Why show a broken filter()?

filter(shade = "sunny") is the single most common typo in this whole unit, because = and == look almost identical and only one of them tests equality. dplyr’s error message actually names the mistake — “you’ve used = instead of ==” — so once you’ve seen it fixed here, the next time R prints it at you it’s a fix, not a mystery.

`select()` — Picking Columns

# select() keeps the columns you name ------------------

# Keep only shade and mass
leaf_df %>% select(shade, mass_g) %>% head()

# A tibble: 6 × 2
  shade mass_g
  <chr>  <dbl>
1 sunny   0.4 
2 sunny   0.48
3 sunny   0.34
4 sunny   0.65
5 sunny   0.27
6 sunny   0.43

# Drop the paper_mass_g column (use minus sign)
leaf_df %>% select(-paper_mass_g) %>% head()

# A tibble: 6 × 7
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16

# Keep columns that start with a word
leaf_df %>% select(starts_with("mass")) %>% head()

# A tibble: 6 × 1
  mass_g
   <dbl>
1   0.4 
2   0.48
3   0.34
4   0.65
5   0.27
6   0.43

Why select() matters:
- Real datasets often have 50–100+ columns
- You rarely need all of them
- select() keeps your workspace clean
Useful helpers:
- starts_with("x") — columns starting with “x”
- ends_with("_mm") — columns ending with “_mm”
- contains("mass") — columns containing “mass”

📖 R4DS §3.3

`mutate()` — Creating New Columns

Note

🔮 Predict first: mass_g * 1000 converts grams to milligrams. Before running: will mutate() replace mass_g or add a new column? How many columns will the result have?

# mutate() adds a new column to the data frame ---------

# Convert grams to milligrams
leaf_df %>%
  mutate(mass_mg = mass_g * 1000) %>%
  head()

# A tibble: 6 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16
# ℹ 2 more variables: paper_mass_g <dbl>, mass_mg <dbl>

# Add a size category based on mass
leaf_df %>%
  mutate(size_class = if_else(mass_g > 0.5, "large",
                              "small")) %>%
  head()

# A tibble: 6 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16
# ℹ 2 more variables: paper_mass_g <dbl>, size_class <chr>

# mutate() can create several new columns at once
leaf_df %>%
  mutate(
    mass_mg    = mass_g * 1000,
    petiole_cm = petiole_mm / 10 # petiole length in cm
  ) %>%
  head()

# A tibble: 6 × 10
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
4 <NA>    <NA>    12345 sunny   0.65         35         0.15
5 <NA>    <NA>    12345 sunny   0.27         34         0.11
6 <NA>    <NA>    12345 sunny   0.43         40         0.16
# ℹ 3 more variables: paper_mass_g <dbl>, mass_mg <dbl>,
#   petiole_cm <dbl>

mutate() keeps all existing columns and adds new ones
Use if_else(condition, value_if_true, value_if_false) to create categories

Note

✅ Watch the column count

mutate() never removes columns — it adds to the right of your data frame, so leaf_df gains one column per new variable you create.

To overwrite a column instead of adding one, reuse the same name on the left: mutate(mass_g = round(mass_g, 2))

📖 R4DS §3.3

`arrange()` — Sorting Rows

# arrange() sorts rows by one or more columns ----------

# Lightest leaves first
leaf_df %>% arrange(mass_g)

# A tibble: 53 × 8
   twig_id leaf_id teams     shade mass_g petiole_mm thickness_mm
   <chr>   <chr>   <chr>     <chr>  <dbl>      <dbl>        <dbl>
 1 twig_6  l06     fighting… sunny  0.193       29           0.13
 2 <NA>    4       Oscar_Ka… sunny  0.230       35.7         0.36
 3 <NA>    <NA>    12345     sunny  0.27        34           0.11
 4 <NA>    6       Oscar_Ka… sunny  0.282       46.9         0.4 
 5 <NA>    5       Oscar_Ka… sunny  0.304       51.3         0.43
 6 <NA>    <NA>    12345     sunny  0.34        68           0.14
 7 <NA>    <NA>    12345     shady  0.34        58           0.15
 8 <NA>    <NA>    12345     shady  0.34        45           0.13
 9 <NA>    <NA>    leaf_ogl… shady  0.344       48.3         0.67
10 <NA>    5       Oscar_Ka… shady  0.344       37.0         0.25
# ℹ 43 more rows
# ℹ 1 more variable: paper_mass_g <dbl>

# Heaviest leaves first (use desc() to reverse)
leaf_df %>% arrange(desc(mass_g))

# A tibble: 53 × 8
   twig_id leaf_id teams     shade mass_g petiole_mm thickness_mm
   <chr>   <chr>   <chr>     <chr>  <dbl>      <dbl>        <dbl>
 1 <NA>    <NA>    still_lo… sunny  1.08      84             0.68
 2 twig_2  l09     fighting… shady  0.920      0.383        NA   
 3 <NA>    <NA>    leaf_ogl… shady  0.8       41.2           0.5 
 4 <NA>    <NA>    leaf_ogl… sunny  0.791     53.9           0.23
 5 twig_2  l10     fighting… shady  0.782     67             0.27
 6 twig_5  l04     fighting… sunny  0.76      79             0.16
 7 twig_4  l01     fighting… sunny  0.737     75             0.24
 8 <NA>    <NA>    leaf_ogl… sunny  0.688     45.3           0.19
 9 <NA>    <NA>    leaf_ogl… shady  0.68      61.2           0.47
10 <NA>    <NA>    still_lo… sunny  0.665     64             0.69
# ℹ 43 more rows
# ℹ 1 more variable: paper_mass_g <dbl>

# Sort by shade, then by mass within each shade
leaf_df %>% arrange(shade, desc(mass_g))

# A tibble: 53 × 8
   twig_id leaf_id teams     shade mass_g petiole_mm thickness_mm
   <chr>   <chr>   <chr>     <chr>  <dbl>      <dbl>        <dbl>
 1 twig_2  l09     fighting… shady  0.920      0.383        NA   
 2 <NA>    <NA>    leaf_ogl… shady  0.8       41.2           0.5 
 3 twig_2  l10     fighting… shady  0.782     67             0.27
 4 <NA>    <NA>    leaf_ogl… shady  0.68      61.2           0.47
 5 <NA>    <NA>    leaf_ogl… shady  0.61      59.4           0.23
 6 <NA>    <NA>    still_lo… shady  0.607     82             0.32
 7 <NA>    <NA>    12345     shady  0.6       68             0.12
 8 twig_1  l07     fighting… shady  0.589     66             0.32
 9 twig_3  l11     fighting… shady  0.585     68             0.41
10 <NA>    <NA>    12345     shady  0.57      73             0.14
# ℹ 43 more rows
# ℹ 1 more variable: paper_mass_g <dbl>

Default: ascending order (smallest first)
desc(column) → descending order (largest first)
Very useful before printing a table or checking outliers

Tip

arrange() is rarely used in the middle of an analysis pipeline — it is most useful at the end when you want to inspect your results in a specific order.

The Full Pipeline

# Chain all the verbs together -------------------------
# Read it top to bottom like a recipe

wrangled_df <- leaf_df %>%
  filter(mass_g > 0) %>% # remove any zeros
  select(shade, mass_g, thickness_mm) %>% # keep only needed cols
  mutate(
    mass_mg = mass_g * 1000,
    size_class = if_else(mass_g > 0.5, "large", "small")
  ) %>%
  arrange(shade, desc(mass_g)) # sort by shade, then mass

head(wrangled_df)

# A tibble: 6 × 5
  shade mass_g thickness_mm mass_mg size_class
  <chr>  <dbl>        <dbl>   <dbl> <chr>     
1 shady  0.920        NA       920. large     
2 shady  0.8           0.5     800  large     
3 shady  0.782         0.27    782. large     
4 shady  0.68          0.47    680  large     
5 shady  0.61          0.23    610  large     
6 shady  0.607         0.32    607. large

Read the pipeline out loud:

Take leaf_df,

then filter to non-zero masses,

then select three columns,

then add two new columns,

then sort by shade and mass.

Note

✅ Compare it to nested calls

Write the same pipeline as nested functions — arrange(mutate(select(filter(leaf_df, mass_g > 0), shade, mass_g, thickness_mm), mass_mg = mass_g * 1000, size_class = if_else(mass_g > 0.5, "large", "small")), shade, desc(mass_g)) — and the first thing that runs is buried in the middle. The pipe lets you read top to bottom in the order R actually executes.

→ ACTIVITY 5 starts now

🛑 Go to Activity 5 — Wrangling

Close the slides. Open the Activity 5 and work through filter, select, mutate, arrange, and the full pipeline. Type every line yourself into your script; predict before you run.

What We Learned Today

filter() — keep rows by condition
select() — keep columns by name
mutate() — add new calculated columns
arrange() — sort rows
Full pipeline: chain all verbs with %>%

References:

📖 R4DS §3 — Data Transformation
📖 R4DS §5 — Data Tidying
🌐 Data Carpentry

Up next — Lecture 06, Summary Statistics:

Mean, median, variance, SD, SE
Counting correctly with sum(!is.na())
group_by() + summarize(), and skimr

Lecture 05 — Wrangling Data

Where we left off (Lecture 04 — GGPlot II)

Goals for Today

How to Use These Slides — Predict · Type · Run

Load Libraries and Data

Take a First Look

The Core Tidyverse Verbs

filter() — Picking Rows

Live Demo — Watch It Break (on purpose)

select() — Picking Columns

mutate() — Creating New Columns

arrange() — Sorting Rows

The Full Pipeline

→ ACTIVITY 5 starts now

What We Learned Today

`filter()` — Picking Rows

`select()` — Picking Columns

`mutate()` — Creating New Columns

`arrange()` — Sorting Rows