Lecture 05 — Wrangling Data

filter, select, mutate, arrange — and chaining them into a pipeline

Bill Perry

2026-11-10

Where we left off (Lecture 04 — GGPlot II)

Setup — R and Positron installed; project folders created (data/, figures/, scripts/)
Loading data — read_excel() and read_csv(); a first look with glimpse()
Pipe — %>% reads as “then”; chains steps together
GGPlot I & II — geoms, facets, stat_summary(), scale_color_manual(), coordinates, and themes

Note

✅ Key idea from Lecture 04

You can already load data and plot it beautifully. Today we learn to reshape and clean the data itself, before it ever reaches ggplot().

Goals for Today

The core tidyverse verbs:

filter() — pick rows by a condition
select() — pick columns by name
mutate() — create new columns
arrange() — sort rows
Chain them all into a pipeline

Tip

🖐 Try it yourself

By the end you will wrangle our leaf data into exactly the columns and rows you need, in one connected pipeline.

Our tools today:

readxl
tidyverse

References:

📖 R4DS Ch 3 — Data Transformation
📖 R4DS Ch 5 — Data Tidying
🌐 Data Carpentry

Naming conventions:

data frames → _df
plots → _plot
models → _model

How to Use These Slides — Predict · Type · Run

This is a shorter lecture than usual — one focused idea, wrangling, done properly. You will switch to the activity once, at the end, and type the code yourself into your R script there.

For every code block, do three things:

Predict — before it runs, say what you think the output will be
Type it out by hand — do not copy-paste
Run it and compare to your prediction

Note

✅ Why bother?

filter(), select(), mutate(), and arrange() look interchangeable on a slide — guessing the output first is what actually forces you to tell them apart.
You will type %>% dozens of times today; typing it now is what makes it automatic later, when you’re chaining five verbs under a deadline instead of reading one line calmly.

Load Libraries and Data

# Load all packages at the top of every script ----------
library(readxl) # reading Excel files
library(tidyverse) # data wrangling + ggplot2
library(janitor) # clean_names()

# Read the leaf data from the data folder ---------------
leaf_df <-
  read_excel("data/2026_09_03_data_sci_leaf_area.xlsx") %>%
  clean_names()

Take a First Look

# glimpse(): one row per column — name, type, first values
glimpse(leaf_df)

Rows: 53
Columns: 8
$ twig_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ leaf_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ teams        <chr> "12345", "12345", "12345", "12345", "12345…
$ shade        <chr> "sunny", "sunny", "sunny", "sunny", "sunny…
$ mass_g       <dbl> 0.4000, 0.4800, 0.3400, 0.6500, 0.2700, 0.…
$ petiole_mm   <dbl> 79.0000, 63.0000, 68.0000, 35.0000, 34.000…
$ thickness_mm <dbl> 0.15, 0.14, 0.14, 0.15, 0.11, 0.16, 0.14, …
$ paper_mass_g <dbl> 0.2100, 0.2100, 0.2100, 0.2400, 0.1500, 0.…

<chr> = character (shade, teams)
<dbl> = numeric (mass_g, petiole_mm, thickness_mm, paper_mass_g)

The Core Tidyverse Verbs

Five functions do most of the work in data wrangling:

Function	What it does
`filter()`	keep rows matching a condition
`select()`	keep columns by name
`mutate()`	add or change a column
`arrange()`	sort rows
`summarize()`	collapse rows to a summary

All take a data frame as first input (via %>%) and return a data frame. We’ll wrangle with the first four today — summarize() is next lecture’s topic.

📖 R4DS §3 — Data Transformation

Note

✅ Why these five?

Almost every wrangling step you’ll write this semester decomposes into some combination of picking rows, picking columns, adding a column, sorting, or collapsing to a summary — that’s filter, select, mutate, arrange, and summarize.

Think of them as building blocks:

filter()   → rows you want
select()   → columns you want
mutate()   → new columns you need
arrange()  → order the rows
summarize()→ collapse to summaries (next lecture)

`filter()` — Picking Rows

Note

🔮 Predict first: We have 53 leaves. Before you run it, guess how many rows filter(shade == "sunny") returns. Write your number down, then run it and check.

# filter() keeps rows where the condition is TRUE -------

# Keep only sunny leaves
leaf_df %>% filter(shade == "sunny") %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
# ℹ 1 more variable: paper_mass_g <dbl>

Common comparison operators:

Symbol	Meaning
`==`	exactly equal
`!=`	not equal
`>` `<`	greater / less than
`>=` `<=`	greater / less than or equal
is.na	return all NA rows
!is.na	return all rows that are NOT NA

`filter()` — Numbers and Two Conditions

# Keep only leaves heavier than 0.5 g
leaf_df %>% filter(mass_g > 0.5) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.65         35         0.15
2 <NA>    <NA>    12345 shady   0.6          68         0.12
3 <NA>    <NA>    12345 shady   0.57         73         0.14
# ℹ 1 more variable: paper_mass_g <dbl>

# Combine two conditions with , (AND)
leaf_df %>%
  filter(shade == "shady", mass_g > 0.5) %>%
  head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345      shady  0.6           68         0.12
2 <NA>    <NA>    12345      shady  0.57          73         0.14
3 twig_1  l07     fighting_… shady  0.589         66         0.32
# ℹ 1 more variable: paper_mass_g <dbl>

A comma between conditions means AND
- both have to be true for the row to stay

Warning

⚠️ Watch out!

= assigns a value.
== tests equality.
filter(shade = "sunny") → error.
filter(shade == "sunny") → correct.

`select()` — Picking Columns

# select() keeps the columns you name ------------------

# Keep only shade and mass
leaf_df %>% select(shade, mass_g) %>% head(2)

# A tibble: 2 × 2
  shade mass_g
  <chr>  <dbl>
1 sunny   0.4 
2 sunny   0.48

# Drop a column with a minus sign - what is left?
leaf_df %>% select(-paper_mass_g) %>% names()

[1] "twig_id"      "leaf_id"      "teams"        "shade"       
[5] "mass_g"       "petiole_mm"   "thickness_mm"

# Keep columns that start with a word
leaf_df %>% select(starts_with("mass")) %>% head(2)

# A tibble: 2 × 1
  mass_g
   <dbl>
1   0.4 
2   0.48

Why it matters: real data sets have 50–100+ columns and you rarely need all of them
Useful helpers:
- starts_with("x") — columns starting with “x”
- ends_with("_mm") — columns ending with “_mm”
- contains("mass") — columns containing “mass”

📖 R4DS §3.3

`mutate()` — Creating New Columns

Note

🔮 Predict first: mass_g * 1000 converts grams to milligrams. Before running: will mutate() replace mass_g or add a new column? How many columns will the result have?

# mutate() adds a new column to the data frame ---------

# Convert grams to milligrams
leaf_df %>%
  mutate(mass_mg = mass_g * 1000) %>%
  head(3)

# A tibble: 3 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
3 <NA>    <NA>    12345 sunny   0.34         68         0.14
# ℹ 2 more variables: paper_mass_g <dbl>, mass_mg <dbl>

mutate() keeps all existing columns and adds new ones

Note

✅ Watch the column count

mutate() never removes columns
- it adds to the right of your data frame
- one new column per variable you create
To overwrite instead, reuse the name on the left:
- mutate(mass_g = round(mass_g, 2))

`mutate()` — Categories and Several at Once

# Add a size category based on mass
leaf_df %>%
  mutate(size_class = if_else(mass_g > 0.5, "large",
                              "small")) %>%
  head(2)

# A tibble: 2 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
# ℹ 2 more variables: paper_mass_g <dbl>, size_class <chr>

# mutate() can create several new columns at once
leaf_df %>%
  mutate(
    mass_mg    = mass_g * 1000,
    petiole_cm = petiole_mm / 10 # petiole length in cm
  ) %>%
  head(2)

# A tibble: 2 × 10
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15
2 <NA>    <NA>    12345 sunny   0.48         63         0.14
# ℹ 3 more variables: paper_mass_g <dbl>, mass_mg <dbl>,
#   petiole_cm <dbl>

Use if_else(condition, value_if_true, value_if_false) to create categories
Separate several new columns with commas, like the arguments they are

📖 R4DS §3.3

`arrange()` — Sorting Rows

# arrange() sorts rows by one or more columns ----------

# Lightest leaves first
leaf_df %>% arrange(mass_g) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 twig_6  l06     fighting_… sunny  0.193       29           0.13
2 <NA>    4       Oscar_Karl sunny  0.230       35.7         0.36
3 <NA>    <NA>    12345      sunny  0.27        34           0.11
# ℹ 1 more variable: paper_mass_g <dbl>

# Heaviest leaves first (use desc() to reverse)
leaf_df %>% arrange(desc(mass_g)) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 <NA>    <NA>    still_loa… sunny  1.08      84             0.68
2 twig_2  l09     fighting_… shady  0.920      0.383        NA   
3 <NA>    <NA>    leaf_ogle… shady  0.8       41.2           0.5 
# ℹ 1 more variable: paper_mass_g <dbl>

Default: ascending order (smallest first)
desc(column) → descending order (largest first)
Very useful before printing a table or checking outliers

Tip

arrange() is rarely used in the middle of a pipeline — it is most useful at the end, when you want to inspect your results in a specific order.

`arrange()` — Sorting by Two Columns

# Sort by shade, then by mass within each shade
leaf_df %>% arrange(shade, desc(mass_g)) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams      shade mass_g petiole_mm thickness_mm
  <chr>   <chr>   <chr>      <chr>  <dbl>      <dbl>        <dbl>
1 twig_2  l09     fighting_… shady  0.920      0.383        NA   
2 <NA>    <NA>    leaf_ogle… shady  0.8       41.2           0.5 
3 twig_2  l10     fighting_… shady  0.782     67             0.27
# ℹ 1 more variable: paper_mass_g <dbl>

The first column sorts the whole data frame
The second breaks ties inside each group of the first
Here: shady leaves first, and within shady, heaviest first

Note

You can mix directions — arrange(shade, desc(mass_g)) sorts shade ascending and mass descending in the same call.

The Full Pipeline

# Chain all the verbs together -------------------------
# Read it top to bottom like a recipe

wrangled_df <- leaf_df %>%
  filter(mass_g > 0) %>% # remove any zeros
  select(shade, mass_g, thickness_mm) %>% # keep only needed cols
  mutate(
    mass_mg = mass_g * 1000,
    size_class = if_else(mass_g > 0.5, "large", "small")
  ) %>%
  arrange(shade, desc(mass_g)) # sort by shade, then mass

head(wrangled_df, 3)

# A tibble: 3 × 5
  shade mass_g thickness_mm mass_mg size_class
  <chr>  <dbl>        <dbl>   <dbl> <chr>     
1 shady  0.920        NA       920. large     
2 shady  0.8           0.5     800  large     
3 shady  0.782         0.27    782. large

Read the pipeline out loud:

Take leaf_df,

then filter to non-zero masses,

then select three columns,

then add two new columns,

then sort by shade and mass.

Note

✅ Compare it to nested calls

Written as nested functions, the first step ends up buried in the middle
The pipe lets you read top to bottom, in the order R actually runs things

→ ACTIVITY 5 starts now

🛑 Go to Activity 5 — Wrangling

Close the slides. Open the Activity 5 and work through filter, select, mutate, arrange, and the full pipeline. Type every line yourself into your script; predict before you run.

What We Learned Today

filter() — keep rows by condition
select() — keep columns by name
mutate() — add new calculated columns
arrange() — sort rows
Full pipeline: chain all verbs with %>%

References:

📖 R4DS §3 — Data Transformation
📖 R4DS §5 — Data Tidying
🌐 Data Carpentry

Up next — Lecture 06, Summary Statistics:

Mean, median, variance, SD, SE
Counting correctly with sum(!is.na())
group_by() + summarize(), and skimr

Lecture 05 — Wrangling Data

Where we left off (Lecture 04 — GGPlot II)

Goals for Today

How to Use These Slides — Predict · Type · Run

Load Libraries and Data

Take a First Look

The Core Tidyverse Verbs

filter() — Picking Rows

filter() — Numbers and Two Conditions

select() — Picking Columns

mutate() — Creating New Columns

mutate() — Categories and Several at Once

arrange() — Sorting Rows

arrange() — Sorting by Two Columns

The Full Pipeline

→ ACTIVITY 5 starts now

What We Learned Today

`filter()` — Picking Rows

`filter()` — Numbers and Two Conditions

`select()` — Picking Columns

`mutate()` — Creating New Columns

`mutate()` — Categories and Several at Once

`arrange()` — Sorting Rows

`arrange()` — Sorting by Two Columns