Lecture 05 — Wrangling Data

filter, select, mutate, arrange — and chaining them into a pipeline

Filter, select, mutate, and arrange with the tidyverse pipe — building one clean wrangling pipeline on the leaf data.

Where we left off (Lecture 04 — GGPlot II)

• Setup — R and Positron installed; project folders created (data/, figures/, scripts/)
• Loading data — read_excel() and read_csv(); a first look with glimpse()
• Pipe — %>% reads as “then”; chains steps together
• GGPlot I & II — geoms, facets, stat_summary(), scale_color_manual(), coordinates, and themes

Note

✅ Key idea from Lecture 04

You can already load data and plot it beautifully. Today we learn to reshape and clean the data itself, before it ever reaches ggplot().

Goals for Today

The core tidyverse verbs:

• filter() — pick rows by a condition
• select() — pick columns by name
• mutate() — create new columns
• arrange() — sort rows
• Chain them all into a pipeline

Tip

🖐 Try it yourself

By the end you will wrangle our leaf data into exactly the columns and rows you need, in one connected pipeline.

Our tools today:

• readxl
• tidyverse

References:

• 📖 R4DS Ch 3 — Data Transformation
• 📖 R4DS Ch 5 — Data Tidying
• 🌐 Data Carpentry

Naming conventions:

• data frames → _df
• plots → _plot
• models → _model

How to Use These Slides — Predict · Type · Run

This is a shorter lecture than usual — one focused idea, wrangling, done properly. You will switch to the activity once, at the end, and type the code yourself into your R script there.

For every code block, do three things:

1. Predict — before it runs, say what you think the output will be
2. Type it out by hand — do not copy-paste
3. Run it and compare to your prediction

Note

✅ Why bother?

• filter(), select(), mutate(), and arrange() look interchangeable on a slide — guessing the output first is what actually forces you to tell them apart.
• You will type %>% dozens of times today; typing it now is what makes it automatic later, when you’re chaining five verbs under a deadline instead of reading one line calmly.

Load Libraries and Data

# Load all packages at the top of every script ----------
library(readxl) # reading Excel files
library(tidyverse) # data wrangling + ggplot2
library(janitor) # clean_names()

# Read the leaf data from the data folder ---------------
leaf_df <-
  read_excel("data/2026_09_03_data_sci_leaf_area.xlsx") %>%
  clean_names()

Take a First Look

# glimpse(): one row per column — name, type, first values
glimpse(leaf_df)

Rows: 53
Columns: 8
$ twig_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ leaf_id      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ teams        <chr> "12345", "12345", "12345", "12345", "12345…
$ shade        <chr> "sunny", "sunny", "sunny", "sunny", "sunny…
$ mass_g       <dbl> 0.4000, 0.4800, 0.3400, 0.6500, 0.2700, 0.…
$ petiole_mm   <dbl> 79.0000, 63.0000, 68.0000, 35.0000, 34.000…
$ thickness_mm <dbl> 0.15, 0.14, 0.14, 0.15, 0.11, 0.16, 0.14, …
$ paper_mass_g <dbl> 0.2100, 0.2100, 0.2100, 0.2400, 0.1500, 0.…

• <chr> = character (shade, teams)
• <dbl> = numeric (mass_g, petiole_mm, thickness_mm, paper_mass_g)

The Core Tidyverse Verbs

Five functions do most of the work in data wrangling:

Function	What it does
filter()	keep rows matching a condition
select()	keep columns by name
mutate()	add or change a column
arrange()	sort rows
summarize()	collapse rows to a summary

All take a data frame as first input (via %>%) and return a data frame. We’ll wrangle with the first four today — summarize() is next lecture’s topic.

📖 R4DS §3 — Data Transformation

Note

✅ Why these five?

Almost every wrangling step you’ll write this semester decomposes into some combination of picking rows, picking columns, adding a column, sorting, or collapsing to a summary — that’s filter, select, mutate, arrange, and summarize.

Think of them as building blocks:

filter()   → rows you want
select()   → columns you want
mutate()   → new columns you need
arrange()  → order the rows
summarize()→ collapse to summaries (next lecture)

filter() — Picking Rows

Note

🔮 Predict first: We have 53 leaves. Before you run it, guess how many rows filter(shade == "sunny") returns. Write your number down, then run it and check.

# filter() keeps rows where the condition is TRUE -------

# Keep only sunny leaves
leaf_df %>% filter(shade == "sunny") %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21

Common comparison operators:

Symbol	Meaning
==	exactly equal
!=	not equal
> <	greater / less than
>= <=	greater / less than or equal
is.na	return all NA rows
!is.na	return all rows that are NOT NA

filter() — Numbers and Two Conditions

# Keep only leaves heavier than 0.5 g
leaf_df %>% filter(mass_g > 0.5) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.65         35         0.15         0.24
2 <NA>    <NA>    12345 shady   0.6          68         0.12         0.23
3 <NA>    <NA>    12345 shady   0.57         73         0.14         0.28

# Combine two conditions with , (AND)
leaf_df %>%
  filter(shade == "shady", mass_g > 0.5) %>%
  head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345        shady  0.6           68         0.12        0.23 
2 <NA>    <NA>    12345        shady  0.57          73         0.14        0.28 
3 twig_1  l07     fighting_mo… shady  0.589         66         0.32        0.257

• A comma between conditions means AND
  • both have to be true for the row to stay

Warning

⚠️ Watch out!

• = assigns a value.
• == tests equality.
• filter(shade = "sunny") → error.
• filter(shade == "sunny") → correct.

Live Demo — Watch It Break (on purpose)

I will type this wrong on purpose:

# One equals sign — a very common mistake
leaf_df %>% filter(shade = "sunny")

R stops and tells us:

Error in `filter()`:
! We detected a named input.
i This usually means that you've used `=` instead of `==`.

Now the fix — two equals signs:

leaf_df %>% filter(shade == "sunny")

Tip

✅ Why show a broken filter()?

filter(shade = "sunny") is the single most common typo in this whole unit, because = and == look almost identical and only one of them tests equality. dplyr’s error message actually names the mistake — “you’ve used = instead of ==” — so once you’ve seen it fixed here, the next time R prints it at you it’s a fix, not a mystery.

select() — Picking Columns

# select() keeps the columns you name ------------------

# Keep only shade and mass
leaf_df %>% select(shade, mass_g) %>% head(2)

# A tibble: 2 × 2
  shade mass_g
  <chr>  <dbl>
1 sunny   0.4 
2 sunny   0.48

# Drop a column with a minus sign - what is left?
leaf_df %>% select(-paper_mass_g) %>% names()

[1] "twig_id"      "leaf_id"      "teams"        "shade"        "mass_g"      
[6] "petiole_mm"   "thickness_mm"

# Keep columns that start with a word
leaf_df %>% select(starts_with("mass")) %>% head(2)

# A tibble: 2 × 1
  mass_g
   <dbl>
1   0.4 
2   0.48

• Why it matters: real data sets have 50–100+ columns and you rarely need all of them
• Useful helpers:
  • starts_with("x") — columns starting with “x”
  • ends_with("_mm") — columns ending with “_mm”
  • contains("mass") — columns containing “mass”

📖 R4DS §3.3

mutate() — Creating New Columns

Note

🔮 Predict first: mass_g * 1000 converts grams to milligrams. Before running: will mutate() replace mass_g or add a new column? How many columns will the result have?

# mutate() adds a new column to the data frame ---------

# Convert grams to milligrams
leaf_df %>%
  mutate(mass_mg = mass_g * 1000) %>%
  head(3)

# A tibble: 3 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
3 <NA>    <NA>    12345 sunny   0.34         68         0.14         0.21
# ℹ 1 more variable: mass_mg <dbl>

• mutate() keeps all existing columns and adds new ones

Note

✅ Watch the column count

• mutate() never removes columns
  • it adds to the right of your data frame
  • one new column per variable you create
• To overwrite instead, reuse the name on the left:
  • mutate(mass_g = round(mass_g, 2))

mutate() — Categories and Several at Once

# Add a size category based on mass
leaf_df %>%
  mutate(size_class = if_else(mass_g > 0.5, "large",
                              "small")) %>%
  head(2)

# A tibble: 2 × 9
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
# ℹ 1 more variable: size_class <chr>

# mutate() can create several new columns at once
leaf_df %>%
  mutate(
    mass_mg    = mass_g * 1000,
    petiole_cm = petiole_mm / 10 # petiole length in cm
  ) %>%
  head(2)

# A tibble: 2 × 10
  twig_id leaf_id teams shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr> <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    12345 sunny   0.4          79         0.15         0.21
2 <NA>    <NA>    12345 sunny   0.48         63         0.14         0.21
# ℹ 2 more variables: mass_mg <dbl>, petiole_cm <dbl>

• Use if_else(condition, value_if_true, value_if_false) to create categories
• Separate several new columns with commas, like the arguments they are

📖 R4DS §3.3

arrange() — Sorting Rows

# arrange() sorts rows by one or more columns ----------

# Lightest leaves first
leaf_df %>% arrange(mass_g) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 twig_6  l06     fighting_mo… sunny  0.193       29           0.13        0.113
2 <NA>    4       Oscar_Karl   sunny  0.230       35.7         0.36       NA    
3 <NA>    <NA>    12345        sunny  0.27        34           0.11        0.15 

# Heaviest leaves first (use desc() to reverse)
leaf_df %>% arrange(desc(mass_g)) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 <NA>    <NA>    still_loadi… sunny  1.08      84             0.68        0.420
2 twig_2  l09     fighting_mo… shady  0.920      0.383        NA          NA    
3 <NA>    <NA>    leaf_oglers  shady  0.8       41.2           0.5         0.3  

• Default: ascending order (smallest first)
• desc(column) → descending order (largest first)
• Very useful before printing a table or checking outliers

Tip

arrange() is rarely used in the middle of a pipeline — it is most useful at the end, when you want to inspect your results in a specific order.

arrange() — Sorting by Two Columns

# Sort by shade, then by mass within each shade
leaf_df %>% arrange(shade, desc(mass_g)) %>% head(3)

# A tibble: 3 × 8
  twig_id leaf_id teams        shade mass_g petiole_mm thickness_mm paper_mass_g
  <chr>   <chr>   <chr>        <chr>  <dbl>      <dbl>        <dbl>        <dbl>
1 twig_2  l09     fighting_mo… shady  0.920      0.383        NA          NA    
2 <NA>    <NA>    leaf_oglers  shady  0.8       41.2           0.5         0.3  
3 twig_2  l10     fighting_mo… shady  0.782     67             0.27        0.342

• The first column sorts the whole data frame
• The second breaks ties inside each group of the first
• Here: shady leaves first, and within shady, heaviest first

Note

You can mix directions — arrange(shade, desc(mass_g)) sorts shade ascending and mass descending in the same call.

The Full Pipeline

# Chain all the verbs together -------------------------
# Read it top to bottom like a recipe

wrangled_df <- leaf_df %>%
  filter(mass_g > 0) %>% # remove any zeros
  select(shade, mass_g, thickness_mm) %>% # keep only needed cols
  mutate(
    mass_mg = mass_g * 1000,
    size_class = if_else(mass_g > 0.5, "large", "small")
  ) %>%
  arrange(shade, desc(mass_g)) # sort by shade, then mass

head(wrangled_df, 3)

# A tibble: 3 × 5
  shade mass_g thickness_mm mass_mg size_class
  <chr>  <dbl>        <dbl>   <dbl> <chr>     
1 shady  0.920        NA       920. large     
2 shady  0.8           0.5     800  large     
3 shady  0.782         0.27    782. large     

Read the pipeline out loud:

• Take leaf_df,
  • then filter to non-zero masses,
    • then select three columns,
      • then add two new columns,
        • then sort by shade and mass.

Note

✅ Compare it to nested calls

• Written as nested functions, the first step ends up buried in the middle
• The pipe lets you read top to bottom, in the order R actually runs things

→ ACTIVITY 5 starts now

🛑 Go to Activity 5 — Wrangling

Close the slides. Open the Activity 5 and work through filter, select, mutate, arrange, and the full pipeline. Type every line yourself into your script; predict before you run.

What We Learned Today

• filter() — keep rows by condition
• select() — keep columns by name
• mutate() — add new calculated columns
• arrange() — sort rows
• Full pipeline: chain all verbs with %>%

References:

• 📖 R4DS §3 — Data Transformation
• 📖 R4DS §5 — Data Tidying
• 🌐 Data Carpentry

Up next — Lecture 06, Summary Statistics:

• Mean, median, variance, SD, SE
• Counting correctly with sum(!is.na())
• group_by() + summarize(), and skimr