Your Final Project Starts Now
Finding real data, asking a real question, and building toward a real answer
Finding Real Data and your own projects
Where We Left Off (Lecture 07)
pivot_longer()— collapsed a 53-column wide file into tidy long data R can actually work withpivot_wider()— turned long data back into a human-readable summary table- Date wrangling — ice seasons cross the new year;
if_else()+ymd()fixed the calendar slice_max()— pulled the peak ice-cover day from each yearlm()again — modeled whether annual maximum ice cover on Lake Superior is declining
✅ Transition
Every dataset we’ve used so far — leaf morphology, Cisco fish, Duluth weather, Lake Superior ice — was handed to you.
Today you start finding your own! That you will clean and analyze!
Goals for Today
- Understand what makes a good final project dataset and question
- See a gallery of data sources — serious ecology/science and fun cryptid options
- Learn the project milestone structure — five deliverables spread over the next five weeks
- Understand what each milestone requires before you submit
- Leave today with at least three dataset candidates in mind
🖐 Today’s action
Before you leave, write down one dataset you’re seriously considering and the question you’d ask with it.
No new R code today.
This lecture is about thinking like a data scientist — asking a question first, then finding the data that can answer it.
📖 R4DS Ch. 1 — The whole game
“The best dataset is the one you can’t stop thinking about.”
What Is the Final Project?
Over the next five weeks you will work independently on a dataset of your choice. The final product is a reproducible report that includes:
- A clear biological or scientific question
- Data downloaded or imported in R (no Excel pre-processing)
- Exploratory visualizations — raw plots, summaries, distributions
- At least one statistical analysis appropriate to your question (regression, t-test, ANOVA, or similar)
- Assumption checks on your statistical model
- A short written interpretation of your results
This is exactly the workflow you’ve been building all semester — applied to something you care about.
What you are NOT required to do:
- Find a perfect, spotless dataset (messy is fine — cleaning is a skill)
- Do something that no one has done before - no example data and code examples
- Use more than one analysis method
What you ARE required to do:
- Have a dataset with at least 50 rows and at least 3 variables
- Be able to state your hypothesis as H₀ and Hₐ before you analyze
- Submit every milestone on time — partial credit only if submitted
The Five Milestones
| # | What’s due | When |
|---|---|---|
| M1 | Dataset proposal — 1 paragraph + link | DATES |
| M2 | Data in R — loaded, glimpsed, first raw plot | DATES |
| M3 | Cleaned data + exploratory figures | DATES |
| M4 | Statistical analysis + assumption checks | DATES |
| M5 | Final Quarto report, knitted to HTML, presentations. | Week Prior to finals |
Each milestone builds directly on the last — M2 cannot happen if M1 isn’t approved.
Why milestones and not one big deadline?
- Professional data projects always have checkpoints
- It forces you to start — the hardest part of any project
- Each piece is graded on its own, so a rough M2 doesn’t sink your M5
- You get feedback from me before the next step, not after everything is done
M1 is due at the START of DATE_____________.
Milestone 1 — What You Need to Submit
A short paragraph (5–8 sentences) containing:
- The name and source URL of your dataset
- How many rows and columns it has (look this up on the source website or download and
glimpse()) - The key variables you plan to use — what is your X? What is your Y?
- Your biological or scientific question in plain English
- Your null and alternate hypotheses (H₀ and Hₐ)
- Which statistical approach you think you’ll use (regression, t-test, ANOVA, or other)
- Sketch of the take home graph
Submit as a Word file to Canvas.
Example M1 paragraph:
“I plan to use the BFRO Bigfoot sighting dataset (kaggle.com, ~5,000 rows), which records the date, state, season, and classification of each sighting. My key variables are season (X) and sighting count (Y). My question is: do Bigfoot sightings peak in summer compared to other seasons? H₀: sighting frequency does not differ by season. Hₐ: sighting frequency differs by season. I plan to use a one-way ANOVA or chi-square test on sighting counts grouped by season.”
One paragraph. That’s it. I’ll respond with approved / revise before M2 is due.
What Makes a Good Project Dataset?
It should have:
- At least 50 rows (more is better — 200–2,000 is ideal for beginners)
- At least one numeric response variable (the thing you want to explain)
- At least one predictor — either numeric (for regression) or categorical with 2+ levels (for t-test/ANOVA)
- A question you can actually answer with the methods we’ve covered
- A graph and analysis you can do
Watch out for:
- Datasets with only counts or percentages (hard to do regression or ANOVA on directly)
- Datasets where every row is a country or a year (too few observations)
- Datasets with hundreds of columns but only 20 rows (more variables than data)
Good question types for this course:
| Question | Method |
|---|---|
| Does Y increase with X? | Regression |
| Does group A differ from group B on Y? | t-test |
| Does Y differ across 3+ groups? | ANOVA |
| Does Y change over time? | Regression on year |
| How does Y depend on both X and group? | ANCOVA (optional) |
Any of these — applied to data you find interesting — is a valid project.
Data Sources: Ecology and Environmental Science
These are peer-reviewed and citable — great for biology-related questions:
- 🌿 GBIF — Global Biodiversity Information Facility
Species occurrence records worldwide, 2+ billion observations
gbif.org— download by species, region, or date range - 🌊 EDI Repository — Environmental Data Initiative
Long-term ecology datasets, many already tidy
edirepository.org - 🌱 NEON — National Ecological Observatory Network
Standardized ecological data across 81 US sites
data.neonscience.org - 📦 Dryad — data behind published papers
Every dataset comes with a real paper you can cite
datadryad.org - 🔬 Zenodo — open research data from labs worldwide
zenodo.org - Figshare - https://figshare.com/
Good for regression or ANOVA questions like:
- Does plant height differ between disturbed and undisturbed sites?
- Does bird species richness decrease with latitude?
- Has the first flowering date of a species shifted over time?
- Is fish abundance correlated with water temperature?
NEON and EDI data are especially good — they’re already in tidy, well-documented CSV format and many come with R vignettes.
Data Sources: Ecology and Environmental Science (cont.)
🧊 NOAA-GLERL Great Lakes Ice
Daily ice cover for all five lakes back to 1973
glerl.noaa.gov/data/ice
Great Lakes Superior, Michigan, Huron, Erie, Ontario — each a different story🐟 LTER Network — Long Term Ecological Research
Fish, invertebrate, and plankton counts from dozens of sites
lternet.edu/data or- https://lternet.edu/using-lter-data/🌡️ Berkeley Earth
Global land temperature anomalies by country and city
berkeleyearth.org/data🐦 eBird — Cornell Lab of Ornithology
500M+ bird observation records globally
ebird.org/science/use-ebird-data
The Great Lakes ice option in particular:
You already know how to load and pivot sup.txt — the exact same code works for mic.txt, hur.txt, eri.txt, and ont.txt.
Possible questions: - Does Lake Erie lose more ice per decade than Lake Superior? - Which lake shows the strongest trend in maximum ice cover? - Has the timing of peak ice changed since 1973?
This is a low-barrier entry since the hard code work is already done.
Data Sources: Fun and Engaging 👽🦶☕
These are popular with students and very workable statistically:
👣 Bigfoot / BFRO Sightings
~5,000 reports with date, state, season, classification (A/B/C), weather
kaggle.com/datasets/mexwell/bigfoot-sightings
ANOVA on sightings by season; regression of count on year; t-test by class type👽 UFO Sightings — NUFORC
88,000+ reports with date, location, shape, duration
kaggle.com/datasets/joebeachcapital/ufo-sightings
How has sighting frequency changed over time? Does duration differ by shape?☕ Coffee Reviews
Cupped coffee scores, country of origin, altitude, processing method
kaggle.com— search “coffee quality dataset”
Does altitude predict quality score? Does processing method matter (ANOVA)?🎮 Video Game Sales
Sales figures by platform, genre, region, year — 16,000 titles
kaggle.com— search “video game sales”
Does genre predict sales? Has the platform landscape shifted over time?
Why these work for this course:
- Clean enough to load with
read_csv()directly - Have clear categorical and numeric variables
- Produce obvious testable hypotheses
- Students actually want to look at the results
You don’t have to study biology to do a biology data science course project. The methods are the same regardless of the data.
Data Sources: General and Exploratory
🌍 Our World in Data
Country-level indicators: health, climate, economics, education
ourworldindata.org/data— all downloadable as CSV
Global trends; great for regression of any indicator over time📊 FiveThirtyEight
Sports, politics, culture — all their story data is public
github.com/fivethirtyeight/data
Dozens of clean CSVs; each has a published story as inspiration🏈 Sports Reference
Player and team stats for NFL, NBA, MLB, NHL
sports-reference.com
Does salary predict performance? Regression all day.🚲 Bike Sharing (UCI ML Repository)
Hourly bike rental counts, weather, season — 17,000 rows
archive.ics.uci.edu
Regression: does temperature predict rentals? ANOVA across seasons?🔬 Figshare
Research data across all fields — often from published papers
figshare.com
And the one your professor can’t stop you from using:
- 🏔️ Kaggle — the world’s largest public dataset repository
kaggle.com/datasets
Free account required to download
Filter by: file type → CSV; size → small (under 10 MB to start)
Google Dataset Search:
datasetsearch.research.google.com
Search any topic and find datasets across dozens of repositories at once.
What Kind of Question Should You Ask?
A good project question has three parts:
- A response variable (Y) — something you can measure or count
- A predictor (X) — something that might explain Y
- A direction — does more X lead to more (or less) Y? Or does Y differ across groups?
Examples of strong questions:
- Do Bigfoot sightings in summer outnumber those in winter? (ANOVA)
- Has the number of UFO sightings increased since the 1990s? (regression on year)
- Does the altitude of a coffee farm predict its cupping score? (regression)
- Does Lake Erie show a stronger ice-loss trend than Lake Superior? (compare two slopes)
- Does bird species richness differ between NEON forest and grassland sites? (t-test)
Avoid questions like:
- “What is interesting about this dataset?” — too vague for H₀/Hₐ
- “Which state has the most UFO sightings?” — descriptive, not inferential
- “Is climate change real?” — not answerable with one dataset
A useful test:
Can you write H₀ and Hₐ in one sentence each? If yes, you have a testable question. If not, keep narrowing.
📖 R4DS Ch. 1.1 — framing a data science question
The Workflow You Already Know
Every milestone maps to something you’ve already done in this class:
| Milestone | Skill you already have |
|---|---|
| M1 — find dataset, state hypotheses | Lecture 01 — null vs. alternate hypotheses |
M2 — load in R, glimpse(), first plot |
Lecture 02/03 — read_csv(), ggplot() |
| M3 — clean + exploratory figures | Lecture 03/04 — filter(), mutate(), boxplots |
| M4 — statistical model + assumption checks | Lecture 05/06 — lm(), residual plots, lm() |
| M5 — Quarto report knitted to HTML | Every lecture — you’ve been doing this all semester - MAYBE!!!!!! |
You are not learning new tools for the final project. You are applying tools you already have to data you chose.
The biggest mistake in final projects:
Waiting until Milestone 3 to actually open the data in R.
If you can’t load the file and run glimpse() in the first week, you will not be able to finish. M2 is your early warning system — take it seriously.
Choosing Wisely — A Practical Checklist
Before you write your M1 proposal, answer these five questions:
If all five are yes — you have a viable dataset. Submit M1.
If any are no — keep looking, or come to office hours.
Practical tip — test your download now:
# Try loading your candidate dataset this week ---
test_df <- read_csv("path/to/your/file.csv")
glimpse(test_df)
dim(test_df)
head(test_df)If read_csv() throws an error, you want to know that before M1, not after.
If the file needs read.table() or special arguments (like our ice data did), that’s fine — but discover it early.
Common Pitfalls and How to Avoid Them
“I can’t find a dataset I like” - Try Google Dataset Search first: datasetsearch.research.google.com - Search for a topic you’re already interested in + the word “data” or “csv” - Come to office hours with a general topic — I can usually point you to a source in five minutes
“My dataset is too messy to analyze” - Messy is fine for this course — cleaning is the first two milestones - If you can read_csv() it and glimpse() it, you can work with it - You don’t need every column — a well-chosen select() is enough
“I can’t figure out what question to ask” - Start with the variables: what is numeric? What is categorical? - Ask: “what would I predict, before looking at the data?” - That prediction is your Hₐ; its opposite is H₀
“My dataset only has two columns” - You need at least one X and one Y — two columns can be enough for regression - But usually: look for a richer version of the same dataset on the same source
“I already started analyzing before M1 was approved” - Submit M1 anyway — describe what you’ve done so far - I’ll either approve it or redirect you before you go further
Office hours exist for this. If you’re stuck on finding or loading a dataset, come in. Don’t wait until the M2 deadline.
What a Strong Final Project Looks Like
From past semesters, strong projects have:
- A clear narrative — the report tells a story from question to answer
- Figures that do the talking — the key result is visible in a plot before the stats confirm it
- Simple, correct statistics — one well-interpreted
lm()beats three confused ones - Honest interpretation — “the trend was not significant (p = 0.12), which may reflect the small sample size” is a perfectly valid finding
- Code that runs — the Quarto file knits to HTML without errors, every time
Things that do not make a strong project: - Many analyses none of which are interpreted - A significant p-value with no plot showing the actual effect - Copied code from Stack Overflow that you can’t explain
The question you will be graded on:
“Does this report tell me something real about this dataset, using tools from this class, in a way I could reproduce?”
That’s it. Not complexity. Not novelty. Clarity + reproducibility.
📖 R4DS Ch. 28 — Quarto formats: making a final report
Summary: What to Do This Week
By end of this week:
- Visit at least three of the dataset sources shown today
- Download a candidate dataset and confirm
read_csv()loads it - Identify your Y variable, your X variable, and write one sentence of hypothesis
- Write your Milestone 1 paragraph (see the template on the next slide)
- Submit M1 to Canvas before the start of Lecture 09
Resources:
- Dataset source list (this slide deck, downloadable)
- Office hours — come with a topic or a link, leave with a plan
- Canvas discussion board — post a candidate dataset and get peer feedback
Milestone 1 is due: start of Lecture 09
It is short. It is low stakes. It is not optional.
The purpose is to make sure you have a working dataset before you write any analysis code. Every semester, students who skip or delay M1 struggle at M4 and M5.
Two class periods from now. Write it this weekend.
Milestone 1 Template
Copy this into a Word doc or text file and fill in the blanks:
“I plan to use the [dataset name] dataset, available at [URL]. It contains approximately [n rows] observations and [n columns] variables. The key variables I plan to use are [X variable] as my predictor and [Y variable] as my response. My research question is: [your question in plain English]. My null hypothesis is that [H₀ in one sentence], and my alternate hypothesis is that [Hₐ in one sentence]. I plan to use [regression / t-test / ANOVA / other] to test this. I have confirmed I can load this file into R using
read_csv()[or describe what function you used].”
What happens after you submit:
- I read all M1 proposals within 48 hours
- I reply with one of:
- ✅ Approved — proceed to M2
- 🔄 Revise — one specific thing to change, resubmit by [date]
- ❌ Not viable — I’ll suggest an alternative source
Most proposals get approved. The revision requests are almost always about the hypothesis being too vague.