Tidy Data

]

# Week 2: .fancy[Tidy Data]

### <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M243.4 2.587C251.4-.8625 260.6-.8625 268.6 2.587L492.6 98.59C506.6 104.6 514.4 119.6 511.3 134.4C508.3 149.3 495.2 159.1 479.1 160V168C479.1 181.3 469.3 192 455.1 192H55.1C42.74 192 31.1 181.3 31.1 168V160C16.81 159.1 3.708 149.3 .6528 134.4C-2.402 119.6 5.429 104.6 19.39 98.59L243.4 2.587zM256 128C273.7 128 288 113.7 288 96C288 78.33 273.7 64 256 64C238.3 64 224 78.33 224 96C224 113.7 238.3 128 256 128zM127.1 416H167.1V224H231.1V416H280V224H344V416H384V224H448V420.3C448.6 420.6 449.2 420.1 449.8 421.4L497.8 453.4C509.5 461.2 514.7 475.8 510.6 489.3C506.5 502.8 494.1 512 480 512H31.1C17.9 512 5.458 502.8 1.372 489.3C-2.715 475.8 2.515 461.2 14.25 453.4L62.25 421.4C62.82 420.1 63.41 420.6 63.1 420.3V224H127.1V416z"/></svg> EMSE 4575: Exploratory Data Analysis
### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M224 256c70.7 0 128-57.31 128-128s-57.3-128-128-128C153.3 0 96 57.31 96 128S153.3 256 224 256zM274.7 304H173.3C77.61 304 0 381.6 0 477.3c0 19.14 15.52 34.67 34.66 34.67h378.7C432.5 512 448 496.5 448 477.3C448 381.6 370.4 304 274.7 304z"/></svg> John Paul Helveston
### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M96 32C96 14.33 110.3 0 128 0C145.7 0 160 14.33 160 32V64H288V32C288 14.33 302.3 0 320 0C337.7 0 352 14.33 352 32V64H400C426.5 64 448 85.49 448 112V160H0V112C0 85.49 21.49 64 48 64H96V32zM448 464C448 490.5 426.5 512 400 512H48C21.49 512 0 490.5 0 464V192H448V464z"/></svg> September 07, 2022

]

---

# Week 2: .fancy[Tidy Data]

## 1. Tidy Data
## 2. Tidy Data Wrangling

## BREAK

## 3. Tidy Data Visualization
## 4. Data Provenance & Curation
## 5. Writing a Research Question

---

# Week 2: .fancy[Tidy Data]

## 1. .orange[Tidy Data]
## 2. Tidy Data Wrangling

## BREAK

## 3. Tidy Data Visualization
## 4. Data Provenance & Curation
## 5. Writing a Research Question

---

## .center[Federal R&D Spending by Department]

```
#> # A tibble: 6 × 15
#>    year   DHS   DOC   DOD   DOE   DOT   EPA   HHS Interior  NASA   NIH   NSF Other  USDA    VA
#>   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1  1976     0   819 35696 10882  1142   968  9226     1152 12513  8025  2372  1191  1837   404
#> 2  1977     0   837 37967 13741  1095   966  9507     1082 12553  8214  2395  1280  1796   374
#> 3  1978     0   871 37022 15663  1156  1175 10533     1125 12516  8802  2446  1237  1962   356
#> 4  1979     0   952 37174 15612  1004  1102 10127     1176 13079  9243  2404  2321  2054   353
#> 5  1980     0   945 37005 15226  1048   903 10045     1082 13837  9093  2407  2468  1887   359
#> 6  1981     0   829 41737 14798   978   901  9644      990 13276  8580  2300  1925  1964   382
```

---

## .center[Federal R&D Spending by Department]

# "Wide" format

]]

# "Long" format

```
#> # A tibble: 6 × 3
#>   department  year rd_budget_mil
#>   <chr>      <dbl>         <dbl>
#> 1 DOD         1976         35696
#> 2 NASA        1976         12513
#> 3 DOE         1976         10882
#> 4 HHS         1976          9226
#> 5 NIH         1976          8025
#> 6 NSF         1976          2372
```

]]

---

## .center[Federal R&D Spending by Department]

# "Wide" format

```
#> [1] 42 15
```

]]

# "Long" format

```
#> [1] 588   3
```

]]

---

# .center[Tidy data = "Long" format]

- Each **variable** has its own **column**
- Each **observation** has its own **row**

---

# Tidy data

- Each **variable** has its own **column**
- Each **observation** has its own **row**

]

]

---

# "Long" format

]]

# "Wide" format

]]

---

# .center[**Do the names describe the values?**]

## **Yes**: "Long" format

]]

## **No**: "Wide" format

```
#> # A tibble: 6 × 8
#>    year   DHS   DOC   DOD   DOE   DOT   EPA   HHS
#>   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1  1976     0   819 35696 10882  1142   968  9226
#> 2  1977     0   837 37967 13741  1095   966  9507
#> 3  1978     0   871 37022 15663  1156  1175 10533
#> 4  1979     0   952 37174 15612  1004  1102 10127
#> 5  1980     0   945 37005 15226  1048   903 10045
#> 6  1981     0   829 41737 14798   978   901  9644
```

]]

---

# **Quick practice 1**: "long" or "wide" format?

**Description**: Tuberculosis cases in various countries

```
#> # A tibble: 6 × 4
#>   country      year  cases population
#>   <chr>       <dbl>  <dbl>      <dbl>
#> 1 Afghanistan  1999    745   19987071
#> 2 Afghanistan  2000   2666   20595360
#> 3 Brazil       1999  37737  172006362
#> 4 Brazil       2000  80488  174504898
#> 5 China        1999 212258 1272915272
#> 6 China        2000 213766 1280428583
```
]

---

# **Quick practice 2**: "long" or "wide" format?

**Description**: Word counts in LOTR trilogy

```
#> # A tibble: 9 × 4
#>   Film                       Race   Female  Male
#>   <chr>                      <chr>   <dbl> <dbl>
#> 1 The Fellowship Of The Ring Elf      1229   971
#> 2 The Fellowship Of The Ring Hobbit     14  3644
#> 3 The Fellowship Of The Ring Man         0  1995
#> 4 The Return Of The King     Elf       183   510
#> 5 The Return Of The King     Hobbit      2  2673
#> 6 The Return Of The King     Man       268  2459
#> 7 The Two Towers             Elf       331   513
#> 8 The Two Towers             Hobbit      0  2463
#> 9 The Two Towers             Man       401  3589
```
]

---

# **Quick practice 3**: "long" or "wide" format?

**Description**: Word counts in LOTR trilogy

```
#> # A tibble: 15 × 4
#>    Film                       Race   Gender Word_Count
#>    <chr>                      <chr>  <chr>       <dbl>
#>  1 The Fellowship Of The Ring Elf    Female       1229
#>  2 The Fellowship Of The Ring Elf    Male          971
#>  3 The Fellowship Of The Ring Hobbit Female         14
#>  4 The Fellowship Of The Ring Hobbit Male         3644
#>  5 The Fellowship Of The Ring Man    Female          0
#>  6 The Fellowship Of The Ring Man    Male         1995
#>  7 The Return Of The King     Elf    Female        183
#>  8 The Return Of The King     Elf    Male          510
#>  9 The Return Of The King     Hobbit Female          2
#> 10 The Return Of The King     Hobbit Male         2673
#> 11 The Return Of The King     Man    Female        268
#> 12 The Return Of The King     Man    Male         2459
#> 13 The Two Towers             Elf    Female        331
#> 14 The Two Towers             Elf    Male          513
#> 15 The Two Towers             Hobbit Female          0
```

---

# Reshaping data with

## `pivot_longer()` and `pivot_wider()`

---

background-color: #fff

# Reshaping data

## `pivot_longer()`<br>`pivot_wider()`

]

]

---

## .center[From "long" to "wide" with `pivot_wider()`]

---

## .center[From "long" to "wide" with `pivot_wider()`]

```r
head(fed_spend_long)
```

]

```r
fed_spend_wide <- fed_spend_long %>%
    pivot_wider(
*       names_from = department,
*       values_from = rd_budget_mil)

head(fed_spend_wide)
```

```
#> # A tibble: 6 × 15
#>    year   DOD  NASA   DOE   HHS   NIH   NSF  USDA Interior   DOT   EPA   DOC   DHS    VA Other
#>   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1  1976 35696 12513 10882  9226  8025  2372  1837     1152  1142   968   819     0   404  1191
#> 2  1977 37967 12553 13741  9507  8214  2395  1796     1082  1095   966   837     0   374  1280
#> 3  1978 37022 12516 15663 10533  8802  2446  1962     1125  1156  1175   871     0   356  1237
#> 4  1979 37174 13079 15612 10127  9243  2404  2054     1176  1004  1102   952     0   353  2321
#> 5  1980 37005 13837 15226 10045  9093  2407  1887     1082  1048   903   945     0   359  2468
#> 6  1981 41737 13276 14798  9644  8580  2300  1964      990   978   901   829     0   382  1925
```

]

---

## .center[From "wide" to "long" with `pivot_longer()`]

---

## .center[From "wide" to "long" with `pivot_longer()`]

```r
head(fed_spend_wide)
```

]

```r
fed_spend_long <- fed_spend_wide %>%
    pivot_longer( 
*       names_to = "department",
*       values_to = "rd_budget_mil",
*       cols = DOD:Other)

head(fed_spend_long)
```

```
#> # A tibble: 6 × 3
#>    year department rd_budget_mil
#>   <dbl> <chr>              <dbl>
#> 1  1976 DOD                35696
#> 2  1976 NASA               12513
#> 3  1976 DOE                10882
#> 4  1976 HHS                 9226
#> 5  1976 NIH                 8025
#> 6  1976 NSF                 2372
```

]

---

## Can also set `cols` by selecting which columns _not_ to use

```r
names(fed_spend_wide)
```

```
#>  [1] "year"     "DOD"      "NASA"     "DOE"      "HHS"      "NIH"      "NSF"      "USDA"     "Interior" "DOT"      "EPA"      "DOC"      "DHS"      "VA"       "Other"
```

]

```r
fed_spend_long <- fed_spend_wide %>%
    pivot_longer(
        names_to = "department", 
        values_to = "rd_budget_mil",
*       cols = -year)

head(fed_spend_long)
```

]

---

# Your turn: Reshaping Data

Open the `practice.Rmd` file.

Run the code chunk to read in the following two data files:

- `pv_cell_production.xlsx`: Data on solar photovoltaic cell production by country
- `milk_production.csv`: Data on milk production by state

Now modify the format of each:

- If the data are in "wide" format, convert it to "long" with `pivot_longer()`
- If the data are in "long" format, convert it to "wide" with `pivot_wider()`

---

# Week 2: .fancy[Tidy Data]

## 1. Tidy Data
## 2. .orange[Tidy Data Wrangling]

## BREAK

## 3. Tidy Data Visualization
## 4. Data Provenance & Curation
## 5. Writing a Research Question

---

# Why do we need tidy data?

(a quick explanation with cute graphics, by [Allison Horst](https://github.com/allisonhorst/stats-illustrations))

---

class: center
background-image: url("images/horst_tidydata_1.jpg")
background-size: contain

---

class: center
background-image: url("images/horst_tidydata_2.jpg")
background-size: contain

---

class: center
background-image: url("images/horst_tidydata_3.jpg")
background-size: contain

---

# Tidy data wrangling

Compute the total R&D spending in each year

```r
head(fed_spend_wide)
```

---

# Tidy data wrangling

Compute the total R&D spending in each year

**Approach 1**: Create new `total` by adding each variable

```r
fed_spend_wide %>%
  mutate(total = DHS + DOC + DOD + DOE + DOT + EPA + HHS + Interior + NASA + NIH + NSF + Other + USDA + VA) %>%
  select(year, total)
```

```
#> # A tibble: 42 × 2
#>     year  total
#>    <dbl>  <dbl>
#>  1  1976  86227
#>  2  1977  91807
#>  3  1978  94864
#>  4  1979  96601
#>  5  1980  96305
#>  6  1981  98304
#>  7  1982  95448
#>  8  1983  95010
#>  9  1984 105371
#> 10  1985 114818
#> # … with 32 more rows
#> # ℹ Use `print(n = ...)` to see more rows
```

---

# Tidy data wrangling

Compute the total R&D spending by department in each year

**Approach 2**: Reshape first, then summarise

```r
fed_spend_long <- fed_spend_wide %>%
    pivot_longer(
        names_to = "department", 
        values_to = "rd_budget_mil",
        cols = -year)

head(fed_spend_long)
```

]

```r
fed_spend_long %>%
    group_by(year) %>%
    summarise(total = sum(rd_budget_mil))
```

]

---

# Tidy data wrangling

Compute the total R&D spending by department in each year

**Approach 2**: Reshape first, then summarise

```r
total <- fed_spend_wide %>%
    pivot_longer(
        names_to = "department", 
        values_to = "rd_budget_mil",
        cols = -year) %>% 
    group_by(year) %>%
    summarise(total = sum(rd_budget_mil))
```

]

```r
head(total)
```

```
#> # A tibble: 6 × 2
#>    year total
#>   <dbl> <dbl>
#> 1  1976 86227
#> 2  1977 91807
#> 3  1978 94864
#> 4  1979 96601
#> 5  1980 96305
#> 6  1981 98304
```

]

---

# Your turn: Tidy Data Wrangling

Open the `practice.Rmd` file.

Run the code chunk to read in the following two data files:

- `gapminder.csv`: Life expectancy in different countries over time
- `gdp.csv`: GDP of different countries over time

Now convert the data into a tidy (long) structure, then create the following summary data frames:

- Mean life expectancy in each year.
- Mean GDP in each year.

---

# .fancy[Break]

---

# Week 2: .fancy[Tidy Data]

## 1. Tidy Data
## 2. Tidy Data Wrangling

## BREAK

## 3. .orange[Tidy Data Visualization]
## 4. Data Provenance & Curation
## 5. Writing a Research Question

---

# Tidy data vizualization

Make a bar chart of total R&D spending by agency

```r
head(fed_spend_wide)
```

]

]

---

# Tidy data vizualization

Make a bar chart of total R&D spending by agency

```r
ggplot(fed_spend_wide) +
* geom_col(aes(x = rd_budget_mil, y = department)) +
  theme_bw() +
  labs(
      x = "R&D Spending ($Millions)",
      y = "Federal Agency"
  )
```

```
#> Error in FUN(X[[i]], ...): object 'rd_budget_mil' not found
```

]

]

---

# Tidy data vizualization

Make a bar chart of total R&D spending by agency

```r
fed_spend_wide %>%
*   pivot_longer(
*       names_to = "department",
*       values_to = "rd_budget_mil",
*       cols = -year) %>%
  ggplot() +
  geom_col(aes(x = rd_budget_mil, y = department)) +
  theme_bw() +
  labs(
      x = "R&D Spending ($Millions)",
      y = "Federal Agency"
  )
```

]

]

---

# Your turn: Tidy Data Visualization

Run the code chunk to read in the two data files, then convert the data into a tidy (long) structure to create the following charts:

]

]

---

# Week 2: .fancy[Tidy Data]

## 1. Tidy Data
## 2. Tidy Data Wrangling

## BREAK

## 3. Tidy Data Visualization
## 4. .orange[Data Provenance & Curation]
## 5. Writing a Research Question

---

### Data provenance - It matters where you get your data

**Validity**:

- Is this data trustworthy? Is it authentic?
- Where did the data come from?
- How has the data been changed / managed over time?
- Is the data complete?

**Comprehension**:

- Is this data accurate?
- Can you explain your results?
- Is this the right data to answer your question?

**Reproducibility**:

- I should be able to fully replicate your results from your raw data and code.

---

## <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M500.3 443.7l-119.7-119.7c27.22-40.41 40.65-90.9 33.46-144.7C401.8 87.79 326.8 13.32 235.2 1.723C99.01-15.51-15.51 99.01 1.724 235.2c11.6 91.64 86.08 166.7 177.6 178.9c53.8 7.189 104.3-6.236 144.7-33.46l119.7 119.7c15.62 15.62 40.95 15.62 56.57 0C515.9 484.7 515.9 459.3 500.3 443.7zM79.1 208c0-70.58 57.42-128 128-128s128 57.42 128 128c0 70.58-57.42 128-128 128S79.1 278.6 79.1 208z"/></svg> **Document your source like a museum curator**

**Example**: View `README.md` file in the `data` folder

Whenever you download data, you should **at a minimum** record the following:

- The name of the file you are describing.
  - The date you downloaded it.
  - The original name of the downloaded file (in case you renamed it).
  - The url to the site you downloaded it from.
  - The source of the _original_ data (sometimes different from the site you downloaded it from).
  - A short description of the data, maybe how they were collected (if available).
  - A dictionary for the data (e.g. a simple markdown table describing each variable).

---
class: inverse

# Your turn

Documentation in the "data/README.md" file is missing for the following data sets:

- wildlife_impacts.csv: [source](https://github.com/rfordatascience/tidytuesday/tree/master/data/2019/2019-07-23)
- north_america_bear_killings.txt: [source](https://data.world/makeovermonday/2019w21)
- uspto_clean_energy_patents.xlsx: [source](https://www.nsf.gov/statistics/2018/nsb20181/report/sections/industry-technology-and-the-global-marketplace/global-trends-in-sustainable-energy-research-and-technologies)

Go to the above sites and add the following information to the "data/README.md" file:

- The name of the downloaded file.
- The web address to the site you downloaded the data from.
- The source of the _original_ data (if different from the website).
- A short description of the data and how they were collected.
- A dictionary for the data (hint: the site might already have this!).

---

# Week 2: .fancy[Tidy Data]

## 1. Tidy Data
## 2. Tidy Data Wrangling

## BREAK

## 3. Tidy Data Visualization
## 4. Data Provenance & Curation
## 5. .orange[Writing a Research Question]

---

# Writing a research question

Follow [these guidelines](https://writingcenter.gmu.edu/guides/how-to-write-a-research-question) - your question should be:

- **Clear**: your audience can easily understand its purpose without additional explanation.

- **Focused**: it is narrow enough that it can be addressed thoroughly with the data available and within the limits of the final project report.

- **Concise**: it is expressed in the fewest possible words.

- **Complex**: it is not answerable with a simple "yes" or "no," but rather requires synthesis and analysis of data.

- **Arguable**: its potential answers are open to debate rather than accepted facts (do others care about it?)

---

# Writing a research question

**Bad question: Why are social networking sites harmful?**

- Unclear: it does not specify _which_ social networking sites or state what harm is being caused; assumes that "harm" exists.

**Improved question: How are online users experiencing or addressing privacy issues on social networking sites such as Facebook and Twitter?**

- Specifies the sites (Facebook and Twitter), type of harm (privacy issues), and who is harmed (online users).

---

# Writing a research question

**Example from previous classes**:

- [Genders in the Workforce](https://eda.seas.gwu.edu/showcase/2021-Spring/gender_pay_gap.html): How has the US gender wage gap changed over time for different occupations and age groups?
- [NFL Suspensions](https://eda.seas.gwu.edu/showcase/2021-Spring/nfl_suspensions.html): What factors contribute to the severity of disciplinary actions towards NFL players from 2002-2014?

**Other good examples**: See the [Example Projects Page](https://eda.seas.gwu.edu/2022-Fall/help/example-projects.html) page

---

# Use [this link](https://docs.google.com/spreadsheets/d/1JdgZWdnPPidLa809Tjos0nU7F4pgeogpsWPKzRxMzjo/edit?usp=sharing) to start forming teams