Data transformation

---

## Reminders

- Assignment 1: due soon!
  - submit link to your repo on Canvas
  - check general rubric and assignment-specific rubric on our website
  
  
- Yesterday: github, visualization and ggplot2
  - review assigned readings
  - practice practice practice
  
  
- Assignment 2 up: 
  - covers data visualization, transformation, and exploration

---
# Editing your files: Script and RMD structure

* [Consider how to structure your R scripts](https://bookdown.org/yih_huynh/Guide-to-R-Book/r-conventions.html)
* [Rmarkdown (intermediate)](https://posit.co/blog/r-markdown-tips-tricks-2-cleaning-up-your-code/)
* [Rmarkdown resources](https://bookdown.org/yihui/rmarkdown-cookbook/)

---

# Programming as a form of problem-solving

---

## `penguins`

---

## `penguins`

``` r
penguins
```

```
## # A tibble: 344 × 8
##    species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##    <fct>   <fct>              <dbl>         <dbl>             <int>       <int>
##  1 Adelie  Torgersen           39.1          18.7               181        3750
##  2 Adelie  Torgersen           39.5          17.4               186        3800
##  3 Adelie  Torgersen           40.3          18                 195        3250
##  4 Adelie  Torgersen           NA            NA                  NA          NA
##  5 Adelie  Torgersen           36.7          19.3               193        3450
##  6 Adelie  Torgersen           39.3          20.6               190        3650
##  7 Adelie  Torgersen           38.9          17.8               181        3625
##  8 Adelie  Torgersen           39.2          19.6               195        4675
##  9 Adelie  Torgersen           34.1          18.1               193        3475
## 10 Adelie  Torgersen           42            20.2               190        4250
## # ℹ 334 more rows
## # ℹ 2 more variables: sex <fct>, year <int>
```

---

## What is the average body mass of an Adelie penguin?

1. First we need to identify the **input**, or the data were going to analyze: `penguins` data frame
1. Next we need to **select** only the observations classified as species Adelie penguins
1. Finally we need to calculate the average value, or **mean**, of the variable `body_mass_g`

``` r
head(penguins, 5)
```

```
## # A tibble: 5 × 8
##   species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
##   <fct>   <fct>              <dbl>         <dbl>             <int>       <int>
## 1 Adelie  Torgersen           39.1          18.7               181        3750
## 2 Adelie  Torgersen           39.5          17.4               186        3800
## 3 Adelie  Torgersen           40.3          18                 195        3250
## 4 Adelie  Torgersen           NA            NA                  NA          NA
## 5 Adelie  Torgersen           36.7          19.3               193        3450
## # ℹ 2 more variables: sex <fct>, year <int>
```

]

``` r
library(palmerpenguins)
data("penguins")
penguins_adelie <- filter(.data = penguins, species == "Adelie")
summarize(.data = penguins_adelie, avg_mass = mean(body_mass_g, na.rm = TRUE))
```

```
## # A tibble: 1 × 1
##   avg_mass
##      <dbl>
## 1    3701.
```

]
]

---

## What is the average body mass of a penguin for each species?

1. First we need to identify the **input**, or the data were going to analyze: our `penguins` data frame
1. Next we need to **group** the observations by species
1. Finally we need to calculate the average value, or **mean**, of the variable `body_mass_g`

]

``` r
library(palmerpenguins)
data("penguins")
penguins_species <- group_by(.data = penguins, species)
summarize(.data = penguins_species, avg_mass = mean(body_mass_g, na.rm = TRUE))
```

```
## # A tibble: 3 × 2
##   species   avg_mass
##   <fct>        <dbl>
## 1 Adelie       3701.
## 2 Chinstrap    3733.
## 3 Gentoo       5076.
```

]
]

---

# Data transformation with dplyr

---

Documentation: **https://dplyr.tidyverse.org/**

---

## Flow for data transformation

1. The first argument is a data frame
--

1. Subsequent arguments describe what to do with the data frame

--
1. The result is a **new data frame**

`dplyr` contains useful functions for transforming and manipulating data frames.

These functions can be thought of as **verbs that define actions** to be performed on the data: 
the noun is the data, and the verb is acting on the noun.

All of the dplyr verbs (all the verbs in the wider tidyverse) work similarly.

---

## Key functions in `dplyr`

`function()`  | *Action performed*
--------------|--------------------------------------------------------
`filter()`    | Picks observations from the data frame based on their values (operates on rows)
`arrange()`   | Changes the order of observations based on their values
`select()`    | Picks columns/variables from the data frame based on their names (operates on columns)
`rename()`    | Changes the name of columns in the data frame
`mutate()`    | Creates new columns from existing ones
`group_by()`  | Changes the unit of analysis from the complete data frame to individual groups
`summarize()` | Collapses the data frame to a smaller number of rows which summarize the larger data

These are the basic verbs you will commonly use to transform your data. By **combining them together**, you can perform powerful data manipulation tasks.

---

## American vs. British English

<blockquote class="twitter-tweet" data-width="550" data-lang="en" data-dnt="true" data-theme="light"><p lang="en" dir="ltr">The holy grail: "For consistency, aim to use British (rather than American) spelling." <a href="https://twitter.com/hashtag/rstats?src=hash&ref_src=twsrc%5Etfw">#rstats</a> <a href="http://t.co/7qQSWIowcl">http://t.co/7qQSWIowcl</a>. Colour is right!</p>— Hadley Wickham (@hadleywickham) <a href="https://twitter.com/hadleywickham/status/405707093770244097?ref_src=twsrc%5Etfw">November 27, 2013</a></blockquote>

]

![](https://media.giphy.com/media/8KnfG3knLExpu/giphy.gif)

]

* US `summarize()` = UK `summarise()`
* US `color()` = UK `colour()`

---

## Look again at key functions in `dplyr`

`function()`  | Action performed
--------------|--------------------------------------------------------
`filter()`    | Picks observations from the data frame based on their values (operates on rows)
`arrange()`   | Changes the order of observations based on their values
`select()`    | Picks columns/variables from the data frame based on their names (operates on columns)
`rename()`    | Changes the name of columns in the data frame
`mutate()`    | Creates new columns from existing ones
`group_by()`  | Changes the unit of analysis from the complete data frame to individual groups
`summarize()` | Collapses the data frame to a smaller number of rows which summarize the larger data

---

# dplyr in action: trying out functions:

* filter(data, criteria) (other rows discarded)
* arrange(.data, by, how) (sort your table by value(s))
* select() (select the columns you wish to keep DANGER!)
* rename(.data, newvarname = oldvarname) (rename variables)
* mutate() (create variable based on prior variable(s))
* group_by() (get ready to do math/mutations/etc based on some kind of aggregate factor)
* summarize() (do that math!)

---

# dplyr in action: trying out functions:

``` r
penguins2<-filter(penguins, sex == "female")
arrange(penguins, bill_length_mm) # notice: what did this do?
favorite_penguins <- select(penguins, sex) #how is this different from filter?
oops <-rename(penguins, KINDofPENGUIN = species) #what is wrong here -- what mistakes did I make?
penguins$bill_in <- mutate(penguins, bill_depth_in = bill_depth_mm/25.4) 
penguins_species <- group_by(.data = penguins, species)
summarize(
  .data = penguins_species,
  avg_bill = mean(bill_length_mm, na.rm = TRUE),
  avg_mass = mean(body_mass_g, na.rm = TRUE)
)
```

## What is the average bill length and body mass for each Adelie penguin by sex?

---
### What is the average bill length and body mass for each Adelie penguin by sex?
.panelset[
.panel[.panel-name[Filter, then group by]

``` r
library(palmerpenguins)
data("penguins")

penguins_adelie <- filter(.data = penguins, species == "Adelie")
penguins_adelie_sex <- group_by(.data = penguins_adelie, sex)
summarize(
  .data = penguins_adelie_sex,
  avg_bill = mean(bill_length_mm, na.rm = TRUE),
  avg_mass = mean(body_mass_g, na.rm = TRUE)
)
```

```
## # A tibble: 3 × 3
##   sex    avg_bill avg_mass
##   <fct>     <dbl>    <dbl>
## 1 female     37.3    3369.
## 2 male       40.4    4043.
## 3 <NA>       37.8    3540
```

]

``` r
library(palmerpenguins)
data("penguins")

penguins_sex <- group_by(.data = penguins, sex)
penguins_sex_adelie <- filter(.data = penguins_sex, species == "Adelie")
summarize(
  .data = penguins_sex_adelie,
  avg_bill = mean(bill_length_mm, na.rm = TRUE),
  avg_mass = mean(body_mass_g, na.rm = TRUE)
)
```

```
## # A tibble: 3 × 3
##   sex    avg_bill avg_mass
##   <fct>     <dbl>    <dbl>
## 1 female     37.3    3369.
## 2 male       40.4    4043.
## 3 <NA>       37.8    3540
```

]
]

---

## Saving transformed data (`<-`)

``` r
filter(diamonds, cut == "Ideal")
```

```
## # A tibble: 21,551 × 10
##    carat cut   color clarity depth table price     x     y     z
##    <dbl> <ord> <ord> <ord>   <dbl> <dbl> <int> <dbl> <dbl> <dbl>
##  1  0.23 Ideal E     SI2      61.5    55   326  3.95  3.98  2.43
##  2  0.23 Ideal J     VS1      62.8    56   340  3.93  3.9   2.46
##  3  0.31 Ideal J     SI2      62.2    54   344  4.35  4.37  2.71
##  4  0.3  Ideal I     SI2      62      54   348  4.31  4.34  2.68
##  5  0.33 Ideal I     SI2      61.8    55   403  4.49  4.51  2.78
##  6  0.33 Ideal I     SI2      61.2    56   403  4.49  4.5   2.75
##  7  0.33 Ideal J     SI1      61.1    56   403  4.49  4.55  2.76
##  8  0.23 Ideal G     VS1      61.9    54   404  3.93  3.95  2.44
##  9  0.32 Ideal I     SI1      60.9    55   404  4.45  4.48  2.72
## 10  0.3  Ideal I     SI2      61      59   405  4.3   4.33  2.63
## # ℹ 21,541 more rows
```

]

``` r
diamonds_ideal <- filter(.data = diamonds, cut == "Ideal")
```

]

``` r
(diamonds_ideal <- filter(.data = diamonds, cut == "Ideal"))
```

]
]

---

## Syntactic column names

Normally within tidyverse functions, we can refer to column names directly. For example:

``` r
count(x = diamonds, color)
```

```
## # A tibble: 7 × 2
##   color     n
##   <ord> <int>
## 1 D      6775
## 2 E      9797
## 3 F      9542
## 4 G     11292
## 5 H      8304
## 6 I      5422
## 7 J      2808
```

Here `color` is a column of the `diamonds` data frame. It has a syntactic name, so we can refer to it directly. This won't work for any column name that is non-syntactic.

---

### Syntactic column names

``` r
count(x = diamonds, color)
```

```
## # A tibble: 7 × 2
##   color     n
##   <ord> <int>
## 1 D      6775
## 2 E      9797
## 3 F      9542
## 4 G     11292
## 5 H      8304
## 6 I      5422
## 7 J      2808
```

A **syntactic name** is what R consider a **valid name** and consists of letters, digits, . and `_` but it can’t begin with `_` (or other symbols) or a with a digit

]

### Non-syntactic names

* `Social conservative`
* `7-point ideology`
* `_id`

A **non-syntactic name** is name that doesn’t follow these rules or a name that uses a reserved word in R like TRUE, NULL, if, and function (see the complete list in ?Reserved)

You **do not want to deliberately create non-syntactic names**, BUT you need to understand how they work because you’ll come across them, for ex. with data that has been created outside R

]

---

## Referring to non-syntactic names

### Use backticks

``` r
count(x = diamonds, `color`)
```

```
## # A tibble: 7 × 2
##   color     n
##   <ord> <int>
## 1 D      6775
## 2 E      9797
## 3 F      9542
## 4 G     11292
## 5 H      8304
## 6 I      5422
## 7 J      2808
```

]

### Don't use quotation marks

``` r
count(x = diamonds, "color")
```

```
## # A tibble: 1 × 2
##   `"color"`     n
##   <chr>     <int>
## 1 color     53940
```

The word “color” has been duplicated 53940 times and tabulated using the count() function. That's a weird behavior and not what we intended!

]

Note that color is actually a syntactic name, so it does not need backticks (it can have them though). The point of this example is to illustrate that with non-syntactic names you want to use backticks rather than ""

---

## Referring to non-syntactic names

Non-syntactic name without backticks
```
7-point ideology <- c("communism", "anarchism", "fascism")
Error: unexpected symbol in "7-point ideology"
```
Non-syntactic name with backticks

``` r
`7-point ideology` <- c("communism", "anarchism", "fascism")
```

Non-syntactic name without backticks
```
if <- 10
Error: unexpected assignment in "if <-"
```

Non-syntactic name with backticks

``` r
`if` <- 10
```

---

# Pipes (`%>%`)

---

## Piping (`%>%`)

> Pipes are a special operator in R. They allow you to write a sequence of multiple operations by   **passing the result of one function to another one, in sequence**

Pipes simplify your code and make your operations more intuitive, BUT they are not the only way to write your R code... in fact for a long time they did not even exist in R! [See here to learn more](https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe/)

> You will see them two ways (%>% and |>): both are correct. The book favors |> and I favor %>%

Let' start by looking at ways you can write the same code with and without pipes

---

## Piping (`%>%`)

Imagine we have the following task:
**using the penguins dataset, calculate the average body mass for Adelie penguins on different islands**

First, we break down the problem into smaller steps:
1. Filter penguins to only keep observations where the species is Adelie
1. Group the filtered penguins by island
1. Summarize the grouped and filtered penguins by calculating the average body mass

#### How do we implement the code?

---

### Option 1: save each step in a new data frame

``` r
data(penguins)

penguins_1 <- filter(penguins, species == "Adelie")
penguins_2 <- group_by(penguins_1, island)
penguins_3 <- summarize(penguins_2, body_mass = mean(body_mass_g, na.rm = TRUE))
print(penguins_3)
```

```
## # A tibble: 3 × 2
##   island    body_mass
##   <fct>         <dbl>
## 1 Biscoe        3710.
## 2 Dream         3688.
## 3 Torgersen     3706.
```
This is valid code. The drawback is that we have to name each intermediate object. Here each intermediate object is named with a number to the end, which is not good self-documentation. We have to remember that the data exists in each intermediate step and remember to reference the correct one.

---

### Option 2: replace the original data frame

``` r
data(penguins)

penguins <- filter(penguins, species == "Adelie")
penguins <- group_by(penguins, island)
(penguins <- summarize(penguins, body_mass = mean(body_mass_g, na.rm = TRUE)))
```

```
## # A tibble: 3 × 2
##   island    body_mass
##   <fct>         <dbl>
## 1 Biscoe        3710.
## 2 Dream         3688.
## 3 Torgersen     3706.
```

Instead of creating intermediate objects, we can replace the original data frame with the modified form. This is also valid code. The drawback is that if we make an error in the middle of the operation, we need to rerun the entire operation from the beginning.

---

### Option 3: function composition

``` r
data(penguins)

summarize(group_by(filter(penguins, species == "Adelie"), island), 
          body_mass = mean(body_mass_g, na.rm = TRUE))
```

```
## # A tibble: 3 × 2
##   island    body_mass
##   <fct>         <dbl>
## 1 Biscoe        3710.
## 2 Dream         3688.
## 3 Torgersen     3706.
```

This is valid code. The drawback is that this code is hard to read, since we need to read it from the inside out, and prone to errors.

---

### Option 4: pipes!

``` r
penguins %>%
  filter(species == "Adelie") %>%
  group_by(island) %>%
  summarize(body_mass = mean(body_mass_g, na.rm = TRUE))
```

```
## # A tibble: 3 × 2
##   island    body_mass
##   <fct>         <dbl>
## 1 Biscoe        3710.
## 2 Dream         3688.
## 3 Torgersen     3706.
```

Notice the clearer syntax and the focus on **actions**, not objects.

Pipes **chains a series of functions together**: they automatically pass the output from the first function the next one as the input, producing code that is easily readable by humans.

<!--

## Piping (`%>%`)

.panelset.sideways[
.panel[.panel-name[No pipes]

``` r
by_dest <- group_by(
  .data = flights,
  dest
)

delays <- summarise(
  .data = by_dest,
  count = n(),
  dist = mean(distance, na.rm = TRUE),
  delay = mean(arr_delay, na.rm = TRUE)
)

delays <- filter(
  .data = delays,
  count > 20,
  dest != "HNL"
)
```

]

``` r
delays <- flights %>%
  group_by(dest) %>%
  summarize(
    count = n(),
    dist = mean(distance, na.rm = TRUE),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>%
  filter(
    count > 20,
    dest != "HNL"
  )
```

]

]
-->

---

## Errors to avoid with pipes: example 1

#### Invalid code

```
# all flights departed from NYC in 2013
library(nycflights13)
data(flights)

delays <- flights %>% 
  by_dest <- group_by(dest) %>% 
  delay <- summarize(
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>% 
  d <- filter(count > 20, dest != "HNL")
```

]

#### Correct code

``` r
# all flights departed from NYC in 2013
library(nycflights13)
data(flights)

delays <- flights %>%
  group_by(dest) %>%
  summarize(
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>%
  filter(count > 20, dest != "HNL")
```

]

Don’t assign anything within the pipes. That is, **don't not use <- inside the piped operation** for intermediate steps. Only use this at the beginning if you want to save the output

---

## Errors to avoid with pipes: example 2

#### Invalid code

```
delays <- flights %>%
  group_by(dest)
  summarize(
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  )
  filter(count > 20, dest != "HNL")
```

]

#### Correct code

``` r
delays <- flights %>%
  group_by(dest) %>%
  summarize(
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>%
  filter(count > 20, dest != "HNL")
```

]

Remember to **add the pipe %>% at the end of each line involved in the piped operation**

---

## Errors to avoid with pipes: example 3

#### Invalid code

```
delays <- flights %>% 
  group_by(.data = flights, dest) %>% 
  summarize(.data = flights,
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>% 
  filter(.data = flights, 
          count > 20, dest != "HNL")
```
]

#### Correct code

``` r
delays <- flights %>%
  group_by(dest) %>%
  summarize(
    count = n(),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>%
  filter(count > 20, dest != "HNL")
```

]

When you use pipes, **don’t reference the data frame with each function**: just the first time at the beginning of the pipe sequence

---

## Practice transforming data

---
### Exercise: built-in datasets
(group work)
* Accept assignment (individually): [https://classroom.github.com/a/oX3fYgiL](https://classroom.github.com/a/oX3fYgiL)
* Summarize penguins in interesting way
* Call `data()` in R
* Look over list of data options
* Choose a dataset
* Summarize the dataset in a meaningful way (use AT LEAST two of the key functions from slide #12)
* Commit and push! (not graded -- for practice only!)

---

## Assignment 2 overview

---

## Tips to tackle a coding assignment
  
- start early and divide it up in multiple sessions
- you are stepping outside your comfort zone: some problems (and some frustration) are expected... be patient with yourself
- jumping right into coding is a bad habit: 
   * take some time to review the readings and class notes
   * take some time to think at how you can approach the assignment into small steps
- form a study group!

**got stack?** 
- take a break and come back with fresh eyes
- write down the problem with pen and paper and/or explain it out loud to yourself, your roommate, your dog, cat, fish, etc.
- 15-min rule, google/stackoverflow, ask a classmate and/or the instructional staff
- review our website FAQ on how to ask questions
  
---
# NEW IMPROVED CLONING!

I have an additional (non-shell/terminal!) way to clone!

* Accept the assignment
* go to your new repo
* copy the link
* In RStudio: (top left) Go to File > New Project > Version Control > Git > (navigate to where you want, and add the relevant details -- folder name, git url!)

---

## Acknowledgments

The content of these slides is derived in part from Sabrina Nardin's and Benjamin Soltoff’s “Computing for the Social Sciences” course materials, licensed under the CC BY NC 4.0 Creative Commons License. Any errors or oversights are mine alone.