```{r setup, include=FALSE}
library(tidyverse)
library(RCurl)
library(ggmap)
library(maps)
library(network)
library(sna)
library(GGally)
```

Here you will find skeletal solutions / suggestions for the problems.

## 1. What predicts hate crimes in America?

The US has an acute problem with hate crimes. In the week following Trump's 2016 election night, the hate crimes famously spiked compared with the long-running average.

Sociologists argue that it is not a uniformly distributed problem, however. They suspect that it could be related to factors like inequality, urban living and median income.

They have data on the average number of hate crimes (per 100,000 people) in USA between 2010 and 2015 for each American state. They also have a measurement for inequality called gini_index, the proportion of people living in metro areas, and the median income of the state.

As you see below, the dataframe is called `hatedata` and its columns are called `median_houshold_income`, `gini_index`, and `share_population_in_metro_areas`.


```{r hate, echo=FALSE, warning=FALSE}
hatedata <- read.csv("hate_crimes.csv", header = T)

hatedata <- hatedata %>% select(state, median_household_income, gini_index, share_population_in_metro_areas, avg_hatecrimes_per_100k_fbi)

#head(hatedata)
head(as_tibble(hatedata)) %>% knitr::kable(digits = 5, caption = "Hate crime data")

```


What would be a suitable visualization for sociologists to investigate whether these three factors play a role? Is it feasible to combine them all into one or are they better split into several graphs? What possible problems do you see in your graph that could make analysis more difficult?

```{r, eval=FALSE}
library(gridExtra)

g1 <- ggplot(hatedata, aes(x=gini_index, y=avg_hatecrimes_per_100k_fbi)) + geom_point()
g2 <- ggplot(hatedata, aes(x=median_household_income, y=avg_hatecrimes_per_100k_fbi)) + geom_point()
g3 <- ggplot(hatedata, aes(x=share_population_in_metro_areas, y=avg_hatecrimes_per_100k_fbi)) + geom_point()

grid.arrange(g1,g2,g3, nrow=1)

```


## 2. Do Pulitzers matter for a newspaper's growth?

The finest honour any investigative journalist could aspire to is called the Pulitzer Prize. The owners of the newspaper they work for surely wouldn't mind either, as this could lead to a boost in circulation. But is there empirical evidence to support their assumption that Pulitzers matter for sales?

In the dataset below, called `puldata`, there is information on the size of their circulation in 2004 (` Daily.Circulation..2004`), the percentage by which it grew or declined between 2004 and 2013 (`Change.in.Daily.Circulation`) and finally how many Pulitzer Prize winners worked at the newspaper between those years (`Pulitzer.Prize.Winners.and.Finalists..2004.2014`).

```{r pulitzer, echo=FALSE, warning=FALSE}
puldata <- read.csv("pulitzer-circulation-data.csv", header = T)

puldata <- puldata %>% select(Daily.Circulation..2004, Change.in.Daily.Circulation..2004.2013, Pulitzer.Prize.Winners.and.Finalists..2004.2014)

puldata$Daily.Circulation..2004 <- as.numeric(sub(",", "", puldata$Daily.Circulation..2004, fixed = TRUE))

head(as_tibble(puldata)) %>%  knitr::kable(digits = 5, caption = "Pulitzer and Newspaper data")

```


Is there a relationship, would you say? Is there a way to make the visualisation also show whether the initial 2004 circulation influenced the Pulitzer-fueled growth?

```{r}

ggplot(puldata, aes(x=Pulitzer.Prize.Winners.and.Finalists..2004.2014, y = Change.in.Daily.Circulation..2004.2013, size = Daily.Circulation..2004)) + geom_point()

```

## 3. What is the shape of Trump's twitter popularity?

Now to American politics and their principal arena: the Twitter-verse. 

Trump boasts an impressive following of more than 150 million accounts (perhaps with a few Russian bots included), but follower numbers never give a full story. Political scientists may want to have a closer look at the number of interactions, and specifically how many people like (or 'favorite') his tweets. 

In the `trumpdata` prepared below, there is a `url` giving the link to his tweet, `replies`, `retweets`, and `favorites`.

```{r trump, echo=FALSE, warning=FALSE}
trumpdata <- read.csv("realDonaldTrump.csv", header = T)

trumpdata <- trumpdata %>% select(url,replies, retweets, favorites)

head(as_tibble(trumpdata)) %>% knitr::kable(digits = 5, caption = "Trump Twitter data")

```

How could you show to political scientists the general tendencies of his favorite-numbers? Are they always hovering around the same number? Are some massively more popular than most?

```{r}
ggplot(trumpdata, aes(x=favorites)) + geom_histogram()

```

## 4. Which politicians are most rapped about?

On the same political note, but on a different platform, data from the musical lyrics website [Genius.com](www.genius.com) reveals that politcians are a favourite topic in rap songs. This popularity stretches decades back. 

Clever social scientists studying this phenomenon have applied a technique called *sentiment analysis* to determine whether the rapper's attitude towards the politician is positive, negative or neutral.

Have a look at the `hiphopdata` set below, and its three fields `candidate`, `song`, `artist`, and `sentiment`.
 
```{r hiphop, echo=FALSE, warning=FALSE}
hiphopdata <- read.csv("genius_hip_hop_lyrics.csv", header = T)

hiphopdata <- hiphopdata %>% select(candidate, song, artist, sentiment)

head(as_tibble(hiphopdata)) %>% knitr::kable(digits = 5, caption = "Hiphop Politics Data")

```

Which politicians are given the rap treatment most often, we wonder? And is there a way to visualize conveniently the proportion of different rap sentiments among the lyrical references?


```{r}
ggplot(hiphopdata, aes(x=candidate, fill=sentiment)) + geom_histogram(stat="count")

```


## 5. Who cares about the Oxford comma?

Now for a controversial, English-related debate. As you may know, the *Oxford comma* refers to whether, at the end of a listing, you should add a comma or not before the last item. For example, is it right to say *It's important for a person to be honest, kind and loyal.* or would you rather have it as *It's important for a person to be honest, kind, and loyal.*.

To settle the matter via democratic processes, a poll was conducted that asked people which option they preferred. The pollsters also asked about their education background, thinking that perhaps postgraduates would be more pedantic in the matter.

The data has been prepared below, in `commaperc`, which contains the percentages (`freq`) of people for each `Education` group who preferred one or the other.


```{r comma, echo=FALSE, warning=FALSE}

commadata <- read.csv("comma-survey.csv", header = T)

commadata <- commadata %>% select(In.your.opinion..which.sentence.is.more.gramatically.correct., Education)

commadata <- rename(commadata, Opinion = In.your.opinion..which.sentence.is.more.gramatically.correct.)

commaperc <- commadata %>% group_by(Education, Opinion) %>% summarise (n = n()) %>% mutate(freq = n/sum(n))

head(as_tibble(commaperc)) %>% knitr::kable(digits = 5, caption = "Oxford Comma Opinions")

```

So, postgraduates - are we more pedantic about the Oxford commas? 

```{r}

ggplot(commaperc, aes(x=Education, y= freq, fill=Opinion)) + geom_bar(stat="identity")

```


## 6. Which airlines are the best to fly with?

With the twin Boeing disasters fresh in memory, we are excused to sometimes worry a bit about the safety records of modern airlines. So let's have a look at them.

In the `airdata` dataset below, there is a number of fatalities between 2000-2014 given as a heartless statistic in `fatalities_00_14` for each `airline`. Have a look.

```{r air, echo=FALSE, warning=FALSE}

airdata <- read.csv("airline-safety.csv", header = T)

airdata <- arrange(airdata, desc(fatalities_00_14))

head(as_tibble(airdata)) %>% knitr::kable(digits = 5, caption = "Airport fatality data")
```

This data has already been sorted from high-to-low. We wonder of course whether we could summarise visually which airline we should avoid in the future, assuming that their past incidents has something to say about their future safety.

With so many airlines on display, is there a way to prevent it looking cluttered?

```{r}
ggplot(airdata, aes(y=airline, x=fatalities_00_14)) + geom_point() + geom_segment(aes(y=airline, yend=airline, x=0, xend=fatalities_00_14) ) 

```


## 7. What is the teenagers' drug of choice?

Many communities struggle with youth drug use, which could be carried over to later stages of life. To monitor the shares of people at different ages using drugs, authorities have collected cross-sectional data showing, for each drug, how many percent use it in each age interval.

This data is found in `drugdata`, which has the fields `drug` and `percentage`.


```{r drugs, echo=FALSE, warning=FALSE}

drugdata <- read.csv("drug-use-by-age.csv", header = T)

drugdata <- drugdata %>% select(age, ends_with('use'), -starts_with("alcohol")) 

col_old <- colnames(drugdata)
col_new <- gsub(pattern = ".use",replacement = "", x  = col_old)
colnames(drugdata) <- col_new

drugdata <- drugdata %>% gather(drug, percentage, cocaine:sedative) 

head(as_tibble(drugdata)) %>% knitr::kable(digits = 5, caption = "Drug use data")

```

How could we look into the trends in different drugs' popularity over the life span? How could we minimise the visual confusion in doing so? 

```{r}
ggplot(drugdata, aes(x=age , group=drug, y=percentage)) + geom_line(stat="identity",aes(linetype=drug))

```


## 8. Where are all the women in biopics?

Abraham Lincoln, Winston Churchill, and Freddie Mercury have all been the subject of a biographical motion picture, also known as *biopic*. Every Great Man of History seems to have led a Hollywood actor to an Oscar. But where are all the Women of History?

Film nerds have collected the following data `biopics` which shows the year of release (`year_release`), the gender of the subject (`subject_sex`), the `title`, the `subject` name and the `lead_actor_actress` who played them.

```{r biopics, echo=FALSE , warning=FALSE}

biopicdata <- read.csv("biopics.csv", header = T)

biopicdata <- biopicdata %>% select(year_release, subject_sex, title, subject, lead_actor_actress )


head(as_tibble(biopicdata)) %>% knitr::kable(digits = 5, caption = "Biopic subject sex data")

```

Is there a convenient way for us to visualize how this proportion in gender has changed over the last decades? Are female biopics on the rise or has it stayed constant?

```{r}
ggplot(biopicdata, aes(x=year_release, fill=subject_sex)) + geom_area(stat="count") 

```

## 9. Where do people marry their second cousins?

People born and bred in UK may be surprised to learn that marrying second cousins is a common practice. There are many reasons for doing so: staying close to the family and community, minimising the disruption of marriage, keeping property among consanguines...

We would therefore expect this to have a relationship with factors like poverty. In the data below, a dataframe containing the (`long`,`lat`) coordinates that define the corners of some countries has been merged with data that gives the `Percent` of how many people marry their second cousins.

```{r pressure, echo=FALSE, warning=FALSE}

cousindata <- read.csv("cousin-marriage-data.csv", header = T)

map <- map_data("world")

map2 <- full_join(map, cousindata, by=c('region' = 'Country'))

head(as_tibble(map2)) %>% knitr::kable(digits = 5, caption = "Map data")
```

Moving into the territory of geospatial data visualization, could you figure out a way of showing in an overlay of the world map, how many people in each country engage in this practice?

```{r}

ggplot(map2, aes(x=long, y=lat, group=group, fill=Percent)) + geom_polygon()

```


# Beyond ggplot2 (extension)

## 10. Network data: 

If you ever found yourself watching the romcom *Love, Actually* on Christmas, then you may have marvelled at how the screenwriters managed to weave together such a complex ensemble script. You may wonder if they ever used some kind of network visualization to map out all the character interactions.

Unfortunately, `ggplot` does not provide convenient support for networks, but via a set of libraries called `networks`, `sna` and `GGally` we could nevertheless cobble together a dataframe where the two first columns (`actor` and `actors`) give the characters in an interaction, and the third (`number`) gives the interaction frequency.

```{r cars, echo=FALSE, warning=FALSE}

ladata <- read.csv("love_actually_adjacencies.csv", header = T)

ladata[is.na(ladata)] <- 0

ladata <- ladata %>% 
          gather(actor, number, bill_nighy:alan_rickman) %>%
           select(actors, actor, number) %>% 
          filter(number != 0)

lanet <- as.network(ladata, directed=FALSE, matrix.type = "edgelist")

head(as_tibble(ladata)) %>% knitr::kable(digits = 5, caption = "Love Actually data")
```

Using the function `ggnet2` we could simply feed the network to it. Now you know! (no exercise for this one :) )

```{r}
ggnet2(lanet, label = TRUE, label.size = 5)

```

