So far we’ve been creating the object to work with ourselves. Usually you will work with already existing datasets though (like a file from an experiment, survey etc.). The first thing you need to do to start working on your file is to load it into R session. There are many ways of loading a dataset or file into your R session so that you can use it. How to do it depends mainly on how the data is stored.
When loading a dataset there are generally a few things to consider:
C:/Users/User/Documents specifies the path to the documents folder. A file path can be absolute like the one shown above - its unequivocal. There are also relative file paths that specifies the location of a file relative to current directory. By default R will look in what is called working directory. For simple scripts this by default is set to the documents folder. You can check the current working directory with getwd() function. If you want to change the working directory you can do it with setwd() and pass a string with new directory as argument. This way of working with directories will do for now but it’s not really a good way of managing directories. If you move your script to another folder or share the data and script with other people using setwd() will fail you. Because of this you should avoid manually setting working directory.. One way to deal with this problem is to use here() function from the here package. It uses some heuristics to determine the directory of your script. It can solve the problem of moving or sharing files but remember it uses heuristics so it might not always work. The other solution is to use R projects which automatically set the working directory to the project directory. We’ll learn about them later on.We’ll start with loading what are called flat files: csv (short for comma separated values) and tsv (tab separated values). Both of these are basically plain text files just structures in a ver specific wayThe name kind of gives away how these formats store data: in csv columns are separated by commas and in tsv columns are separated by tabs. You can see an example of a csv file below:
Not very readable right? There are 2 packages with similar functions for loading .csv and .tsv files: utils and readr. We`ll load the gapminder population data saved as a .csv file:
df1 <- read.csv("data/pop.csv")
head(df)
1 function (x, df1, df2, ncp, log = FALSE)
2 {
3 if (missing(ncp))
4 .Call(C_df, x, df1, df2, log)
5 else .Call(C_dnf, x, df1, df2, ncp, log)
6 } Using the readr package is very similar:
library(readr)
df2 <- read_csv("data/pop.csv")Rows: 197 Columns: 302
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (302): country, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 18...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.head(df2)# A tibble: 6 × 302
country `1800` `1801` `1802` `1803` `1804` `1805` `1806` `1807` `1808` `1809`
<chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 Afghani… 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M
2 Angola 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M
3 Albania 400k 402k 404k 405k 407k 409k 411k 413k 414k 416k
4 Andorra 2650 2650 2650 2650 2650 2650 2650 2650 2650 2650
5 UAE 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k
6 Argenti… 534k 520k 506k 492k 479k 466k 453k 441k 429k 417k
# ℹ 291 more variables: `1810` <chr>, `1811` <chr>, `1812` <chr>, `1813` <chr>,
# `1814` <chr>, `1815` <chr>, `1816` <chr>, `1817` <chr>, `1818` <chr>,
# `1819` <chr>, `1820` <chr>, `1821` <chr>, `1822` <chr>, `1823` <chr>,
# `1824` <chr>, `1825` <chr>, `1826` <chr>, `1827` <chr>, `1828` <chr>,
# `1829` <chr>, `1830` <chr>, `1831` <chr>, `1832` <chr>, `1833` <chr>,
# `1834` <chr>, `1835` <chr>, `1836` <chr>, `1837` <chr>, `1838` <chr>,
# `1839` <chr>, `1840` <chr>, `1841` <chr>, `1842` <chr>, `1843` <chr>, …One big difference is that readr by default gives a message about number of rows, columns and types of variables that were loaded. Notice that it loaded every variable as character. That’s because the original file has “k” and “M” to indicate thousands and millions which is interpreted by R as text. We would need to convert those into proper numbers. Loading .tsv files is exactly the same, the only difference is that functions to do it are named read.tsv() and read_tsv().
There’s a bunch of things we can customize when loading data into R. Maybe you don’t need the entire dataset but only a subset of columns? Or you want to manually specify what types of variables you want? Or maybe the dataset you are using does not have variable names? You can specify all of these things as arguments to the loading functions. You can look at the documentation to lean more about them.
Data coming from statistical software can’t be loaded in such a simple way as above. Mainly because it stores more information. Here we’ll focus on .sav format which is used by SPSS. Apart from rows and columns (observations and variables) .sav format stores additional information e.g. on value labels - it is able to attach labels to numbers (like in Likert scales 1 can refer to ‘strongly disagree’). SPSS also allows to specify user-defined missing values (a common practice is to e.g. code missing values with 99).
This means we have to somehow deal with this additional information when loading .sav files. There are essentially two ways to go about it: reduce the amount of information stored or introduce a new type of values that can store this additional information. This first approach is taken by the foreign package. The second one is taken by haven package. Each of these approaches has some advantages and drawbacks. Stripping labels from values is generally easier and keeps consistency in terms of types of objects you are dealing with. You just keep working with numeric, integer, character or boolean values. The downside is that you lose some information and when e.g. saving a data file back to .sav format you can’t restore them. It might also cause problems if some variables need to be reversed beause you will lose information about that. Using haven allows you to keep all the labels but this is achieved by creating a new type of object: haven-labelled. Thanks to this you can keep all the information e.g. when saving back to .sav format. Most of the time you shouldn’t have problems with this new type of object but some statistical procedures from other packages might not like this and refuse to work. If that happens you can strip the labels using zap_labels().
The main functiom in foreign is read.spss(). By default foregin will load data into a list rather than a dataframe. You can load into a data frame by setting the argument to.data.frame to TRUE. Another useful argument is use.value.labels which, if set to TRUE will convert the numerical values stored in .sav into their corresponding labels. This is the way foreign deals with labelled values: you can use either numeric values or their labels. In the documentation of the function you can read about additional arguments that control handling of labels.
Please notice though that the documentation for read.spss() function in foreign states that it was originally developed in 2000 and does not guarantee compatibility with newer versions of SPSS (whcih hasn’t changed much since but still. At least it doesn’t look like windows xp anymore).
To look at loading .sav files we’ll use a trimmed version (the whole file is huge and we don’t need it) of World Value Survey wave 6:
library(foreign)
df_foreign <- read.spss("data/WVS6.sav", to.data.frame = TRUE)Once we have the data loaded lets see what types of values we have.
str(df_foreign[,1:3])'data.frame': 89565 obs. of 3 variables:
$ V1 : Factor w/ 7 levels "1981-1984","1989-1993",..: 6 6 6 6 6 6 6 6 6 6 ...
$ V2 : Factor w/ 61 levels "Not asked in survey",..: 2 2 2 2 2 2 2 2 2 2 ...
$ V2A: Factor w/ 61 levels "Algeria","Azerbaijan",..: 1 1 1 1 1 1 1 1 1 1 ...Haven package deals differently with loading labeled variables. It introduces a new type of variable: haven-labelled data. It is capable of storing both numeric values and labels attached to it by adding an attribute to the variable with labels. You can load a file in haven using read_spss() (notice the underscore).
library(haven)
df_haven <- read_spss("data/WVS6.sav")Now that we have loaded the dataset, lets look at the types of variables we have.
str(df_haven[,1:3])tibble [89,565 × 3] (S3: tbl_df/tbl/data.frame)
$ V1 : dbl+lbl [1:89565] 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6...
..@ label : chr "Wave"
..@ format.spss: chr "F2.0"
..@ labels : Named num [1:7] 1 2 3 4 5 6 7
.. ..- attr(*, "names")= chr [1:7] "1981-1984" "1989-1993" "1994-1998" "1999-2004" ...
$ V2 : dbl+lbl [1:89565] 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, ...
..@ label : chr "Country/region"
..@ format.spss: chr "F4.0"
..@ labels : Named num [1:61] -4 12 31 32 36 51 76 112 152 156 ...
.. ..- attr(*, "names")= chr [1:61] "Not asked in survey" "Algeria" "Azerbaijan" "Argentina" ...
$ V2A: dbl+lbl [1:89565] 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, ...
..@ label : chr "Country/regions [with split ups]"
..@ format.spss: chr "F3.0"
..@ labels : Named num [1:61] 12 31 32 36 51 76 112 152 156 158 ...
.. ..- attr(*, "names")= chr [1:61] "Algeria" "Azerbaijan" "Argentina" "Australia" ...
- attr(*, "label")= chr "filelabel"And lets see what happens if we zap the labels.
df_nolab <- zap_labels(df_haven)
str(df_nolab)tibble [89,565 × 8] (S3: tbl_df/tbl/data.frame)
$ V1 : num [1:89565] 6 6 6 6 6 6 6 6 6 6 ...
..- attr(*, "label")= chr "Wave"
..- attr(*, "format.spss")= chr "F2.0"
$ V2 : num [1:89565] 12 12 12 12 12 12 12 12 12 12 ...
..- attr(*, "label")= chr "Country/region"
..- attr(*, "format.spss")= chr "F4.0"
$ V2A : num [1:89565] 12 12 12 12 12 12 12 12 12 12 ...
..- attr(*, "label")= chr "Country/regions [with split ups]"
..- attr(*, "format.spss")= chr "F3.0"
$ COW : num [1:89565] 615 615 615 615 615 615 615 615 615 615 ...
..- attr(*, "label")= chr "COW Country Code"
..- attr(*, "format.spss")= chr "F2.0"
$ C_COW_ALPHA : chr [1:89565] "ALG" "ALG" "ALG" "ALG" ...
..- attr(*, "label")= chr "Country code CoW alpha"
..- attr(*, "format.spss")= chr "A8"
$ B_COUNTRY_ALPHA: chr [1:89565] "DZA" "DZA" "DZA" "DZA" ...
..- attr(*, "label")= chr "Country code ISO 3166 alpha"
..- attr(*, "format.spss")= chr "A8"
$ V10 : num [1:89565] 2 2 2 2 1 2 2 2 2 1 ...
..- attr(*, "label")= chr "Feeling of happiness"
..- attr(*, "format.spss")= chr "F3.0"
$ V11 : num [1:89565] 1 2 2 1 3 1 2 1 2 1 ...
..- attr(*, "label")= chr "State of health (subjective)"
..- attr(*, "format.spss")= chr "F3.0"
- attr(*, "label")= chr "filelabel"One last thing about the haven package. SPSS has the habit of storing categorical variables as labelled values (e.g. 1 - blue, 2 - brown). This can be a big problem because R will try to treat them as numeric values. This can lead to some ridiculous errors in analysis. To change a variable from haven-labelled to a factors with values taken from labels you can use as_factor() (again notice the underscore instead of dot).
One additional thing you have to take into account when loading data from excel is that it can store a number of sheets in a single file. This has to be taken into account when loading such file into R.
One of the packages available for loading excel data is readxl.
The main difference in excel files is that they can store multiple sheets in one file. Because of that we need to specify which sheet we want to load. You can inspect the names of sheets with excel_sheets().
library(readxl)
excel_sheets("data/pop.xlsx")[1] "pop"Once we know the sheet names we can load the one that interests us. We need to specify the path to the file and which sheet we want to load. There are of course many other arguments that give additional control over what is loaded but they are quite similar to other loading functions. You can read more on them in the documentation of the function. If you need to load multiple sheets you can save the sheet names as an object and then loop over them using lapply().
df_excel <- read_excel("data/pop.xlsx", sheet = "pop")Now we can look at the top of the file:
head(df_excel)# A tibble: 6 × 302
country `1800` `1801` `1802` `1803` `1804` `1805` `1806` `1807` `1808` `1809`
<chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 Afghani… 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M 3.28M
2 Angola 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M 1.57M
3 Albania 400k 402k 404k 405k 407k 409k 411k 413k 414k 416k
4 Andorra 2650 2650 2650 2650 2650 2650 2650 2650 2650 2650
5 UAE 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k 40.2k
6 Argenti… 534k 520k 506k 492k 479k 466k 453k 441k 429k 417k
# ℹ 291 more variables: `1810` <chr>, `1811` <chr>, `1812` <chr>, `1813` <chr>,
# `1814` <chr>, `1815` <chr>, `1816` <chr>, `1817` <chr>, `1818` <chr>,
# `1819` <chr>, `1820` <chr>, `1821` <chr>, `1822` <chr>, `1823` <chr>,
# `1824` <chr>, `1825` <chr>, `1826` <chr>, `1827` <chr>, `1828` <chr>,
# `1829` <chr>, `1830` <chr>, `1831` <chr>, `1832` <chr>, `1833` <chr>,
# `1834` <chr>, `1835` <chr>, `1836` <chr>, `1837` <chr>, `1838` <chr>,
# `1839` <chr>, `1840` <chr>, `1841` <chr>, `1842` <chr>, `1843` <chr>, …There are situations in which you might work with data that does not come from simple tables but is stored in completely different way. One example that we’ll introduce here is the json format. Json is short for Javascript Object Notation and is a common way of storing data in the web.
Json stores data as key - value pairs. These might not approximate tabular format and can be nested and fairly complicated. This type of data is especially common when downloading data directly from the web (e.g. social media data) or from APIs. You can imagine a file that stores information on each user of a website: their username, password and all posts that they have created along with information on each post like their creation date. It might look something like this:
Because json can be a complicated and nested structure the type of data that best approximates it in R is a list. There are ways to ask R to try and handle such list structure and try to convert it into a data frame but it does not always work. Cleaning an unevenly nested json can be a real pain sometimes!
We’ll look at an example of a NASA API that stores information on the number of people currently present on space stations. The package we’ll use to load the data is jsonlite.
library(jsonlite)
nasa <- fromJSON("http://api.open-notify.org/astros.json")Notice how the loaded object looks like. It is a list with 3 elements: number of people as an integer, a dataframe of astronauts and a string “success”.
nasa$people
craft name
1 ISS Oleg Kononenko
2 ISS Nikolai Chub
3 ISS Tracy Caldwell Dyson
4 ISS Matthew Dominick
5 ISS Michael Barratt
6 ISS Jeanette Epps
7 ISS Alexander Grebenkin
8 ISS Butch Wilmore
9 ISS Sunita Williams
10 Tiangong Li Guangsu
11 Tiangong Li Cong
12 Tiangong Ye Guangfu
$number
[1] 12
$message
[1] "success"attention variable in each experimental condition.