Learning Path: R: Powerful Data Analysis with R

Learning Data Analysis with R. 75 lectures 05:59:42 Learning Data Analysis with R -The Course Overview This video provides an overview of the entire course. Importing Data from Tables (read.table) Accessing and importing open access environmental data is a crucial skill for data scientists. This section teaches you how to download data from the Web, import it in R and check it for consistency. • Download open-access data from the USGS website • Import it in R using read.table • Check its structure to start exploring the data Downloading Open Data from FTP Sites Often times, datasets are provided for free, but on FTP, websites and practitioners need to be able to access them. R is perfectly capable of downloading and importing data from FTP sites. • Understand the basics of downloading data in R • Download the data with the download.file function • Learn how to handle compressed formats Fixed-Width Format Not all text files can be opened easily with read.table. The fixed-width format is still popular but requires a bit more work in R. • Understand the fixed-width format • Identify the main stricture of the dataset • Import the file in R Importing with read.lines (The Last Resort) Some data files are simply too difficult to be imported with simple functions. Luckily R provides the readLines function that allows importing of even the most difficult tables. • Understand where we need to use readLines • Read the data in strings • Work with strings to import the file Cleaning Your Data Most open data is generated automatically and therefore may contain NA or other values that need to be removed. R has various functions to deal with this problem. • Reiterate the use of the readLines function • Collect data in data frames • Clean the dataset Loading the Required Packages To follow the exercises in the book viewers would need to install several important packages. This video will explain how to do and where to find information about them. • Check the CRAN website for info about packages • Install/load packages in R • Find additional information Importing Vector Data (ESRI shp and GeoJSON) Vector data are very popular and widespread and require some thoughts before importing. R has dedicated tools to import these data and work with them. • Work with shapefiles • Differences between rgdal and raster • GeoJSON, the format for web developers Transforming from data.frame to SpatialPointsDataFrame Often times, spatial data is provided in tables and needs to be transformed before it can be used for analysis. This can be done simply with the sp package. • Check the table structure to identify coordinates • Transform a table into a spatial object • Plot the data to check if the process was successful Understanding Projections Geographical projections are very important and need to be handled carefully. R provides robust functions to do so successfully. • Understand projections • Identify the data projection if unknown • Set the projection of the file Basic time/dates formats Many datasets have a temporal component and practitioners need to know how to deal with it. R provides functions to do that in a very easy way. • Identify the time variable • The basic Date format • The more advanced POSIXct format Introducing the Raster Format Raster data is fundamentally different from vector data, since its values refer to specific areas (cells) and no single locations. This video will clearly explain this difference and teach users how to import this data in R. • Explain what raster data is • Importing with rgdal • Introducing the raster package Reading Raster Data in NetCDF The NetCDF format is becoming very popular, since it allows to store 4D datasets. This requires some technical skills to be accessed and this video will teach viewers to open and import NetCDF files. • Gather open NetCDF data from the Web • Understand the format • Open it with R Mosaicking Many raster datasets we download from the web are distributed in tiles, meaning a single raster for each subset of the area. To obtain a full raster for the study area we are interested to cover we can create a mosaic. • Download raster DTMs • Understand the process of mosaicking • Create a full DTM Stacking to Include the Temporal Component Mosaicking involves merging rasters based on location. Spatio-temporal datasets include also multiple rasters for the same location but different times. To merge these we need to use the stacking function. • Download NDVI data • Handle the temporal component • Create a stack dataset Exporting Data in Tables Once we complete our analysis we often need to export our results and share them with colleagues. Popular formats are CSV and TXT files, which we learn how to export in this video. • Subset a dataset • Export in CSV • Export in TXT Exporting Vector Data (ESRI shp File) If we work with vector data and we want to share the same format with our co-workers, we need to learn how to export in vector formats. This will be covered here. • Export ESRI shapefiles • Understand the process • Open our results in a GIS Exporting Rasters in Various Formats (GeoTIFF, ASCII Grids) Many raster datasets we download from the Web are distributed in tiles, meaning a single raster for each subset of the area. To obtain a full raster for the study area we are interested in covering, we can create a mosaic. • Download raster DTMs • Understand the process of mosaicking • Create a full DTM Exporting Data for WebGIS Systems (GeoJSON, KML) Nowadays WebGIS applications are extremely popular. However, to use our data for WebGIS, we first need to export them in the correct format. This video will show how to do that. • Export data in GeoJSON • Export in KML • Open our data on Google Maps Preparing the Dataset In the previous volume we explored the basics R functions and syntaxes to import various types of data. In this video we will put these functions together, and overcome some unexpected challenges, to import a full year of NOAA data. • Download the raw data and import them • Find the coordinates and merge two data.frames • Save the cleaned dataset for later use Measuring Spread (Standard Deviation and Standard Distance) Before we can start analyzing our data we first need to properly understand what we are dealing with. The first step we have to take in this direction is describe our data with simple statistical indexes. • Measure central tendency • Measure spread • Summarize our data Understanding Your Data with Plots Numerical summaries are very useful but certainly not ideal to provide us with a direct feeling for the dataset in hands. Plots are much more informative and thus being able to produce them is certainly a crucial skill for data analysts. • Download the EPA data • Produce histograms • Produce density plots Plotting for Multivariate Data For multivariate data we are often interested in assessing correlation between variables. This can be done in R very easily, and ggplot2 can also be used to produce more informative plots. • Assess multiple correlations at once • Plot scatterplots by state • Customize scatterplots to include 3 variables Finding Outliers Detecting outliers is another basic skill that every data analyst should have and master. R provides a lot of technical tools to help us in finding outliers. • Understanding outliers • Finding outliers with standard deviation and mean absolute deviation • Box-plot provides another handy way to detect outliers Manipulating Vector Data - Introduction This Section will be dedicated entirely to manipulating vector data. However, viewers first need to familiarize with some basic concepts, otherwise they may not be able to understand the rest of the section. • Understand the concept of bounding box • Understand the concept of centroid • Subset spatial objects by attribute Re-Projecting Your Data In volume 1 we learned how to set the projection of our spatial data. However, in many cases we have to change this projection to successfully complete our analysis, and this requires some specific knowledge. • Understand that bounding boxes and centroids can be calculated for polygons too • Re-project spatial objects • Calculate area and perimeter of a polygon Intersection In many cases we may be interested in understanding the relation between spatial objects. One of such relations is the intersection, where we first want to know how two objects intersect, and then also extract only the part of one of these object that is included or outside the first. • Test intersections in R • Extract only the part of the object included in the first • Extract only the part of the object outside the first Buffer and Distance Other important GIS operations that users have to master involve creating buffers and calculating distances between objects. • Create a buffer around polygons • Calculate distance between points • Calculate distance between polygons Union and Overlay The last two GIS functions that anybody should master are used to merge different geometries and spatial objects and overlay. • Merge geometries of the same type • Merge different geometries • Overlay and select by location Manipulating Raster Data - Introduction Raster objects are imported in R as rectangular matrixes. Users needs to be aware of this to properly work on these data, otherwise it may create some issues during the data analysis. • Understand raster data • Perform descriptive statistics on ra ster data • Re-projecting raster data Converting Vector/Table Data into Raster In many cases open data are not distributed directly in raster formats and they need to be converted. This can be easily done with the right functions. • Convert data.frames into rasters • Convert spatial data into rasters • Convert rasters into matrix or spatial data Subsetting and Selection Working with raster data often means extracting data for particular locations for further analysis, or crop the data to reduce their size. These are essential skills to master for any data analyst. • Extract values from rasters • Use these data for analysis • Clipping Filtering Sometimes we may need to filter out some values of our raster. It may seem tricky but only because it requires some skills. • Filter temperature data • Aggregate rasters • Disaggregation Raster Calculator Creating new raster by calculating their value is extremely important for spatial data analysis. Doing so is simple but can be difficult to understand at first. • A simple raster calculation • Calculate slope and aspect • Advanced calculation with shaded relief Plotting Basics Syntactically plotting spatial data in R is no different than plotting other types of data. Therefore, users need to know the basics of plotting before they can start making maps. • Plotting symbols • Plotting colors • Save plots Visualizing Spatial Data - Adding Layers Creating multilayer plot can be difficult because we need to take care of several different aspects at once. However, learning that is very easy. • Create multilayer plots • Understanding the layer system • Zooming and saving Color Scale When plotting spatial data we are often interested in using colors to show the values of some variables. This can be done manually but producing the right color scale may be difficult. This issue can be solved employing automatic methods. • Creating a manual color scale • Understanding the plotting window • Automatic color scale Creating Multivariate Plots Creating multivariate plots not only means adding layers, but also using legends so that the viewer understands what the plot is showing. Creating legends in R is tricky because it requires a lot of tweaking, which will be explained here. • Change size and add title • Create simply legend • Add another legend column Handling the Temporal Component Temporal data need to be treated with specific procedures to highlight this additional component. This may be done in different ways depending on the scope of the analysis and R provides the right platform for this. • Extracting the temporal information • Plot multiple images according to specific times • Time distances Interactive Maps - Introduction Being able to plot spatial data on web maps is certainly helpful and a crucial skill to have, but it can be difficult since it requires knowledge of different technologies. R makes this process very easy with dedicated functions that allow us to plot on web GIS services a breeze. • Understand web mapping • Mapping platforms • Required packages Plotting Vector Data on Google Maps Plotting data with the function plotGoogleMaps is not as easy as using the function plot. With a simple step by step guide we can achieve good command of the function, so that users can plot whatever data they choose. • Install plotGoogleMaps • Create your first map • Customize the plotting window Interactive Maps - Adding Layers An interactive map with just one layer is hardly useful for our purposes. Many times we are faced with the challenge of plotting several data at once. This requires some additional work and understanding, but it is definitely not hard in R. • Understand the layer system • Add layers with the right options • Check the result Plotting Raster Data on Google Maps Plotting raster data on Google maps can be tricky. The function plotGoogleMaps does not handle rasters very well and if not done correctly the visualization will fail. This video will show users how to plot rasters successfully. • Download the seismic risk map • Understand the limitations of plotting rasters on Google Maps • Plotting rasters successfully Using Leaflet to Plot on Open Street Maps Plotting on Google Maps is easy but Google Maps are commercial products therefore if we want to use the on our commercial website we would need to pay. OpenStreetMaps are free to use, therefore knowing how to use them is certainly an advantage N • Plot and...