Learning PySpark

A Brief Primer on PySpark 6 lectures 14:55 The Course Overview This video gives an overview of the entire course. Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository A Brief Primer on PySpark 6 lectures 14:55 The Course Overview This video gives an overview of the entire course. Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository The Course Overview This video gives an overview of the entire course. The Course Overview This video gives an overview of the entire course. The Course Overview This video gives an overview of the entire course. The Course Overview This video gives an overview of the entire course. This video gives an overview of the entire course. This video gives an overview of the entire course. Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Brief Introduction to Spark The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface The aim of the video is to explain Spark and its Python interface. • Learn about Spark • Explain its popularity • Touch upon the Python interface Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Apache Spark Stack The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming The aim of this video is to provide a brief overview of Apache Spark stack components. • Apache Spark ecosystem introduction • Introduce SparkSQL and DataFrames • Explain the overview of remaining components – MLlib, GraphX, and streaming Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Spark Execution Process The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph The aim of this video is to briefly review the execution process. • Explain the overview of interactions between the driver and workers • Discuss how Spark decides which tasks run in parallel • Represent the job execution plan as a directed acyclic graph Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Newest Capabilities of PySpark 2.0+ The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming The aim of this video is to briefly review the newest features of Spark 2.0+. • Introduce the phase two of the project Tungsten • Instantiate Spark session instead of three other contexts • Brief overview of the Spark structured streaming Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository Cloning GitHub Repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository The aim of this video is to clone the GitHub repository for the course. Doing this will set everything we need for the following videos. • Find the repository on GitHub • Open terminal • Clone the repository Resilient Distributed Datasets. 11 lectures 38:48 Brief Introduction to RDDs In this video, we will provide a brief overview of one of the fundamental data structures of Spark – the RDDs. • Explain the system requirements • What are RDDs • Learn about Spark’s Python API Creating RDDs In this video, we will learn how to create RDDs in many different ways. • Parallelize a list of integers • Parallelize a list of tuples • Read from files Schema of an RDD In this video, we explore the advantages and disadvantages of RDD’s lack of schema. • Explain how RDDs handles structured data • Explain how RDDs handles unstructured data • Explain how RDDs handles semi-structured highly-heterogeneous data Understanding Lazy Execution Spark is lazy to process data. In this video we will learn why this is an advantage. • Execute a transformation • Execute an action • Track the execution process Introducing Transformations – .map(…) In this video, we will introduce lambdas and the .map(…) transformation. • Define functions to use inside the .map(…) transformation • Use Lambda inside the .map(…) transformation • What is the .map(…) transformation after all Introducing Transformations – .filter(…) In this video, we will learn how to filter data from RDDs. • Use the RDD from the .map(…) video • Subset the number of elements • Use .filter(…) to remove labels row Introducing Transformations – .flatMap(…) In this video, we will explain the difference between .flatMap(…) and .map(…) transformations and we will learn to use it to filter malformed records. • Explain the difference between .map(…) versus .flat Map(…) • How does the .flatMap(…) work • Use the .flatMap(…) to filter malformed records Introducing Transformations – .distinct(…) In this video, we will explore what the .distinct(…) transformation does. • Create an unordered list of integers • Explain the way a distinct items are found • Use the .distinct(…) transformation Introducing Transformations – .sample(…) In this video, we will learn how to sample data from RDDs. • Check how many records are in an RDD • Decide what proportion of records to return • Use the .sample(…) transformation to sample without replacement Introducing Transformations – .join(…) In this video, we will learn how to join two RDDs. • Populate two RDDs with some random data • The first element of the record is the key, the remained is values • Use the .join(…) transformation to join two RDDs Introducing Transformations – .repartition(…) In this video, we will explore how to effectively use repartitioning. • Use data read earlier • Check how many partitions • Use the • Subset the number of elements • Use .filter(…) to remove labels...