Current Slide

Small screen detected. You are viewing the mobile version of SlideWiki. If you wish to edit slides you will need to use a larger device.

RDD Operations

Word Count example
 
 
 
  
SparkContext
val textFile = sparkSession.sparkContext.textFile("hdfs://...")
val wordCounts = textFile.flatMap(line => line.split(" "))
  .filter(!_.isEmpty())
  .map(word => (word1))
  .reduceByKey(_ + _) //(a, b) => a + b
wordCounts.take(10)
HadoopRDD
MapPartittionRDD
MapPartittionRDD
MapPartittionRDD
ShuffledRDD
Value
Directed Acyclic Graph (DAG) for Word Count example

Content Tools

Sources

There are currently no sources for this slide.