Below you will find spark streaming tutorials below are available in both Scala and Python.
But, before diving into the specific tutorials, please ensure you know the following Spark Streaming fundamentals first.
Streaming is an extension of core Apache Spark APIs and provides fault-tolerant stream processing from sources like Kafka, Kinesis, or TCP sockets. Streaming applications run continuously as opposed to the traditional Spark batch processing jobs which are scheduled to run and stop periodically.
Streaming provides API functions such as map, reduce, join, and window for creating the desired output of the continuous running stream.
Functions may be chained together to customize the result. The results of Streaming applications may be written further downstream to object stores, databases, or even back to the origin streaming storage event log such as Kafka or Kinesis where it can be consumed from another application.
When Spark Streaming was first launched in Spark 0.7, it only allowed processing time processing, which is based on the time the data is processed by Spark Streaming, regardless of the timestamps associated with the data.
Beginning with Spark 1.2, Spark Streaming added support for event time processing, which is based on the time an event happened, as indicated by a timestamp linked with the event data.
Event time processing functionality enables Spark Streaming to manage out-of-order events and late-arriving data, which is critical in many real-world use cases.
Spark Streaming included a new API for setting windows and watermarks to support event time processing, allowing you to describe how to group data based on event timestamps and how to handle out-of-order events and late-arriving data.
For more on windows in stream processing
The Spark Streaming windowing and watermarking capabilities are designed to interact with event time processing and handle streaming data based on event time.
To begin, you must understand streaming has undergone two significant types of releases over its short history.
The original release was called Spark Streaming while the second, modern release is called Spark Structured Streaming.
If you are new to streaming with Spark, this means two key takeaways for you. One, as you are learning, make sure to understand which version of streaming tutorial you are reading; i.e. is it Structured Streaming or not? Two, if given a choice, use Structured Streaming for any new development for performance and newer API reasons.
Structured Spark Streaming engine built upon the Spark SQL engine. This means using the more modern Dataset/DataFrame API in Scala, Java, Python or R to build custom filters, aggregations, event-time windows, stream-to-batch, stream-to-stream joins, etc. We cover many examples of building these applications on this site.
Outside of Databricks, which most people know, there are options from both AWS via EMR as well as Azure. Spark Streaming is also used in the execution engine for managed products such as AWS Glue streaming.
Let me know if I’m missing any others.
There are many options for creating continuously running stream processing applications, but if we limit comparison to distributed applications running on more than one node, the most popular alternative to Spark Structured Streaming is Apache Flink.
In this post, let’s explore an example of updating an existing Spark Streaming application to newer Spark Structured Streaming. We will start simple and then move to a more advanced Kafka Spark Structured Streaming examples. My original Kafka Spark Streaming post is three years old now. On the Spark side, the data abstractions have evolved … Read more
If you are looking for Spark with Kinesis example, you are in the right place. This Spark Streaming with Kinesis tutorial intends to help you become better at integrating the two. In this tutorial, we’ll examine some custom Spark Kinesis code and also show a screencast of running it. In addition, we’re going to cover … Read more
Spark Streaming with Kafka is becoming so common in data pipelines these days, it’s difficult to find one without the other. This tutorial will present an example of streaming Kafka from Spark. In this example, we’ll be feeding weather data into Kafka and then processing this data from Spark Streaming in Scala. As the data … Read more
How do you create and automate tests of Spark Streaming applications? In this tutorial, we’ll show an example of one way in Scala. This post is heavy on code examples and has the added bonus of using a code coverage plugin. Are the tests in this tutorial examples unit tests? Or, are the examples integration … Read more
Let’s write a Spark Streaming example which streams from Slack in Scala. This tutorial will show how to write, configure and execute the code, first. Then, the source code will be examined in detail. If you don’t have a Slack team, you can set one up for free. We’ll cover that too. Sound fun? … Read more
Let’s start Apache Spark Streaming with Scala with small steps to build up our skills and confidence. These small steps will create the forward momentum needed when learning new skills. The quickest way to gain confidence and momentum in learning new software development skills is executing code that performs without error. In this post, … Read more
