Spark Structured Streaming with Kafka Example – Part 1

Spark Structured Streaming with Kafka Examples

In this post, let’s explore an example of updating an existing Spark Streaming application to newer Spark Structured Streaming.  We will start simple and then move to a more advanced Kafka Spark Structured Streaming examples. My original Kafka Spark Streaming post is three years old now.  On the Spark side, the data abstractions have evolved […]

Spark Kinesis Example – Moving Beyond Word Count

spark kinesis

If you are looking for Spark with Kinesis example, you are in the right place.  This Spark Streaming with Kinesis tutorial intends to help you become better at integrating the two. In this tutorial, we’ll examine some custom Spark Kinesis code and also show a screencast of running it.  In addition, we’re going to cover […]

Spark Streaming with Kafka Example

Spark Streaming with Kafka

Spark Streaming with Kafka is becoming so common in data pipelines these days, it’s difficult to find one without the other.   This tutorial will present an example of streaming Kafka from Spark.  In this example, we’ll be feeding weather data into Kafka and then processing this data from Spark Streaming in Scala.  As the data […]

Spark Streaming Testing with Scala Example

Spark Streaming Testing

Spark Streaming Testing How do you create and automate tests of Spark Streaming applications?  In this tutorial, we’ll show an example of one way in Scala.  This post is heavy on code examples and has the added bonus of using a code coverage plugin. Are the tests in this tutorial examples unit tests?  Or, are […]

Spark Streaming Example – How to Stream from Slack

Spark Streaming Example

Let’s write a Spark Streaming example which streams from Slack in Scala.  This tutorial will show how to write, configure and execute the code, first.  Then, the source code will be examined in detail.  If you don’t have a Slack team,  you can set one up for free.   We’ll cover that too.  Sound fun?  […]

Spark Streaming with Scala

Spark Streaming with Scala

Let’s start Apache Spark Streaming with Scala with small steps to build up our skills and confidence.  These small steps will create the forward momentum needed when learning new skills.  The quickest way to gain confidence and momentum in learning new software development skills is executing code that performs without error.  Right?  I mean, right!?  […]