Spark ML

Spark MLlib Tutorials

Spark MLlib is a machine learning (ML) library for Apache Spark. The intention is to make building machine learning applications more scalable and easy.

The library provides:

  1. Common learning algorithms such as classification, regression, clustering, and collaborative filtering
  2. Featurization including extraction, transformation, dimensionality reduction, and selection
  3. Pipelines: tools for constructing, evaluating, and tuning ML Pipelines
  4. Persistence: saving and load algorithms, models, and Pipelines
  5. Utilities: linear algebra, statistics, data handling, etc.

Spark ML Tutorials

What are the differences between Spark ML and Spark MLlib?

Essentially, the difference between Spark ML and Spark MLlib comes down to which abstraction is supported. Spark ML has evolved similarly to other Spark extensions and Spark Core with APIs based on RDD, DataFrame, or DataSet. Choosing ML vs. MLlib basically comes down to whether or not you wish to use RDD or DataFrame APIs.

Spark MLlib supports RDD-Based API and is located in the spark.mllib package. MLlib will not add new features to the RDD-based API.

Spark ML supports DataFrame API with implementation located in the org.apache.spark.ml Scala package. As with similar changes in other areas of Spark, there are many benefits of DataFrames including Tungsten and Catalyst optimizations, and uniform APIs across languages.

Spark ML is now the primary API for MLlib and should be used for any new projects.

Installation and Usage

The Spark MLlib tutorials listed below are designed to help you get started with machine learning in Apache Spark. As with other Spark Tutorials on this site, please pay attention to whether the API is RDD or DataFrame/DataSet based.

Spark ML APIs are available as Maven Artifacts similar to Spark Structured Streaming and Spark SQL.

For examples of various usage in Maven and other build tools such as SBT, check out https://search.maven.org/artifact/org.apache.spark/spark-mllib_2.13/3.3.1/jar

Spark Machine Learning Example with Scala

by
Spark Machine Learning Example

In this Apache Spark Machine Learning example, Spark MLlib is introduced and Scala source code analyzed.  This post and accompanying screencast videos demonstrate a custom Spark MLlib Spark driver application.  Then, the Spark MLLib Scala source code is examined.  Many topics are shown and explained, but first, let’s describe a few machine learning concepts. Machine … Read more

Spark Machine Learning – Chapter 11 Machine Learning with MLlib

by
Spark Machine Learning

Spark Machine Learning is contained with Spark MLlib.  Spark MLlib Spark’s library of machine learning (ML) functions designed to run in parallel on clusters.  MLlib contains a variety of learning algorithms. The topic of machine learning itself could fill many books, so instead, this chapter explains ML in Apache Spark. This post is an excerpt … Read more