Apache Spark: what it does, current releases and how to try it

Updated November 2026

Apache Spark is an engine for processing large amounts of data on a cluster of machines: you write SQL or DataFrame code in Python, Scala, Java or R, and Spark splits the work into tasks that run in parallel. As of November 2026 the newest release is Spark 4.3.0 (October 30, 2026), with recent maintenance releases for the 4.2, 4.1, 4.0 and 3.5 lines.

What Spark is

Spark reads data from files (Parquet, CSV, JSON and others), databases and streams, transforms it, and writes the result somewhere else. The same program runs on a laptop with a few megabytes or on hundreds of machines with terabytes; only the cluster settings change. The main pieces, all in one download:

The older RDD-based Spark Streaming (DStreams) API is still documented, and SparkR is deprecated.

Typical uses

Architecture in plain words

Every Spark application has one driver and a set of executors:

Driver
The process that runs your main() function or notebook and creates the Spark session. It turns your code into a plan, cuts the plan into tasks (one unit of work for one executor), and sends the tasks out. It serves a web UI, normally on port 4040, showing jobs, tasks and memory use.
Executors
Processes on worker machines that run tasks in several threads and keep data in memory or on disk between steps. Each application gets its own executors for its whole lifetime.
Cluster manager
Hands out machines to applications. Spark supports its own simple standalone manager, Hadoop YARN, and Kubernetes. Without one, --master "local[N]" runs everything in one process with N threads.

One practical consequence: executors connect back to the driver for the whole run, so the driver must be reachable over the network from the worker nodes. A driver on a laptop behind NAT cannot drive a remote cluster; that is one of the problems Spark Connect solves.

A rough example of the parallelism: a 100 GB Parquet data set read in 128 MB pieces becomes about 800 tasks (100 × 1,024 ÷ 128 = 800). With 20 executors of 4 cores each, 80 tasks run at once, so the read happens in about 10 waves.

Current versions and support

Releases on the Apache download server, as of November 2026:

Apache Spark release lines, as of November 2026
LineLatest releaseReleasedNotes
4.34.3.0October 30, 2026First release on the new quarterly cadence
4.24.2.1October 27, 20264.2.0 released July 14, 2026; 18-month maintenance window
4.14.1.4October 27, 2026Maintenance release
4.04.0.5October 27, 2026Maintenance release
3.53.5.9July 16, 2026Extended LTS, security fixes only, until November 2027

The versioning policy changed in 2026. Starting with 4.3.0, Spark plans a release every three months and a new major version once a year. Major and feature releases get 6 months of bug-fix releases; the last release of each major line is the long-term-support (LTS) release with 18 months. For the 4.x line the LTS will be 4.5.0, not 4.3.0. Critical security patches are backported to all actively maintained branches. The policy page does not give separate end dates for 4.0 and 4.1, so check the news page for new maintenance releases before you plan around them.

Spark 4.3 runs on Java 17, 21 or 25, Scala 2.13 and Python 3.11 or later (R 4.0+ is deprecated). Spark 4 dropped Scala 2.12; Spark 3.5 is built for Scala 2.12 by default with a separate 2.13 build. Code compiled against Spark's Scala API must use the same Scala version as your Spark build.

Install and try it

Python only: pip

The quickest way to try Spark locally is the PySpark package, which bundles Spark itself (it still needs a Java runtime, version 17 or later):

python3 -m venv ~/spark-venv
. ~/spark-venv/bin/activate
pip install pyspark==4.3.0
python - <<'EOF'
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[2]").appName("test").getOrCreate()
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "n"])
df.groupBy("key").sum("n").show()
spark.stop()
EOF

Expected output is a small table with a = 4 and b = 2.

Full distribution

The binary package built with Hadoop 3 client libraries is the usual choice:

curl -O https://dlcdn.apache.org/spark/spark-4.3.0/spark-4.3.0-bin-hadoop3.tgz
curl -O https://dlcdn.apache.org/spark/spark-4.3.0/spark-4.3.0-bin-hadoop3.tgz.sha512
sha512sum -c spark-4.3.0-bin-hadoop3.tgz.sha512
tar -xzf spark-4.3.0-bin-hadoop3.tgz
cd spark-4.3.0-bin-hadoop3

./bin/run-example SparkPi 10          # prints "Pi is roughly 3.14..."
./bin/pyspark --master "local[2]"     # interactive Python shell
./bin/spark-shell --master "local[2]" # interactive Scala shell
./bin/spark-submit examples/src/main/python/pi.py 10

Other packages on the same page: -bin-without-hadoop (bring your own Hadoop libraries), a -connect variant, and the source tarball. Container images are published on Docker Hub as apache/spark and as the Docker Hub library image spark; the Spark project notes that they contain non-ASF software under other licenses.

Security notes

Off by default. The Spark documentation states that security features like authentication are not enabled by default and that no deployment mode is secure by default. Anyone who can submit a job can run arbitrary code on the cluster; the project treats that as by design, not a bug.

Minimum steps for any shared cluster, in conf/spark-defaults.conf:

spark.authenticate              true
spark.network.crypto.enabled    true
spark.network.crypto.cipher     AES/GCM/NoPadding

Vulnerabilities are listed on the project's security page; reports go to security@apache.org (the ASF security team, which involves the Spark PMC). Release announcements appear on the news page and the user mailing list (user-subscribe@spark.apache.org). When moving between minor versions, read the migration guide in the documentation for the target version; SQL behaviour and configuration defaults sometimes change.

Sources