Rust
Native SparkSession or remote SparkConnectSession, with the same lazy DataFrames, Columns, typed FromSparkRow decoding, and Arrow results. The Rust API also runs interactively in browser WebAssembly.
A Rust implementation of Apache Spark with first-class Rust, Python, Java, Scala, SparkR-compatible R, TypeScript/Node, Ruby, and SQL APIs. It runs as a 150 MiB Spark Connect binary or a 136 MiB embedded Python extension in-process — think DuckDB — or scales out across a cluster as a Spark-compatible distributed engine, or runs entirely in your browser via WebAssembly. Same code, same plans, four deployment shapes.
DuckDB set the bar for embedded analytics — a single binary with vectorized execution — but it stops at one machine. Apache Spark defined the API surface and deployment model for distributed compute — but it cannot run inside your process. Outgrowing one has meant rewriting for the other. spark-rust closes that gap: one Rust query engine, four packagings, sharing the same analyzer, optimizer rules, physical plans, spill operators, and MLlib bindings — embedded on a laptop or scaled across a cluster, without switching engines.
pip install sparkrust, then sparkrust.connect(). No JVM, no daemon,
no network — the engine runs inside your Python process and reads parquet directly.
DuckDB-shaped ergonomics with the Spark API surface.
The same spark-connect-server binary on localhost. Point
unmodified PySpark (3.5 and 4.0 dialects), notebooks, or JDBC clients at
sc://localhost:15002 and share one engine across clients.
Same code, same query plans, pointed at a cluster driver. Executors receive encoded query plans, run them with their own Arrow-batched DataFusion, and shuffle over gRPC + Arrow IPC. Unmodified PySpark just works.
Distributed query plans · gRPC shuffleThe same SQL core compiled to wasm32 and run client-side in a Web Worker — no
server, no network, your data never leaves the tab. Spark dialect rewrites and UDFs all
execute in WASM against in-memory Arrow batches.
Prototype in-process on a laptop, then deploy the exact same SQL or DataFrame code to a cluster through Spark Connect — without changing a line.
Embed the engine directly from Rust or Python, or use Spark Connect from Rust, PySpark, Java, Scala, SparkR-compatible R, TypeScript/Node, and Ruby. Rust and TypeScript also run against the embedded browser-WASM engine. SQL is the shared semantic layer across every surface.
Native SparkSession or remote SparkConnectSession, with the same lazy DataFrames, Columns, typed FromSparkRow decoding, and Arrow results. The Rust API also runs interactively in browser WebAssembly.
Embedded sparkrust with lazy relational methods, Arrow/pandas results, and UDFs; or unmodified PySpark 3.5/4.x over Connect.
Spark-shaped DataFrame, typed relational Dataset, Arrow results, DataFrameWriter, catalog operations, TLS/auth, cancellation, and Connect transport.
Java 17 · ConnectScala facade over the JVM transport with DataFrames, typed relational decoding, writers, catalogs, and familiar column/function syntax.
Scala · ConnectSQL, lazy DataFrame verbs, grouping, joins, actions, Arrow LocalRelation upload, writes, and %>% workflows.
Strict TypeScript SDK with gRPC, TLS/auth, Spark Connect plan builders, chunked Arrow IPC decoding, DataFrames, Columns, and actions.
Node 20+ · ConnectNative Ruby DataFrames and Columns with TLS/auth, chunked Arrow IPC decoding, joins, aggregation, actions, and Spark-style horizontal or vertical result display.
Ruby · ConnectRust embedded · Arrow-native
use spark_rust::prelude::*; let spark = SparkSession::builder().build(); let rows = spark.table("sales").await? .filter(col("amount").geq(lit(3_i64)))? .group_by([col("region")]) .agg([sum(col("amount")).alias("total")])? .order_by([col("total").desc()])? .collect_as::<(String, i64)>().await?;
TypeScript Connect or embedded WASM
import { SparkSession, col, lit, sum } from "@spark-rust/connect"; const spark = SparkSession.builder() .remote("sc://localhost:15002").getOrCreate(); const rows = await spark.table("sales") .filter(col("amount").geq(lit(3))) .groupBy("region") .agg(sum(col("amount")).alias("total")) .orderBy(col("total").desc()).collect();
The same filter/aggregate, join, and null/distinct workloads are implemented in the established Rust, Python, Java, Scala, SparkR, TypeScript, and SQL APIs — producing identical results in each; the Ruby SDK has its own runnable DataFrame example. Relational plans can run distributed; local callbacks are named local, and arbitrary remote closures are not implied.
The newest complete campaign uses eight workers and native Hive Metastore metadata. Spark-rust is one uninterrupted same-binary run; the comparator is the per-query median of three complete Spark 4.1.1 + Gluten native-tuned runs on the matched corpus.
| Scale | spark-rust EX-6 | Spark 4.1.1 + Gluten | Result | Per-query wins |
|---|---|---|---|---|
| SF=100 | 330.561s · 99/99 | 419.423s median vector · 99/99 | 21.2% less wall | 56/99 |
| SF=1000 | 1,283.735s · 99/99 | 1,615.885s median vector · 99/99 | 20.6% less wall | 63/99 |
This is the latest production-path benchmark. The single-node DuckDB and exact four-worker Spark 4.2 tables below retain their own hardware/topology contracts; numbers are not mixed across those comparisons.
Serial, one fresh process per query, both engines reading the same local Parquet on the same 32-core AMD EPYC 7763 worker. This July 13–14 run is the current hardware-controlled cross-scale comparison.
| Scale factor | spark-rust | DuckDB 1.5.4 | SR / DuckDB | Coverage |
|---|---|---|---|---|
| SF=1 | 16.4s · 99/99 | 9.1s · 99/99 | 1.81× | 0 row differences |
| SF=10 | 51.4s · 99/99 | 26.0s · 99/99 | 1.98× | 0 row differences |
| SF=100 · warm medians | 253.8s · 99/99 | 130.7s · 99/99 | 1.94× | 0 row differences |
| SF=1000 | 3,350.6s · 99/99 | 6,854.9s over 98 common; Q85 timeout | 0.49× — spark-rust 2.06× faster | 0 row differences |
0 row differences at every scale. At SF=100, query-referenced table registration reduced spark-rust to 253.824s; against DuckDB's 130.697s warm total, the current gap is 1.942×. At 1 TB spark-rust completes Q85, wins the common 98-query aggregate by 2.06×, and completes all 99 queries. Apache Spark 4.2 completed the same SF=1000 suite in 11,375.3s, making spark-rust 3.39× faster on the same worker.
Serial Q1→Q99 production runs on the same four-worker cluster. Both engines use the same worker hosts and the same Azure parquet corpus.
| Scale | spark-rust | Apache Spark 4.2 | Overall | Per-query wins |
|---|---|---|---|---|
| SF=100 | 538.0s · 99/99 | 1,089.6s · 99/99 | 2.03× faster | refresh pending |
| SF=1000 | 4,729.9s · 99/99 | 4,692.5s · 98/99; Q72 timeout | 1.005× on common 98 | 74/98 mutual |
The current SF=100 spark-rust figure is the latest sealed width-8 full-survey A/B ON arm. ON and OFF both completed 99/99 with identical row counts; ON reduced aggregate query wall from 697.7s to 538.0s. Against Spark's exact-topology 1,089.6s result, that is 2.03× faster and 50.6% less wall time.
Spark's exact-topology SF=1000 run completed 98 queries in 4,692.5s; Q72 hit its honest 3,600s cap after pathological spill. The refreshed spark-rust run completes 99/99 in 4,729.9s. On the mutually successful 98 queries the engines are within 0.5% in aggregate, while spark-rust wins 74/98 and is 2.05× faster on the median query. Q72 completes in 14.629s for spark-rust, at least 246× below Spark's timeout cap.
Resource-aware dispatch: scan and aggregation fanout are selected conservatively from query shape and measured state size, with full-suite correctness and memory gates before defaults change.
pyspark (3.5 & 4.0) connects and runs. Opt-in bearer-token auth + TLS.TCLIService (binary + HTTP + TLS + SASL); unmodified beeline / DBeaver / Tableau connect over jdbc:hive2://.spark-gpu) and RAPIDS cuDF GPU-native Parquet scan (spark-cudf), bit-exact vs CPU.A real-time topology page visualizes the cluster live: executors appear as nodes sized by active task count; shuffle edges light up with animated particles showing the direction and volume of in-flight transfers.
The driver accepts executor registrations and per-heartbeat counter snapshots, fanned out via event logs to the history server's dashboard — so you can watch tasks, shuffle transfers, and cluster-wide totals update live as a query runs.
The snippets below show deployment topology. The language section above covers the Rust, Python, Java, Scala, SparkR, TypeScript, and SQL APIs that target those deployments.
1 · In-process Python no daemon · DuckDB-shaped
# pip install sparkrust import sparkrust con = sparkrust.connect() con.sql("CREATE VIEW sales AS SELECT * FROM read_parquet('sales/*.parquet')") df = con.sql("SELECT region, SUM(revenue) AS rev FROM sales GROUP BY region") df.show() # PySpark-style tabular print rows = df.collect() # list of tuples table = df.arrow() # pyarrow.Table (zero-copy ready) pdf = df.toPandas() # pandas DataFrame
2 · Single-node Spark Connect server laptop-scale
# same spark-connect-server binary, run locally import pyspark.sql spark = pyspark.sql.SparkSession.builder.remote("sc://localhost:15002").getOrCreate() df = spark.read.parquet("s3://my-bucket/sales/") df.groupBy("region").agg({"revenue": "sum"}).show()
3 · Distributed Spark Connect cluster same code, point at a driver
# identical pipeline, now executed across N executors spark = pyspark.sql.SparkSession.builder.remote("sc://driver.example.com:15002").getOrCreate()
4 · In-browser WebAssembly zero install · runs in the tab
// same SQL core compiled to wasm32, loaded in a Web Worker import init, { run_sql_json } from "./pkg/spark_wasm.js"; await init(); // fetch + instantiate the .wasm engine const rows = '[{"region":"west","revenue":10},{"region":"west","revenue":5}]'; const out = run_sql_json( "SELECT region, SUM(revenue) AS rev FROM data GROUP BY region", rows); // no server, no network — your data never leaves the tab
No JVM, no bundled deps, no installer — dynamically linked against the same
standard system libs. The Connect server and Python extension are different packagings of the
same Rust engine. Current stripped Linux x86-64 release builds, measured at commit
84c06534:
| Artifact | Size |
|---|---|
spark-connect-server — full engine + Connect protocol, default features off | 150.26 MiB |
sparkrust._sparkrust.abi3.so — in-process Python ext | 135.70 MiB |
spark_wasm_bg.wasm — browser engine after wasm-opt | 25.13 MiB |
spark_wasm_bg.wasm.gz — deployed transfer | 7.31 MiB |
duckdb-1.5.4 — reference CLI | 59 MiB |
For context, Apache Spark 4.2's distribution is ~330 MB compressed (~1 GB extracted) plus the JVM — reflecting its much broader feature surface.
spark-rust uses a local fork of DataFusion
— the Rust query engine that also powers Polars, ParadeDB, InfluxDB v3 and Sail — for
Arrow-columnar vectorized execution, SQL parsing, optimization, and Parquet I/O. On top we add
the Spark-shaped pieces: a Spark-dialect SQL layer, analyzer rewrites, spill operators,
skew-join split, the gRPC shuffle service, the Connect server, MLlib over Connect, an
Apache Iceberg / Delta / Hudi lakehouse layer (including Hudi COW/MOR reads and
atomic local COW writes), a wasm32 build of the SQL core that
runs Spark SQL client-side in the browser, and the topology dashboard.
Coming soon — the natural next direction once the core engine ships.
Ask for queries in English, get them planned and run against your data.
The engine learns from prior runs — cost-model population, plan-shape histograms, adaptive bloom-filter gating — and applies it to new queries automatically.
Ask why a query is slow; get a plain-language summary of the physical plan and per-operator metrics.
Build the source image for embedded Python and Spark SQL plus Scala 2.13, SparkR-compatible R, Rust, TypeScript, and Ruby over a private Connect service. The notebook suite covers language APIs, TPC-H/TPC-DS, ML, streaming, concurrency, Delta Liquid Clustering, and seven-catalog federation.
Every code cell runs against its real kernel in the playground gate. Persistent
polyglot magics, live per-cell progress, anchored Run All status, and bounded output
make long courses observable and repeatable. The public image is not published yet;
build examples/playground locally and open JupyterHub on port 8000.
The exact same Rust engine, compiled to WebAssembly. Your data never leaves the tab.
spark-rust is currently in private development; open-source release is planned for the near future.