Rust
Native SparkSession or remote SparkConnectSession, with the same lazy DataFrames, Columns, typed FromSparkRow decoding, and Arrow results. The Rust API also runs interactively in browser WebAssembly.
A Rust implementation of Apache Spark with first-class Rust, Python, Java, Scala, SparkR-compatible R, TypeScript/Node, and SQL APIs. It runs as a 150 MiB Spark Connect binary or a 136 MiB embedded Python extension in-process — think DuckDB — or scales out across a cluster as a Spark-compatible distributed engine, or runs entirely in your browser via WebAssembly. Same code, same plans, four deployment shapes.
DuckDB set the bar for embedded analytics — a single binary with vectorized execution. Apache Spark defined the API surface and deployment model for distributed compute. spark-rust adds the same engine in multiple lanes: one Rust query engine, four packagings, sharing the same analyzer, optimizer rules, physical plans, spill operators, and MLlib bindings.
pip install sparkrust, then sparkrust.connect(). No JVM, no daemon,
no network — the engine runs inside your Python process and reads parquet directly.
DuckDB-shaped ergonomics with the Spark API surface.
The same spark-connect-server binary on localhost. Point
unmodified PySpark (3.5 and 4.0 dialects), notebooks, or JDBC clients at
sc://localhost:50051 and share one engine across clients.
Same code, same query plans, pointed at a cluster driver. Executors receive encoded query plans, run them with their own Arrow-batched DataFusion, and shuffle over gRPC + Arrow IPC. Unmodified PySpark just works.
Distributed query plans · gRPC shuffleThe same SQL core compiled to wasm32 and run client-side in a Web Worker — no
server, no network, your data never leaves the tab. Spark dialect rewrites and UDFs all
execute in WASM against in-memory Arrow batches.
Prototype in-process on a laptop, then deploy the exact same SQL or DataFrame code to a cluster through Spark Connect — without changing a line.
Embed the engine directly from Rust or Python, or use Spark Connect from Rust, PySpark, Java, Scala, SparkR-compatible R, and TypeScript/Node. Rust and TypeScript also run against the embedded browser-WASM engine. SQL is the shared semantic layer across every surface.
Native SparkSession or remote SparkConnectSession, with the same lazy DataFrames, Columns, typed FromSparkRow decoding, and Arrow results. The Rust API also runs interactively in browser WebAssembly.
Embedded sparkrust with lazy relational methods, Arrow/pandas results, and UDFs; or unmodified PySpark 3.5/4.x over Connect.
Spark-shaped DataFrame, typed relational Dataset, Arrow results, DataFrameWriter, catalog operations, TLS/auth, cancellation, and Connect transport.
Java 17 · ConnectScala facade over the JVM transport with DataFrames, typed relational decoding, writers, catalogs, and familiar column/function syntax.
Scala · ConnectSQL, lazy DataFrame verbs, grouping, joins, actions, Arrow LocalRelation upload, writes, and %>% workflows.
Strict TypeScript SDK with gRPC, TLS/auth, Spark Connect plan builders, chunked Arrow IPC decoding, DataFrames, Columns, and actions.
Node 20+ · ConnectRust embedded · Arrow-native
use spark_rust::prelude::*; let spark = SparkSession::builder().build(); let rows = spark.table("sales").await? .filter(col("amount").geq(lit(3_i64)))? .group_by([col("region")]) .agg([sum(col("amount")).alias("total")])? .order_by([col("total").desc()])? .collect_as::<(String, i64)>().await?;
TypeScript Connect or embedded WASM
import { SparkSession, col, lit, sum } from "@spark-rust/connect"; const spark = SparkSession.builder() .remote("sc://localhost:15002").getOrCreate(); const rows = await spark.table("sales") .filter(col("amount").geq(lit(3))) .groupBy("region") .agg(sum(col("amount")).alias("total")) .orderBy(col("total").desc()).collect();
See the same filter/aggregate, join, and null/distinct workloads in Rust, Python, Java, Scala, SparkR, TypeScript, and SQL. Relational plans can run distributed; local callbacks are named local, and arbitrary remote closures are not implied.
Serial, one fresh process per query, both engines reading the same local Parquet on the same 32-core AMD EPYC 7763 worker. This July 13–14 run is the current hardware-controlled cross-scale comparison.
| Scale factor | spark-rust | DuckDB 1.5.4 | SR / DuckDB | Coverage |
|---|---|---|---|---|
| SF=1 | 16.4s · 99/99 | 9.1s · 99/99 | 1.81× | 0 row differences |
| SF=10 | 51.4s · 99/99 | 26.0s · 99/99 | 1.98× | 0 row differences |
| SF=100 | 367.2s · 99/99 | 136.9s · 99/99 | 2.68× | 0 row differences |
| SF=1000 | 3,350.6s · 99/99 | 6,854.9s over 98 common; Q85 timeout | 0.49× — spark-rust 2.06× faster | 0 row differences |
0 row differences at every scale. DuckDB leads the smaller scales; at 1 TB spark-rust completes Q85, wins the common 98-query aggregate by 2.06×, and completes all 99 queries. Apache Spark 4.2 completed the same SF=1000 suite in 11,375.3s, making spark-rust 3.39× faster on the same worker.
Serial Q1→Q99 production runs on the same four-worker cluster. Both engines use the same worker hosts and the same Azure parquet corpus.
| Scale | spark-rust | Apache Spark 4.2 | Overall | Per-query wins |
|---|---|---|---|---|
| SF=100 | 538.0s · 99/99 | 1,089.6s · 99/99 | 2.03× faster | refresh pending |
| SF=1000 | 7,773.0s · 99/99 | 4,692.5s · 98/99; Q72 timeout | coverage differs | 61/98 mutual |
The current SF=100 spark-rust figure is the latest sealed width-8 full-survey A/B ON arm. ON and OFF both completed 99/99 with identical row counts; ON reduced aggregate query wall from 697.7s to 538.0s. Against Spark's exact-topology 1,089.6s result, that is 2.03× faster and 50.6% less wall time.
Spark's exact-topology SF=1000 run completed 98 queries in 4,692.5s; Q72 hit its honest 3,600s cap after pathological spill. spark-rust completes 99/99 in 7,773.0s. Over the mutually successful 98 queries Spark is 1.653× faster in aggregate, while spark-rust wins 61/98 and is 1.56× faster on the median query. Two recent query-plan improvements reduced Q35 to 12.642s and Q45 to 12.079s; those focused timings await a refreshed full-99 run before changing the aggregate headline.
pyspark (3.5 & 4.0) connects and runs. Opt-in bearer-token auth + TLS.TCLIService (binary + HTTP + TLS + SASL); unmodified beeline / DBeaver / Tableau connect over jdbc:hive2://.spark-gpu) and RAPIDS cuDF GPU-native Parquet scan (spark-cudf), bit-exact vs CPU.A real-time topology page visualizes the cluster live: executors appear as nodes sized by active task count; shuffle edges light up with animated particles showing the direction and volume of in-flight transfers.
The driver accepts executor registrations and per-heartbeat counter snapshots, fanned out via event logs to the history server's dashboard — so you can watch tasks, shuffle transfers, and cluster-wide totals update live as a query runs.
The snippets below show deployment topology. The language section above covers the Rust, Python, Java, Scala, SparkR, TypeScript, and SQL APIs that target those deployments.
1 · In-process Python no daemon · DuckDB-shaped
# pip install sparkrust import sparkrust con = sparkrust.connect() con.sql("CREATE VIEW sales AS SELECT * FROM read_parquet('sales/*.parquet')") df = con.sql("SELECT region, SUM(revenue) AS rev FROM sales GROUP BY region") df.show() # PySpark-style tabular print rows = df.collect() # list of tuples table = df.arrow() # pyarrow.Table (zero-copy ready) pdf = df.toPandas() # pandas DataFrame
2 · Single-node Spark Connect server laptop-scale
# same spark-connect-server binary, run locally import pyspark.sql spark = pyspark.sql.SparkSession.builder.remote("sc://localhost:50051").getOrCreate() df = spark.read.parquet("s3://my-bucket/sales/") df.groupBy("region").agg({"revenue": "sum"}).show()
3 · Distributed Spark Connect cluster same code, point at a driver
# identical pipeline, now executed across N executors spark = pyspark.sql.SparkSession.builder.remote("sc://driver.example.com:50051").getOrCreate()
4 · In-browser WebAssembly zero install · runs in the tab
// same SQL core compiled to wasm32, loaded in a Web Worker import init, { run_sql_json } from "./pkg/spark_wasm.js"; await init(); // fetch + instantiate the .wasm engine const rows = '[{"region":"west","revenue":10},{"region":"west","revenue":5}]'; const out = run_sql_json( "SELECT region, SUM(revenue) AS rev FROM data GROUP BY region", rows); // no server, no network — your data never leaves the tab
No JVM, no bundled deps, no installer — dynamically linked against the same
standard system libs. The Connect server and Python extension are different packagings of the
same Rust engine. Current stripped Linux x86-64 release builds, measured at commit
84c06534:
| Artifact | Size |
|---|---|
spark-connect-server — full engine + Connect protocol, default features off | 150.26 MiB |
sparkrust._sparkrust.abi3.so — in-process Python ext | 135.70 MiB |
spark_wasm_bg.wasm — browser engine after wasm-opt | 25.13 MiB |
spark_wasm_bg.wasm.gz — deployed transfer | 7.31 MiB |
duckdb-1.5.4 — reference CLI | 59 MiB |
For context, Apache Spark 4.2's distribution is ~330 MB compressed (~1 GB extracted) plus the JVM — reflecting its much broader feature surface.
spark-rust uses a local fork of DataFusion
— the Rust query engine that also powers Polars, ParadeDB, InfluxDB v3 and Sail — for
Arrow-columnar vectorized execution, SQL parsing, optimization, and Parquet I/O. On top we add
the Spark-shaped pieces: a Spark-dialect SQL layer, analyzer rewrites, spill operators,
skew-join split, the gRPC shuffle service, the Connect server, MLlib over Connect, an
Apache Iceberg / Delta / Hudi lakehouse layer (including Hudi COW/MOR reads and
atomic local COW writes), a wasm32 build of the SQL core that
runs Spark SQL client-side in the browser, and the topology dashboard.
Coming soon — the natural next direction once the core engine ships.
Ask for queries in English, get them planned and run against your data.
The engine learns from prior runs — cost-model population, plan-shape histograms, adaptive bloom-filter gating — and applies it to new queries automatically.
Ask why a query is slow; get a plain-language summary of the physical plan and per-operator metrics.
The exact same Rust engine, compiled to WebAssembly. Your data never leaves the tab.
spark-rust is currently in private development; open-source release is planned for the near future.