Sparkless / Robin Sparkless¶
PySpark-compatible DataFrames without the JVM. The sparkless Python package is the primary adoption path for testing and local development. The robin-sparkless Rust crate powers it (and can be embedded directly).
Install Python package: pip install "sparkless>=4,<5"
Evaluating Sparkless? Read Before you adopt first — UDF limits, parity gaps, and production caveats.
Choose your path¶
| Persona | Start here |
|---|---|
| Python user / tester | Getting started (Python) · Package README |
| Rust embedder | Quickstart · User guide · docs.rs |
| Contributor | CONTRIBUTING.md |
| Enterprise / production | Before you adopt · Production deployment |
Python (Sparkless v4)¶
- Getting started — Install from PyPI, quick start, core features
- Testing guide — Dual-mode pytest with
sparkless.testing - Migration — From PySpark or Sparkless 3.x
- FAQ · Troubleshooting (Python)
- Python API reference (starter)
- PySpark differences · Parity status
Rust (robin-sparkless)¶
Robin Sparkless provides a PySpark-like API in Rust using Polars for execution. Sparkless v4 calls it via PyO3.
| Feature | Description |
|---|---|
| Core | SparkSession, DataFrame; lazy by default. ExprIr (engine-agnostic) and Column API (Polars-backed) |
| Engine | Polars for fast, native execution |
| Optional | SQL, Delta Lake, JDBC |
Product history¶
Sparkless 3.x (Polars Python backend) and 4.x (this repo, Rust backend) — see Product history.
License¶
MIT