Why it matters

Arrow underpins fast PySpark. Understanding shapes performance.

Advertisement

The architecture

Zero-copy transfer between languages.

Columnar layout.

Arrow benefitsColumnar in-memorylanguage-neutralZero-copyno serialization overheadEnablesPandas UDFs, RArrow eliminates pickle serialization overhead
Arrow role.
Advertisement

How it works end to end

spark.sql.execution.arrow.pyspark.enabled=true.

Batch size: spark.sql.execution.arrow.maxRecordsPerBatch.

Also used in: Pandas UDFs, toPandas, R DataFrames.