Why it matters

Spark ML enables ML at big data scale. Understanding shapes ML pipelines on Spark.

Advertisement

The architecture

DataFrame-based API. Estimators + Transformers + Pipelines.

Spark ML abstractionsTransformerDF → DFEstimatorfit → ModelPipelinecompose stagesSimilar to scikit-learn API; distributed under the hood
ML pipeline model.
Advertisement

How it works end to end

Algorithms: linear/logistic regression, random forest, gradient-boosted trees, k-means, ALS for recommender.

Feature engineering: OneHotEncoder, StringIndexer, VectorAssembler, StandardScaler.

Cross-validation: CrossValidator, TrainValidationSplit.