spark-performance

Workload-aware ML framework for PySpark performance prediction and configuration recommendation, validated through real benchmark execution and out-of-distribution evaluation.

Advanced PySpark interview project with Spark performance optimization and Databricks compatibility.

Drop-in Apache Spark performance analyzer — attach via spark.extraListeners, get actionable recommendations

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.

Local-first Spark event-log analysis and evidence-backed performance optimization for VS Code and GitHub Copilot.

A comprehensive PySpark learning repository covering Spark Architecture, RDDs, DataFrames, Spark SQL, AQE, Performance Tuning, Optimizations, DAGs, and Real-World Data Engineering Concepts.

Plataforma de engenharia agentic e orientada a evidências para análise, diagnóstico e otimização de workloads Spark, cobrindo AWS Glue, EMR, Databricks, Iceberg, streaming, performance, custos, governança e runtime.