This repository showcases how to setup a Scala Spark job on Docker and in Dataproc and execute a Broadcast Join technique.
-
Updated
Jun 6, 2026 - Shell
#
broadcast-join
Here are 8 public repositories matching this topic...
PySpark ETL & analytics pipeline for taxi trip ETA, partitioned Parquet, windowed aggregations and performance patterns.
performance etl analytics bigdata pyspark sparksql parquet partioning partition-pruning broadcast-join
-
Updated
Sep 17, 2025 - Jupyter Notebook
Diagnoses Spark data skew from the job's own event log - no cluster, no JVM - and tells key skew apart from a straggler by comparing task time against bytes read. Then measures the fixes against each other on real Spark 4: across 16 cells, salting won 0. Broadcast 6.5x, AQE 1.5x, salting 0.81x.
python benchmarking big-data apache-spark distributed-computing pyspark data-engineering performance-tuning profiling shuffle query-optimization observability salting spark4 broadcast-join adaptive-query-execution data-skew spark-event-log
-
Updated
Aug 19, 2026 - Python
Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.
-
Updated
Aug 18, 2026 - Python
This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.
pyspark data-engineering parquet partitioning databricks etl-pipeline bucketing delta-lake broadcast-join spark-optimization spark-performance adaptive-query-execution
-
Updated
Mar 3, 2026 - Python
Battle-tested Apache Spark tuning patterns with reproducible benchmarks. 10 techniques (partition pruning, broadcast joins, AQE, skew handling, Z-ORDER, and more) — each paired with measured before/after speedups runnable on a laptop.
python emr performance big-data spark apache-spark optimization distributed-computing pyspark data-engineering benchmarks performance-tuning databricks spark-sql delta-lake zorder partition-pruning broadcast-join adaptive-query-execution skew-handling
-
Updated
Apr 23, 2026 - Python
End- to-End Performance-optimized sales data pipeline using Medallion Architecture with broadcast joins, fact/dimension modeling, Autoloader & incremental processing
pipeline optimization incremental pyspark data-engineering joins notebooks databricks performance-optimization etl-pipeline delta-lake incremental-processing medallion-architecture broadcast-join auto-loader
-
Updated
Aug 4, 2026 - Python
🚖 Ingest and analyze NYC yellow taxi data with a streamlined ETL pipeline, featuring data cleaning, analytics, and business-ready outputs.
performance etl analytics bigdata pyspark sparksql parquet partioning partition-pruning broadcast-join
-
Updated
Oct 13, 2026 - Jupyter Notebook
Add this topic to your repo
To associate your repository with the broadcast-join topic, visit your repo's landing page and select "manage topics."