Welcome to the code repository for Open Lakehouses for BI to AI with Apache Hudi, published by Packt. This repository contains hands-on examples, tutorials, and code samples that demonstrate Apache Hudi's capabilities for building robust data lakes.
Open Lakehouses for BI to AI with Apache Hudi is a comprehensive guide to Apache Hudi, covering everything from basic concepts to advanced production patterns. The book provides practical examples and real-world scenarios to help you master Hudi for your data engineering needs.
The code is organized by chapter. Each folder is self-contained, with its own README, sample data, and run scripts.
| Folder | Chapter | Runs on |
|---|---|---|
chapter02 |
Chapter 2: Apache Hudi Essentials and Quickstart | Spark 3.5, Hudi 1.2.0; Docker optional |
chapter03 |
Chapter 3: Apache Hudi ingestion patterns and write workflows | Spark 3.5, Hudi 1.2.0 |
chapter04 |
Chapter 4: Querying Hudi tables | Spark 3.5, Hudi 1.2.1; Python for ai_ml/ |
chapter05 |
Chapter 5: Managing Tables Using Table Services | Spark 3.5, Hudi 1.2.1 |
chapter06 |
Chapter 6: Building a Real-Time Marketplace Payout Lakehouse with Apache Hudi | Spark 3.5, Hudi 1.2.1, Docker (Kafka) |
chapter07 |
Chapter 7: Building a Streaming and CDC Lakehouse with Apache Hudi and Flink | Docker only (Flink 1.20, Hudi 1.2.0) |
chapter09 |
Chapter 9: Advanced Lakehouse Capabilities | Spark 3.5 and 4.1, Flink 1.20, Hudi 1.2.1 |
chapter10 |
Chapter 10: Designing Production Lakehouse Pipelines | Design files and configuration for review (Kafka, S3, DynamoDB, Glue); Hudi 1.2.1 |
The Spark examples are written in Scala, and every chapter with Spark code also has PySpark versions of them:
- Chapter 2: PySpark notebooks in
chapter02/docker/notebooks. - Chapters 3 and 4: a
.pyfile next to each.scalafile, with the same name. - Chapters 5 and 9: a
pyspark/folder next tospark/, with run scripts inscripts/. - Chapter 6: a
pyspark/folder. The silver job runs Hudi'sSqlQueryBasedTransformerinstead of the Scala transformer, so a Python-only setup does not need sbt.
Each chapter's README shows how to run them. A few examples stay Scala-only because they need a
JVM class: Chapter 9's custom pre-write validator, its low-level write-client examples, and the
examples for sections that were cut from the chapter (see chapter09/README.md). The Flink
examples (Chapter 7, and Chapter 9's record-level index job) are Flink SQL.
Each chapter's README lists the exact versions it was tested with. In general you need:
- Apache Spark 3.5 with Scala 2.12 (tested on 3.5.6). Chapter 9's VARIANT examples also need Spark 4.1 with Scala 2.13.
- Java 11 or 17
- Apache Hudi 1.2.0 or later. The scripts pull the bundles via
--packagesor download them. - Apache Flink 1.20 for Chapter 9's Flink examples (Chapter 7 runs Flink inside Docker)
- Docker and Docker Compose for Chapters 6 and 7 (optional for Chapter 2)
- Python 3.8 or later for the PySpark versions and Chapter 4's
ai_ml/examples (3.10 or later for Chapter 9's VARIANT examples, because PySpark 4.1 requires it)
- At least 4 GB RAM for Spark; Chapter 7's Docker stack (Flink, Kafka, MySQL) needs 6-8 GB
- 2+ CPU cores recommended
- ~2 GB disk space for sample data and tables; Chapter 3's HoodieStreamer example needs at least 5 GB free
- Clone this repository
- Navigate to the chapter you want to explore
- Follow the chapter-specific README for detailed setup instructions
- Each chapter is self-contained with its own dataset and examples
Found an issue or want to improve the examples? Contributions are welcome! Please:
- Fork the repository
- Create a feature branch
- Make your changes with clear commit messages
- Submit a pull request
This code is provided as supplementary material for Open Lakehouses for BI to AI with Apache Hudi and is released under the MIT License.
For questions about the book or code examples:
- Check the Issues page
- Refer to the Apache Hudi documentation
- Visit the Apache Hudi community
Happy learning with Apache Hudi! 🎉