Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Open Lakehouses for BI to AI with Apache Hudi

Welcome to the code repository for Open Lakehouses for BI to AI with Apache Hudi, published by Packt. This repository contains hands-on examples, tutorials, and code samples that demonstrate Apache Hudi's capabilities for building robust data lakes.

📖 About the Book

Open Lakehouses for BI to AI with Apache Hudi is a comprehensive guide to Apache Hudi, covering everything from basic concepts to advanced production patterns. The book provides practical examples and real-world scenarios to help you master Hudi for your data engineering needs.

🗂️ Repository Structure

The code is organized by chapter. Each folder is self-contained, with its own README, sample data, and run scripts.

Folder Chapter Runs on
chapter02 Chapter 2: Apache Hudi Essentials and Quickstart Spark 3.5, Hudi 1.2.0; Docker optional
chapter03 Chapter 3: Apache Hudi ingestion patterns and write workflows Spark 3.5, Hudi 1.2.0
chapter04 Chapter 4: Querying Hudi tables Spark 3.5, Hudi 1.2.1; Python for ai_ml/
chapter05 Chapter 5: Managing Tables Using Table Services Spark 3.5, Hudi 1.2.1
chapter06 Chapter 6: Building a Real-Time Marketplace Payout Lakehouse with Apache Hudi Spark 3.5, Hudi 1.2.1, Docker (Kafka)
chapter07 Chapter 7: Building a Streaming and CDC Lakehouse with Apache Hudi and Flink Docker only (Flink 1.20, Hudi 1.2.0)
chapter09 Chapter 9: Advanced Lakehouse Capabilities Spark 3.5 and 4.1, Flink 1.20, Hudi 1.2.1
chapter10 Chapter 10: Designing Production Lakehouse Pipelines Design files and configuration for review (Kafka, S3, DynamoDB, Glue); Hudi 1.2.1

Scala and PySpark

The Spark examples are written in Scala, and every chapter with Spark code also has PySpark versions of them:

  • Chapter 2: PySpark notebooks in chapter02/docker/notebooks.
  • Chapters 3 and 4: a .py file next to each .scala file, with the same name.
  • Chapters 5 and 9: a pyspark/ folder next to spark/, with run scripts in scripts/.
  • Chapter 6: a pyspark/ folder. The silver job runs Hudi's SqlQueryBasedTransformer instead of the Scala transformer, so a Python-only setup does not need sbt.

Each chapter's README shows how to run them. A few examples stay Scala-only because they need a JVM class: Chapter 9's custom pre-write validator, its low-level write-client examples, and the examples for sections that were cut from the chapter (see chapter09/README.md). The Flink examples (Chapter 7, and Chapter 9's record-level index job) are Flink SQL.

📋 Prerequisites

Each chapter's README lists the exact versions it was tested with. In general you need:

Software Requirements

  • Apache Spark 3.5 with Scala 2.12 (tested on 3.5.6). Chapter 9's VARIANT examples also need Spark 4.1 with Scala 2.13.
  • Java 11 or 17
  • Apache Hudi 1.2.0 or later. The scripts pull the bundles via --packages or download them.
  • Apache Flink 1.20 for Chapter 9's Flink examples (Chapter 7 runs Flink inside Docker)
  • Docker and Docker Compose for Chapters 6 and 7 (optional for Chapter 2)
  • Python 3.8 or later for the PySpark versions and Chapter 4's ai_ml/ examples (3.10 or later for Chapter 9's VARIANT examples, because PySpark 4.1 requires it)

Hardware Requirements

  • At least 4 GB RAM for Spark; Chapter 7's Docker stack (Flink, Kafka, MySQL) needs 6-8 GB
  • 2+ CPU cores recommended
  • ~2 GB disk space for sample data and tables; Chapter 3's HoodieStreamer example needs at least 5 GB free

🚀 Getting Started

  1. Clone this repository
  2. Navigate to the chapter you want to explore
  3. Follow the chapter-specific README for detailed setup instructions
  4. Each chapter is self-contained with its own dataset and examples

🤝 Contributing

Found an issue or want to improve the examples? Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes with clear commit messages
  4. Submit a pull request

📄 License

This code is provided as supplementary material for Open Lakehouses for BI to AI with Apache Hudi and is released under the MIT License.

📧 Support

For questions about the book or code examples:


Happy learning with Apache Hudi! 🎉

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages