Skip to content
AMD-EcosystemPublic

Latest commit

 

History

44,396 Commits

Folders and files

Repository files navigation

 hipDF - GPU DataFrames on AMD GPUs


Note

This README is derived from the original RAPIDSAI project's README. More care is necessary to remove/modify parts that are only applicable to the original version.

Note

This ROCm™ port is derived work based on the NVIDIA RAPIDS® cuDF project (version 26.06). It aims to follow the latter's directory structure and API naming as closely as possible to minimize porting friction for users that are interested in using both projects.

Resources

RAPIDS resources:

Overview

Built based on the Apache Arrow columnar memory format, hipDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.

hipDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of HIP programming.

hipDF is composed of multiple libraries including:

  • libcudf: A HIP C++ library with Apache Arrow compliant data structures and fundamental algorithms for tabular data.
  • pylibcudf: A Python library providing Cython bindings for libcudf.
  • cudf: A Python library providing
    • A DataFrame library mirroring the pandas API
    • A zero-code change accelerator, cudf.pandas, for existing pandas code.
  • cudf-polars: A Python library providing a GPU engine for Polars
  • dask-cudf: A Python library providing a GPU backend for Dask DataFrames

Quick Start

Note

Currently, a docker image is not available for AMD GPUs.

Installation

Caution

Incompatibility notice: Mixing RAPIDS and AMD-DS packages/installations is not supported. To avoid conflicts, strictly separate and isolated environments must be maintained if it is required to install both RAPIDS and AMD-DS packages on the same system.

Note

We support only AMD GPUs. Use the RAPIDS package for NVIDIA GPUs.

ROCm/GPU Requirements

  • ROCm HIP SDK compilers version 7.2.1
  • GCC 14 toolchain or later (for building from source)
  • Build requirements: rocthrust-dev, rocm-llvm-dev, hipcub (Ubuntu)
  • Runtime requirements: rocm-llvm-dev (Ubuntu)
  • Officially supported architecture (gfx90a, gfx942, gfx950).
  • Ubuntu 24.04+
  • Python 3.10 or later

Examples

The following examples showcase reading a parquet file, dropping missing rows with a null value, and performing a groupby aggregation on the data.

cudf

import cudf and the APIs are largely similar to pandas.

import cudf

df = cudf.read_parquet("data.parquet")
df.dropna().groupby(["A", "B"]).mean()

cudf.pandas

With a Python file containing pandas code:

import pandas as pd

df = pd.read_parquet("data.parquet")
df.dropna().groupby(["A", "B"]).mean()

Use cudf.pandas by invoking python with -m cudf.pandas

$ python -m cudf.pandas script.py

If running the pandas code in an interactive Jupyter environment, call %load_ext cudf.pandas before importing pandas.

In [1]: %load_ext cudf.pandas

In [2]: import pandas as pd

In [3]: df = pd.read_parquet("data.parquet")

In [4]: df.dropna().groupby(["A", "B"]).mean()

cudf-polars

Using Polars' lazy API, call collect with engine="gpu" to run the operation on the GPU

import polars as pl

lf = pl.scan_parquet("data.parquet")
lf.drop_nulls().group_by(["A", "B"]).mean().collect(engine="gpu")

Install from AMD PyPI

See install instructions.

Build/Install from Source

See build instructions.

Open GPU Data Science

The AMD Data Science suite of open source software libraries aims to enable execution of end-to-end data science and analytics pipelines entirely on AMD GPUs. It relies on ROCm HIP primitives for low-level compute optimization, but exposing that GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.

Questions and Discussion

For bug reports or feature requests, please file an issue on the GitHub issue tracker.

For questions or discussion about cuDF and GPU data processing, feel free to post in the RAPIDS Slack workspace.

Apache Arrow on GPU

The GPU version of Apache Arrow is a common API that enables efficient interchange of tabular data between processes running on the GPU. End-to-end computation on the GPU avoids unnecessary copying and converting of data off the GPU, reducing compute time and cost for high-performance analytics common in artificial intelligence workloads. As the name implies, hipDF uses the Apache Arrow columnar data format on the GPU. Currently, a subset of the features in Apache Arrow are supported.

Contributing

hipDF is open to contributions from the community! Please see our guide for contributing to hipDF for more information.

Releases

Packages

Used by

Contributors

Languages