Note
This README is derived from the original RAPIDSAI project's README. More care is necessary to remove/modify parts that are only applicable to the original version.
Note
This ROCm™ port is derived work based on the NVIDIA RAPIDS® cuDF project (version 26.06). It aims to follow the latter's directory structure and API naming as closely as possible to minimize porting friction for users that are interested in using both projects.
RAPIDS resources:
- cudf (Python) documentation
- libcudf (C++/CUDA) documentation
- RAPIDS Community: Get help, contribute, and collaborate.
Built based on the Apache Arrow columnar memory format, hipDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
hipDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of HIP programming.
hipDF is composed of multiple libraries including:
- libcudf: A HIP C++ library with Apache Arrow compliant data structures and fundamental algorithms for tabular data.
- pylibcudf: A Python library providing Cython bindings for libcudf.
- cudf: A Python library providing
- A DataFrame library mirroring the pandas API
- A zero-code change accelerator, cudf.pandas, for existing pandas code.
- cudf-polars: A Python library providing a GPU engine for Polars
- dask-cudf: A Python library providing a GPU backend for Dask DataFrames
Note
Currently, a docker image is not available for AMD GPUs.
Caution
Incompatibility notice: Mixing RAPIDS and AMD-DS packages/installations is not supported. To avoid conflicts, strictly separate and isolated environments must be maintained if it is required to install both RAPIDS and AMD-DS packages on the same system.
Note
We support only AMD GPUs. Use the RAPIDS package for NVIDIA GPUs.
- ROCm HIP SDK compilers version 7.2.1
- GCC 14 toolchain or later (for building from source)
- Build requirements:
rocthrust-dev,rocm-llvm-dev,hipcub(Ubuntu) - Runtime requirements:
rocm-llvm-dev(Ubuntu) - Officially supported architecture (gfx90a, gfx942, gfx950).
- Ubuntu 24.04+
- Python 3.10 or later
The following examples showcase reading a parquet file, dropping missing rows with a null value, and performing a groupby aggregation on the data.
import cudf and the APIs are largely similar to pandas.
import cudf
df = cudf.read_parquet("data.parquet")
df.dropna().groupby(["A", "B"]).mean()With a Python file containing pandas code:
import pandas as pd
df = pd.read_parquet("data.parquet")
df.dropna().groupby(["A", "B"]).mean()Use cudf.pandas by invoking python with -m cudf.pandas
$ python -m cudf.pandas script.pyIf running the pandas code in an interactive Jupyter environment, call %load_ext cudf.pandas before
importing pandas.
In [1]: %load_ext cudf.pandas
In [2]: import pandas as pd
In [3]: df = pd.read_parquet("data.parquet")
In [4]: df.dropna().groupby(["A", "B"]).mean()Using Polars' lazy API, call collect with engine="gpu" to run
the operation on the GPU
import polars as pl
lf = pl.scan_parquet("data.parquet")
lf.drop_nulls().group_by(["A", "B"]).mean().collect(engine="gpu")See install instructions.
See build instructions.
The AMD Data Science suite of open source software libraries aims to enable execution of end-to-end data science and analytics pipelines entirely on AMD GPUs. It relies on ROCm HIP primitives for low-level compute optimization, but exposing that GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.
For bug reports or feature requests, please file an issue on the GitHub issue tracker.
For questions or discussion about cuDF and GPU data processing, feel free to post in the RAPIDS Slack workspace.
The GPU version of Apache Arrow is a common API that enables efficient interchange of tabular data between processes running on the GPU. End-to-end computation on the GPU avoids unnecessary copying and converting of data off the GPU, reducing compute time and cost for high-performance analytics common in artificial intelligence workloads. As the name implies, hipDF uses the Apache Arrow columnar data format on the GPU. Currently, a subset of the features in Apache Arrow are supported.
hipDF is open to contributions from the community! Please see our guide for contributing to hipDF for more information.