Skip to content

Tracking issue for the "ptx-kernel" ABI #38788

Open
Open
Tracking issue for the "ptx-kernel" ABI#38788
Labels
B-unstableBlocker: Implemented in the nightly compiler and unstable.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCO-NVPTXTarget: the NVPTX LLVM backend for running rust on GPUs, https://llvm.org/docs/NVPTXUsage.htmlS-tracking-design-concernsStatus: There are blocking design concerns.S-tracking-needs-summaryStatus: It's hard to tell what's been done and what hasn't! Someone should do some investigation.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Description

@japaric
japaric
opened on Jan 2, 2017

Here's a suggestion for an update to the tracking issue to include concerns. Partially copied for japaric's original post and added concerns from and links to relevant issues.

If you have the possibility you should take a look @RDambrosio016

Feature gate #![feature(abi_ptx)]

This ABI is intended to be used when generating code for device (GPU) targets like nvptx64-nvidia-cuda. It is used to generate kernels ("global functions") that work as an entry point from host (cpu) code. Functions that do not use the "ptx-kernel" ABI are "device functions" and only callable from kernels and device functions. Device functions are specifically not usable from host (cpu) code.

Public API

The following code

#![no_std]
#![feature(abi_ptx)]

#[no_mangle]
pub extern "ptx-kernel" fn foo() {}

Produces

.version 3.2
.target sm_30
.address_size 64

	// .globl	foo

.visible .entry foo()
{
	ret;
}

Steps / History

Unresolved Questions

  • Resolve what kind of stability guarantees can be made about the generated ptx.
    • The ABI of kernels have been previously changed for a major version bump and the ptx-interoperability doc is still outdated.
    • PTX is an ISA with many versions. The newest is major version 7. Do we need to reserve the possibility of breaking things when moving to a new major version?
    • Figure out what llvm does in relations to the nvptx64-nvidia-cuda target and the __global__ modifier.
  • What kind of types should be allowed to use as arguments in kernels. Should it be a hard error to use these types or only a warning (Global and device kernels are unsound rust-cuda/wg#11)
    • The most important part is to find a minimal but useful subset of Rust types that can be used in kernels. raw pointers, primitive types and #[repr(C)] types seems like a good start (no slices, tuples, references, etc).
    • Using mutable references is almost certain UB except for a few unusable special cases (spawning a single thread only)
    • There are many convenient types in Rust which do not have a stable ABI (&[T], (T, U), etc). Are there some types that do not have a stable representation but can be relied on having an identical representation for sequential compilation with a given rustc version? If so are there any way we could pass them safely between host and device code compiled with the same rustc version?
  • This unstable feature is one of the last stoppers to using nvptx64-nvidia-cuda on stable Rust. The target seems to still have a few bugs (NVPTX backend metabug #38789). Should this feature be kept unstable to avoid usage of nvptx64-nvidia-cuda until it has been verified to be usable.
  • How should shared be supported? Is it necessary to do that from the go?

Notes

  • It is not possible to emulate kernels with #[naked] functions as the .entry directive needs to be emited for nvptx kernels.

Activity

  1. japaric
    added
    O-NVPTXTarget: the NVPTX LLVM backend for running rust on GPUs, https://llvm.org/docs/NVPTXUsage.html
    B-unstableBlocker: Implemented in the nightly compiler and unstable.
    on Jan 2, 2017
  2. sfackler
    added
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    on Jan 3, 2017
  3. Mark-Simulacrum
    added
    C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFC
    on Jul 22, 2017

  4. steveklabnik commented on Mar 12, 2019

    @steveklabnik
    steveklabnik
    on Mar 12, 2019
    Contributor

    Triage: not aware of any movement on stabilizing this.

  5. lahwran commented on Mar 7, 2020

    @lahwran
    lahwran
    on Mar 7, 2020

    would be very interested in this! any of y'all know what needs doing to bring this back to life?

    edit: following links through the metabug, and then through a project that mentioned the metabug, looks like there are in fact folks still interested. cool!

  6. kjetilkjeka commented on Feb 23, 2022

    @kjetilkjeka
    kjetilkjeka
    on Feb 23, 2022
    Contributor

    I'm curious if I might have found a codegen bug related to this feature. It is related to how the struct is assumed to be passed to the kernel.

    When compiling the following Rust code 
    #![no_std]
#![feature(abi_ptx, stdsimd)]

#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
    unreachable!()
}

#[repr(C)]
pub struct Foo{
    array: [f32; 9],
}

#[no_mangle]
pub unsafe extern "ptx-kernel" fn add(a: *mut f32, b: Foo) {
    *a = b.array[5];
}
    Rustc will produce the following

    When compiling with rustc +nightly --target nvptx64-nvidia-cuda <filename> -C target-cpu=sm_52 --crate-type cdylib 

    //
// Generated by LLVM NVPTX Back-End
//

.version 4.1
.target sm_52
.address_size 64

	// .globl	add

.visible .entry add(
	.param .u64 add_param_0,
	.param .u64 add_param_1
)
{
	.reg .f32 	%f<2>;
	.reg .b64 	%rd<5>;

	ld.param.u64 	%rd1, [add_param_0];
	ld.param.u64 	%rd2, [add_param_1];
	cvta.to.global.u64 	%rd3, %rd2;
	cvta.to.global.u64 	%rd4, %rd1;
	ld.global.f32 	%f1, [%rd3+32];
	st.global.f32 	[%rd4], %f1;
	ret;

}
    On the other hand, when compiling the following C++ code 
    struct foo {
    float array[9];
};

extern "C" __global__ void add( float *a, struct foo b) {
    *a = b.array[5];
}
    I get the following output for nvcc

    When compiling with nvcc -ptx <filename> 

    //
// Generated by NVIDIA NVVM Compiler
//
// Compiler Build ID: CL-30794723
// Cuda compilation tools, release 11.6, V11.6.55
// Based on NVVM 7.0.1
//

.version 7.6
.target sm_52
.address_size 64

	// .globl	add

.visible .entry add(
	.param .u64 add_param_0,
	.param .align 4 .b8 add_param_1[36]
)
{
	.reg .f32 	%f<2>;
	.reg .b64 	%rd<3>;


	ld.param.u64 	%rd1, [add_param_0];
	cvta.to.global.u64 	%rd2, %rd1;
	ld.param.f32 	%f1, [add_param_1+20];
	st.global.f32 	[%rd2], %f1;
	ret;

}
    I get the following output for clang

    When compiling with clang++ --cuda-device-only -nocudalib --cuda-gpu-arch=sm_52 <filename> -S -o <outfile>

    //
// Generated by LLVM NVPTX Back-End
//

.version 4.1
.target sm_52
.address_size 64

	// .globl	add

.visible .entry add(
	.param .u64 add_param_0,
	.param .align 4 .b8 add_param_1[36]
)
{
	.local .align 8 .b8 	__local_depot0[48];
	.reg .b64 	%SP;
	.reg .b64 	%SPL;
	.reg .f32 	%f<11>;
	.reg .b64 	%rd<8>;

	mov.u64 	%SPL, __local_depot0;
	cvta.local.u64 	%SP, %SPL;
	mov.b64 	%rd2, add_param_1;
	ld.param.u64 	%rd1, [add_param_0];
	ld.param.f32 	%f1, [add_param_1];
	ld.param.f32 	%f2, [add_param_1+8];
	ld.param.f32 	%f3, [add_param_1+12];
	ld.param.f32 	%f4, [add_param_1+16];
	ld.param.f32 	%f5, [add_param_1+20];
	ld.param.f32 	%f6, [add_param_1+24];
	ld.param.f32 	%f7, [add_param_1+28];
	ld.param.f32 	%f8, [add_param_1+32];
	ld.param.f32 	%f9, [add_param_1+4];
	add.u64 	%rd3, %SP, 0;
	or.b64  	%rd4, %rd3, 4;
	st.f32 	[%rd4], %f9;
	st.f32 	[%SP+32], %f8;
	st.f32 	[%SP+28], %f7;
	st.f32 	[%SP+24], %f6;
	st.f32 	[%SP+20], %f5;
	st.f32 	[%SP+16], %f4;
	st.f32 	[%SP+12], %f3;
	st.f32 	[%SP+8], %f2;
	st.f32 	[%SP+0], %f1;
	cvta.to.global.u64 	%rd5, %rd1;
	cvta.global.u64 	%rd6, %rd5;
	st.u64 	[%SP+40], %rd6;
	ld.f32 	%f10, [%SP+20];
	ld.u64 	%rd7, [%SP+40];
	st.f32 	[%rd7], %f10;
	ret;

}
    Here's the different compiler versions I'm using 
    $ rustc +nightly --version
rustc 1.61.0-nightly (68369a041 2022-02-22)
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0
$ clang++ --version
clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

    I would be happy to try to figure out whats going on if anyone can confirm that this is a bug? If anyone would be able to point me in the right direction that would be a nice bonus as well.

  7. pnkfelix commented on Mar 17, 2022

    @pnkfelix
    pnkfelix
    on Mar 17, 2022
    Contributor

    Visiting as part of a T-compiler backlog bonanza prepass.

    This at very least is in a "needs summary" state. But based on @RDambrosio016 's comment from 10 days ago, there may even be design concerns? (At least depending on what scope of potential parameter types we want this ABI to cover at the outset.)

    @rustbot label: S-tracking-design-concerns S-tracking-needs-summary

  8. rustbot
    added
    S-tracking-design-concernsStatus: There are blocking design concerns.
    S-tracking-needs-summaryStatus: It's hard to tell what's been done and what hasn't! Someone should do some investigation.
    on Mar 17, 2022

  9. kjetilkjeka commented on Mar 17, 2022

    @kjetilkjeka
    kjetilkjeka
    on Mar 17, 2022
    Contributor

    @pnkfelix I'm currently working on fixing bugs in ptx-kernel in #94703 and would like to help out providing a summary. I have also discovered some design concerns that I can contribute to the summary.

    Do you have a template or example of how it should look like?

  10. kjetilkjeka commented on Mar 25, 2022

    @kjetilkjeka
    kjetilkjeka
    on Mar 25, 2022
    Contributor

    Here's a suggestion for an update to the tracking issue to include concerns. Partially copied for japaric's original post and added concerns from and links to relevant issues.

    If you have the possibility you should take a look @RDambrosio016

    Feature gate #![feature(abi_ptx)]

    This ABI is intended to be used when generating code for device (GPU) targets like nvptx64-nvidia-cuda. It is used to generate kernels ("global functions") that work as an entry point from host (cpu) code. Functions that do not use the "ptx-kernel" ABI are "device functions" and only callable from kernels and device functions. Device functions are specifically not usable from host (cpu) code.

    Public API

    The following code

    #![no_std]
#![feature(abi_ptx)]

#[no_mangle]
pub extern "ptx-kernel" fn foo() {}

    Produces

    .version 3.2
.target sm_30
.address_size 64

	// .globl	foo

.visible .entry foo()
{
	ret;
}

    Steps / History

    Unresolved Questions

    • Resolve what kind of stability guarantees can be made about the generated ptx.
      • The ABI of kernels have been previously changed for a major version bump and the ptx-interoperability doc is still outdated.
      • PTX is an ISA with many versions. The newest is major version 7. Do we need to reserve the possibility of breaking things when moving to a new major version?
      • Figure out what llvm does in relations to the nvptx64-nvidia-cuda target and the __global__ modifier.
    • What kind of types should be allowed to use as arguments in kernels. Should it be a hard error to use these types or only a warning (Global and device kernels are unsound rust-cuda/wg#11)
      • The most important part is to find a minimal but useful subset of Rust types that can be used in kernels. raw pointers, primitive types and #[repr(C)] types seems like a good start (no slices, tuples, references, etc).
      • Using mutable references is almost certain UB except for a few unusable special cases (spawning a single thread only)
      • There are many convenient types in Rust which do not have a stable ABI (&[T], (T, U), etc). Are there some types that do not have a stable representation but can be relied on having an identical representation for sequential compilation with a given rustc version? If so are there any way we could pass them safely between host and device code compiled with the same rustc version?
    • This unstable feature is one of the last stoppers to using nvptx64-nvidia-cuda on stable Rust. The target seems to still have a few bugs (NVPTX backend metabug #38789). Should this feature be kept unstable to avoid usage of nvptx64-nvidia-cuda until it has been verified to be usable.
    • How should shared be supported? Is it necessary to do that from the go?

    Notes

    • It is not possible to emulate kernels with #[naked] functions as the .entry directive needs to be emited for nvptx kernels.
  11. GuillaumeGomez
    added 2 commits that reference this issue on Apr 26, 2022

    Rollup merge of rust-lang#94703 - kjetilkjeka:nvptx-kernel-args-abi2,…

    f1fbc71

    Rollup merge of rust-lang#94703 - kjetilkjeka:nvptx-kernel-args-abi2,…

    fe49981
  12. matthiaskrgr
    added a commit that references this issue on Oct 27, 2023

    Rollup merge of rust-lang#117247 - kjetilkjeka:nvptx_direct_passmode_…

    c3d56be
  13. rust-timer
    added a commit that references this issue on Oct 27, 2023

    Unrolled build for rust-lang#117247

    8eed35e
  14. github-actions
    added a commit that references this issue on Oct 28, 2023

    Rollup merge of #117247 - kjetilkjeka:nvptx_direct_passmode_exception…

    0f87438

  15. workingjubilee commented on Jan 14, 2025

    @workingjubilee
    workingjubilee
    on Jan 14, 2025
    Member

    It seems likely that the way this ABI will work is going to be similar in its broad strokes between most GPU targets. Much like extern "C" varies on a per-target basis, it's not a crime to have an ABI string that works slightly differently on a platform-dependent basis. So, as there is some interest in Rust supporting multiple GPU targets, we should probably have just one extern "gpu-kernel" ABI, and see how far we can get with it. That way we don't have to keep reimplementing the same features (argument and return type validation, not being callable from device functions, etc.) for every single platform.

  16. workingjubilee
    added a commit that references this issue on Feb 6, 2025

    Rollup merge of rust-lang#133932 - bjorn3:fix_ptx_kernel_abi, r=wesle…

    cc0f3ef
  17. rust-timer
    added a commit that references this issue on Feb 6, 2025

    Unrolled build for rust-lang#133932

    c90d535
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    B-unstableBlocker: Implemented in the nightly compiler and unstable.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCO-NVPTXTarget: the NVPTX LLVM backend for running rust on GPUs, https://llvm.org/docs/NVPTXUsage.htmlS-tracking-design-concernsStatus: There are blocking design concerns.S-tracking-needs-summaryStatus: It's hard to tell what's been done and what hasn't! Someone should do some investigation.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions