The problem was not related to PowerPC architecture. I needed to pass the fatbin file to the host-side compilation command with -Xclang -fcuda-include-gpubinary -Xclang axpy.fatbin to replicate the whole compilation behavior.

Here is the corrected Makefile:

BIN_FILE=axpy
SRC_FILE=$(BIN_FILE).cu

main: $(BIN_FILE)

# Host Side
$(BIN_FILE).ll: $(SRC_FILE) $(BIN_FILE).fatbin
    clang++ -stdlib=libc++ -Wall -Werror $(BIN_FILE).cu -march=ppc64le --cuda-host-only -relocatable-pch \
        -Xclang -fcuda-include-gpubinary -Xclang $(BIN_FILE).fatbin -S -g -c -emit-llvm

$(BIN_FILE).o: $(BIN_FILE).ll
    llc -march=ppc64le $(BIN_FILE).ll -o $(BIN_FILE).s
    clang++ -c -Wall $(BIN_FILE).s -o $(BIN_FILE).o

# GPU Side
$(BIN_FILE)-cuda-nvptx64-nvidia-cuda-sm_70.ll: $(SRC_FILE)
    clang++ -x cuda -stdlib=libc++ -Wall -Werror $(BIN_FILE).cu --cuda-device-only \
        --cuda-gpu-arch=sm_70 -S -g -emit-llvm

$(BIN_FILE).ptx: $(BIN_FILE)-cuda-nvptx64-nvidia-cuda-sm_70.ll
    llc -march=nvptx64 -mcpu=sm_70 -mattr=+ptx64 $(BIN_FILE)-cuda-nvptx64-nvidia-cuda-sm_70.ll -o $(BIN_FILE).ptx

$(BIN_FILE).ptx.o: $(BIN_FILE).ptx
    ptxas -m64 --gpu-name=sm_70 $(BIN_FILE).ptx -o $(BIN_FILE).ptx.o

$(BIN_FILE).fatbin: $(BIN_FILE).ptx.o
    fatbinary --64 --create $(BIN_FILE).fatbin --image=profile=sm_70,file=$(BIN_FILE).ptx.o \
        --image=profile=compute_70,file=$(BIN_FILE).ptx -link

$(BIN_FILE)_dlink.o: $(BIN_FILE).fatbin
    nvcc $(BIN_FILE).fatbin -gencode arch=compute_70,code=sm_70 \
        -dlink -o $(BIN_FILE)_dlink.o -lcudart -lcudart_static -lcudadevrt

# Link both object files together (either nvcc or clang works here):
$(BIN_FILE): $(BIN_FILE).o $(BIN_FILE)_dlink.o
    nvcc $(BIN_FILE).o $(BIN_FILE)_dlink.o -o $(BIN_FILE) -arch=sm_70 -lc++

Figure 1 in this link includes the creation steps of the fatbinary file.

Hi @dahubley, Codeplay is supporting and maintaining the CUDA backend for Nvidia GPU support in DPC++ for Windows and Linux. You can find instructions for how to build DPC++ with the CUDA backend and how to use it, and there are some video guides on our web page (https://www.codeplay.com/solutions/oneapi/for-cuda/).

On a related note, we are also contributing to a HIP backend for AMD GPU support in DPC++, which is currently experimental and supported on Linux only. There are instructions for how to build with the HIP backend on the DPC++ getting started guide (https://intel.github.io/llvm-docs/GetStartedGuide.html#build-dpc-toolchain-with-support-for-hip-amd).

We are planning to make a binary package available in the first quarter of 2022. It won't be part of the Intel oneAPI releases but we will make it available to download via the Codeplay website and will update it on a regular basis. This binary release will also come with more documentation and guides to help you get the most out of it.

If you would like to talk to us directly about our plans or have any questions you can also email us at sycl@codeplay.com

clang –x=CUDA in the public LLVM trunk is something experimental done outside of NVIDIA; you should contact the llvm-dev alias with questions.

NVCC doesn't support emitting LLVM IR.

Thanks to contributions from Google and others, Clang now supports building CUDA. Command line parameters are slightly different from nvcc, though. According to the official documentation, assuming your file is named axpy.cu, the basic usage is:

$ clang++ axpy.cu -o axpy --cuda-gpu-arch=<GPU arch>  \
    -L<CUDA install path>/<lib64 or lib>              \
    -lcudart_static -ldl -lrt -pthread

Note that using Clang for compiling CUDA still requires that you have the proprietary CUDA runtime from the NVIDIA CUDA toolkit installed.