LLVM Compiler/Installation
Installing Pre-build Binaries
[edit | edit source]on Ubuntu
- sudo apt-get install clang llvm
Installing from github master
[edit | edit source]Hardware/Operating system
- Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
- Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card https://developer.nvidia.com/cuda-gpus .
- Linux (ubuntu 18.04)
Prerequisites
[edit | edit source]sudo apt update sudo apt install build-essential sudo apt install cmake sudo apt install -y libelf-dev libffi-dev sudo apt install -y pkg-config
Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda
After installation, export two environment variables, vim ~/.bashrc to add two lines
- export PATH=$PATH:/usr/local/cuda-10.2/bin/
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/
Get source packages
[edit | edit source]Assuming you are in your home directory
cd
git clone https://github.com/llvm/llvm-project.git
Build using GCC
[edit | edit source]mkdir build cd build/ cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \ -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37 \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75 \ -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -G "Unix Makefiles" ../llvm-project/llvm time make -j time make -j install
Explanation for configuration options
- -DCMAKE_C_COMPILER=gcc // the C compiler used to compile clang/llvm, GCC
- -DCMAKE_CXX_COMPILER=g++ // the C++ compiler used to compile clang/llvm: G++
- -DLLVM_TARGETS_TO_BUILD=X86;PowerPC;NVPTX;AMDGPU // explicitly specify target devices to support, Intel ,Nvidia, IBM, and AMD CPUs or GPUs
- -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 // default GPU computing capability version to support, https://developer.nvidia.com/cuda-gpus lists such information.
- -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 // all GPU computing capability versions to build in libomptarget
Other optional options
- -DGCC_INSTALL_PREFIX=${GCC_PATH}
- -DCMAKE_C_COMPILER=${GCC_PATH}/bin/gcc
- -DCMAKE_CXX_COMPILER=${GCC_PATH}/bin/g++
- -DCMAKE_Fortran_COMPILER=${GCC_PATH}/bin/gfortran
- -DCUDA_PATH= // this option should be automatically set if cuda is in your search path
- -DCUDA_TOOLKIT_ROOT_DIR= // this option should be automatically set if cuda is in your search path
- -DOPENMP_ENABLE_LIBOMPTARGET=ON // this should be on by default
- -DLIBOMP_FORTRAN_MODULES=ON
- -DBUILD_SHARED_LIBS=OFF // turn off shared libs
More examples
- cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" -DCMAKE_INSTALL_PREFIX=/Users/abc/llvm-research/inst-10.0.1 -DCMAKE_BUILD_TYPE=Debug ../llvm/
- ninja -j8
- ninja install
Rebuild using Clang
[edit | edit source]add the path to the installed clang, vim ~/.bashrc
export PATH=~/llvm/bin:$PATH export LD_LIBRARY_PATH=~/llvm/lib:$LD_LIBRARY_PATH cd build-openmp cd build-openmp/ cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \ -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37 \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75 \ -DCMAKE_C_COMPILER=clang \ -DCMAKE_CXX_COMPILER=clang++ \ -G "Unix Makefiles" ../llvm-project/llvm make -j make -j install
Optional options, explictly turn on bitcode lib, and the compiler/linker to build it
- -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true
- -DLIBOMPTARGET_NVPTX_CUDA_COMPILER=${PREFIX}/bin/clang
- -DLIBOMPTARGET_NVPTX_BC_LINKER=${PREFIX}/bin/llvm-link
Installing from releases
[edit | edit source]Hardware/Operating system
- Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
- Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card here.
- Linux (ubuntu 18.04)
Prerequisites
[edit | edit source]sudo apt update sudo apt install build-essential sudo apt install cmake sudo apt install -y libelf-dev libffi-dev sudo apt install -y pkg-config
Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
Assuming you current path is
- /home/ubuntu/omp5-gpu-llvm
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda
After installation, export two environment variables, vim ~/.bashrc to add two lines
- export PATH=$PATH:/usr/local/cuda-10.2/bin/
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/
Get source packages
[edit | edit source]Three steps to download, untar, and put them into the right locations.
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/clang-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/openmp-10.0.0.src.tar.xz wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/compiler-rt-10.0.0.src.tar.xz tar xf llvm-10.0.0.src.tar.xz tar xf clang-10.0.0.src.tar.xz tar xf openmp-10.0.0.src.tar.xz tar xf compiler-rt-10.0.0.src.tar.xz mv clang-10.0.0.src llvm-10.0.0.src/tools/clang mv openmp-10.0.0.src llvm-10.0.0.src/projects/openmp mv compiler-rt-10.0.0.src llvm-10.0.0.src/projects/compiler-rt
In the end, the directory layout should look like
- llvm-10.0.0.src
- tools/clang
- projects/openmp
- projects/compiler-rt
Build the Compiler with OpenMP offloading support
[edit | edit source]You need to know the Compute Capability version of your GPU. https://developer.nvidia.com/cuda-gpus lists such information. For example, some typical GPUs and their CC versions are:
- Tesla K80 3.7, sm_37
- Tesla P100 6.0, sm_37
- Tesla V100 7.0, sm_37
mkdir build cd build # this step is to generate a make file using cmake. picking gcc/g++ as the compiler cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \ -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37 \ -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 ../llvm-10.0.0.src # the screen output of the step above should show the following info: # -- Found LIBOMPTARGET_DEP_CUDA_DRIVER: /usr/lib/x86_64-linux-gnu/libcuda.so # -- LIBOMPTARGET: Building offloading runtime library libomptarget. # -- LIBOMPTARGET: Building CUDA offloading plugin. # -- LIBOMPTARGET: Building x86_64 offloading plugin. make -j6 make install -j6
After the installation, you should expand your PATH and LD_LIBRARY_PATH again
- export PATH=$PATH:/home/ubuntu/omp5-gpu-llvm/install/bin
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/omp5-gpu-llvm/install/lib
Rebuild the OpenMP runtime libraries
[edit | edit source]You should use the freshly installed clang to rebuild the OpenMP runtime library
cd /home/ubuntu/omp5-gpu-llvm mkdir build-openmp cd build-openmp cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \ -DCMAKE_C_COMPILER=$(pwd)/../install/bin/clang \ -DCMAKE_CXX_COMPILER=$(pwd)/../install/bin/clang++ \ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 \ ../llvm-10.0.0.src/projects/openmp make -j6 make install -j6
Test the installation
[edit | edit source]save the following code into a file named ongpu.c
#include <stdio.h> #include <omp.h> int main() { int runningOnGPU = 0; /* Test if GPU is available using OpenMP4.5 */ #pragma omp target map(from:runningOnGPU) { if (omp_is_initial_device() == 0) runningOnGPU = 1; } /* If still running on CPU, GPU must not be available */ if (runningOnGPU) printf("### Able to use the GPU! ### \n"); else printf("### Unable to use the GPU, using CPU! ###\n"); return 0; }
Compile and run it
clang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda ongpu.c ./a.out ### Able to use the GPU! ###
Troubleshooting
[edit | edit source]error while loading libomp.so
[edit | edit source]./a.out: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory solution
- export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathTo/installed-llvm/lib
External Links
[edit | edit source]- OpenMP 5 GPU offloading instructions [Fabio]: https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions--%5BFabio%5D
- Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs: https://hpc-wiki.info/hpc/Building_LLVM/Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs
- OpenMP 5 GPU offloading instructions with Clang 11 [Italo] https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions-with-Clang-11-%5BItalo%5D
- a docker file showing LLVM installation: https://github.com/freeCompilerCamp/play-with-compiler/blob/master/dockerfiles/dind/Dockerfile.llvm10