In this software package, we have implemented scalable multi-GPU kernel designs for four widely-used BLAS kernels: SpMV, SpTrsv, SpTrans and SpMM on NVIDIA DGX-1 and DGX-2 type of scale-up system, which is the typical configuration for state-of-the-art HPC like OLCF Summit and LLNL Sierra supercomputers.