| Function | Purpose | | :--- | :--- | | cublasLtCreate | Initialize the library handle. | | cublasLtMatmulDescCreate | Create the GEMM operation descriptor. | | cublasLtMatrixLayoutCreate | Define dimensions and memory layout for matrices. | | cublasLtMatmulPreferenceCreate | Define constraints for kernel selection (workspace). | | cublasLtMatmulAlgoGetHeuristic | Find the best kernel for the grouped problem. | | cublasLtMatmul | Execute the grouped matrix multiplication. |
Enter – a game changer for batched, variable-sized matmul operations. cublaslt grouped gemm documentation
Configure cublasLtMatmulDesc_t with the desired compute precision (e.g., CUDA_R_16F ) and epilogue functions (like ReLU or bias addition). | Function | Purpose | | :--- |
In legacy cuBLAS , grouped GEMM often requires specific function pointers (e.g., cublasGemmGrouped ). In cuBLASLt , grouped functionality is invoked via the generic cublasLtMatmul but is configured using or by treating the inputs as an array of problem descriptors. | Enter – a game changer for batched,
NVIDIA reports speedups of up to 1.2x in MoE generation phases when using grouped APIs over standard batched alternatives.
If you're working with (e.g., in LLM inference, attention mechanisms, or recommendation systems), you’ve likely hit the overhead of launching many separate GEMM kernels.