Mixtral training

[ChristianPala]

Hey folks,

when using:
TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ

or the more recent and recommended:
casperhansen/mixtral-instruct-awq

with the training script provided in the examples, I get a runtime exception:

Exception has occurred: RuntimeError

CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
File "/home/christian.pala/models-zoo/models_zoo/main.py", line 5, in main cli.main(prog="models-zoo") File "/home/christian.pala/models-zoo/models_zoo/main.py", line 9, in main()RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)

Is adapter training not supported for mixtral?