Support for Albert and Swin/ViT

[larry-fuy]

Just curious whether there is a plan to support Albert and Swin/ViT. currently I am playing with a model for multimodal learning which involves language models like Albert and visual transformers like Swin and ViT. If there is no immediate plan for this due to tight hands, I am wondering whether there is any documents to guide adding support of new models or customized models so I could help?

[mrwyattii]

@larry-fuy there are no immediate plans to support these models, but we are very welcome to outside contributions. Our documentation could certainly be improved in terms of describing how to add new model support, so let me provide a starting point here and I would be happy to help in the process.

In general, to support a model we need to have support for kernel injection on the DeepSpeed-inference side (but we can also run models without kernel injection if we only want to use ZeRO). For example, we support automatic kernel injection with Bert, GPT-J, etc. models with DeepSpeed-inference. Kernel injection can also be achieved with custom injection policies. Check out this example from our unit tests on how to do this:
https://github.com/microsoft/DeepSpeed/blob/62c071e016aca6dbf12695bdb680721dcf3500ad/tests/unit/inference/test_inference.py#L408

Once we have this support on the DeepSpeed-inference side, adding the model to MII should be as easy as expanding the list of supported model types and tasks.

Please let me know if I can clarify anything further.