Topics
- MSK - Things to consider
- MSK Connect
- MSK Serverless
- Monitoring
- Kafka Log Retention and Cleanup Policies
- Mirror Maker 2
- Other References
- Lack of .NET, node and python support for MSK IAM auth
- IAM role and Kafka ACL mapping is challenging, especially cross account access
- Cross account Lambda using a MSK as event source (currently not supported)
- Auto storage scaling can be a trap - team may run a perf test without changing topic retention, once storage got scaled up, can only recreate the cluster, and it's a challenge to update all endpoint references/integrations (custom domain name not supported)
- Currently mTLS is not supported
- Limitations and things to consider - The Pro’s and Con’s of using AWS MSK Serverless, Sinnappu J., 2022-07-18
- MSK Serverless supports only IAM for authentication and authorization.
- MSK Serverless (and Provisioned) AWS_MSK_IAM’ supports Java (or any other JVM language) via aws-msk-iam-auth module. No GoLang and Python libraries available to connect to MSK serverless using the 'AWS_MSK_IAM' mechanism.
- MSK Serverless cluster has a maximum write throughput, read throughput, and number of partitions allowed - (not for very large workloads and high intensity events based use cases)
- Monitoring an Amazon MSK Cluster (AWS Developer Guide)
- Monirtoring Consumer-Lag with LinkedIn Burrow
- Monitoring with Prometheus
- Ref-1: https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html
- Ref-2: https://medium.com/@sunny_81705/kafka-log-retention-and-cleanup-policies-c8d9cb7e09f8, 2019-07-28
- Ref-3: https://www.allprogrammingtutorials.com/tutorials/configuring-messages-retention-time-in-kafka.php
- See example mskconfig.properties