Add authentication to add-cluster command options

[maowerner]

Currently the authentication options are not supported when using kaf config add-cluster.

The examples are very helpful for setting up a cluster with authentication and I got everything working fine by editing the configuration file directly, but it could be more beginner-friendly.

Are there any pitfalls to do this? As far as I can see, it should be sufficient to add more options in

kaf/cmd/kaf/config.go

Line 172 in 4fef082

cfg.Clusters = append(cfg.Clusters, &config.Cluster{

I could offer to contribute this myself :)

[maowerner]

I looked into the Kafka security protocol https://kafka.apache.org/31/javadoc/org/apache/kafka/common/security/auth/SecurityProtocol.html
Basically the options that would be required are

• plaintext
• sasl_plaintext
• sasl_ssl
• ssl

This is not reflected in Sarama https://github.com/Shopify/sarama/blob/main/config.go where SSL and TLS can both be optionally configured.

In the example files there is also a mechanism property which is a lot more approachable than SASL with all its optional flags.

I basically see three ways the interface could work:

1. Add a new flag --security-protocol (plaintext|sasl_plaintext|sasl_ssl|ssl)

• pros
  • As securityProtocol is a struct in Kafka it is very easy to make the transition from the Kafka docu to kaf
• cons
  • I personally do not find this very intuitive if you don't come from the Kafka docu.
  • Either all sasl options must be presented or the sasl mechanism introduces yet another hierarchy

2. Add new boolean flags --sasl and --tls

• pros: All possible combinations are valid security protocols. This is quite a neat.
• cons:
  • Either all sasl options must be presented or the sasl mechanism introduces yet another hierarchy
  • Not future-proof. If at some point Kafka offers another mechanism it might be difficult to reflect.

3. Add new flags --authentication (sasl-plain|sasl-oauthbearer|sasl-scram-sha-512|tls) and --encryption

• pros:
  • I personally find it very intuitive to think of authentication and encryption as two different concerns, even if I can address both with TLS. Also authentication and encryption are less technical than sasl and tls and therefore beginner-friendly
  • There is no nested hierarchy required as the mechanism is directly exposed
• cons:
  • Not all possible combinations are a obviously valid securityProtocols. If authentication is left out but encryption is set or authentication = tls but encryption is left out, the securityProtocol should be ssl but it is not clear to a beginner, that traffic is still authenticated and encrypted.

I personally prefer option 3. One thing I like a lot about kaf is how much more intuitive it makes it to interact with Kafka - especially for programmers that have not worked with Kafka before. And I think two large reasons for this are better naming and less option compared to e.g. the Kafka CLI.

Authentication and Encryption are an entry barrier that I would like to help make as low as possible. I think having a flag for each of them would serve the purpose.

[birdayz]

Thanks so much for working on this and bringing these ideas!

I tend to agree. Some config options of kafka - especially in the SASL area - are hard to understand. So i share your approach of making it as accessible as possible - even if things are presented differently.

What exactly would --encryption do? Is it a boolean? I didn't quite understand this one.
It would turn on TLS?

[maowerner]

Glad to here this :)

If authentication is presented this way, it must be possible to differentiate between sasl_plaintext and sasl_ssl. I indeed thought of a boolean. If it is set, then I would also require the TLS parameters. And even though it is more consistent with authentication (sasl-plain|sasl-oauthbearer|sasl-scram-sha-512|tls) something like encryption (tls) might be confusing because there is only one option.

Do you feel like booleans --encrypt or --encrypted or even longer options like--enable-encryption or --enable-tls would be clearer?

I personally think --encrypted to turn on encryption would be the most intuitive.

[birdayz]

The name should explicitly about "tls", i.e. --tls or --enable-tls. --encryption sounds way too generic for what it is.

What about this:
--enable-tls will enable TLS. It will unlock usage of certain flags for client cert,key, server ca, skip cert validation, and whatever else is possible with TLS.
--sasl-mechanism, with all the existing sasl mechanism. Some options will be mutually exclusive with the TLS setting, i.e. SASL/PLAIN is incompatible with --enable-tls.

from what i see, this is more like your approach 1).
it should be pretty much in line with how kaf works at the moment, i think.

[jackcipher]

@maowerner @birdayz
FYI
the following flags have been throughly tested.

      --sasl-enable                                              Connect using SASL
      --sasl-jaas-config-file string                             Location of JAAS config file with SASL username and password
      --sasl-method string                                       SASL method to use (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 (default "PLAIN")
      --sasl-password string                                     SASL user password
      --sasl-plugin-command string                               Path to authentication plugin binary
      --sasl-plugin-enable                                       Use plugin for SASL authentication
      --sasl-plugin-log-level string                             Log level of the auth plugin (default "trace")
      --sasl-plugin-mechanism string                             SASL mechanism used for proxy authentication: PLAIN or OAUTHBEARER (default "OAUTHBEARER")
      --sasl-plugin-param stringArray                            Authentication plugin parameter
      --sasl-plugin-timeout duration                             Authentication timeout (default 10s)
      --sasl-username string                                     SASL user name
      --tls-ca-chain-cert-file string                            PEM encoded CA's certificate file
      --tls-client-cert-file string                              PEM encoded file with client certificate
      --tls-client-key-file string                               PEM encoded file with private key for the client certificate
      --tls-client-key-password string                           Password to decrypt rsa private key
      --tls-enable                                               Whether or not to use TLS when connecting to the broker
      --tls-insecure-skip-verify                                 It controls whether a client verifies the server's certificate chain and host name
      --tls-same-client-cert-enable                              Use only when mutual TLS is enabled on proxy and broker. It controls whether a proxy validates if proxy client certificate exactly matches brokers client cert (tls-client-cert-file)