Test against OpenSSL 3

[davidhorstmann-arm]

Our current "OpenSSL Next" is OpenSSL 1.1.1. This will reach the end of its life in 6 months:

https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/

With this in mind, we should enable compatibility testing with OpenSSL 3.

[mpg]

Note: 3.0 or higher is also necessary in order to test FFDH in TLS 1.3, see #7627 (comment)

(Btw, our current "base" version is 1.0.2 which is already EOL. We might want another issue to upgrade the base version.) (Also, do we still need "legacy" to be that old? Probably yes for RC4 and 3DES ciphersuites in 2.28. This should be documented. Should probably be yet another issue.)

[mpg]

Note: 3.0 or higher is also necessary in order to test FFDH in TLS 1.3, see #7627 (comment)

I'm moving this issue in the same EPIC as FFDH support, as we want interop testing on the CI.

[mpg]

Note: this means currently all the cases that require OpenSSL 3.x are never executed in the CI. See #2691 - but also #5389 and #5390. CC @gilles-peskine-arm FYI.

[gilles-peskine-arm]

Should we replace OPENSSL_NEXT a by 3.x version (currently 1.1.1), or are you aware of any reason why we might need both 1.1.1 and 3.x?

[mpg]

I'm not aware of any reason we would need both, so I think just replacing OPENSSL_NEXT with 3.x should be good. Of course, with this kind of thing, we never know for sure until we try...

[gowthamsk-arm]

I ran ssl-opt.sh and compat.sh against a full config build using OpenSSL 3.0 in Ubuntu 22.04.

For the ssl-opt.sh, there seem to be 5 failures

keyUsage cli 1.3: DigitalSignature+KeyEncipherment, RSA: OK ............ SERVER START TIMEOUT FAIL
keyUsage cli 1.3: KeyEncipherment, RSA: fail ........................... SERVER START TIMEOUT FAIL
keyUsage cli 1.3: KeyAgreement, RSA: fail .............................. SERVER START TIMEOUT FAIL
keyUsage cli-auth 1.3: RSA, DigitalSignature: OK ....................... FAIL
keyUsage cli-auth 1.3: RSA, KeyEncipherment: fail (soft) ............... FAIL
And the skipped test cases were reduced from 96 to 41 when running against OpenSSL 3.0.

And for compat.sh, although there seem to be no failures many tests are skipped. (against OpenSSL 1.1.1 the skipped tests were 0)
PASSED (1200 / 1200 tests (106 skipped))

I'm currently investigating the failures. Will post further updates.

[gilles-peskine-arm]

We're currently working on switching OPENSSL_NEXT from 1.1.1 to 3.1.2. This breaks a number of tests though, so one way or another we need a transition period where each branch can decide whether it uses 1.1.1 or 3.1.2.

An alternative could be to introduce an OPENSSL_NEXT_NEXT, and gradually migrate components from OPENSSL_NEXT to OPENSSL_NEXT_NEXT, at which point we could get rid of OPENSSL_NEXT.

[gilles-peskine-arm]

I've been thinking about how we can upgrade versions of OpenSSL and GnuTLS on the CI. Currently, the docker files decide which version is xxx, xxx_LEGACY and xxx_NEXT. Changes to docker files must not break any recent-ish commit to Mbed TLS: not just the maintained branches, but also branches forked not too long ago, that people are still working on, in particular active pull requests. So it's not good if docker file changes can break working branches. The docker files should only decide what is available, and a file in each branch should decide what is actually used.

This hasn't been an issue until now because we have not upgraded OpenSSL or GnuTLS since the current Jenkins/Docker-based CI was put in place.

Therefore my plan is that the dockerfiles will provide e.g. OPENSSL_1_1_1, OPENSSL_3_1_2, etc. And it'll be up to all.sh to decide that OPENSSL_NEXT is OPENSSL_1_1_1 one day and OPENSSL_3_1_2 the next day, together with any necessary changes in e.g. ssl-opt.sh and compat.sh. We might move away from _LEGACY and _NEXT names at some point, but that's a future consideration.

The dockerfiles will keep providing the current bare/LEGACY/NEXT names for backward compatibility for a few months.

[mpg]

Note: we currently have OpenSSL 3.0 and 3.1 installed in the CI's docker images: https://github.com/Mbed-TLS/mbedtls-test/blob/master/resources/docker_files/ubuntu-16.04/Dockerfile#L196

So this is now about whether we are using them when we should.

[mpg]

Also, I was wondering what level of interop testing we currently have. So, I grabbed the outcomes.csv file from a random PR and grepped around.

% egrep 'component_test_tls13;(ssl-opt|tls13-).*1\.3' outcomes.csv | egrep -i 'open|gnu.*;PASS;$'
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Test openssl tls1_3 feature;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Test gnutls tls1_3 feature;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: minimal feature sets - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: minimal feature sets - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: alpn - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: alpn - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: server alpn - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: server alpn - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:gnutls: srv max TLS 1.0;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:gnutls: srv max TLS 1.1;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:gnutls: srv max TLS 1.2;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:openssl: srv max TLS 1.0;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:openssl: srv max TLS 1.1;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Not supported version check:openssl: srv max TLS 1.2;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, no client certificate - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, no client certificate - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, no server middlebox compat - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, no server middlebox compat - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp256r1_sha256 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp256r1_sha256 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp384r1_sha384 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp384r1_sha384 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp521r1_sha512 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, ecdsa_secp521r1_sha512 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha256 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha256 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha384 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha384 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha512 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, rsa_pss_rsae_sha512 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, client alg not in server list - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication, client alg not in server list - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, no server middlebox compat - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, ecdsa_secp256r1_sha256 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, ecdsa_secp384r1_sha384 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, ecdsa_secp521r1_sha512 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, rsa_pss_rsae_sha256 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, rsa_pss_rsae_sha384 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, rsa_pss_rsae_sha512 - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Client authentication - opaque key, client alg not in server list - openssl;SKIP;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: HRR check, ciphersuite TLS_AES_128_GCM_SHA256 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: HRR check, ciphersuite TLS_AES_256_GCM_SHA384 - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: HRR check, ciphersuite TLS_AES_128_GCM_SHA256 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: HRR check, ciphersuite TLS_AES_256_GCM_SHA384 - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - openssl;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - openssl with client authentication;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - gnutls;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - gnutls with client authentication;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - openssl with sni;PASS;
Linux-x86_64;component_test_tls13;ssl-opt;TLS 1.3: Server side check - gnutls with sni;PASS;

So it looks like we are getting some level of interop testing for 1.3 with both OpenSSL and GnuTLS as we stand.

(The question came up while discussing the priority level of this issue.)

[mpg]

So this is now about whether we are using them when we should.

I'll note we currently have a function requires_openssl_3_x that tests if OPENSSL_NEXT is 3.x. That makes no sense with the new scheme. Test cases that require OpenSSL 3.x should use OPENSSL_3 not OPENSSL_NEXT.