Support Arduino ESP32 alpha/RC v3.0.0 based on ESP-IDF v5.1

[sgryphon]

PlatformIO already supports overriding platform packages to use an alternative or development version of a package, however a missing library dependency means this doesn't work for arduino-esp32 alpha v3.0.0.

This PR adds a reference to framework-arduinoespressif32-libs SDK (aka esp32-arduino-libs), so that the latest arduino-esp32 alpha can be used. But only if you override platform packages to the alpha 3.0.0 version, which references libs, and also point at the libs location.

The change is backwards compatible, i.e. does not break any existing code, as the library framework-arduinoespressif32-libs is only included if it is referenced in an overridden arduino-esp32 manifest.

This fixes issue #1225 and #1211, i.e. if you are getting the error "KeyError: 'framework-arduinoespressif32-libs':" due to the missing library (when you reference an alpha v3 version of arduino-esp32)

To use, configure your project to reference the latest arduino-esp32, which will include a reference to framework-arduinoespressif32-libs. You also need to override the location of the libs (as they are not currently in the PlatformIO repository).

platform = https://github.com/sgryphon/platform-espressif32.git#sgryphon/add-esp32-arduino-libs
platform_packages =
    platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#master
    platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1

If you use a older version of the platform package, e.g. the default, then libs are not needed, and won't be loaded. So the following also works fine (old version of platform package, no libs), or any override of the esp32-arduino platform package that does not include the libs (i.e. pre alpha v3) will also work:

platform = https://github.com/sgryphon/platform-espressif32.git#sgryphon/add-esp32-arduino-libs
platform_packages =

This change simply allows the arduino-esp32 platform package to be configured with alpha 3.0.0, if you want to test some advanced development (without affecting existing support).

Future work: The library dependency needs to be added to the PlatformIO Registry, and when 3.0.0 is released/stable, the dependency can be made Optional = False as the default.

Note: This change doesn't necessarily mean your code will work, only that the missing dependency will be there. Alpha 3.0.0 has breaking changes, so existing code/libraries may not be compatible. The esp32-arduino package does have some basic examples that do work, and are run as part of their integrations tests. https://github.com/espressif/arduino-esp32/blob/e4d6a8abf9fab7422e21f72d3aeb2e9c9484ab36/.github/scripts/on-push.sh#L92

[philm001]

Does this work for all ESP32 boards? Or just certain ones

[sgryphon]

Does this work for all ESP32 boards? Or just certain ones

This change isn't directly related to boards, but to PlatformIO setup. There is no board specific stuff.

All it will do is allow you to get past the error "KeyError: 'framework-arduinoespressif32-libs':", when setting up the platform, even before compiling anything.

Once past that error, you application will probably fail to compile as there are many breaking changes in Arduino ESP32 alpha v3 (and it is alpha).

e.g. Even a very basic example on my M5Core2 failed -- neither the M5Core2 or M5Unified libraries would even compile

However, removing the include reference to any M5 stuff, and just using core Arduino (e.g. start serial and output a counter), does build and run, so core platform stuff works.

This change to allow PlatformIO to at least set up the dependencies, will then allow peeps to test whether their libraries, etc will work with v3.0.0 and update them as necessary.

There is a migration guide https://docs.espressif.com/projects/arduino-esp32/en/latest/migration_guides/2.x_to_3.0.html

Note that the new libs will only be loaded if you override platform packages for arduino-esp32 to point to an alpha 3 version (that references it).

UPDATE: I was able to get some of M5Core2 working, using M5Unified, and then patching esp32-hal-ledc.h to include freertos/FreeRTOS.h and freertos/semphr. i.e. this change (allowing libs to be set up) then allows me to work out the incompatibility with alpha 3.0.0 and look for a fix.

[sgryphon]

Does this work for all ESP32 boards? Or just certain ones

As mentioned, this change (allowing the PlatformIO integration to set up the libs dependency) doesn't have anything directly to do with boards ... but what might be relevant is what the actual library supports.

The libs referenced in the example configuration is for ESP-IDF v5.1 from https://github.com/espressif/esp32-arduino-libs

Inside that ZIP file are folders for esp32, esp32c3, esp32c6, esp32h2, esp32s2, and esp32s3

Note that this change (in the setup scripts) doesn't directly reference this library ... all the change does is allow you to add the reference to your platform.io config (well, it does define a default of 5.1.0, but the library hasn't be added to the PlatformIO registry so that won't do anything anyway -- which is why it is optional and only configured if you reference an alpha platform that includes it).

You could also configure libs to point at a custom build, e.g. something based on ESP-IDF 5.2 or 5.3.

This change is about giving you the ability to make that reference, not what is necessarily what is in the reference. As mentioned alpha 3.0.0 has a bunch of breaking changes, so your app might not work or even compile.

[Jason2866]

For C6 and H2 custom boards.json are needed. The existing ones (if there are any) don't have the entry to support Arduino as framework.

[connornishijima]

Sure wish I knew what was holding back 3.0.0/IDF 5.1.x adoption in platformIO, thank you VERY much for figuring this setup out.

[sgryphon]

Sure wish I knew what was holding back 3.0.0/IDF 5.1.x adoption in platformIO, thank you VERY much for figuring this setup out.

Probably the main reason is that 3.0.0 is only RC1, and not officially released yet.

PlatformIO does allow overriding of the URLs, so you can usually easily point to an updated / RC version (or even your own custom one). That has always been supported.

In this case it is a bit different, because the base script needs to account for a new libs component (what this PR does).

They could merge this PR now; it doesn't break anything existing, and enables support for the alpha/RC version. But it is also reasonable for them not to merge in something that is only for an unreleased thing (e.g. what happens if it changes, e.g. 3 components or merges back into 1).

[thijstriemstra]

Probably the main reason is that 3.0.0 is only RC1, and not officially released yet.

I highly doubt this. #1225 clearly states espressif dropped platformio starting from 3.0. We'll see...

[songpola]

Probably the main reason is that 3.0.0 is only RC1, and not officially released yet.

I highly doubt this. #1225 clearly states espressif dropped platformio starting from 3.0. We'll see...

They updated that they didn't drop PlatformIO support entirely.
They just stated that the PlatformIO supports are not maintained by them officially, but are maintained by the PlatformIO community.

[sgryphon]

#1225 clearly states espressif dropped platformio starting from 3.0. We'll see...

#1225 is a post by Platform IO ... yes, it claims that Espressif has discontinued support, but Espressif has said no such thing. If you follow the link to the "official statement" it is a pull request updating their documentation.

Initially it removed details about PlatformIO from the Espressif documentation because it confused users (Espressif has no control over PlatformIO or anyway to provide support for it); if you follow down the PR, that was then changed (before merging) to keep the details and instead just having a warning that PlatformIO issues need to be reported to PlatformIO (not Espressif).

You can see the current (RC1) version of the page: https://github.com/espressif/arduino-esp32/blob/3.0.0-rc1/docs/en/installing.rst#installing-using-platformio

No where is there any mention of dropping Platform IO -- it's still right there.

[robertlipe]

Thank you for putting this together. I was steered here by LeCyberDucky after half-venting my frustration at the anger I had after discovering I'm running a six-year old G++ with something that "Uses ESP-IDF 5.1.3", not realizing that it actually uses an ancient version it squirrels away instead of using IDF's real compiler. So I started down the path of trying this and I smash into a "there's no possible way this can be this broken" issue before my first list of code is compiled. It's possible that I'm suffering from a configuration issue and it's possible that I'm posting in the wrong place (I respond well to guidance), but would you expect the following line of code to work?

~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-g++ --version $(cat ~/.platformio/packages/framework-arduinoespressif32-libs/esp32/flags/cpp_flags)

It's a trick question. The answer is, "no, no it doesn't."...
xtensa-esp32-elf-g++: error: unrecognized command line option '-std=gnu++2b'; did you mean '-std=gnu++2a'?

This, of course, breaks every compilation:
$ ~/.platformio/penv/bin/pio run -v --target upload

xtensa-esp32-elf-g++ -o .pio/build/demo/src/colordata.cpp.o -c -std=gnu++23 -Wno-frame-address -std=gnu++2b -fexceptions -fno-rtti -g3 -Ofast -ffunction-sections -fdata-sections -Os -mlongcalls -ffunction-sections -fdata-sections -Wno-error=unused-function -Wno-error [ pages and pages of -I spew, surely totally blowing out the OS's negative dnode cache, but I'm not there to fight that battle. ]

The offender is, of course, that gnu++2b when the tools are expecting 2a. Can't override it with build_flags because user build_flags come before and not after all this.

This seems to obvious to have not been noticed. I won't guess I'm the first person to try to build C++ code, so I'm guessing I've done something special to deserve this. Any ideas what?

Thank you!

[robertlipe]

Oh. Of course. I even said the answer myself. The problem is Platformio is being dumb. (Why should I have guessed this?)
If I could get Platformio to run ESPRESSIF's g++ instead of PLATFORMIO's g++, things would be better:

~/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20230928/xtensa-esp-elf/bin/xtensa-esp32-elf-g++ --version $(cat ~/.platformio/packages/framework-arduinoespressif32-libs/esp32/flags/cpp_flags)
xtensa-esp-elf-g++ (crosstool-NG esp-13.2.0_20230928) 13.2.0

I should have added (and paid attention to) --version sooner:

~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-g++ --version $(cat ~/.platformio/packages/framework-arduinoespressif32-libs/esp32/flags/cpp_flags)
xtensa-esp32-elf-g++: error: unrecognized command line option '-std=gnu++2b'; did you mean '-std=gnu++2a'?
xtensa-esp32-elf-g++ (crosstool-NG esp-2021r2-patch5) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.

Platformio's G++ is the one from the stone age.

This is a result of the following change in my platformio.ini
@@ -253,6 +253,11 @@ board_build.embed_files = site/dist/index.html.gz
site/dist/index.js.gz
site/dist/favicon.ico.gz
board_build.embed_txtfiles = config/timezones.json
+platform = https://github.com/sgryphon/platform-espressif32.git#sgryphon/add-esp32-arduino-libs
+platform_packages =

• platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#master
• platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1
  +board_build.arduino.upstream_packages = no`

Somehow, I'm missing the correct magic chant to tell it to use the ACTUAL G++ and not the ancient version.

Since I did admittedly go offstream on that last line without really understanding it, without it, it does report:
Error! Failed to extract upstream toolchainconfigurations:
Bad package version idf--c432c692fa
You can disable this feature via the board_build.arduino.upstream_packages = no setting in your platformio.ini file.

EDITED:
OK. The opening post led me astray. Maybe. I used
platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#master
But it seems the correct branch (url?) is
platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.0-rc1

With that, I get the correct compiler and everything explodes WONDERFULLY.

No, that might SOUND like a bad thing, but now that nothing is compiling, I know I'm using the "right" compiler and all this crazy volatile usage I've been seeing in arduino code all these years is coming home to roost.

Now I can move on to the real task at hand of mopping up the mess. But if my correction is correct, please consider updating the prose in that first post to save the next person some pain. If my correction isn't correct, well, please correct me. :-)

[sgryphon]

#master is volatile, so could end up with issues. It was working around the time of #3.0.0-rc1 (about 30 commits / 3 weeks ago).

Which version of arduino-esp32 and the libs you use may vary -- the platform script changes are simply to allow you to make the references. I'm glad you were able to find a branch that worked for you.

I will check to see if it is something about the current master, the platform scripts (may need rebasing), or another issue, e.g. changes in alpha/rc 3.0.0 may simply not be compatible, and update the notes accordingly.

[Jason2866]

It is a bug here https://github.com/espressif/arduino-esp32/blob/5a023568a640f7b55f40381a28389d05df3af9c5/package/package_esp32_index.template.json#L45
there is the branch missing
Using Arduino branch idf-release/v5.1 should work

[robertlipe]

That looks probable, Jason. Thanx for the analysis assistance and the upcoming inevitable fix. Glad to help move this forward, even if only in a small way.

…

On Thu, May 2, 2024, 7:17 AM Jason2866 ***@***.***> wrote: It is a bug here https://github.com/espressif/arduino-esp32/blob/5a023568a640f7b55f40381a28389d05df3af9c5/package/package_esp32_index.template.json#L45 there is the branch missing — Reply to this email directly, view it on GitHub <#1281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD32IBBQKCPNBTPLFHJTZAIVHJAVCNFSM6AAAAABB2NJKICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGM2TMNZZGM> . You are receiving this because you commented.Message ID: ***@***.***>

[connornishijima]

In case it helps anyone, here is my current platformio.ini that successfully compiles Arduino 3.0.0-rc1 (with IDF v5.1.3-470-g2fd087b246-dirty behind the scenes)

All I had to do was manually make the changes defined in sgryphon's PR to my platformio files while using this specific set of URLs:

platform = https://github.com/platformio/platform-espressif32.git#develop
platform_packages = 
    platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.0-rc1
    platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1

[Jason2866]

@connornishijima Yep, that's the simplest and easiest solution.

[Hedda]

FYI, Espressif has released "stable" version 3.0.0 Arduino ESP32 Core based on ESP-IDF v5.1.4 with support for ESP32-H2/ESP32-C6

https://github.com/espressif/arduino-esp32/releases/tag/3.0.0

Official documentation in arduino-esp32 v3.0.0 from Espressif still have instructions for installing using PlatformIO:

• https://github.com/espressif/arduino-esp32/blob/3.0.0/docs/en/installing.rst#installing-using-platformio

(Arduino ESP32 core v3.0.0 was first announced over 7-months ago but this updated version is now finally been marked as stable).

• Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1225

PS: Also means can report non-PlatformIO problems and feature requests related to upstream ESP32-H2/ESP32-C6 as new issues:

https://github.com/espressif/arduino-esp32/issues/

[djfm]

In case it helps anyone, here is my current platformio.ini that successfully compiles Arduino 3.0.0-rc1 (with IDF v5.1.3-470-g2fd087b246-dirty behind the scenes)

All I had to do was manually make the changes defined in sgryphon's PR to my platformio files while using this specific set of URLs:

platform = https://github.com/platformio/platform-espressif32.git#develop
platform_packages = 
    platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.0-rc1
    platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1

In case anyone is interested, as of right now the platformio.ini lines suggested in the quoted comment are working fine without any changes to the PlatformIO code itself contrary to what had been suggested at the time (i.e., this is cool), thank you @connornishijima.

I've had other issues with arduino-esp32#3.0.0-rc1 though, but right now #3.0.7 seems to be working fine - I mean, my program works as before, but I have not yet gotten to using the specific features for which I wanted to upgrade the platform, we'll see about that 🤞.