The future of NonOS - please consider open-sourcing closed libraries

[Misiu]

I've asked this in #205, but because it was closed not everyone had the option to see it and answer it.

@xcguang I'd like to ask about what are Espressif plans for NonOS SDK?
Many people are wondering what will happen to it because many awesome projects depend on it, I think about Arduino Core for ESP8266.
There is a discussion started by @devyte about the future of this awesome project.
There is a question asked by @TD-er:

If Espressif is actually putting NonOS to EOL, is there still any reason for them to keep their core libraries closed?

I'd like to ask @igrr @kedars @me-no-dev @projectgus and @Spritetm (because You buys are listed in People section on Github) about Your thoughts about opening those closed libraries.

If NonOS will be discontinued at some point why don't allow other projects (NodeMCU and Arduino Core) to be able to still function without a mass code rewrite.

Projects like Arduino Core and NodeMCU and countless others are the reason why ESP8266 is so popular.
People (at least this was my case) don't choose chip because it looks nice and has feature x, but because there is a great community around it, there are tons of examples and libraries and there are people that spend their free time to make those projects, examples and are dedicated to helping newbies like me.

As @devyte mentioned in this issue one of the option to still maintain Arduino Core is to stick with NonOS and reverse engineer the closed libs. If Espressif would open those libs then that mass effort could be spent on creating new features or fixing bugs instead of recreating parts that already work.

[kriegste]

It might not be an option for the rf code, i.e. some libs like libphy.a (for legal reasons). But for others this should be possible, I hope...

[me-no-dev]

My honest opinion (that has nothing to do with the fact that I am listed in the repo) is that 8266 cores (Arduino, Node-MCU and such) should move to bugfix stage and efforts should go to newer chips that can do much more than what 8266 can do. ESP32 is not much more expensive than 8266 and price will go down if people use it. The new single-core cheaper chip (that should make 8266 obsolete) is also around the corner and will have many cool things.

Another thing is that 8266 NON-OS has an API that is not at all compatible with things like IDF, which means that special effort is necessary to support it. The RTOS flavour is moving towards IDF and that makes it much easier to support and fix on the long run.

I think also that it is fair to ask free open projects maintainer to put a bit of work and switch to RTOS. That will make porting to new chips either possible out of the box or with minimal changes.

As for opening Non-OS (except a few libs for obvious reasons), I am all for it.

[davydnorris]

My 0.02 on the matter...

Right now I would love to move to the ESP32 but, while not expensive, it's still over double the price of the ESP8266. More importantly it's more power hungry, and that's a huge issue for me. I am hoping the single core upgrade is more lean, but for now I am very happy with the ESP8266.

As far as the new RTOS SDK goes, I will need to port my code but the big worry I have is that I'm doing a lot of audio processing in real time and I have less control over the way time is allocated in the RTOS compared to the NonOS SDK. It may not be a problem, but right now I just don't know, and the effort to port my code may be wasted if I can't get the performance I need.

[eriksl]

For me the obligatory use of RTOS is exactly the reason I don't want to use the ESP32. I am not looking for a hobbyist playground environment, I am looking for a microcontroller. And a microcontroller is controlled by hardware, not some sort of API layer (that doesn't meet my needs and has bugs too).

So that's why, as well, I'd like to see the SDK code open source. Actually, I don't even need that. If enough information is publicised about the hardware, we could create and maintain something like it in the community. Then, understandably there should be a small closed source part that controls the wlan module. But now most of the closed code has nothing to do with wlan, and lots of low-level hardware information not related to wlan is still not revealed. Espressif should take ATmel or MicroChip as an example, they produce 200-300 page datasheets that contain EVERYTHING about the hardware. Nothing secret and properly documented. They can be downloaded by everyone, no NDA's. Not a few examples thrown over a fence like Espressif tends to do.

[Misiu]

@xcguang can we have an official statement? Countless projects use https://github.com/esp8266/Arduino and all of them use espressif chips, so helping with improving the tools = more projects = more chips == more money for espressif 🙂
As @eriksl wrote, probably some wlan code must stay closed, but please open the rest.

[davydnorris]

I know there are several of us who would volunteer our time to maintain this NonOS repo. @someburner and I talked about it a couple of months back.

I would even be happy to sign an NDA if it meant we could maintain this port.

[xcguang]

Thanks very much for your attention to this project. I will initiate a discussion again in the next internal meeting. If there is any progress, I will update it here as soon as possible.

[MaBecker]

Espruino for ESP82xx is also based on NonOS.

[me-no-dev]

I really do not agree with @eriksl up there. You just can not compare ESP and AVR... whatever without WiFi on it. Even in NonOS you do not have the whole CPU time. You have to yield and have no clue when you will get the execution back. Arguing that NonOS is more MCU than RTOS is totally wrong in this case IMO. ESP32 are twice the price because you do not buy it as much... it's not espressif's price, it's board manufacturer's price.
As for documentation: while, yes, the ESP8266 is not well documented, everything after it is. When I wrote the first hardware drivers for ESP8266, months before we were able to compile C++ on it and run Arduino, all we had was a header with list of registers and some code, i think only for SPI something. Lack of documentation has never stopped ESP8266 before, you have nothing to complain for ESP32 and on.

[artua]

@xcguang fingers crossed

[xcguang]

Hi all，
We decided to continue to maintain the project, but focus on the system and WiFi, and not add any feature.

[TD-er]

@xcguang Do you also have a rough estimate on how long you plan to maintain the project?
For example X months/years after the last ESP82xx is produced?

Also what will be the focus on?

• Stability
• Security fixes
• Compatibility (WiFi/build environment/etc)
• Something else?

[d-a-v]

WPA2 Enterprise, (TTLS / PAP) 😁 ?
We already have MSCHAPv2 (password encrypted).
PAP (unencrypted) is mandatory in networks where user directories like OpenLDAP are used for authentication.
That would be a nice WiFi addition especially in universities using the eduroam network.

[uzi18]

@xcguang nice to hear it, but it will not be endless, so maybe we can ask to open it one by one and/or add no-wifi mode?

[davydnorris]

@xcguang this is good news! Offer to assist still applies and I know @someburner and others would be interested.

I would particularly like to get the lwip and mbedtls libs at least to same version as RTOS, and to refactor like @eriksl and @d-a-v have attempted so that they can then be regularly merged with the public releases. This would mean opening up some of the lower level libraries in order to get to the data structures to upgrade to lwIP2 - would Espressif consider this?

I have also found some serious problems with the MQTT library and am about to start building one that will work with both NonOS and RTOS releases. I thought the new RTOS MQTT lib was going to be great but unfortunately it's too locked into the RTOS, and from my experience the receive side of messaging systems work better with event driven or callback approaches instead of polling a socket, but if the link to transport was done as event/callback then it wouldn't matter what you used underneath

Anyway, I am happy to help out

[eriksl]

Completely agree with @davydnorris. Please refactor the open source bits so they're truly open source and can be updated with the current versions. Maybe it's possible to open up some Espressif own sources too.

[Misiu]

@xcguang thank you for the good news, but as many wrote I'm sure that this won't last forever.
There are many situations and issues that would benefit from opening some closed sourced libraries.
For example, this issue. Idea is to be able to use ESP8266 without WIFI, so in theory, all WIFI related code could be skipped.
Bu as @devyte wrote here this will be hard with closed libraries.
many projects would benefit from this, like Tasmota. I'm sure @Jason2866 will agree on that.

I know that there are some libraries that probably will never get open, but I'm sure some parts can.
The community is your friend not your enemy. With open-sourced libraries, the community can fix and optimize them.

[Misiu]

@xcguang any updates about opening some parts? There is an idea to allow compiling code without WiFi parts (when wifi is not needed). This would free a lot of heap, but without your (Espressif) support it will be very hard. More details here: esp8266/Arduino#6680 (comment)

[someburner]

Yeah honestly why don't you guys just say it's licensing issues and it cannot be released? And if it's not due to licensing then what is the rationale? If wifi libs were open it would make this chip so much more attractive for production use- router compatibility issues could be solved by community and would be rock solid presumably.

[eriksl]

Besides this, I repeat, I cannot believe all of the closed stuff is either licensed/NDA or FCC-sensitive.

A (large imho) part of it doesn't even need to be opened up if more specs are released. With specs the same story, I can't believe that the not-revealed specs are all either NDA or FCC-sensitive (think SPI or timer registers for example, I am not talking about the WiFi stuff at all).

But Espressif has made up their minds already.

[Jason2866]

The great time of the ESP82xx is over. Espressif could extend this time with releasing
infos or libraries to the community.
New competitors and new possibilitys will come. See Tuya platform. No long time ago there
was only the ESP82xx. Now there are a few other chips in the Tuya universe...
For DIY maybe interesting W600-PICO

[cnlohr]

For everything esp82xx needs now, this NONOS SDK system is sufficient and continue to be sufficient moving forward. More projects in my purview are using ESP-NOW in broadcast mode and that's been rock solid since 1.5.2. I think the 8266 will still be a mainstay for me, using the NONOS_SDK.

Most of my frustration surrounding this is just by how much FreeRTOS gimps the chips. They have so much power when unbridled, but it's getting more and more difficult to truly access that power. The threads queues semaphores aren't the right pattern for a lot of projects, and it's frustrating that Espressif has chosen to force everyone to that system.

Many different platforms have the ability to use them as either a microcontroller or a OS. It's not really a personal preference but the right-tool-for-the-right job. Whether you have a main loop or you have noodles of tasks squirreling around everywhere.

My expectation is that in time, people will have systems for less and less onus on FreeRTOS and allow for better interface functions, making the system look-and-feel like a microcontroller again. I still need to get around to releasing my TCP/IP stack which is ~2x performance when using websockets. Honestly, the only thing I did was just use callbacks, interrupts and minimize context switches and queues. By butchering the paradigm, I think it's reasonable that most of FreeRTOS's drawbacks might be mitigated.

[davydnorris]

Agree with @cnlohr - for me right now the 8266 with NONOS gives me a stack of computing power without chewing the current too much, to the point where it's a cheaper and better solution than any of the other wireless protocols that people think are cool. Many of the cities I work with ask about whether we will use LoRaWAN and I say "Why? I just use your free wifi. It's way better performing and 25% of the price of a LoRa solution".

Right now I am doing some really quite expensive DSP type work on chip - if I move to the RTOS SDK I won't be able to and I'll have to add something like a TI DSP, which will double my power and expense budget, so I'm not moving from NONOS unless I have to, and if that day comes I may have to look at other SoCs

[eriksl]

General consensus, I think, that RTOS may be suitable for many developers. I think especially for not-so-advanced developers or firms that need to have a limited development span to remain profitable.
But others (like me (we?)) have the time (and the urge) to make it all ourselves to properly make use of all the raw power that's in the ESP8266. Every piece of SDK library code that is obligatory to use is one piece too much. We are not interested in a sort of Arduino environment where everything is "easy" (and inherently inefficient).
But I already learned that we, as individual hobbyist or small-company developers, are in no way interesting for Espressif. Simply too small market share to have any influence on ESP8266 sales, so why bother. I understand their standpoint, really, but it makes me sad.

[Lan-Hekary]

For everything esp82xx needs now, this NONOS SDK system is sufficient and continue to be sufficient moving forward. More projects in my purview are using ESP-NOW in broadcast mode and that's been rock solid since 1.5.2. I think the 8266 will still be a mainstay for me, using the NONOS_SDK.

@cnlohr
I'm wondering about using ESP-NOW Broadcasting with SDK v1.5.4 ..
I can't use it .. and the official document here says it's supported form v2.1 ..

I am asking about this because I am using the ESPUSB with ESP-NOW ..
and I need to be able to broadcast messages but I can't upgrade to newer SDK or the USB won't work.