RESOLVED: Arduino-cli upload issue "panic: send on closed channel" with ESP32

Development Tools Arduino Command Line Tools
1

I have two ESP32 boards: one of our own design, one an ESP32 Dev-Kit.

From my PC I can upload to both boards, no problem, using ESP-IDF (which we have used with both of these boards for years) and arduino-cli.

However, on the PC that runs all of our tests, I cannot upload to our own-design board using arduino-cli: this board/PC works absolutely fine if I upload to the board with ESP-IDF and the ESP32 Dev-Kit is absolutely fine on the test PC with both ESP-IDF and arduino-cli but our own-design board, nada, never works with arduino-cli on the test PC. The own-design board hosts one of our nina-w10 modules, which is a standard supported ESP32 thing, and of course it works on my PC

The error we get appears to be a Golang assert "panic: send on closed channel":

arduino-cli upload -p COM55 --fqbn esp32:esp32:nina_w10 -v --input-file z:\_jenkins_work\build\app.ino.bin z:\_jenkins_work\build
Error initializing instance: loading platform release esp32:esp32@2.0.0: loading boards: skipping loading of boards esp32:esp32:atmegazero_esp32s2, esp32:esp32:deneyapmini: malformed custom board options
panic: send on closed channel

goroutine 23 [running]:
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1.1(0xc0006e2780, 0xc00060ff50)
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:153 +0x5e
created by github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:151 +0x1b8

I've verified that I can connect a serial terminal on COM55 and the board is there, all is fine, but arduino-cli doesn't like it for some reason. I don't believe that the skipped platforms are the issue since those are present in the success case also. The same version of arduino-cli is used on both PCs (0.19.0 Commit: 56419ecd Date: 2021-09-02T14:47:35Z).

Can anyone suggest what might be upsetting arduino-cli and how I might fix it? What is discoverymanager in any case and why do I need it (after all, I've specified the board type and COM port, there is nothing to discover), can I stop it getting in the way somehow?

2

It's worth noting that the test PC has a load of boards hanging off it which are absolutely nothing to do with Arduino [they are for other platforms] and are simultaneously in use for other purposes so if the discovery manager is trying to access them that would probably not be good...

3

Hi @robmeades could you try the new Release Candidate we just released? We made some modifications in the discovery manager and probably it solves your issue.

The discovery manager has been introduced recently in the Arduino CLI and, as its name implies, manages all the pluggable discoveries that are running. The discoveries are a new kind of tool that platform developers can create to add support for different board and protocols. By default the Arduino CLI has two builtin discoveries, serial-discovery and mdns-discovery, the former supports boards connect via a serial port, the latter boards connect on the local network.

You can find more detailed information in the official specifications:

4

Ooo, great, will go get it and report back.

5

Hmmm, different error message with, arduino-cli alpha Version: 0.19.1-rc2 Commit: e7db62f8 Date: 2021-09-24T08:51:07Z, see below. Anything else I can try? Perfectly happy to help debug etc.:

V:\jenkins\workspace\ubxlib_priv_feature_arduino_rmea\ubxlib\port\platform\common\automation>arduino-cli upload -p COM55 -l serial --fqbn esp32:esp32:nina_w10 -v --input-file z:\_jenkins_work\build\app.ino.bin z:\_jenkins_work\build
Error initializing instance: loading platform release esp32:esp32@2.0.0: loading boards: skipping loading of boards esp32:esp32:deneyapmini, esp32:esp32:atmegazero_esp32s2: malformed custom board options
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).QuitAll(0xc000b3aa90, 0xc00059da4c, 0x2, 0x2)
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:197 +0xd5
github.com/arduino/arduino-cli/cli/arguments.(*Port).GetPort.func1(0xc000b3aa90)
        /home/build/cli/arguments/port.go:97 +0x36
github.com/arduino/arduino-cli/cli/arguments.(*Port).GetPort(0x142bd60, 0xc000b38ae0, 0x0, 0xc0017bf3b0, 0x0, 0x0)
        /home/build/cli/arguments/port.go:111 +0x451
github.com/arduino/arduino-cli/cli/upload.run(0xc000b1d340, 0xc000b1be00, 0x1, 0xa)
        /home/build/cli/upload/upload.go:100 +0x268
github.com/spf13/cobra.(*Command).execute(0xc000b1d340, 0xc000b1bcc0, 0xa, 0xa, 0xc000b1d340, 0xc000b1bcc0)
        /go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000940dc0, 0x1, 0x1, 0x0)
        /go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:887
main.main()
        /home/build/main.go:31 +0xea

goroutine 21 [chan send]:
github.com/arduino/arduino-cli/cli.preRun.func1(0xc000b1d340)
        /home/build/cli/cli.go:169 +0xc9
created by github.com/arduino/arduino-cli/cli.preRun
        /home/build/cli/cli.go:159 +0x225

goroutine 12 [chan send]:
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1.1(0xc003124660, 0xc000b3aa90)
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:153 +0x5e
created by github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:151 +0x1b8

goroutine 50 [chan receive]:
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1.1(0xc000070ae0, 0xc000b3aa90)
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:152 +0x76
created by github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).StartSyncAll.func1
        /home/build/arduino/discovery/discoverymanager/discoverymanager.go:151 +0x1b8
6

@robmeades we tried a fix, could you try the build from this PR?

7

Woohoo, you did it, thank you!

arduino-cli upload -p COM55 --fqbn esp32:esp32:nina_w10 -v --input-file z:\_jenkins_work\build\app.ino.bin z:\_jenkins_work\build
Error initializing instance: loading platform release esp32:esp32@2.0.0: loading boards: skipping loading of boards esp32:esp32:deneyapmini, esp32:esp32:atmegazero_esp32s2: malformed custom board options
"C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\tools\esptool_py\3.1.0/esptool.exe" --chip esp32 --port "COM55" --baud 921600  --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0xe000 "C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/partitions/boot_app0.bin" 0x1000 "z:\_jenkins_work\build/app.ino.bootloader.bin" 0x10000 "z:\_jenkins_work\build/app.ino.bin" 0x8000 "z:\_jenkins_work\build/app.ino.partitions.bin"
esptool.py v3.1
Serial port COM55
Connecting....
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:96:af:24
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 2MB
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00001000 to 0x00005fff...
Flash will be erased from 0x00010000 to 0x00086fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Compressed 8192 bytes to 47...
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.1 seconds (effective 450.4 kbit/s)...
Hash of data verified.
Flash params set to 0x0210
Compressed 16544 bytes to 11405...
Wrote 16544 bytes (11405 compressed) at 0x00001000 in 0.5 seconds (effective 259.0 kbit/s)...
Hash of data verified.
Compressed 486672 bytes to 286321...
Wrote 486672 bytes (286321 compressed) at 0x00010000 in 8.5 seconds (effective 459.9 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 118...
Wrote 3072 bytes (118 compressed) at 0x00008000 in 0.1 seconds (effective 308.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
1 Like
8

Awesome! Glad we solved the issue, the fix will be available in the next 0.19.1 release, it will be out next week probably. For the time being the you can use 0.19.1-rc3 if you want, I just pushed so it's still building but shouldn't take more than 20 minutes.

9

I'll just stick with what I've got and update in a week or two to your latest and greatest. Thanks again!

10

Actually, there appears to be a subsequent issue (I can raise it in a new topic if you prefer). While the fix works well from the command-line, we actually call arduino-cli from a Python script as a sub-process and that works fine for the ESP32 Dev Kit board on COM8 (see first output below) but it fails for our own-build ESP32 NINA-W10 board on COM55 (see second output below). This is with your arduino-cli alpha Version: test-1472-f32845aa5d758b3f8d64b00700cab22f46af36dd-git-snapshot Commit: f32845aa Date: 2021-09-24T14:41:43Z. Is there anything I can do to workaround this (or help you fix it)? It is worth noting that we call all of our test builds/runs in parallel, hence more than one instance of arduino-cli may be active [on different boards] at the same time:

Success case:

2021-09-24_17:21:57 u_run_arduino_11.1: arduino-cli upload -p COM8 --fqbn esp32:esp32:esp32doit-devkit-v1 -v --input-file z:\_jenkins_work\11.1\build\app.ino.bin z:\_jenkins_work\11.1\build
2021-09-24_17:21:57 u_run_arduino_11.1: arduino-cli, pid 5012 started with guard time 300 second(s)
2021-09-24_17:21:59 u_run_arduino_11.1: Error initializing instance: loading platform release esp32:esp32@2.0.0: loading boards: skipping loading of boards esp32:esp32:atmegazero_esp32s2, esp32:esp32:deneyapmini: malformed custom board options
2021-09-24_17:22:09 u_run_arduino_11.1: "C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\tools\esptool_py\3.1.0/esptool.exe" --chip esp32 --port "COM8" --baud 921600  --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0xe000 "C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/partitions/boot_app0.bin" 0x1000 "z:\_jenkins_work\11.1\build/app.ino.bootloader.bin" 0x10000 "z:\_jenkins_work\11.1\build/app.ino.bin" 0x8000 "z:\_jenkins_work\11.1\build/app.ino.partitions.bin"
2021-09-24_17:22:10 u_run_arduino_11.1: esptool.py v3.1
2021-09-24_17:22:10 u_run_arduino_11.1: Serial port COM8
2021-09-24_17:22:11 u_run_arduino_11.1: Connecting....
2021-09-24_17:22:11 u_run_arduino_11.1: Chip is ESP32-D0WDQ6 (revision 1)
2021-09-24_17:22:11 u_run_arduino_11.1: Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
2021-09-24_17:22:11 u_run_arduino_11.1: Crystal is 40MHz
2021-09-24_17:22:11 u_run_arduino_11.1: MAC: c4:4f:33:08:36:c1
2021-09-24_17:22:11 u_run_arduino_11.1: Uploading stub...
... etc.

Fail case:

2021-09-24_17:21:56 u_run_arduino_22: arduino-cli upload -p COM55 --fqbn esp32:esp32:nina_w10 -v --input-file z:\_jenkins_work\22\build\app.ino.bin z:\_jenkins_work\22\build
2021-09-24_17:21:56 u_run_arduino_22: arduino-cli, pid 11064 started with guard time 300 second(s)
2021-09-24_17:21:58 u_run_arduino_22: Error initializing instance: loading platform release esp32:esp32@2.0.0: loading boards: skipping loading of boards esp32:esp32:atmegazero_esp32s2, esp32:esp32:deneyapmini: malformed custom board options
2021-09-24_17:21:58 u_run_arduino_22: "C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\tools\esptool_py\3.1.0/esptool.exe" --chip esp32 --port "COM55" --baud 921600  --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0xe000 "C:\Users\mlblab\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/partitions/boot_app0.bin" 0x1000 "z:\_jenkins_work\22\build/app.ino.bootloader.bin" 0x10000 "z:\_jenkins_work\22\build/app.ino.bin" 0x8000 "z:\_jenkins_work\22\build/app.ino.partitions.bin"
2021-09-24_17:21:59 u_run_arduino_22: esptool.py v3.1
2021-09-24_17:21:59 u_run_arduino_22: Serial port COM55
2021-09-24_17:21:59 u_run_arduino_22: Connecting...Traceback (most recent call last):
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 425, in command
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 388, in write
2021-09-24_17:21:59 u_run_arduino_22:   File "serial\serialwin32.py", line 325, in write
2021-09-24_17:21:59 u_run_arduino_22: serial.serialutil.SerialTimeoutException: Write timeout
2021-09-24_17:21:59 u_run_arduino_22: 
2021-09-24_17:21:59 u_run_arduino_22: During handling of the above exception, another exception occurred:
2021-09-24_17:21:59 u_run_arduino_22: 
2021-09-24_17:21:59 u_run_arduino_22: Traceback (most recent call last):
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 4582, in <module>
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 4575, in _main
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 4074, in main
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 121, in get_default_connected_device
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 629, in connect
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 596, in _connect_attempt
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 486, in sync
2021-09-24_17:21:59 u_run_arduino_22:   File "esptool.py", line 451, in command
2021-09-24_17:21:59 u_run_arduino_22:   File "serial\serialutil.py", line 372, in timeout
2021-09-24_17:21:59 u_run_arduino_22:   File "serial\serialwin32.py", line 222, in _reconfigure_port
2021-09-24_17:21:59 u_run_arduino_22: serial.serialutil.SerialException: Cannot configure port, something went wrong. Original message: PermissionError(13, 'Access is denied.', None, 5)
2021-09-24_17:21:59 u_run_arduino_22: [11052] Failed to execute script esptool
11

I've just run the Python script that calls arduino-cli but only for the own-build ESP32 NINA-W10 board and that has worked, so the issue seems to be simultaneity, we are not able to run two copies of arduino-cli at the same time successfully. Unfortunately all our test scripts start at the same time, running in parallel on different boards, employing different tools (ST, SES, west, nRF, JLink etc. as well as five instances of ESP-IDF), to minimise execution time.

Any idea what the cross-connection might be in this case?

12

Mmmh, that's certainly an unrelated issues from the one before. :thinking:

Nonetheless am not sure this is 100% the CLI's fault, have you tried running the ESP32 upload tools in parallel? It could also happen because the address changes after the serial port touch.

13

We do routinely run the ESP-IDF stuff in parallel, though in that case the download step is performed through a call to idf.py rather than calling esptool.py directly. It might be that idf.py does something which makes the ESP-IDF python tools perfectly happy to run multiple sessions in parallel whereas esptool.py called directly does not.

I can fix this locally in our automation scripts by locking the ESP-IDF toolchain just for the download part, so I have a workaround; I guess just be aware if anyone else tries to run multiple copies of arduino-cli in parallel (not unusual in an automated regression test environment) they may hit this issue. It may be worth you adding a "multiple instances" test in whatever setup you have to test arduino-cli itself, just to flush these things out before a user does.

14

We're aware the arduino-cli doesn't behave that well when multiple instances are running in parallel as of now since they all use the same environment, but I wouldn't expect your case to be one of the problematic ones. Thanks for the heads up anyway.

Also for future issues would be great if you could open them directly on the arduino-cli Github repo, it eases our job of tracking them. Hopefully you'll have no further problems but you never know. :sweat_smile:

15

Will raise issues on arduion-cli Github in future. In fact, I raised one on Friday which I will move there now.

1 Like
16

Thanks, appreciate it. :pray:

17

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.