Exceptionally long duration of "Downloading index: ..." process

[DanMan32]

Describe the problem

I know a number of people have reported this problem for months but still it isn't solved.

When starting Arduino IDE 2.2.1, notification area shows "Downloading index: library_index.tar.bz2"
Progress bar stays at 50% for a really long time, like 1/2 hour.

I know it isn't an issue of accessing the Internet, as I am writing this connected to the Internet with Arduino stuck at that stage.
Even if I didn't have an Internet connection at the time, IDE should time out after maybe a minute and move on with its life.

While it is in this state, can't compile, as compile progress seems to wait for the index download to finish

Once I get past this, then successive launches of Arduino seems to be good either for the rest of the day or as long as I don't log out of Windows, haven't figured that part out yet.

To reproduce

Start Arduino IDE 2.2.1 for first time in a particular day

Expected behavior

Not get stuck for 10's of minutes with status "Downloading index: library_index.tar.bz2"

Arduino CLI version

0.34.0

Operating system

Windows

Operating system version

Windows 11

Additional context

It downloads instantly if I use my web browser to download the file:

https://downloads.arduino.cc/libraries/library_index.tar.bz2

---

The fault does not occur If I run Arduino CLI directly:

C:\Program Files\Arduino IDE\resources\app\lib\backend\resources\arduino-cli core update-index

It hesitated about 10 seconds, with seemingly successful results:

Downloading index: package_index.tar.bz2 downloaded

---

Looking at the terminal logs, the Downloading Index seems to take a long time doing these while the fault is occurring:

2023-09-29T04:39:13.570Z daemon INFO time="2023-09-29T00:39:13-04:00" level=info msg="Resolving include AutoConnect.h for arch esp32"
time="2023-09-29T00:39:13-04:00" level=info msg="  found better lib" lib=AutoConnect prio=6B1
2023-09-29T04:39:29.092Z daemon INFO time="2023-09-29T00:39:29-04:00" level=info msg="Resolving include WiFi.h for arch esp32"
time="2023-09-29T00:39:29-04:00" level=info msg="  found better lib" lib=WiFi prio=6B0
2023-09-29T04:39:45.007Z daemon INFO time="2023-09-29T00:39:45-04:00" level=info msg="Resolving include WebServer.h for arch esp32"
time="2023-09-29T00:39:45-04:00" level=info msg="  found better lib" lib=WebServer prio=6B0
[71480:0929/003958.469:ERROR:gpu_init.cc(523)] Passthrough is not supported, GL is disabled, ANGLE is
2023-09-29T04:40:01.671Z daemon INFO time="2023-09-29T00:40:01-04:00" level=info msg="Resolving include DNSServer.h for arch esp32"
time="2023-09-29T00:40:01-04:00" level=info msg="  found better lib" lib=DNSServer prio=6B0
2023-09-29T04:40:17.592Z daemon INFO time="2023-09-29T00:40:17-04:00" level=info msg="Resolving include EEPROM.h for arch esp32"
time="2023-09-29T00:40:17-04:00" level=info msg="  found better lib" lib=EEPROM prio=6B0
2023-09-29T04:40:34.431Z daemon INFO time="2023-09-29T00:40:34-04:00" level=info msg="Resolving include PageBuilder.h for arch esp32"
time="2023-09-29T00:40:34-04:00" level=info msg="  found better lib" lib=PageBuilder prio=6B1
2023-09-29T04:40:53.157Z daemon INFO time="2023-09-29T00:40:53-04:00" level=info msg="Resolving include FS.h for arch esp32"
time="2023-09-29T00:40:53-04:00" level=info msg="  found better lib" lib=FS prio=6B0
2023-09-29T04:41:12.586Z daemon INFO time="2023-09-29T00:41:12-04:00" level=info msg="Resolving include SPIFFS.h for arch esp32"
time="2023-09-29T00:41:12-04:00" level=info msg="  found better lib" lib=SPIFFS prio=6B0
2023-09-29T04:41:33.667Z daemon INFO time="2023-09-29T00:41:33-04:00" level=info msg="Resolving include SD.h for arch esp32"
time="2023-09-29T00:41:33-04:00" level=info msg="  found better lib" lib=SD prio=6A7
time="2023-09-29T00:41:33-04:00" level=info msg="  found better lib" lib=SD prio=6B0
2023-09-29T04:41:54.968Z daemon INFO time="2023-09-29T00:41:54-04:00" level=info msg="Resolving include SPI.h for arch esp32"
time="2023-09-29T00:41:54-04:00" level=info msg="  found better lib" lib=SPI prio=6B0
2023-09-29T04:42:15.144Z daemon INFO time="2023-09-29T00:42:15-04:00" level=info msg="Resolving include Preferences.h for arch esp32"
time="2023-09-29T00:42:15-04:00" level=info msg="  found better lib" lib=Preferences prio=6B0
2023-09-29T04:42:36.738Z daemon INFO time="2023-09-29T00:42:36-04:00" level=info msg="Resolving include Ticker.h for arch esp32"
time="2023-09-29T00:42:36-04:00" level=info msg="  found better lib" lib=Ticker prio=6B0
2023-09-29T04:42:58.281Z daemon INFO time="2023-09-29T00:42:58-04:00" level=info msg="Resolving include ArduinoJson.h for arch esp32"
time="2023-09-29T00:42:58-04:00" level=info msg="  found better lib" lib=ArduinoJson prio=6A7
2023-09-29T04:43:28.337Z daemon INFO time="2023-09-29T00:43:28-04:00" level=info msg="Resolving include HTTPClient.h for arch esp32"
2023-09-29T04:43:28.340Z daemon INFO time="2023-09-29T00:43:28-04:00" level=info msg="  found better lib" lib=HTTPClient prio=6B0
2023-09-29T04:43:58.304Z daemon INFO time="2023-09-29T00:43:58-04:00" level=info msg="Resolving include WiFiClientSecure.h for arch esp32"
time="2023-09-29T00:43:58-04:00" level=info msg="  found better lib" lib=WiFiClientSecure prio=6B0
2023-09-29T04:44:35.127Z daemon INFO time="2023-09-29T00:44:35-04:00" level=info msg="Resolving include HTTPUpdate.h for arch esp32"
time="2023-09-29T00:44:35-04:00" level=info msg="  found better lib" lib=HTTPUpdate prio=6B0
2023-09-29T04:45:07.932Z daemon INFO time="2023-09-29T00:45:07-04:00" level=info msg="Resolving include Update.h for arch esp32"
2023-09-29T04:45:07.933Z daemon INFO time="2023-09-29T00:45:07-04:00" level=info msg="  found better lib" lib=Update prio=6B0
2023-09-29T04:45:39.087Z daemon INFO time="2023-09-29T00:45:39-04:00" level=info msg="Resolving include pcf8574.h for arch esp32"
time="2023-09-29T00:45:39-04:00" level=info msg="  found better lib" lib=pcf8574 prio=6A7
2023-09-29T04:46:11.881Z daemon INFO time="2023-09-29T00:46:11-04:00" level=info msg="Resolving include Wire.h for arch esp32"
time="2023-09-29T00:46:11-04:00" level=info msg="  found better lib" lib=Wire prio=6B0
2023-09-29T04:46:49.620Z daemon INFO time="2023-09-29T00:46:49-04:00" level=info msg="Resolving include Adafruit_ADS1X15.h for arch esp32"
time="2023-09-29T00:46:49-04:00" level=info msg="  found better lib" lib="Adafruit ADS1X15" prio=6A7
2023-09-29T04:47:21.647Z daemon INFO time="2023-09-29T00:47:21-04:00" level=info msg="Resolving include Adafruit_I2CDevice.h for arch esp32"
time="2023-09-29T00:47:21-04:00" level=info msg="  found better lib" lib="Adafruit BusIO" prio=3EB
2023-09-29T04:47:57.713Z daemon INFO time="2023-09-29T00:47:57-04:00" level=info msg="Resolving include ESPTelnet.h for arch esp32"
time="2023-09-29T00:47:57-04:00" level=info msg="  found better lib" lib="ESP Telnet" prio=3EB

Additional reports

• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• "Downloading index" process stalls arduino-ide#1330 (comment)
• Exceptionally long duration of "Downloading index: ..." process #2347 (comment)
• Cache the library/package indexes arduino-ide#2506

Related

• Hanging Downloading index: package_index.tar.bz2 arduino-ide#2418
• "Downloading index" process stalls arduino-ide#1330 (comment)
• "Downloading index" process stalls arduino-ide#1330 (comment)
• "Downloading index" process stalls arduino-ide#1330 (comment)
• "Downloading index" process stalls arduino-ide#1330 (comment)
• Slow build process with multiple libraries arduino-ide#2462 (comment)
• At start of the IDE downloading the 39Mb library_index.json takes minutes to hours.  arduino-ide#2501
• At start of the IDE downloading the 39Mb library_index.json takes minutes to hours.  arduino-ide#2501 (comment)
• https://forum.arduino.cc/t/downloading-index-package-teensy-json/1229463/4
• https://forum.arduino.cc/t/ide-hangs-on-startup-error-downloading-index/1261030/9
• https://forum.arduino.cc/t/compilation-is-delayed-by-downloading-index-package-index-tar-bz2-process/1300983/1
• https://forum.arduino.cc/t/compilation-is-delayed-by-downloading-index-package-index-tar-bz2-process/1300983/3
• https://forum.arduino.cc/t/compilation-is-delayed-by-downloading-index-package-index-tar-bz2-process/1300983/5
• https://forum.arduino.cc/t/compilation-is-delayed-by-downloading-index-package-index-tar-bz2-process/1300983/6
• https://forum.arduino.cc/t/compilation-is-delayed-by-downloading-index-package-index-tar-bz2-process/1300983/7
• https://forum.arduino.cc/t/downloading-index-package-rp2040-index-json/1303866
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/1
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/2
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/3
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/6
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/14
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/17
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/19
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/21
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/22
• https://forum.arduino.cc/t/after-upgrading-to-v2-3-3-stuck-downloading-index/1305575/38
• https://forum.arduino.cc/t/library-and-board-updates-stall-after-2-3-3-auto-update/1305826/7
• https://forum.arduino.cc/t/ide-2-3-3-stuck-trying-to-download-teensy/1306012/1
• https://forum.arduino.cc/t/ide-2-3-3-stuck-trying-to-download-teensy/1306012/7
• https://forum.arduino.cc/t/package-esp32-index-json-hangs-up-loading/1306784/1
• https://forum.arduino.cc/t/package-esp32-index-json-hangs-up-loading/1306784/4
• https://forum.arduino.cc/t/package-esp32-index-json-hangs-up-loading/1306784/6
• https://forum.arduino.cc/t/download-index-package-lgt8fx-index-json-stuck/1307014
• https://forum.arduino.cc/t/downloading-index-package-adafruit-index-json-error/1307083/1
• https://forum.arduino.cc/t/downloading-index-package-adafruit-index-json-error/1307083/10
• https://forum.arduino.cc/t/ide-freezes-after-updating-to-2-3-3/1307890/7

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[RL-Fields]

I am also having this issue. The only "fix" I have found is to open Arduino with the internet disconnected as per the instructions in this thread with a similar issue arduino/arduino-ide#1483

I also found a direct download URL of the bz2 file, which downloaded instantly, but am not sure what to do with it once downloaded
https://downloads.arduino.cc/libraries/library_index.tar.bz2

I'm on Windows 10 and Arduino 2.2.1

[Helienzo]

I also have this issue on Arduino IDE 2.2.1 running on windows 10.

The downloading Index bar freezes at roughly 60% and never finishes. I tried to reinstall and clean out all temp, hidden, library and other folders related to Arduino.

It sometimes works for one start after the cleanup but as soon as I restart the application the problem returns.

The only way to get the IDE to work is to start it without internet connection.

[umbynos]

Does this happen also with the CLI?

[umbynos]

Could you please install the arduino-cli . And try to run arduino-cli update?

[njordan77]

did download the cli command (exe file) and executed within the ARD2 folder....not sure if this is good or not, seemed to download two index files.

EDIT:
didn't change anything. still stuck with an ESP32 index file in the GUI

[umbynos]

Yeah, It's expected that is downloading two index files.
Question is: Is the duration of the download of the index with the cli command reasonable? Or did you experience the same problem of the IDE2 GUI?

Do you have some antivirus software running?

[njordan77]

@speed....the download of these two index files is finished within 1-2secs. so no delay at all.

yes, Win11 and Bitdefender Security.
AND its a zip-file version, so i did not install via setup.

[umbynos]

Could you please try to temporarily disable the antivirus and update the indexes using the IDE GUI?

[niiiksh]

Could you please try to temporarily disable the antivirus and update the indexes using the IDE GUI?

I’ve disabled Windows Security and still downloading index for a very long time! What a bug did you test your program with Windows 11 with latest update?

[ATOMNFT]

I am also having this issue. The only "fix" I have found is to open Arduino with the internet disconnected as per the instructions in this thread with a similar issue arduino/arduino-ide#1483

I also found a direct download URL of the bz2 file, which downloaded instantly, but am not sure what to do with it once downloaded https://downloads.arduino.cc/libraries/library_index.tar.bz2

I'm on Windows 10 and Arduino 2.2.1

put files inside C:\Users\User\AppData\Local\Arduino15\

[ubidefeo]

Please, everyone, help us track this issue and hunt down the cause.

Downloading files to Arduino15 will of course work, because that's what the IDE and CLI do behind the scenes, but we'd love to be able to find this issue which we can't reproduce.

If anyone assumes that's the solution we'll have an outstanding bug.
Thank you for your collaboration
u.

[LouDnl]

Same issue here, unable to use the IDE. Tried putting file and contents manually in said location, same issue.

The notification is even still there if I completely delete the Arduino15 folder from AppData. Everything else gets downloaded and installed just fine while this message is still here.

After a while it magically dissapeared

[ATOMNFT]

I also encounter this issue. What are the steps necessary to solve this issue?

Not sure if this helps. But I have rectified the issue using these entries in additional board manager. Since I have added these the error/hang has not taken place again. When the message pops up in lower right corner it stays for a max of like 3-4 mins then its gone. Everything updates as it should and no compiling/flashing issues.

http://arduino.esp8266.com/stable/package_esp8266com_index.json
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

[magn2998]

Im on Windows and fixed this by trying several things.
Most didn't work, but will list all in case they all need to be done before working.

1. Uninstalled Windows Store version and downloaded from Arduino webpage - Didn't solve it
2. Ran the "arduino-cli core update-index" cmd from AppData/Local/Programs/Arduino IDE/resources/app/lib/backend/resources - Didn't solve it
3. Ran the "arduino-cli update" from same location as above - Didn't solve it
4. Ran application as administrator - Solved it

Maybe this is a simple "Run as admin" solution?
This was also done over a longer period (but below 30 min.) so maybe time just solved it...

[dantogno]

I had this issue. It turned out there was a windows security / admin permission popup in the background that did not properly come up over the rest of the UI. After clicking OK (on several subsequent popups) things worked.

[niiiksh]

I had this issue. It turned out there was a windows security / admin permission popup in the background that did not properly come up over the rest of the UI. After clicking OK (on several subsequent popups) things worked.

I don't think it is popup because for me there is no popup by windows.
And the type of problem is not imperfection as someone set. It is a bug that prevents from starting the build.

[goedzo]

I ran into the same issue on IDE 2.3.3. Run as administrator didn't work. But what did work for me was deleting the folder:
C:\Users\xxx\AppData\Local\Arduino15\packages

When I re-opened arduino, it just asked me to redownload the boards, and it did this without any problems.

[manimal1882]

both goedzo (#2347 (comment)) and kingdo1 (#2347 (comment)) solutions work well.
I prefer the first, but rather than deleting directories, I transferred them to another location within the C drive, so I could compare the oldest directories and the ones created by IDE, and I found some interesting differences. In this way, I also was able to restore the libraries installed some time ago

[wburchard]

I can confirm the bug still exists in version 2.3.3 (running on Windows 11 Pro) after I had replaced the CH32V packages with the latest from main. After reading through the above comments, I unplugged and then reattached the board. This fixed the problem.

Perhaps adding code to the IDE to check if a board is attached while attempting to update packages and suggesting to the user to unplug then reattach the board as a suggestion? While not the ideal solution, doing this would likely eliminate a lot of frustration until the root cause is identified and fixed.

[Karniverse]

Disconnecting the board and reconnecting again resolves the issue for me.

im using arduino 2.3.3 and arduino leonardo pro micro, had same issue but this method worked for me

[CNSNGopenSourceProjects]

Same problem using Windows 10 Pro and Arduine IDE 2.3.3.
On previous versions the downloading index wass slow, but finished after a long time.
On this new IDE version, its never end.

I used to have on URL additional boards manager the github direction for ESP32 and RP2040.
The downloading index used to hang on RP2040.
I removed the RP20240.
Now it hangs while downloading ESP32.
It means the problem is on downloading index code. May be a long timeouit for bad downloading commands.

[CNSNGopenSourceProjects]

I disconnected the network and the Arduino IDE works, after show the error message."Error downloading index: https://downloads.arduino.cc/libraries/library..."

[CNSNGopenSourceProjects]

Try to disconnect the network when starting the Arduino IDE.

[sofar]

Using flatpak (3.3.5) here. Unable to run the cli - entering the namespace is already difficult enough as is - it's entirely dead unless I disconnect my internet before starting the IDE. I let it sit for 48 hours and it still didn't finish downloading, or compiling my sketch.

[Timppa11]

Disconnecting the board from the computer worked for me. Arduino IDE 2.3.3 Windows 10 Arduino Uno wifi rev.2. It's a bit counterintuitive. No one knows the real reason why it is like this?

[goedzo]

Disconnecting the board from the computer worked for me. Arduino IDE 2.3.3 Windows 10 Arduino Uno wifi rev.2. It's a bit counterintuitive. No one knows the real reason why it is like this?

Can confirm this is indeed the case. When my board "Nordic nRF52840 DK" is connected, I get the endless downloading index. Then I disconnect the board from USB, it boots.

[wburchard]

Maybe the connection to the board is causing a file lock to be held that blocks the download?  If the download timeout is set to infinite, that would cause this.

[Timppa11]

Seems that the Arduino IDE developers have bigger things to do!  Should be an easy fix to check if the board is connected and act accordingly.

[PaulStoffregen]

@umbynos - Perhaps Arduino CLI could perform package index downloads in 2 steps. First download the entire file to a large buffer in memory. Then after every byte is received into the buffer, write the entire file to disk as a single large write operation.

Why try this? Well, anti-malware software on Windows often looks for filesystem access as fuzzy matching against heuristic patterns. Maybe some of these download problems people report could be due to Windows Defender, Norton, Macafee or other software installing "filter drivers" which intercept and analyze patters of filesystem access. By putting all the data into a memory buffer before attempting to write even the first byte to disk, Arduino CLI could avoid a pattern of many small writes and (hopefully) have a better chance of avoiding interference. Receive all into a memory buffer would also allow Arduino CLI / IDE to give the user better on-screen feedback. The download could be confirmed complete. Then another message could confirm it's been written to disk. Giving end users better on screen feedback might help future troubleshooting, if this speculation about Windows filter drivers interfering with disk access doesn't turn out to be the entire problem.

[Timppa11]

Antivirus/firewall software does not report anything odd is happening + why would removing the board make any difference?

[Babel17]

I had this issue just now with IDE 2.3.3 and resetting the board resolved it.

By which I mean I pressed the reset button, not just an unplug.

[qdot]

Just hit this with 2.3.3. Closing IDE, deleting AppData contents (c:\Users\[your user name]\AppData\Roaming\arduino-ide, there may also be a Arduino IDE directory that I also deleted), and reopening it seems to have unwedged it, at the cost of losing some of my settings.

[aaronw2]

I'm having this issue with Arduino IDE 2.3.3 on Linux.
arduino-cli update runs instantly. It's only the IDE.

[aaronw2]

I fixed it by closing the serial monitor. My board is outputting a lot of data on the serial port.

[leo-smi]

I fixed it by closing the serial monitor. My board is outputting a lot of data on the serial port.

WORKED FOR ME, THANKS.

[d-luxxx]

I ran into the same issue on IDE 2.3.3. Run as administrator didn't work. But what did work for me was deleting the folder: C:\Users\xxx\AppData\Local\Arduino15\packages

When I re-opened arduino, it just asked me to redownload the boards, and it did this without any problems.

Created an account to say thanks and this worked....

[tonbor]

I ran into the same issue on IDE 2.3.3. Run as administrator didn't work. But what did work for me was deleting the folder: C:\Users\xxx\AppData\Local\Arduino15\packages
When I re-opened arduino, it just asked me to redownload the boards, and it did this without any problems.

Created an account to say thanks and this worked....

And the second day?

[per1234]

I fixed it by closing the serial monitor.

Thanks for sharing this observation @aaronw2. I am able to reproduce the bug of the "Downloading index:" notification never completing, and blocking other operations when Serial Monitor is open.

I bisected the regression and found that it was relatively recent, after the time this issue was originally submitted. So it seems there might be multiple bugs that have similar symptoms. I created a dedicated issue for the bug related to Serial Monitor:

#2719

My board is outputting a lot of data on the serial port.

I found that the amount of data is not relevant, and in fact it will still occur even if the board isn't outputting any data at all. It is only the fact of the Serial Monitor being open and connected to a port.