Compilation fails with spurious "No such file or directory" error if line number of #include directives are changed while editor is "dirty"

[Curvedair2024]

Describe the problem

An Arduino IDE editor tab can be in one of two states:

• Saved: the contents of the file on disk matches the content in the editor buffer
  • The editor will always be in this state if the "Auto save" feature is enabled.
• Dirty: the contents of the file on disk differs from the content staged in the editor buffer.
  • This is indicated by the presence of a ⬤ at the right side of the editor tab.

Regardless of the state, when a compilation is triggered, it is the code in the editor buffer that must be compiled.

At the start of the sketch compilation process, Arduino CLI performs a "library discovery" procedure, where the preprocessor is run tentatively on the sketch program repeatedly. If preprocessing fails, Arduino CLI parses the text of the error message. If it is a "No such file or directory" error from an #include directive, the header filename is extracted from that directive and the searches installed libraries are searched to find the one that provides that header file. The library is added to the compiler's "search path", then the process repeated until either the preprocessing passes, or else a library can't be discovered for an #include directive (see the flowchart in the "Additional context" section of the issue if this isn't clear).

🐛 Sketch compilation fails if the sketch code staged in the editor buffer differs from the file on disk in a way that causes one #include directive to be present in the staged code at the line number of a different #include directive in the file on disk.

To reproduce

1. Select File > Preferences from the Arduino IDE menus.
2. Uncheck the box next to "Auto save".
3. Click the "OK button.
4. Select File > New Sketch from the Arduino IDE menus.
   A new sketch will open in an Arduino IDE window.
5. Change the sketch code to the following:

   #include <EEPROM.h>
   #include <SPI.h>
   #include <SoftwareSerial.h>
   #include <Wire.h>
   void setup() {}
   void loop() {}

   ⓘ The fault will occur with any sketch that contains multiple consecutive #include directives.
6. Select File > Save As... from the Arduino IDE menus.
7. Save the sketch to any convenient location.
8. Select Tools > Board > Arduino AVR Boards > Arduino Uno from the Arduino IDE menus.
   ⓘ The fault will occur with any board. The "Arduino Uno" was picked arbitrarily for the sake of the demonstration.
9. Select Sketch > Verify/Compile from the Arduino IDE menus.
   🙂 The sketch compiles without errors.
10. Add a blank line at the top of the sketch.
    ❗ Do not save the sketch. It must be in a "dirty" state to reproduce the fault.
11. Select Sketch > Verify/Compile from the Arduino IDE menus.
    🐛 Compilation fails spuriously:

    C:\Users\per\Documents\Arduino\Foo\Foo.ino:2:10: fatal error: EEPROM.h: No such file or directory
     #include <SPI.h>
               ^~~~~~~   
    

    ⓘ Note that the #include directive for EEPROM.h is at line 1 in the file on disk, while it is at line 2 in the staged code, which is the position of the #include directive for SPI.h in the file on disk.
12. Add another blank line at the top of the sketch.
    ❗ Do not save the sketch. It must be in a "dirty" state to reproduce the fault.
13. Select Sketch > Verify/Compile from the Arduino IDE menus.
    🐛 Compilation fails spuriously:

    C:\Users\per\Documents\Arduino\Foo\Foo.ino:2:10: fatal error: EEPROM.h: No such file or directory
     #include <SoftwareSerial.h>
               ^~~~~~~~~~
    

14. Add another blank line at the top of the sketch.
    ❗ Do not save the sketch. It must be in a "dirty" state to reproduce the fault.
15. Select Sketch > Verify/Compile from the Arduino IDE menus.
    🐛 Compilation fails spuriously:

    C:\Users\per\Documents\Arduino\Foo\Foo.ino:2:10: fatal error: EEPROM.h: No such file or directory
     #include <Wire.h>
               ^~~~~~~~
    

16. Add another blank line at the top of the sketch.
17. Select Sketch > Verify/Compile from the Arduino IDE menus.
    🙂 The sketch compiles without errors.
    ⓘ Note that the #include directive for EEPROM.h is at line 1 in the file on disk, while it is at line 5 in the staged code, which is the position of the setup function definition in the file on disk.

Expected behavior

Library discovery always works correctly, even when a sketch is in a "dirty" state.

Arduino IDE version

Original report

arduino/[email protected]

Last verified with

arduino/arduino-ide@aa9b10d

Operating system

Windows

Operating system version

• Windows 10
• Windows 11

Additional context

The fault does not occur if the sketch is not in a "dirty" state when compiled, whether the non-dirty state is achieved by the IDE's auto save, or by saving manually.

---

Blank lines were added in the demo for the sake of simplicity, but the fault occurs regardless of what the content of the added line is.

---

I didn't bisect the regression, but the fault does not occur with Arduino IDE 1.8.19.

Workaround

Select File > Save from the Arduino IDE menus.

Library discovery overview

In case the text description of library discovery provided in the introduction is not clear, this flowchart might make it easier to understand:

%%{
  init: {
    'flowchart': {
      'curve': 'monotoneY'
    },
    'theme': 'base',
    'themeVariables': {
      'clusterBkg': '#ffffff',
      'edgeLabelBackground': '#ffffff',
      'lineColor': '#000000',
      'primaryBorderColor': '#000000',
      'primaryColor': '#f0f0f0',
      'primaryTextColor': '#000000'
    }
  }
}%%
flowchart TB
  subgraph main[" "]
    direction TB

    compileCommand(["<code>compile</code><br />command"])
    selectSketchFile[/"Select unprocessed<br />sketch file"/]
    done{"All<br />processed?"}
    preprocess[/"C++ preprocess"/]
    preprocessorError{"Error?"}
    parseError[/"Parse error<br />message"/]
    includeError{"<code>#include</code><br />error?"}
    headerAvailable{"A library<br />provides<br />header?"}
    addToSearchPath["Add library<br />to search path"]

    compile[["Compilation"]]
    failure(["Error"])

    compileCommand --> selectSketchFile
    selectSketchFile --> done
    done -- "Yes" --------> compile

    done -- "No" --> preprocess
    preprocess --> preprocessorError
    preprocessorError -- "Yes" --> parseError
    parseError --> includeError
    includeError -- "Yes" --> headerAvailable
    headerAvailable -- "Yes" --> addToSearchPath
    addToSearchPath --> preprocess

    preprocessorError -- "No" --> selectSketchFile

    includeError -- "No" ----> failure

    headerAvailable -- "No" --> compile
  end

Loading

Additional reports

• https://forum.arduino.cc/t/ide-2-1-bug-i-m-only-adding-a-new-blank-line/1144525
• https://forum.arduino.cc/t/ide-2-2-1-possible-compiler-bug/1208608
• https://forum.arduino.cc/t/blank-lines-inserted-at-start-of-sketch-causes-library-errors/1283385
• https://forum.arduino.cc/t/library-not-found-if-called-in-different-line-in-sketch/1023831
• https://forum.arduino.cc/t/odd-compile-error/1307366

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[dankeboy36]

I could reproduce the bug using the CLI, it's unrelated to the IDE code-base.

I used the HEAD (faa6359) of the CLI repo instead of 0.36.0-rc.1. It may be clear to the maintainers, but I did not see this mentioned in the bug report or an existing CLI issue.

Steps in a terminal:

git rev-parse --short HEAD
faa63596

./arduino-cli version
arduino-cli  Version: git-snapshot Commit: faa63596 Date: 2024-08-01T18:39:18Z

tree ./my_sketch      
./my_sketch
└── my_sketch.ino

1 directory, 1 file

cat ./my_sketch/my_sketch.ino 
#include <EEPROM.h>
#include <SPI.h>
#include <SoftwareSerial.h>
#include <Wire.h>
void setup() {}
void loop() {}

./arduino-cli daemon
Daemon is now listening on 127.0.0.1:50051
{"IP":"127.0.0.1","Port":"50051"}

---

Open a new terminal:

Create

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Create

{
  "instance": {
    "id": 1
  }
}

Init

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Init

Compile

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}, "fqbn": "arduino:avr:uno", "sketchPath": "./my_sketch/my_sketch.ino"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile

✅ compile_output.txt

Compile with dirty state (with sourceOverride but original content)

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}, "fqbn": "arduino:avr:uno", "sketchPath": "./my_sketch/my_sketch.ino", "sourceOverride": {"my_sketch.ino": "#include <EEPROM.h>\n#include <SPI.h>\n#include <SoftwareSerial.h>\n#include <Wire.h>\nvoid setup() {}\nvoid loop() {}\n"}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile

✅ compile_output__source_override_original.txt

Compile with dirty state (with sourceOverride start with a new line)

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}, "fqbn": "arduino:avr:uno", "sketchPath": "./my_sketch/my_sketch.ino", "sourceOverride": {"my_sketch.ino": "\n#include <EEPROM.h>\n#include <SPI.h>\n#include <SoftwareSerial.h>\n#include <Wire.h>\nvoid setup() {}\nvoid loop() {}\n"}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile

❌ compile_output__source_override_edit1.txt

{
  "errStream": "L1VzZXJzL2Frb3Mua2l0dGEvRGVza3RvcC9kZXYvYXJkdWluby1jbGkvbXlfc2tldGNoL215X3NrZXRjaC5pbm86MjoxMDogZmF0YWwgZXJyb3I6IEVFUFJPTS5oOiBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5CiAjaW5jbHVkZSA8U1BJLmg+CiAgICAgICAgICBefn5+fn5+ICAgCmNvbXBpbGF0aW9uIHRlcm1pbmF0ZWQuCg=="
}

Decoded error message:

/Users/akos.kitta/Desktop/dev/arduino-cli/my_sketch/my_sketch.ino:2:10: fatal error: EEPROM.h: No such file or directory
 #include <SPI.h>
          ^~~~~~~   
compilation terminated.

Compile with dirty state (with sourceOverride start with two new lines)

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}, "fqbn": "arduino:avr:uno", "sketchPath": "./my_sketch/my_sketch.ino", "sourceOverride": {"my_sketch.ino": "\n\n#include <EEPROM.h>\n#include <SPI.h>\n#include <SoftwareSerial.h>\n#include <Wire.h>\nvoid setup() {}\nvoid loop() {}\n"}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile

❌ compile_output__source_override_edit2.txt

{
  "errStream": "L1VzZXJzL2Frb3Mua2l0dGEvRGVza3RvcC9kZXYvYXJkdWluby1jbGkvbXlfc2tldGNoL215X3NrZXRjaC5pbm86MzoxMDogZmF0YWwgZXJyb3I6IEVFUFJPTS5oOiBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5CiAjaW5jbHVkZSA8U29mdHdhcmVTZXJpYWwuaD4KICAgICAgICAgIF5+fn5+fn5+fn4KY29tcGlsYXRpb24gdGVybWluYXRlZC4K"
}

Decoded error message:

/Users/akos.kitta/Desktop/dev/arduino-cli/my_sketch/my_sketch.ino:3:10: fatal error: EEPROM.h: No such file or directory
 #include <SoftwareSerial.h>
          ^~~~~~~~~~
compilation terminated.