Improved user error messages for import

[till213]

Received question:

"Im having some issues getting the addon to import . csv files. Even the ones exported from the addon itself. Same with the KML files, except for flightaware downloaded ones that actually work. Is there anything I might be missing about csv files importing? What is the correct file format? I tried using the same as the exported files but it doesnt seem to work."

Reply:

In general error handling for any kind of data is currently very crude indeed: while each data field is validated whether it is in the expected format (e.g. whether a given value is really a number etc.) no actual error details such as line number or the nature of the parsing error (e.g. "value xyz on line 123 is not a number") are currently given.

Whether a given file can be successfully imported depends on the format (import plugin). Here are some general points:

CSV

Sky Dolly format: Sky Dolly currently only imports the CSV file that has been produced with the exact same Sky Dolly release. CSV files that have been generated with previous Sky Dolly release usually cannot be imported and result in a "can't import" error message.

The reason: I keep adding / refactoring simulation variables, and this is reflected in the CSV data. However I do not keep track of "version info" (the CSV format itself is a "row by row" format and it not suitable for "structured data"). In fact, I plan to deprecate this format (only the Sky Dolly CSV variant, not the others) exactly for this reason and replace it with a "logbook export/import" format (= the exact same database format - SQLite - as used for the "logbook")

KML

The Keyhole Markup Language format is also not very suitable for data exchange: it is only meant for visualising geometric data, but not actual exchange of data between applications.

So the current import plugin needs to know exactly what the data source of any given KML is, in order to know how to interpet the various KML elements ("which element contains the flight track, which element contains the timestamps" etc.).

So upon import it is crucial that you select the proper data source:

• flightaware
• flightradar24
• Generic KML

In the later case an attempt is made to interpret the KML elements in a generic way, that is to detect "line segments" in a generic manner. But especially this may fail, because there are multiple ways to represent line (and other) data in KML. Again, this format is only meant for "data visualisation" rather than data exchange.

GPX

This is a pretty simple format and unlike KML indeed meant for data exchange of "GPS track data". With the default values for Waypoints and Position you should be able to import pretty much any GPX file that has a recorded GPS track.

However depending on the use case - e.g. importing flight plans from Little Navmap, as exported as GPX - you can define the elements that contain the expected track data (waypoint, route or track). But also in this case: if the imported GPX data does not match with those settings you get a crude "can't import file" error message.

IGC

This is the only data format that is rigorously defined - at least in theory. I have seen IGC (International Gliding Commission) files (examples) that deviate from the specification. Not using GPS altitude data with the WGS84 reference ellipsoid (but using "pressure altitude" instead, for example) is only one of the more harmless deviations (the IGC standard clearly specifies that all altitude data be with the WGS84 reference ellipsoid).

To summarise:

• Make sure to only import CSV Sky Dolly data that has been exported from the exact same Sky Dolly release - previous CSV files are not supported
• KML: Make sure to select the proper data source (flightaware, flightradar24)
• GPX: This should work with default settings for most GPS files that come from a "GPX tracker device" - but otherwise make sure to select the proper elements (track, waypoint or route - yes, some knowledge about the GPX data format is required here)
• IGC: This should always work

If you think you have a file (e.g. Sky Dolly CSV) that should be correctly imported, but fails, then please attach it to this issue and I have a look at it!

I keep this issue open for implementing a more "user friendly error reporting", but that is somewhat "down the road".

[scondas]

I think you should consider csv import using names in the header row as the keys rather than strict column position. That way, if new columns are inserted into a format, Sky Dolly can be more resilient (see Flightradar24 granular vs regular CSV as an example of this problem) In practice, I have found it much easier to interact with imports using Tacview than with Sky Dolly. https://www.tacview.net/documentation/csv/en/

[scondas]

I also think you should consider preserving and displaying actual event date/time that you get from an import. Maybe the recorded time could show up in the display variables dialog?

[till213]

I think you should consider csv import using names in the header row as the keys rather than strict column position

Well, in fact the more recent CSV parser implementation that is now (since around 0.13, IIRC) in place supports just that - and I believe the Flightradar24 CSV parser is already making use of that (that is, parsing the actual column header names and creating a "map").

However I actually plan to drop the Sky Dolly CSV format completely: it is a pain to maintain whenever I add new simulation variables, it is already mis-using the CSV format by trying to "squeeze an object hierarchy" (position data, engine data, primary flight controls, ... into a flat "row by row" format, and so far I never bothered about backwards compatibility either (so a "Sky Dolly CSV" format written with some prior Sky Dolly version was often simply rejected, due to a parser error - also of course because the columns were expected to be "in rigid order").

Fun fact: "Sky Dolly CSV" was the first "ad hoc" persistence format, before Sky Dolly 0.6 introduced the SQLite "logbook" (which was always planned up front).

Next the use case for the "Sky Dolly CSV" was for "aircraft data exchange" between different logbooks - but also this use case will become superfluous with the introduction of another exchange format - very soon ;)

[till213]

I also think you should consider preserving and displaying actual event date/time that you get from an import. Maybe the recorded time could show up in the display variables dialog?

I am in fact refactoring the whole "import" plugin architecture (not radically, but including taking "recording / creation dates into account, where possible (depending on format)".

Also note that not every format does have actual "date formats": e.g. specifically the "Sky Dolly CSV" does not (it only includes the timestamps in milliseconds, starting from 0). But again, the Sky Dolly CSV format will soon be dropped completely...

[scondas]

I can see how the Sky Dolly CSV format has limitations, but as you re-factor please consider the following use case. I import in Flightradar24 data and let it Augment the data, export out the flight as Sky Dolly CSV, alter the CSV to reflect what really happened (where the Augmentation didn't make appropriate estimates), re-import, then replay the event. Here is a result: https://youtu.be/Jp1NgT6Q3FE . It would be very useful for my case if you maintain an export/import format that is human editable. That said, if that is not in your plans, I will figure something out (write my own editor, fork, etc)

[till213]

Interesting video :)

May I ask what kind of manual edits you do to the "Sky Dolly CSV"? Because if it the altitude (that is usually way off currently when you import real-world flight data from flightradar24.com and/or flightaware.com, as those altitudes are usually "standard pressure altitudes" and do not correlate with the actual world scenery in MSFS) then there is another change that I have in mind ;)

Also, the "flight augmentation" is currently very primitive: it essentially assumes an aircraft like an A320 and simply hardcodes "events" like "gear down 5 minutes before end of replay", the bank angle is simply "guesstimated" and such.

And also with this regards I thought of making the "flight augmentation" configurable, e.g. in JSON format, where you could - per aircraft type (jet, propeller, helicopter, ...) or even per "aircraft title" (e.g. "Asobo A320neo") - configure your own "augmentation events", including relative time-offsets like "3 minutes before touch-down".

But that also requires yet another functionality - "flight analysis" - that would determine aircraft acceleration (given the "GPS path"), take-off and landing "touch points" and all that.

That's all a lot of work ahead (and I need to brush up my Newtonion physics skills a bit, with all this acceleration stuff) ;)

But I can already tell that I really want to get rid of the "Sky Dolly CSV" format, as much as I appreciate your use case: because with the upcoming "16 engine support" it is no fun to keep up with all those columns (even with your suggested change to allow "flexible column headers").

However I can reveal that the new import & export format will be: no big surprise, the Sky Dolly logbook format :) The same migration steps that are done when opening a "normal logbook" will be applied each time you import a "logbook file".

And as such it is also kind of "human readable and editable": you can open and edit those "logbooks" with e.g.

https://sqlitebrowser.org/

And isn't it more fun to update a given set of data with

UPDATE position
SET altitude = ...
WHERE ...

instead of hacking around with some CSV text editor or some spreadsheet editor? :) And if needed editors like SQLiteBrowser also allow you to edit the tables "like in Excel" - if you prefer manually editing individual table cells.

I once published even a "mini-tutorial" here, if you got curious:

https://forums.flightsimulator.com/t/sky-dolly-v0-15-free-flight-recorder-replay-location-management/405367/29

[scondas]

"May I ask what kind of manual edits you do to the "Sky Dolly CSV"?"

So far I have only manipulated engine spool up (throttle position) and Lighting, I will likely be adjusting pitch for my next iteration.

I never thought about editing the SQL tables directly. I'll take a look. Thanks!/

[scondas]

Also, I am now working on reconstructing with weather (this is work in progress and not public): https://youtu.be/MNHJuUNnnGE

[scondas]

Lastly, it appears that the bank Automation calculation is poor with highly sampled FR24 data (~1 sample per second),. In my current effort, I will likely fix that via an Excel algorithm applied to the exported Sky Dolly CSV and then re-import. I haven't figured out my exact plan yet. I am definitely using Sky Dolly differently that the design point, and I can deal with that.

[till213]

Lastly, it appears that the bank Automation calculation is poor with highly sampled FR24 data

Well, just say it: the current bank angle calculation (in the "flight augmentation" code /that is applied whenever we import data from sources such as flightradar24 that only contain a GPS flight track with timestamps, but nothing much else) is plain garbage ;)

As far as I remember I simply calculate the angle betwen the current and next line segment and multiply this with some constant, capped at some (assumed) maximum like 45° or so - that's as silly as it gets ;) In fact, the whole "flight augmentation" is currently merely a "placeholder" for some (future) more sophisticated "flight analysis & augmentation", but that's still a bit away (behind other features that are asked more from the community, like "jitter-free AI aircraft replay").

That's what I meant above with calculating (well, estimating) the lateral acceleration (given the GPS track with timestamps) and all that, because this allows us to calculate (estimate better) the bank angle, e.g.:

https://www.omnicalculator.com/physics/bank-angle#in-the-sky-the-angle-of-banking-of-an-aircraft

(and other methods - I have yet to do some more thorough research on that topic).

[scondas]

If you are interested, you are welcome to look at/use the bankAngle code I ported into my private baseline of Sky Dolly (from some python code I use). It doesn't adjust for wind or vertical accelerations, but you might still find it useful. I have an updated SkyMathTest.cpp as well.

[scondas]

I have been doing some more thinking on this. I hope in your refactoring you leave some capability to import more complete flight information that is available in CVS formats such as Garmin G1000 log ( that include roll, pitch, yaw, flap positions, etc.). In the current baseline i could reformat that log into sky dolly format and then import https://www.reddit.com/r/flying/comments/6jgntl/find_garmin_g1000_sample_csv_file/?utm_source=share&utm_medium=ios_app&utm_name=iossmf. I realize I could write my own plug-in.

[till213]

I hope in your refactoring you leave some capability to import more complete flight information that is available in CVS formats such as Garmin G1000 log

Absolutely: as I previously mentioned already the import format that is going to replace the "Sky Dolly CSV" format is the "Sky Dolly logbook" (*.sdlog) format itself - that is as complete as it gets + with the added benefit of "data migration" that essentially comes "for free" = using the same migration logic that is already in place and that is applied each time when opening a logbook with Sky Dolly itself :)

And a Garmin G1000 log import (in CSV format) sounds like an interesting idea! I'll have a look at your provided link and see whether I can find some documentation (besides the sample CSV "with one line" that seems to be provided on reddit).

Of course we are talking about "future work" here, not something that I'd tackle right away (myself) ;)

I realize I could write my own plug-in.

I suggest you wait until the upcoming v0.16, as there will be quite substantial changes in the import API ("interface") - including simplification of the API itself: many "updateExtendedXYZ()" methods will be removed (not necessary anymore), as I introduced a new "FlightData" class that is essentially a "value only" class (struct) and allows the plugins to directly set all desired "extended data" (such as flight creation time, flight number etc.) directly in the FlightData (the original Flight class inherits from QObject and as such is neither "copyable" nor "moveable" - hence the extraction of its data into the new FlightData struct that allows to "move" the imported data around.