embedded but escaped double-quotes cause error

[bchill]

Hello,

I am using 5.6.2.

With this data in /tmp/file.csv

"name","address"
"John","2 Main St."
"Dave","100 Main St \"Apt 2\""

I get

$ mlr --quote-original --csv sort -f name /tmp/file.csv
mlr: unmatched double quote at line 2.

I am clearly missing something.

Is there some other option I need?

[aborruso]

Hi @bchill if you read CSV rfc, you have:

If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote

Then you have a wrong CSV, it should be

"name","address"
"John","2 Main St."
"Dave","100 Main St ""Apt 2"""

If you run mlr --quote-original --csv sort -f name on it, you will have

name,address
"Dave","100 Main St ""Apt 2"""
"John","2 Main St."

[johnkerl]

@bchill -- @aborruso is correct regarding RFC-4180 CSV:

• https://miller.readthedocs.io/en/latest/file-formats.html#csv-tsv-asv-usv-etc
• https://datatracker.ietf.org/doc/html/rfc4180

That said, Miller is rather strict about following the RFC, but also claims to be a data-cleaning tool and non-standard CSV is abundant in the world ... so this is room for improvement in Miller. :)

[bchill]

Indeed, the use of \ as a modifier (escape) is highly prevalent.

I have control over the creation of the file. I'll fix it.

Thanks to you both you!

[johnkerl]

I have control over the creation of the file. I'll fix it.

Lucky you! :D