emoji in atlantic puzzle do not appear in saved .puz file

[alexdej]

As reported in alexdej/puzpy#30 emoji from the Atlantic daily puzzle from 2023-11-13 do not appear in the resulting .puz file. Instead it appears to have 0s which break parsing of the file and cause AcrossLite to crash. Not sure if this is an issue with xword-dl or with puz.py.

atlantic-20231113.zip

[afontenot]

Edit: this is now outdated, see the below comment.

[afontenot]

Incidentally, I don't see any issue here with the latest git commit of xword-dl and 0.2.4 of puzpy. Probably some change of behavior since that release is the proximate cause.

[afontenot]

Had another look at this. Three thoughts:

• I said above that I couldn't reproduce the issue with some specific versions, but actually I don't see any issue with your file either. The clue strings are null terminated - the emoji are not replaced by nulls. I do not see a crash with the Gnome Crosswords app, using your file. The crasher with AcrossLite is probably the result of a blank (empty string) clue, not a broken puzzle file.

• The ideal way to fix this is probably to change xword-dl to generate 2.0 puz files by default, but even releasing the current HEAD would fix the AcrossLite issue because emoji are currently decoded to :string_representation: rather than being replaced with empty strings.

• There appears to be no way to set the puzzle version with puzpy when creating a new puzzle. You might want to add a version argument to the initializer. When setting the version manually in xword-dl with code like the following, I can save emoji:

        puzzle = puz.Puzzle()
        puzzle.version = b'2.0'
        puzzle.fileversion = b'2.0\0'
        puzzle.encoding = 'UTF-8'

[thisisparker]

Yep, @afontenot has this right I think. The latest released version of xword-dl doesn't know what to do with emoji, because they can't be encoded with the text encoding scheme available in puz files before v2, and so if a clue contains only emoji the resulting file would get bit crashy.

I have attempted to fix that with #157 but haven't released to PyPI since then. HEAD should have it. If you encounter any issues like that with the code at HEAD I want to know about it. In the meantime, yes, I am overdue for a release!

I have to think about the implications of generating puz v2 files; my understanding is that client support is not fully there but maybe I could have it be an option somehow.