Support non-ASCII characters in function arguments

[seisman]

Description of proposed changes

Support non-ASCII characters in PyGMT arguments, so that users don't have to use octal codes.
See #2204 for context.

Working example

import pygmt

fig = pygmt.Figure()
fig.basemap(region=[0, 10, 0, 5], projection="x1c", frame="WSEN+tABC¥±°″≥×≈∇⇔∑DEF")
fig.show()

it produces:

Notes to maintainers
The symbols and ZapfDingbats charset can be obtained from https://unicode.org/Public/MAPPINGS/VENDORS/ADOBE/zdingbat.txt and https://unicode.org/Public/MAPPINGS/VENDORS/ADOBE/symbol.txt. The following script is used to generate the list of characters, but manually editing are necessary.

with open("zdingbat.txt", "r") as fin:
    for line in fin:
        if line.startswith("#"):
            continue
        hexval = line.split()[0].strip()
        print(chr(int(hexval, base=16)), end="")
        i += 1
        if i % 16 == 0:
            print()

TODO

• Support all non-ASCII characters in the ISOLatin1+ character set
• Support non-ASCII characters in the Symbols character set
• Support non-ASCII characters in the ZapfDingbats character set
• Check PS_CHAR_ENCODING so it also works with Standard+ character set?
• Check arguments that are not processed by build_arg_string (e.g., text in Figure.text)
• Add a gallery example or a tutorial?
• update any existing examples

Closes #2204 Closes #2000

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
• If adding new functionality, add an example to docstrings or tutorials.
• Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

• /format: automatically format and lint the code
• /test-gmt-dev: run full tests on the latest GMT development version

[seisman]

This new feature is almost done and is ready for more testings. Please let me know what you think.

[weiji14]

Looking good! Could you also change this line in the colorbar gallery example to use degrees perhaps?

pygmt/examples/gallery/embellishments/colorbar.py

Line 43 in dde23f1

frame=["x+lTemperature", r"y+l\260C"],

Also, do we need to test different PS_CHAR_ENCODING settings like Standard, Standard+, ISOLatin1, ISOLatin1+, and ISO-8859-x? Currently the mapping seems to assume ISOLatin1, but if a user sets PS_CHAR_ENCODING to another encoding, the output might be different.

[seisman]

Looking good! Could you also change this line in the colorbar gallery example to use degrees perhaps?

pygmt/examples/gallery/embellishments/colorbar.py

Line 43 in dde23f1

frame=["x+lTemperature", r"y+l\260C"],

Done in e1b43b2.

Also, do we need to test different PS_CHAR_ENCODING settings like Standard, Standard+, ISOLatin1, ISOLatin1+, and ISO-8859-x? Currently the mapping seems to assume ISOLatin1, but if a user sets PS_CHAR_ENCODING to another encoding, the output might be different.

ISOLatin1+ is an extension of ISOLatin1, which means ISOLatin1+ supports more characters than ISOLatin (https://docs.generic-mapping-tools.org/latest/cookbook/octal-codes.html). Thus, it does no harm to support ISOLatin1+ only characters even if users set ISOLatin1 encoding.

For Standard+ and other ISO-8859-x encodings, yes, we can definitely support them. But it also means we have to carefully check their differences and maintain the corresponding mapping dictionary from character to octal codes.

[seisman]

Actually, Python supports many different encodings, thus it's possible to implement this feature without manually maintaining the big dictionary, just like what I already do at line 166 for ISOLatin1+ characters \240 to \377. But I don't have enough knowledge about Python encodings now.

[seisman]

Update:

Can be done using:

for i in range(160, 256):
    print(i, chr(i), chr(i).encode("iso-8859-1").decode("iso-8859-5"))

[seisman]

Here is the improved version:

for code in [*range(0o040, 0o200), *range(0o240, 0o400)]:
    char = codecs.decode(bytes([code]), "iso8859-5", errors="replace")

[seisman]

Symbols \140, \275 and \276 don't exist in the unicode table, so they're incorrectly shown.

Some symbols of \36x and \37x are shown as boxes in the GitHub web GUI, but they're shown correctly in Vim, so these symbols should be good.

[seisman]

Check PS_CHAR_ENCODING so it also works with Standard+ character set?

Already answered in #2584 (comment).

For other ISO-8859-x encodings, it's possible to generate the big mapping dictionary automatically using the following code:

{chr(i).encode("iso-8859-1").decode("iso-8859-5"): "\\" + format(i, "o") for i in range(160, 256)}

but we have to check the PS_CHAR_ENCODING parameter to decide which dictionary to use. However, adding from pygmt.clib.session import Session to pygmt.helpers.utils results in circular imports.

Check arguments that are not processed by build_arg_string (e.g., text in Figure.text)

Yes, we can also translate the strings passed by the text parameter, but we can do very little to plaintext files (we can, but it means we have to open and read the whole plaintext file, which is not efficient). So not sure what we should do here.

Add a gallery example or a tutorial?

Better to do it in a separate PR.

[seisman]

@GenericMappingTools/pygmt-maintainers Any comments and suggestions?

[michaelgrund]

Looks good to me!

[weiji14]

Haven't checked every individual symbol, but I trust that they are correct 🙂

Check arguments that are not processed by build_arg_string (e.g., text in Figure.text)

Yes, we can also translate the strings passed by the text parameter, but we can do very little to plaintext files (we can, but it means we have to open and read the whole plaintext file, which is not efficient). So not sure what we should do here.

Ok, to handle ASCII characters in fig.text in a separate PR.