DMD template is in incorrect format and causes JSONDecoder exception.

[rasjani]

https://github.com/winedarksea/AutoTS/blame/7a843d050f62c1b4046c0b08020838e8eb95388a/autots/templates/general.py#L467

when running following:

from autots import AutoTS
model = AutoTS(forecast_length=10, frequency='infer',ensemble='simple', drop_data_older_than_periods=200)

following backtrace is thrown:

Traceback (most recent call last):
  File "/Users/rasjani/src/omat/tmp/tst.py", line 2, in <module>
    model = AutoTS(forecast_length=10, frequency='infer',ensemble='simple', drop_data_older_than_periods=200)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rasjani/src/omat/tmp/venv/lib/python3.11/site-packages/autots/evaluator/auto_ts.py", line 392, in __init__
    full_params = json.loads(row['TransformationParameters'])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

Root cause:

value stored to TransformationParameters is not valid json..

[rasjani]

Also lines https://github.com/winedarksea/AutoTS/blame/7a843d050f62c1b4046c0b08020838e8eb95388a/autots/templates/general.py#L472

and https://github.com/winedarksea/AutoTS/blame/7a843d050f62c1b4046c0b08020838e8eb95388a/autots/templates/general.py#L473

look like they will not work as they are now written.

[winedarksea]

@rasjani Thanks for noticing this, fixed now! 08c1d9a

[rasjani]

I have no skin in this game, i came to file this bug report from stackoverflow[0] when someone asked about this issue. However, i want to ask a question; why wont you just use python dicts in autots/templates/general.py and if you need to store those as json in to the dataframe, use json.dumps() instead of json.loads() ? That way you actually get proper error detection and valid json without doing bunch of string escaping that leads to errors like this ?

[0] - https://stackoverflow.com/questions/78470381/why-do-i-get-jsondecodeerror-extra-data-line-1-column-4-char-3

[winedarksea]

Thanks for being a relay here for the information. I do use json.dumps and loads in the proper import_template and export_template commands but general.py is a legacy thing, used to be a json file. I have added a more exacting unittest so that should catch future problems here.