Unable to assign nested structure into a null value

[lkoniecz]

Operating System

1. Name/Distribution: macOS
2. Version: Sonoma 14.3

Version of Python and packages in use at the time of the issue.

1. Distribution: PyPy
2. Python Version: 3.11.5
3. Version of yamlpath installed: 3.8.1
4. Version of ruamel.yaml installed: ruamel.yaml 0.17.21, ruamel.yaml.clib 0.2.8

Minimum sample of YAML (or compatible) data necessary to trigger the issue

foo:

Complete steps to reproduce the issue when triggered via:

yaml-set --change=foo.bar.abcd --value="value" myfile.yaml

Expected Outcome

foo:
   bar:
       abcd: value

Actual Outcome

foo: value

Screenshot(s), if Available

[wwkimball]

You've certainly found an edge-case in the code but I'm not sure my response is going to meet your expectation. I'm not even sure what should happen in this case because what you're attempting to do is to overwrite null with a map/hash. If your null were any other scalar, you'd get an appropriate error stating that yamlpath is unwilling to convert a scalar into a complex type. On the other hand, if your null were already a map/hash -- even an empty one -- you'd get the outcome you expect.

This is the expected behavior when attempting to convert a scalar into a complex data type:

$ echo "foo: var" | yaml-set --change=foo.bar.key --value=value
CRITICAL:  Cannot add PathSegmentTypes.KEY subreference to scalars at 'bar' in 'foo.bar.key'.

An error is expected because converting data types is an unsafe operation. If I really wanted to convert a string into a map/hash, I would instead do this:

$ echo "foo: var" | yaml-set --change=foo --delete | yaml-set --change=foo.bar.key --value=value
---
foo:
  bar:
    key: value

As you can see, I have first deleted the scalar, then added an entirely new map/hash in its place.

On the other hand, when your parent is a map, you get what you're after, like this:

$ echo "foo: {}" | yaml-set --change=foo.bar.key --value=value
---
foo:
  bar:
    key: value

$ echo -e "foo:\n  some: value" | yaml-set --change=foo.bar.key --value=value
---
foo:
  some: value
  bar:
    key: value

As you can see, the original target can be an empty or populated map/hash. As long as the data-types match, yamlpath will make the expected change.

But null is a weird case with undefined behaviors. Did the data author mean for null at any given node to be the absence of a scalar, map/hash, or list/array? There's no self-evident, programmatic way to know.

In my experience, I've seen both. In Puppet/Hiera, I often see null used in place of future scalar values where null is used deliberately to trigger certain code responses in the Puppet class that receives it. In Docker, I often see volumes defined in name-only, leaving what would obviously be a map/hash as null; if other specific details are needed later, say for different profiles, they the map/hash is added.

In the case of null being used as a placeholder for a future scalar value, yamlpath should emit the error you see above. It's unsafe to convert data-types (usually indicative of a problem in the code attempting to make the change secondary to a misunderstanding of the data or its consumer, or a bug in the code). In the case of null being used as a placeholder for a future map/hash however, such a change should be allowed. The change from null to map/hash is exactly what the data and code authors agree in advance should happen.

For now, the behavior you see from yamlpath is undefined. Anthon (the author of ruamel.yaml, on which yamlpath is based) and I didn't write yamlpath's change code with awareness of null values (this was a collaborative effort). The only null-sensitive code in yamlpath is from the perspective of a query.

I'm going to convert this issue into a discussion to solicit more opinions on what yamlpath should do in this edge case.

The question up for discussion becomes:

When it comes to changing the value, should yamlpath treat null as a scalar, a complex type, or both?

[lkoniecz]

Thanks for such a comprehensive answer. I am the owner of the yaml file and have nothing against initialising the variable to an empty map, {} that is. In such case it works correctly.

I wasn't aware of the typing underneath.

going further however
can I do this?
yaml-set --change=foo.bar.0.abcd --value="value" myfile.yaml

foo:
   bar:
     - abcd: value

but getting

foo:
   bar:
     '0':
        abcd: value

is it double? not necessarily in one assignment

regards

[wwkimball]

In YAML Paths, array indexes are best indicated with square brackets, like [0], especially when working with implicit structure. Bare number indexing works only when dealing with explicit structure (pre-existing list/array elements). Without the square brackets, the segment is treated as a map/hash key when constructing structure on-the-fly because it's impossible to know whether a non-bracketed number is really meant to be a list/array index versus a number-like map/hash key.

To accomplish your goal, add the square brackets, like so:

$ echo | yaml-set --change=foo.bar[0].abcd --value=value
---
foo:
  bar:
    - abcd: value