DM-47335: Fix dataset query missing implied dimensions

[dhirving]

Revert serialization change from DM-46776 in SerializedDataCoordinate, which caused it to drop values for implied dimensions.

That change broke the transfer of DatasetRefs from server to client, since it drops the values for implied dimensions in the data ID. Per the documented behavior of dataset queries, the returned refs will have hasFull() = True and this behavior is relied on in tutorial notebooks.

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes
• (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.37%. Comparing base (6cb0823) to head (7d49ac0).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1112   +/-   ##
=======================================
  Coverage   89.37%   89.37%           
=======================================
  Files         362      362           
  Lines       48288    48301   +13     
  Branches     5872     5872           
=======================================
+ Hits        43155    43168   +13     
  Misses       3718     3718           
  Partials     1415     1415           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TallJimbo]

I can see how this broke RemoteButler query results, but I'm curious how this could have changed dataset type definitions, since initializing a DimensionGroup with the required subset and the full set should always yield the same result, unless the dimension universe has changed.

I'm not at all against the revert that was just merged, but I think long term this is a good reason to have the wire format for queries differ from long-term persistence and other cases where the consumer is another butler, because I do still think the latter should be only saving the required subset.

[dhirving]

Looks like you're right -- I didn't realize that the DatasetType constructor would automatically add all implied dimensions. (The unit test data that is used everywhere explicitly adds in band even though it's theoretically implied by physical_filter.)

So I can probably un-revert the SerializedDatasetType change.