Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Dataverse API removes metadatablocks if optional params are omitted #11130

Open
qqmyers opened this issue Jan 2, 2025 · 8 comments · May be fixed by #11144
Open

Update Dataverse API removes metadatablocks if optional params are omitted #11130

qqmyers opened this issue Jan 2, 2025 · 8 comments · May be fixed by #11144
Labels
Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) Status: Needs Input Applied to issues in need of input from someone currently unavailable Type: Bug a defect

Comments

@qqmyers
Copy link
Member

qqmyers commented Jan 2, 2025

According to https://guides.dataverse.org/en/latest/api/native-api.html#update-a-dataverse-collection,
Prepare a JSON file containing the fields for the properties you want to update. You do not need to include all the properties, only the ones you want to update. However, the changes in #11026 added consequences for not adding some optional fields:

When it comes to omitting these fields in the JSON:

Omitting facetIds or metadataBlockNames causes the Dataverse collection to inherit the corresponding configuration from its parent.

Omitting inputLevels removes any existing custom input levels in the Dataverse collection.

Omitting the entire metadataBlocks object in the request JSON would exclude the three sub-objects, resulting in the application of the two changes described above.

In combination, this means that trying to use the API to edit the dataverseType, for example, by just sending the properties to update, as of 6.5, now removes the metadatablocks associated with the dataverse/collection and sets it to not be the metadata root.

For child collections, this presumably causes them to start inheriting from the root. For the root collection, the case where this bug was seen, it causes Internal Server Errors when creating datasets, and puts the root collection in a state where its metadatablocks can't be edited in the UI.

What steps does it take to reproduce the issue?

Use the API above to make a change to dataverseType, see that the subcollection now inherits metadatablocks/the root collection is broken if you changed it, see that creating a dataset causes an error.

Which version of Dataverse are you using?
6.5/develop

Any related open or closed issues to this bug report?

For those with access, see discussion at https://iqss.slack.com/archives/C03R1E7T4KA/p1735848411767949

Screenshots:

No matter the issue, screenshots are always welcome.

To add a screenshot, please use one of the following formats and/or methods described here:

Are you thinking about creating a pull request for this issue?
Help is always welcome, is this bug something you or your organization plan to fix?

@qqmyers qqmyers added the Type: Bug a defect label Jan 2, 2025
@qqmyers qqmyers added the Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) label Jan 2, 2025
@qqmyers qqmyers moved this to Ready for Triage in IQSS Dataverse Project Jan 2, 2025
@scolapasta scolapasta moved this from Ready for Triage to This Sprint 🏃‍♀️ 🏃 in IQSS Dataverse Project Jan 6, 2025
@stevenwinship stevenwinship self-assigned this Jan 7, 2025
@stevenwinship stevenwinship moved this from This Sprint 🏃‍♀️ 🏃 to In Progress 💻 in IQSS Dataverse Project Jan 7, 2025
@GPortas GPortas added the Status: Needs Input Applied to issues in need of input from someone currently unavailable label Jan 7, 2025
@GPortas GPortas self-assigned this Jan 7, 2025
@stevenwinship
Copy link
Contributor

stevenwinship commented Jan 7, 2025

First pass at handling this issue - Add ignoreMissingData query parameter.

When it comes to omitting these fields in the JSON:

- Omitting ``facetIds`` or ``metadataBlockNames`` causes the Dataverse collection to inherit the corresponding configuration from its parent.
- Omitting ``inputLevels`` removes any existing custom input levels in the Dataverse collection.
- Omitting the entire ``metadataBlocks`` object in the request JSON would exclude the three sub-objects, resulting in the application of the two changes described above.

To prevent these omissions from being removed and inherited from the parent you can add the query parameter ignoreMissingData=true

.. code-block:: bash

  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  export SERVER_URL=https://demo.dataverse.org
  export DV_ALIAS=dvAlias

  curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/dataverses/$DV_ALIAS?ignoreMissingData=true" --upload-file dataverse-complete.json

@GPortas
Copy link
Contributor

GPortas commented Jan 7, 2025

@qqmyers @stevenwinship @g-saracca

Regarding using one flag or two as an endpoint parameter. In JSF, facets are selected independently from metadata blocks. In fact, there are two separate flags in the database to manage this inheritance: https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/Dataverse.java#L183-L184

And look at JSF to see how we can select facets even from metadata blocks that aren’t selected:

Image

I’m not sure if this behavior in JSF is intentional, but that’s how we replicated it in the SPA and the API.

For this reason, if we decide to modify the endpoint to explicitly indicate inheritance using flags, I would recommend providing two independent flags: one for inheriting metadata blocks and another for facets. This is because inheriting metadata blocks does not imply inheriting facets, as they are independent concepts (I understand that we aim to preserve this behavior to align with the goal of replicating JSF).

@GPortas
Copy link
Contributor

GPortas commented Jan 7, 2025

First pass at handling this issue - Add ignoreMissingData query parameter.

When it comes to omitting these fields in the JSON:

- Omitting ``facetIds`` or ``metadataBlockNames`` causes the Dataverse collection to inherit the corresponding configuration from its parent.
- Omitting ``inputLevels`` removes any existing custom input levels in the Dataverse collection.
- Omitting the entire ``metadataBlocks`` object in the request JSON would exclude the three sub-objects, resulting in the application of the two changes described above.

To prevent these omissions from being removed and inherited from the parent you can add the query parameter ignoreMissingData=true

.. code-block:: bash

  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  export SERVER_URL=https://demo.dataverse.org
  export DV_ALIAS=dvAlias

  curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/dataverses/$DV_ALIAS?ignoreMissingData=true" --upload-file dataverse-complete.json

That could be another valid approach. Possibly simpler!

@stevenwinship
Copy link
Contributor

If we use 2 flags (1 for metadataBlocks and 1 for facetIds) how do we handle inputLevels? Do we need another flag or can we group inputLevels with facetIds or metadataBlocks?

@GPortas
Copy link
Contributor

GPortas commented Jan 7, 2025

If we use 2 flags (1 for metadataBlocks and 1 for facetIds) how do we handle inputLevels? Do we need another flag or can we group inputLevels with facetIds or metadataBlocks?

If you inherit metadata blocks from the parent, there is no need to set input levels manually. The input levels of the parent will automatically be applied when retrieving the metadata blocks and fields of the new Dataverse. Input levels should be sent if you are not checking the 'inherit metadata blocks from parent' and, of course, you want to customize input levels for a metadata block.

@qqmyers
Copy link
Member Author

qqmyers commented Jan 7, 2025

FWIW: I'm not sure I like the polarity of a ignoreMissingData=true flag - seems like a inheritMetadataSettingsFromParentIfMissing=true flag would be safer (though longer - not sure if there's a short way to say it). Even so, those are both implicit and cause the same Json to have different effects -
seems like a

metadataBlocks {
  "inheritMetadataBlocksFromParent":true,
  "inerhertFacetsFromParent":"true"
}

as an option in Json versus including the metadataBlockNames/inputLevels and facetIds fields would be clearer?

@g-saracca
Copy link
Contributor

I agree with @GPortas that we follow the JSF behaviour, we just need now an extra tweak. I like @qqmyers idea about

metadataBlocks {
"inheritMetadataBlocksFromParent":true,
"inheritFacetsFromParent":"true"
}

This will mean that if inheritMetadataBlocksFromParent is true, even sending metadataBlockNames and inputLevels with some values, these two properties will be skipped. The same if inheritFacetsFromParent is true, even sending facetIds with values, they will be skipped.
Is that ok?
So that will be the only way to switch the collection to start using metadata blocks and/or facets from parent.

@qqmyers
Copy link
Member Author

qqmyers commented Jan 8, 2025

I'd suggest documenting it to say you can send the relevant flag or the corresponding list, but not both. In code, we could just check for the flags first and (as we often have done) ignore any other json that is irrelevant. If it's easy to reject the call when the user sends both a flag and a list - even better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) Status: Needs Input Applied to issues in need of input from someone currently unavailable Type: Bug a defect
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants