Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make metadata same accross roles #760

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

nikokaoja
Copy link
Collaborator

@nikokaoja nikokaoja commented Nov 18, 2024

  • removed Domain rules
  • removed Asset rules
  • removed Asset loader
  • replace prefix with space in information metadata
  • added external_id to Information metadata
  • namespace now automatically generated not anymore settable
  • disable conversion of "reference" between roles since reference is over-used
  • disable number of tests until we refactor rules
  • rules aspect added as field to metadata
  • linking of rules between roles enabled through optional metadata fields (.physical, .logical, .conceptual)

Copy link

github-actions bot commented Nov 18, 2024

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
14415 9452 66% 60% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
cognite/neat/_graph/loaders/init.py 57% 🟢
cognite/neat/_rules/_shared.py 97% 🟢
cognite/neat/_rules/analysis/init.py 100% 🟢
cognite/neat/_rules/exporters/_rules2excel.py 87% 🟢
cognite/neat/_rules/importers/_base.py 87% 🟢
cognite/neat/_rules/importers/_dtdl2rules/dtdl_importer.py 73% 🟢
cognite/neat/_rules/importers/_rdf/_base.py 93% 🟢
cognite/neat/_rules/importers/_rdf/_imf2rules/_imf2metadata.py 100% 🟢
cognite/neat/_rules/importers/_rdf/_inference2rules.py 93% 🟢
cognite/neat/_rules/importers/_rdf/_owl2rules/_owl2metadata.py 100% 🟢
cognite/neat/_rules/importers/_rdf/_shared.py 86% 🟢
cognite/neat/_rules/importers/_spreadsheet2rules.py 79% 🟢
cognite/neat/_rules/models/init.py 100% 🟢
cognite/neat/_rules/models/_base_rules.py 88% 🟢
cognite/neat/_rules/models/dms/_rules.py 92% 🟢
cognite/neat/_rules/models/dms/_rules_input.py 96% 🟢
cognite/neat/_rules/models/dms/_validation.py 96% 🟢
cognite/neat/_rules/models/information/_rules.py 86% 🟢
cognite/neat/_rules/models/information/_rules_input.py 97% 🟢
cognite/neat/_rules/transformers/init.py 100% 🟢
cognite/neat/_rules/transformers/_converters.py 60% 🟢
cognite/neat/_rules/transformers/_verification.py 95% 🟢
cognite/neat/_session/_base.py 69% 🟢
cognite/neat/_store/_provenance.py 77% 🟢
cognite/neat/_workflows/steps/data_contracts.py 0% 🟢
cognite/neat/_workflows/steps/lib/current/rules_exporter.py 0% 🟢
TOTAL 81% 🟢

updated for commit: 9b0017d by action🐍

@nikokaoja nikokaoja marked this pull request as ready for review November 18, 2024 20:20
@nikokaoja nikokaoja requested a review from a team as a code owner November 18, 2024 20:20
@@ -37,6 +38,7 @@
)


@pytest.mark.skip(reason="Skipping this test until refactoring of rules is completed")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that we are in a non-releaseable state?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary file. Should not be committed. Suggest adding something like ~*.xlsx to .gitignore.

@doctrino doctrino self-requested a review November 19, 2024 07:14
Copy link
Collaborator

@doctrino doctrino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly clarification questions

@@ -125,6 +150,34 @@ class BaseMetadata(SchemaModel):
"""

role: ClassVar[RoleTypes]
aspect: ClassVar[DataModelAspect]
space: SpaceType = Field(SchemaCompleteness.partial, alias="prefix")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong default value?


@abstractmethod
def get_prefix(self) -> str:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets remove this and replace it with .prefix. It is only used in one location

Unlike namespace, the identifier does not end with "/" or "#".
"""
return DEFAULT_NAMESPACE[f"data-model/verified/{self.aspect}/{self.space}/{self.external_id}/{self.version}"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this! Very good idea to have aspect as part of the unique identifier.

license: str | None = None
rights: str | None = None
physical: str | None = None
conceptual: str | None = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the thinking here to use these two fields to link the different input rules?

metadata["namespace"] = default
def fix_space(metadata: dict, default: str = "neat") -> dict:
if space := metadata.get("space", None):
if not isinstance(space, str) or not PATTERNS.space_compliance.match(space):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A consequence now is that Information rules needs to use a prefix that is compliant with the regex for CDF space?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants