feat: extforce converter-boundary condition (#717) (unst-8490)

* test initializing the `ExtOldModel`

* remove unused imported modules

* remove unused imported modules

* reformat the test_ext_old_to_new.py and test the printed messages

* reformat the test_ext_old_to_new.py and test the printed messages

* add `InitialCondInterpolationMethod` class and test

* add `InitialCondFileType` class and test

* fix initial condition tests

* create `test_meteo_forcing_file_type` to test `MeteoForcingFileType`

* change tests to check for the parameter values not the parameter name

* test `MeteoInterpolationMethod` in the `test_meteo_interpolation_methods`

* reformat and test the Meteo class

* clean tests that were moved to the TestMeteo test class

* move `test_missing_required_fields` to the separate `TestMeteo` class

* solve the compare floating point values

* rename the old `TestMeteo` to `TestExtModel`

* reformat the `TestExtModel`

* add the cases for time_series/boundary condition files as forcing files to the `TestMeteo`

* correct field names in the `InitialCondition` class

* test the `InitialConditions` class for all possible behaviours

* fix issue in assigning mutable object (list) as a default

* test `construct_filemodel_new_or_existing` function

* reformat the test_ext_old_to_new.py file to use fixtures instead of the tests.utils module

* import the used function directly at the top of the file do not use "module.function"

* name test files by their respective modules not inner functions or classes

* reformat test file

* move check values to the conftest file

* test the `_read_ext_old_data` function

* test the `InitialConditionConverter` class

* reformat test

* move the initial condition fields tests from the test_ext.py to the test_inifield

* remove duplicated initial condition fields classes

* autoformat: isort & black

* move the InitialCondition tests to the tests_inifield.py

* fix use mutable as default value

* create and test `InitialConditionConverter`

* autoformat: isort & black

* user snake case parameter names

* integrate the Initial_Condition converter to the `ext_old_to_new`

* autoformat: isort & black

* delete test results

* fix test error

* clean

* autoformat: isort & black

* remove ignored tests

* use default_factory=list instead of [] to prevent using mutable objects as default value

* use fixtures instead of variables declared in the `tests.utils` module

* convert relative imports to absolute imports

* exclude python=3.12.5 bcause of black warning about memory safety issue.

* create separate group for the docs dependencies

* create `ExternalForcingConverter` class to include all converter functionality

* create class method for the `ExternalForcingConverter` from the old external forcing `ExtOldModel`

* autoformat: isort & black

* convert the `ext_old_to_new` function into a method called `update` in the `ExternalForcingConverter` class

* first step to move out the `construct_filemodel_new_or_existing` calling from the `update` method

* create setter and getter properties for each model, and make abstract version of the `update` method, separate the `save` functionality

* reformat the converter tests

* separate tests for the update, save, and add default paths to the models in the constructor method

* fix the error of using mutables as a default value.

* create separate test class for the update method in the converter

* test files for the update meteo test

* rename the meteo_converter into converters and merge the boundary coverter to it

* move the initial_condition_converter to the converters module

* move the initial_condition_converter to the converters module

* autoformat: isort & black

* add the `BoundaryConditionConverter` to the converter_factory.py

* create a `converters.BoundaryConditionConverter` class

* fix using mutables as default value

* fix using mutables as default value

* use the getattr to fetch attributes from the object

* create subfolders in the test directory

* get rid of relative paths

* add init file to the polyline test dir

* add tests for different polyline cases (basic case, and with label)

* add tests for different polyline cases (basic case, and with label)

* split the test_ext.py to a separate testing modules

* fix error in Boundary object instantiation in the `BoundaryConditionConverter`

* test the `ext.models.boundary` with an existing polyline

* test the `ExternalForcingConverter.update` with only boundary condition data

* replace float point comparison with np.isclose

* replace float point comparison with np.isclose

* remove unused imported function

* add try, except clause to avoid PermissionError in Windows

* update MetroForcingFileType class with updates types

* silence dimr and serializer tests

* autoformat: isort & black

* fix error in using mutables as default value

* remove the `initialsalinitytopuse` quantity from the `ExtOldInitialConditionQuantity` class.

* add missing initial condition quantities and add `__missing__` method to accept tracer quantities

* adjust the `tracer` quantity to `initialtracer`

* add list of old initial conditions quantities that comes with a suffix and test them

* update docstring

* remove duplicate `InitialConditions` class (duplicate if `InitialField`)

* fix floating point comparison

* fix floating point comparison

* remove un-used parameter `postfix and fix sonar warnings`

* remove the `locationtype` and convert the value of the extrapolation to yes/no from 1/0

* the `ExternalForcingConverter` class can be instantiated by `ExtOldModel` or a path to external forcing file and the `_read_old_file` not is used inside the constructor to read the external forcing file if path is given to the converter

* rename boundary polylines files to `boundary-polyline-` prefix

* update converter command line

* return the correct return type hint

---------

Co-authored-by: MAfarrag <[email protected]>

hydrolib/core/dflowfm/bc/models.py

@@ -792,7 +792,7 @@ class ForcingModel(INIModel):
     general: ForcingGeneral = ForcingGeneral()
     """ForcingGeneral: `[General]` block with file metadata."""
 
-    forcing: List[ForcingBase] = []
+    forcing: List[ForcingBase] = Field(default_factory=list)
     """List[ForcingBase]: List of `[Forcing]` blocks for all forcing
     definitions in a single .bc file. Actual data is stored in
     forcing[..].datablock from [hydrolib.core.dflowfm.ini.models.DataBlockINIBasedModel.datablock]."""

hydrolib/core/dflowfm/ini/models.py

@@ -281,7 +281,7 @@ class DataBlockINIBasedModel(INIBasedModel):
         datablock (Datablock): (class attribute) the actual data columns.
     """
 
-    datablock: Datablock = []
+    datablock: Datablock = Field(default_factory=list)
 
     _make_lists = make_list_validator("datablock")
 

hydrolib/core/dflowfm/polyfile/models.py

@@ -3,6 +3,8 @@
 
 from typing import Callable, List, Optional, Sequence
 
+from pydantic.v1 import Field
+
 from hydrolib.core.basemodel import BaseModel, ModelSaveSettings, ParsableFileModel
 
 
@@ -81,7 +83,7 @@ class PolyFile(ParsableFileModel):
     """Poly-file (.pol/.pli/.pliz) representation."""
 
     has_z_values: bool = False
-    objects: Sequence[PolyObject] = []
+    objects: Sequence[PolyObject] = Field(default_factory=list)
 
     def _serialize(self, _: dict, save_settings: ModelSaveSettings) -> None:
         from .serializer import write_polyfile

hydrolib/tools/ext_old_to_new/__init__.py

@@ -1 +1 @@
-from .main_converter import ext_old_to_new, ext_old_to_new_from_mdu
+from .main_converter import ext_old_to_new_from_mdu

hydrolib/tools/ext_old_to_new/converter_factory.py

@@ -1,14 +1,15 @@
 from hydrolib.core.dflowfm.extold.models import (
+    ExtOldBoundaryQuantity,
     ExtOldInitialConditionQuantity,
     ExtOldMeteoQuantity,
 )
-from hydrolib.tools.ext_old_to_new.initial_condition_converter import (
+from hydrolib.tools.ext_old_to_new.base_converter import BaseConverter
+from hydrolib.tools.ext_old_to_new.converters import (
+    BoundaryConditionConverter,
     InitialConditionConverter,
+    MeteoConverter,
 )
 
-from .base_converter import BaseConverter
-from .meteo_converter import MeteoConverter
-
 
 def __contains__(cls, item):
     try:
@@ -40,7 +41,9 @@ def create_converter(quantity) -> BaseConverter:
         """
         if __contains__(ExtOldMeteoQuantity, quantity):
             return MeteoConverter()
-        if __contains__(ExtOldInitialConditionQuantity, quantity):
+        elif __contains__(ExtOldInitialConditionQuantity, quantity):
             return InitialConditionConverter()
+        elif __contains__(ExtOldBoundaryQuantity, quantity):
+            return BoundaryConditionConverter()
         else:
             raise ValueError(f"No converter available for QUANTITY={quantity}.")

hydrolib/tools/ext_old_to_new/converters.py

@@ -0,0 +1,185 @@
+from hydrolib.core.basemodel import DiskOnlyFileModel
+from hydrolib.core.dflowfm.bc.models import ForcingModel
+from hydrolib.core.dflowfm.ext.models import Boundary, Meteo
+from hydrolib.core.dflowfm.extold.models import ExtOldForcing
+from hydrolib.core.dflowfm.inifield.models import InitialField, InterpolationMethod
+from hydrolib.tools.ext_old_to_new.enum_converters import (
+    oldfiletype_to_forcing_file_type,
+    oldmethod_to_averaging_type,
+    oldmethod_to_interpolation_method,
+)
+
+from .base_converter import BaseConverter
+
+
+class MeteoConverter(BaseConverter):
+    def __init__(self):
+        super().__init__()
+
+    def convert(self, forcing: ExtOldForcing) -> Meteo:
+        """Convert an old external forcing block with meteo data to a Meteo
+        forcing block suitable for inclusion in a new external forcings file.
+
+        This function takes a forcing block from an old external forcings
+        file, represented by an instance of ExtOldForcing, and converts it
+        into a Meteo object. The Meteo object is suitable for use in new
+        external forcings files, adhering to the updated format and
+        specifications.
+
+        Args:
+            forcing (ExtOldForcing): The contents of a single forcing block
+            in an old external forcings file. This object contains all the
+            necessary information, such as quantity, values, and timestamps,
+            required for the conversion process.
+
+        Returns:
+            Meteo: A Meteo object that represents the converted forcing
+            block, ready to be included in a new external forcings file. The
+            Meteo object conforms to the new format specifications, ensuring
+            compatibility with updated systems and models.
+
+        Raises:
+            ValueError: If the forcing block contains a quantity that is not
+            supported by the converter, a ValueError is raised. This ensures
+            that only compatible forcing blocks are processed, maintaining
+            data integrity and preventing errors in the conversion process.
+        """
+        meteo_data = {
+            "quantity": forcing.quantity,
+            "forcingfile": forcing.filename,
+            "forcingfiletype": oldfiletype_to_forcing_file_type(forcing.filetype),
+            "forcingVariableName": forcing.varname,
+        }
+        if forcing.sourcemask != DiskOnlyFileModel(None):
+            raise ValueError(
+                f"Attribute 'SOURCEMASK' is no longer supported, cannot "
+                f"convert this input. Encountered for QUANTITY="
+                f"{forcing.quantity} and FILENAME={forcing.filename}."
+            )
+        meteo_data["interpolationmethod"] = oldmethod_to_interpolation_method(
+            forcing.method
+        )
+        if meteo_data["interpolationmethod"] == InterpolationMethod.averaging:
+            meteo_data["averagingtype"] = oldmethod_to_averaging_type(forcing.method)
+            meteo_data["averagingrelsize"] = forcing.relativesearchcellsize
+            meteo_data["averagingnummin"] = forcing.nummin
+            meteo_data["averagingpercentile"] = forcing.percentileminmax
+
+        meteo_data["extrapolationAllowed"] = bool(forcing.extrapolation_method)
+        meteo_data["extrapolationSearchRadius"] = forcing.maxsearchradius
+        meteo_data["operand"] = forcing.operand
+
+        meteo_block = Meteo(**meteo_data)
+
+        return meteo_block
+
+
+class BoundaryConditionConverter(BaseConverter):
+
+    def __init__(self):
+        super().__init__()
+
+    def convert(self, forcing: ExtOldForcing) -> Boundary:
+        """Convert an old external forcing block with meteo data to a boundary
+        forcing block suitable for inclusion in a new external forcings file.
+
+        This function takes a forcing block from an old external forcings
+        file, represented by an instance of ExtOldForcing, and converts it
+        into a Meteo object. The Boundary object is suitable for use in new
+        external forcings files, adhering to the updated format and
+        specifications.
+
+        Args:
+            forcing (ExtOldForcing): The contents of a single forcing block
+            in an old external forcings file. This object contains all the
+            necessary information, such as quantity, values, and timestamps,
+            required for the conversion process.
+
+        Returns:
+            Boundary: A Boindary object that represents the converted forcing
+            block, ready to be included in a new external forcings file. The
+            Boundary object conforms to the new format specifications, ensuring
+            compatibility with updated systems and models.
+
+        Raises:
+            ValueError: If the forcing block contains a quantity that is not
+            supported by the converter, a ValueError is raised. This ensures
+            that only compatible forcing blocks are processed, maintaining
+            data integrity and preventing errors in the conversion process.
+        """
+        data = {
+            "quantity": forcing.quantity,
+            "locationfile": forcing.filename.filepath,
+            "forcingfile": ForcingModel(),
+        }
+
+        new_block = Boundary(**data)
+
+        return new_block
+
+
+class InitialConditionConverter(BaseConverter):
+
+    def __init__(self):
+        super().__init__()
+
+    def convert(self, forcing: ExtOldForcing) -> InitialField:
+        """Convert an old external forcing block with Initial condition data to a IinitialField
+        forcing block suitable for inclusion in a new inifieldfile file.
+
+
+        This function takes a forcing block from an old external forcings
+        file, represented by an instance of ExtOldForcing, and converts it
+        into a InitialField object. The InitialField object is suitable for use in new
+        iniFieldFile, adhering to the updated format and specifications.
+
+        Args:
+            forcing (ExtOldForcing): The contents of a single forcing block
+            in an old external forcings file. This object contains all the
+            necessary information, such as quantity, values, and timestamps,
+            required for the conversion process.
+
+        Returns:
+            Initial condition field definition, represents an `[Initial]` block in an inifield file.
+
+        Raises:
+            ValueError: If the forcing block contains a quantity that is not
+            supported by the converter, a ValueError is raised. This ensures
+            that only compatible forcing blocks are processed, maintaining
+            data integrity and preventing errors in the conversion process.
+
+        References:
+            [Sec.D](https://content.oss.deltares.nl/delft3dfm1d2d/D-Flow_FM_User_Manual_1D2D.pdf#subsection.D)
+        """
+        block_data = {
+            "quantity": forcing.quantity,
+            "datafile": forcing.filename,
+            "datafiletype": oldfiletype_to_forcing_file_type(forcing.filetype),
+        }
+        if block_data["datafiletype"] == "polygon":
+            block_data["value"] = forcing.value
+
+        if forcing.sourcemask != DiskOnlyFileModel(None):
+            raise ValueError(
+                f"Attribute 'SOURCEMASK' is no longer supported, cannot "
+                f"convert this input. Encountered for QUANTITY="
+                f"{forcing.quantity} and FILENAME={forcing.filename}."
+            )
+        block_data["interpolationmethod"] = oldmethod_to_interpolation_method(
+            forcing.method
+        )
+        if block_data["interpolationmethod"] == InterpolationMethod.averaging:
+            block_data["averagingtype"] = oldmethod_to_averaging_type(forcing.method)
+            block_data["averagingrelsize"] = forcing.relativesearchcellsize
+            block_data["averagingnummin"] = forcing.nummin
+            block_data["averagingpercentile"] = forcing.percentileminmax
+        block_data["operand"] = forcing.operand
+
+        if hasattr(forcing, "extrapolation"):
+            block_data["extrapolationmethod"] = (
+                "yes" if forcing.extrapolation == 1 else "no"
+            )
+
+        new_block = InitialField(**block_data)
+
+        return new_block