Copy Imputer

[icedoom888]

Is your feature request related to a problem? Please describe.

Within the Alpine domain, converting from Model Levels to Pressure Levels causes the raise of many NaN values on the lower Pressure Levels. This is due to steep orography.
The missing values change dynamically in time.

Describe the solution you'd like

Add an imputer that copies missing information from another field.
When a value is missing at a certain Pressure Level it would be useful to copy the value of the Pressure Level above.
This technique ensures constant imputing in areas with missing Pressure Levels and will not cause a big delta in values with respect to vertical changes.

Describe alternatives you've considered

No response

Additional context

Here attached is an exemple config for such imputer:

mean_imputer:
  default: "none"

  "mean":
   - 10u
   - 10v
   - 2d
   - 2t
   - msl
   - sp

   - q_100
   - q_150
   - q_200
   - q_250
   - q_300
   - q_400
   - q_50
   - q_500

   - t_100
   - t_150
   - t_200
   - t_250
   - t_300
   - t_400
   - t_50
   - t_500
  
   - u_100
   - u_150
   - u_200
   - u_250
   - u_300
   - u_400
   - u_50
   - u_500
  
   - v_100
   - v_150
   - v_200
   - v_250
   - v_300
   - v_400
   - v_50
   - v_500
  
   - w_100
   - w_150
   - w_200
   - w_250
   - w_300
   - w_400
   - w_50
   - w_500
  
   - z_100
   - z_150
   - z_200
   - z_250
   - z_300
   - z_400
   - z_50
   - z_500
  
copy_imputer:
  default: "none"
  
  "q_500":
    - q_600
  "q_600":
    - q_700
  "q_700":
    - q_850
  "q_850":
    - q_925

  "t_500":
    - t_600
  "t_600":
    - t_700
  "t_700":
    - t_850
  "t_850":
    - t_925


  "u_500":
    - u_600
  "u_600":
    - u_700
  "u_700":
    - u_850
  "u_850":
    - u_925

  "v_500":
    - v_600
  "v_600":
    - v_700
  "v_700":
    - v_850
  "v_850":
    - v_925
  

  "w_500":
    - w_600
  "w_600":
    - w_700
  "w_700":
    - w_850
  "w_850":
    - w_925


  "z_500":
    - z_600
  "z_600":
    - z_700
  "z_700":
    - z_850
  "z_850":
    - z_925

processors:
  mean_imputer:
     _target_: anemoi.models.preprocessing.imputer.DynamicInputImputer
     _convert_: all
     config: ${data.mean_imputer}

  copy_imputer:
     _target_: anemoi.models.preprocessing.imputer.DynamicCopyImputer
     _convert_: all
     config: ${data.copy_imputer}
  
  normalizer:
    _target_: anemoi.models.preprocessing.normalizer.InputNormalizer
    _convert_: all
    config: ${data.normalizer}

Organisation

MeteoSwiss

[HCookie]

Could these issues with nan values within the orography be resolved by ignoring them in the calculation of the loss with the loss_weights_mask? Or are there other issues with the nan's?

[icedoom888]

@HCookie the dynamic nature of these NaNs makes ignoring them in the loss impossible. If Nans are ignored, at inference time you cannot mask them out using a constant NaN map (as it was done before). Therefore you need the model to learn how to predict the imputed values. Making it necessary to have meaningful values and not statistic based values or constant values.

[HCookie]

@icedoom888 Thanks for the clarification, makes sense.