French regions #4853

corradio · 2022-12-12T20:42:36Z

Issue

Description

French regional data pulling directly from ECO2MIX. Uses the same data as displayed in the RTE app.
⚠️ Lumps all thermal into unknown as thermal breakdown is unknown. This affects the country as well, as it is computed as an aggregation of subzones.

Outstanding issues

Working production parser
Regional geometries, see web/third_party_maps/*.geojson [need help]
Update bounding boxes
Delete geojson files
Parser needs to support fetching exchanges
Shallow data validation (total production is conserved across aggregation)
Deeper data validation

Data validation

TBD.

Preview

…dded

This reverts commit d088702.

…dules This reverts commit dc6c7e2, reversing changes made to 7558f23.

corradio · 2022-12-15T09:44:56Z

Seems like we're mostly done! Would still require data validation though

VIKTORVAV99 · 2022-12-15T10:16:42Z

I would suggest we keep the 3rd party maps in the PR (and repository) for now until we are happy with both the zone and aggregated geometries.

VIKTORVAV99

Looks good for the most part, just some small notes and one question.

In the current return objects it returns nan, should this not be Null/None if it's missing?

parsers/ECO2MIX.py

VIKTORVAV99 · 2022-12-16T10:52:11Z

parsers/ECO2MIX.py

+    "France": "FR",
+    **{
+        subzone.replace("FR-", ""): subzone
+        for subzone in ZONES_CONFIG["FR"].get("subZoneNames", [])


This creates some type errors as the Literal "FR" can't be assigned to the ZoneKey type but should not affect the functionality.

VIKTORVAV99 · 2022-12-16T10:53:36Z

parsers/ECO2MIX.py

+}
+
+
+def query(url_type_arg, session=None, target_datetime=None):


Could be smart to add types here (and the other functions) for better linting and DX but it's technically not required.

parsers/ECO2MIX.py

Co-authored-by: Viktor Andersson <[email protected]>

corradio · 2022-12-16T12:02:19Z

Looks good for the most part, just some small notes and one question.

In the current return objects it returns nan, should this not be Null/None if it's missing?

From what I know, nans will be turned to NULL before being inserted in the database (I tested locally without our backend). I can change that if needed though.

VIKTORVAV99 · 2022-12-16T12:09:16Z

Looks good for the most part, just some small notes and one question.
In the current return objects it returns nan, should this not be Null/None if it's missing?

From what I know, nans will be turned to NULL before being inserted in the database (I tested locally without our backend). I can change that if needed though.

Then it should be fine to keep as is. 🙂

config/exchanges/BE_FR-ARA.yaml

…rib into olc/split-FR

unitrium · 2022-12-22T09:26:05Z

Just putting some thoughts here as we are trying to achieve something similar in #4894.
Do we want to make the FR zone an aggregated zone ? Does this make sense since we have the capability of getting the data directly at a national level ?

In a perfect world our aggregated data should match the data from the national feed but high chances of having some differences.
I'm thinking we should keep the national data feed as well and add something to the country config like contryView: true to have it only displayed for the country level view as a replacement.

unitrium · 2022-12-22T09:29:11Z

config/zones/FR.yaml

@@ -290,3 +290,16 @@ sources:
  : link: https://unece.org/sites/default/files/2022-04/LCA_3_FINAL%20March%202022.pdf#page=37,
      https://proceedings.windeurope.org/biplatform/rails/active_storage/disk/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaDdDRG9JYTJWNVNTSWhORFJ0ZDJJMWVUbG9OMll6TVRaaGEza3lkamgxZG1aM056WnZZZ1k2QmtWVU9oQmthWE53YjNOcGRHbHZia2tpQVk1cGJteHBibVU3SUdacGJHVnVZVzFsUFNKWGFXNWtaWFZ5YjNCbExWZHBibVF0Wlc1bGNtZDVMV2x1TFVWMWNtOXdaUzB5TURJeExYTjBZWFJwYzNScFkzTXVjR1JtSWpzZ1ptbHNaVzVoYldVcVBWVlVSaTA0SnlkWGFXNWtaWFZ5YjNCbExWZHBibVF0Wlc1bGNtZDVMV2x1TFVWMWNtOXdaUzB5TURJeExYTjBZWFJwYzNScFkzTXVjR1JtQmpzR1ZEb1JZMjl1ZEdWdWRGOTBlWEJsU1NJVVlYQndiR2xqWVhScGIyNHZjR1JtQmpzR1ZBPT0iLCJleHAiOiIyMDIyLTExLTAyVDE1OjU0OjAzLjEyNFoiLCJwdXIiOiJibG9iX2tleSJ9fQ==--c25280a7345257bd91bfaf6d64ddb75c55eef799/Windeurope-Wind-energy-in-Europe-2021-statistics.pdf?content_type=application%2Fpdf&disposition=inline%3B+filename%3D%22Windeurope-Wind-energy-in-Europe-2021-statistics.pdf%22%3B+filename%2A%3DUTF-8%27%27Windeurope-Wind-energy-in-Europe-2021-statistics.pdf
 timezone: Europe/Paris
+subZoneNames:


Just wondering, how will the flowtracing pipeline react?
It will see data from France from both directly measured and also compute the aggregation data. So I guess we will have two datapoints for the same timestamp. One will probably replace the other as we are indexing by datetime. Since the aggregation is computed last, it will probably be the aggregation that will be in the database.

Could you check how it works for Sweden? It should both have working parsers and Aggregates the subzones.

Note: for Sweden the source is exactly the same as we sum the bidding zones in the ENTSO-E parser.

Ok thanks! I didn't know we were producing the SE data like this.
So the way I see it, the last step of the flowtracing pipeline computes spatial aggregations, which I think would be inserted after the flowtracing of parsed data. Therefore if a flowtraced data from parsed data exists with the same timestamp as the flowtraced data coming from the spatial aggregation step, it will be overwritten.

So as it stands I think having a main zone defined like this would most likely overwrite the data obtained through the parsers and have it replaced with data obtained through the aggregation step. Is it the intended behavior ?

Our aggregations should give the same as the national level (especially in the case of Sweden since we are doing it at parser level). But there might be differences we would need to explain.

So as it stands I think having a main zone defined like this would most likely overwrite the data obtained through the parsers and have it replaced with data obtained through the aggregation step.

We could remove the parser level aggregation for Sweden and see what happens? ENTSOE-E has country level data as well but it was missing solar and gas but it would be nice to do this anyway as it would save us a few API calls.

@unitrium yes it's correct that the country-wide data will be overwritten by the aggregation if the subzones pass the quality criterion (for now we just check if all subzones have been flowtraced). We could change the quality check to make sure the country load is equal to the aggregated load.

netlify · 2023-02-08T09:25:37Z

✅ Deploy Preview for phenomenal-syrniki-c5ceaa ready!

Name	Link
🔨 Latest commit	`471abc1`
🔍 Latest deploy log	https://app.netlify.com/sites/phenomenal-syrniki-c5ceaa/deploys/63e36a61830a640008a6907b
😎 Deploy Preview	https://deploy-preview-4853--phenomenal-syrniki-c5ceaa.netlify.app/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

VIKTORVAV99 · 2023-02-08T09:29:46Z

@corradio I just merged master into this branch and will try and get the geometries updated today but is there anything else blocking this from getting merged?

corradio · 2023-02-08T09:55:17Z

Mmh I think the only thing missing is proper data validation as our FR data is under high scrutiny.

…

On Wednesday, February 8, 2023, Viktor Andersson ***@***.***> wrote: @corradio <https://github.com/corradio> I just merged master into this branch and will try and get the geometries updated today but is there anything else blocking this from getting merged? — Reply to this email directly, view it on GitHub <#4853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMUIKB4U254QGCNPCG5TJDWWNRRLANCNFSM6AAAAAAS4MM2CI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Olivier Corradi Founder, CEO Njalsgade 76 2300 Copenhagen S electricitymaps.comlinkedin.com/electricitymaps <https://www.linkedin.com/company/electricitymaps>[image: Electricity Maps]

VIKTORVAV99 · 2023-10-13T14:14:57Z

Is this still blocked?

If so I think we should close the PR until it becomes unblocked.

VIKTORVAV99 · 2023-12-29T11:13:27Z

Closing this for now as there is more and more merge conflicts, it would most likely be easier to start from scratch than resolving all the merge conflicts.

Ailin Huang and others added 25 commits August 4, 2020 16:45

parser for French regions

c2eb0f8

fixing import error on lib.validation

77955b9

added french regions to zones.json

973490b

fixed alpabethical order of zones

04f205d

bounding boxes rechecked + exchanges added in exchanges.json

70712ba

geojson for regions added

4b73904

geojson imported to generate-geometries script and zone definitions a…

46e355f

…dded

moved FR-COR

d088702

map generated

58a675f

lib validation corrected

7558f23

merge master and upgrade installations

dc6c7e2

err

6485299

Revert "moved FR-COR", uncommitting build/*

46daf1c

This reverts commit d088702.

Revert "merge master and upgrade installations" - uncommiting node_mo…

42c4b8b

…dules This reverts commit dc6c7e2, reversing changes made to 7558f23.

Merge branch 'master' into France_regional_parser

2064538

Merge remote-tracking branch 'origin/master' into France_regional_parser

afff915

revert changes to unintended files

551a684

j rocks

61c1e80

Merge

59b01b7

fix polygons and zones, fixes #2954

486e409

wip

63cd078

Merge

484b4c6

Rewrite parsers

050ac10

Merge

31c7a88

Update ECO2MIX

ae235df

corradio marked this pull request as draft December 12, 2022 20:42

github-actions bot added exchange config Pull request or issue for exchange configurations frontend 🎨 parser zone config Pull request or issue for zone configurations labels Dec 12, 2022

update-world

322843b

VIKTORVAV99 mentioned this pull request Dec 15, 2022

France: mismatch between electricitymaps and RTE data #4870

Open

VIKTORVAV99 linked an issue Dec 15, 2022 that may be closed by this pull request

France: mismatch between electricitymaps and RTE data #4870

Open

corradio marked this pull request as ready for review December 16, 2022 07:56

VIKTORVAV99 reviewed Dec 16, 2022

View reviewed changes

corradio and others added 3 commits December 16, 2022 13:00

Update parsers/ECO2MIX.py

6e5fd4c

Co-authored-by: Viktor Andersson <[email protected]>

Update parsers/ECO2MIX.py

e4edc62

Co-authored-by: Viktor Andersson <[email protected]>

Update parsers/ECO2MIX.py

899f7b4

Co-authored-by: Viktor Andersson <[email protected]>

VIKTORVAV99 reviewed Dec 16, 2022

View reviewed changes

config/exchanges/BE_FR-ARA.yaml Outdated Show resolved Hide resolved

corradio added 2 commits December 19, 2022 09:39

Remove unused exchanges

88aa263

Merge branch 'olc/split-FR' of github.com:tmrowco/electricitymap-cont…

444a0ab

…rib into olc/split-FR

unitrium reviewed Dec 22, 2022

View reviewed changes

VIKTORVAV99 mentioned this pull request Jan 13, 2023

Country aggregation incomplete/inconsistent #4951

Closed

Merge remote-tracking branch 'upstream/master' into olc/split-FR

471abc1

VIKTORVAV99 added the BLOCKED Waiting on other task or external factors label Feb 8, 2023

VIKTORVAV99 mentioned this pull request Mar 12, 2023

Aggregate Corsica and continental France in country mode #5190

Open

ghost mentioned this pull request Mar 28, 2023

updated FR parser to enable historical backfill #5240

Merged

2 tasks

VIKTORVAV99 mentioned this pull request Aug 2, 2023

Clean up and modernize the test parser a bit. #5692

Merged

2 tasks

VIKTORVAV99 closed this Dec 29, 2023

VIKTORVAV99 deleted the olc/split-FR branch March 2, 2024 17:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

French regions #4853

French regions #4853

corradio commented Dec 12, 2022 •

edited

Loading

corradio commented Dec 15, 2022

VIKTORVAV99 commented Dec 15, 2022

VIKTORVAV99 left a comment

VIKTORVAV99 Dec 16, 2022

VIKTORVAV99 Dec 16, 2022

corradio commented Dec 16, 2022

VIKTORVAV99 commented Dec 16, 2022

unitrium commented Dec 22, 2022

unitrium Dec 22, 2022

VIKTORVAV99 Dec 22, 2022

unitrium Dec 22, 2022

unitrium Dec 22, 2022

unitrium Dec 22, 2022

VIKTORVAV99 Dec 22, 2022

corradio Jan 2, 2023

netlify bot commented Feb 8, 2023

VIKTORVAV99 commented Feb 8, 2023

corradio commented Feb 8, 2023 via email

VIKTORVAV99 commented Oct 13, 2023

VIKTORVAV99 commented Dec 29, 2023

		}


		def query(url_type_arg, session=None, target_datetime=None):

French regions #4853

French regions #4853

Conversation

corradio commented Dec 12, 2022 • edited Loading

Issue

Description

Outstanding issues

Data validation

Preview

corradio commented Dec 15, 2022

VIKTORVAV99 commented Dec 15, 2022

VIKTORVAV99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corradio commented Dec 16, 2022

VIKTORVAV99 commented Dec 16, 2022

unitrium commented Dec 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Feb 8, 2023

✅ Deploy Preview for phenomenal-syrniki-c5ceaa ready!

VIKTORVAV99 commented Feb 8, 2023

corradio commented Feb 8, 2023 via email

VIKTORVAV99 commented Oct 13, 2023

VIKTORVAV99 commented Dec 29, 2023

corradio commented Dec 12, 2022 •

edited

Loading