Collection of European TSO data sources for improved real-time resolution

[alixunderplatz]

EUROPEAN TSO DATA SOURCE COLLECTION
VERSION 1.2 - 2018-04-24

Note: I will edit and update this text based on my progress!
Countries so far: AT, CH, CZ, HU, PL, RO, SK, EE, LV, LT
Added on 2018-04-24: EE, LT, LV, Baltic states summary

Hi everyone

I want to create a library for first-hand data sources for the European area.
These direct sources are hosted by the Transmission System Operators (TSOs) and they have a better time resolution (refreshing every few seconds to 15-minutes) than ENTSO-E.
This will mainly improve the real-time exchanges, since this is something that most TSOs publish with a high frequency. Exchanges are also varying much stronger over time than generation due to their instant reaction on events in the grid and changes in consumption/generation/load flow on the entire continent.
Of course, there is a lot of work to do for writing individual parsers, but it will be worth doing it!

Every entry below gives a link to the desired information for a region/country and a brief description.
Select the arrows to expand the country-specific data.

---

AT AUSTRIA

 

TSO: APG with page "Markttransparenz"

https://www.apg.at/de/markt/Markttransparenz/

Exchange data:
https://www.apg.at/de/markt/Markttransparenz/Uebertragung/Lastfluesse

This data is updated for 15 min intervals with a delay of ca. 15 - 30 min on map, table, graph and also available as XML or CSV download.
The interval of ENTSO-E data is only hourly. The data is also available faster on APG's website!

Generation data:

Generation data has an interval of 15 min like on ENTSO-E. Maybe availability on APG is also faster.

Site is also available in English language.
Pros: Exchange has better time resoution. Data may be published with smaller delay.
Cons: ???
 

---

CH Switzerland

 

TSO: swissgrid with page "Aktuelle Netzkennzahlen" / "Current grid key figures"

https://www.swissgrid.ch/swissgrid/en/home/reliability/griddata/current_data.html

Exchange data:

Updated every ~15 seconds.

Pros: fast update interval for live-exchange data
Cons: only some text in the images is accessible --> unless I'm wrong, we'd have to use OCR for exchange numbers in the image; no detailed generation data available, so we'd have to keep ENTSO-E
 

---

CZ CZECH REPUBLIC

 

TSO: ČEPS with page "All data"

http://www.ceps.cz/en/all-data#Generation

Generation data:
choosing "Display data for: generation"

It is available in 15 min intervals as "generation per production type". Updated every 15 min with a delay of ~30 min. Also available as TXT, XLS and XML download. Solar and Wind are available as high-resolution 1-minute data with a delay of ~15 min. Change yellow box on top for the other data.

Exchange data:

High resolution 1-min data for "crossborder power flows" with a delay of ~5 min. Also available as XML, XLS or TXT download. Choose the latest hour for the most recent data!

Pros: very fresh data for all generation types and exchanges; high 1-min resolution for fluctuating PV and Wind and Exchanges;
Cons: ???
By the way: CÊPS have something named "Web services": http://www.ceps.cz/en/web-services
Maybe there is sort of an API behind it?
 

---

HU HUNGARY

 

TSO: Mavir with page "Hungarian Power System actual data"

https://www.mavir.hu/web/mavir-en/hungarian-power-system-actual-data

image-url:
https://spectrum.mavir.hu/webonelines/1669_en.jpg

Exchange data: is available as .jpg which is refreshing in irregular intervals of ~1-2 min with delay of up to 2 min.

Pros: fresh data for all relevant exchanges with high resolution
Cons: We'd have to use OCR to extract the data. Maybe there is more and easily accessible data on the website, but the other links for generation etc. don't seem to work on any of my browsers.
 

---

PL POLAND

 

PSE-Operator with page "MAPA KSE"

https://www.pse.pl/dane-systemowe/funkcjonowanie-kse/raporty-dobowe-z-pracy-kse

This map and the charts below are refreshing every ~15 seconds.
The data is up to ~1,5 hours ahead of Entso-E. This is especially crucial for fluctuating wind and hydro generation because they lower the polish real-time emissions by a lot. We could take the combined hydro value from PSE and split it for previous values with entso-e data, if necessary.

ZAPOTRZEBOWANIE [MW] = DEMAND	24 462
GENERACJA [MW] = GENERATION	23 411
el. cieplne = THERMAL	21 548
el. wodne = HYDRO (incl. Pumped)	256
el. wiatrowe = WIND	1 585
SALDO WYMIANY CAŁKOWITEJ [MW] = BALANCE	1 090 IMPORT
CZĘSTOTLIWOŚĆ [Hz] = FREQUENCY	49,987

Pros: short refreshing interval; live wind energy data; accessible live data for all relevant exchanges
Cons: thermal and hydro given, but no seperation by type (coal/gas/biomass and usual hydro/pumped storage)
 

---

RO ROMANIA

 

TSO: Transelectrica with page "Starea Sistemului Național în timp real"

http://www.transelectrica.ro/en/web/tel/home

Exchange data:

http://www.transelectrica.ro/widget/web/tel/sen-harta/-/harta_WAR_SENOperareHartaportlet

It is updated about twice a minute with a delay of ~1.5 min. Detailed data for every single power line that is going to neighbouring countries, so one would have to find out to which country they are leading (easy).
Consum = consumption, Producţie = generation, carbune = coal, hidrocarburi = gas, hidro = hydro, nuclear = nuclear, eolian = wind, fotovoltaic = solar, biomasa = biomass, sold = balance (?)

Generation data:
There is a seperate page with graphs for generation. There is a 10-min interval for the data, which is published with a delay of 10 minutes. It is also avaiable as table or as XLSX below the graph.

Pros: live and high resolution exchanges, 10-min data for generation by type
Cons: ???
 

---

SK SLOVAKIA

 

TSO: SEPS with data platform damasenergy

https://dae.sepsas.sk/public/?CHART_VIEW_CODE=SYS_DE-SS_PUB-PUB-DATAFLOW-SYSTEM_PRODUCTION_AND_LOAD

Maybe one needs to create an account and log in to access all the data. Most of the sections under "publication" won't work on my browsers. Maybe it is just loading really slow. Try it on your system!

Generation data: probably available somewhere on the site

Exchange data:
There is this accessible map as an image:
https://dae.sepsas.sk/public/?UC_CODE=UC701SystemStateMap#-SS_PUB-PUB-DATAFLOW-SYSTEM_STATE_MAP_VIEW
or
https://dae.sepsas.sk/SPECIFIC/UseCases/701Publishing/MapGeneratorHandler.ashx

Blue values are measured live in ~5 min intervals. We'd have to use OCR to extract the exchange data from the page, unless the database on damasenergy isn't accessible somehow.

Pros: data with 5-min interval, probably database available with all required info
Cons: map is an image (--> OCR!)
By the way: All data sources from surrounding countries are available in this library!
 

---

EE ESTONIA

 

TSO: Elering

There is a dashboard for the real-time electricity exchanges on their front page:
https://dashboard.elering.ee/en
It's showing the flow for every interconnector to the neighbours, so the two in the North are with Finland, the two figures in the east with Russia, and the number in the south with Latvia.

The same info can be found in another graph at https://dashboard.elering.ee/en/transmission/cross-border?start=2018-04-15+21%3A00%3A00&end=2018-04-18+21%3A00%3A00&period=day

The "production and consumption" tab shows the entire hourly generation, the only seperation that can be made is by subtracting wind power from total generation and showing it in another category.

The best part is probably that there is an API, returning data in XML format!!! Find the documentation and further links here:
https://dashboard.elering.ee/documentation.html
Latest system key figures:
https://dashboard.elering.ee/api/system/latest

Note: When I checked (on 24 April 2018), the exchanges map was stuck on 19 April and generation data was also unavailable after that date. The reason is probably here:

 

---

LV Latvia

 

TSO: AST with "power system state" dashboard

Everything that is required can be found on the power system state page:
http://www.ast.lv/en/content/power-system-state

The generation breakdown per type is better than for Estonia and Lithuania.
The "other" category combines what is recently mapped as "other" and "biomass" on ENTSO-E, and thermal has the same values as "gas" on ENTSO-E. Resolution is hourly as well, so no advantage in comparison to ENTSO-E data.

 

---

LT Lithuania

 

TSO: Litgrid

http://www.litgrid.eu/index.php/dashboard/630

Same as for Estonia, only "total generation" and "wind power" generation are given, no further breakdown. However this data comes in 2 minute intervals, which is superb! It could at least be used for wind.
National consumption has a high interval of about 10 min.
Lithuania owns a pumped storage plant, but there is no further info on its generation or consumption on that site.

Cross-border power flows are also disclosed in a very high resolution, which is irregular for the different exchanges (2 min to 15 min). Data can be downloaded as .xls-file.
http://www.litgrid.eu/index.php/dashboard/cross-border-power-flows/642#

 

---

BALTIC STATES SUMMARY (EE, LV, LT)

 

TSO: AST with data dashboard for Baltic and Nordic states

There is another dashboard on the website of Latvian TSO "AST", which is probably based on the same data as the statnett dashboard. Here's the link:
http://www.ast.lv/en/content/power-system-state-0

Looks nice and has a very high resolution for all of the Baltic and Nordic states:

.XLS and .CSV downloads are available. For some earlier dates, resolution is 1 hour, now the interval is a few minutes, which is desirable!
 

---

TO BE CONTINUED ...

[systemcatch]

Exceptional! We can switch to some of these data sources if they're better or even run 2 parsers for a zones generation/exchange then use one as the main with the other as the backup for the map in the case the main source goes down. @corradio any thoughts on doing that?

[corradio]

Thanks @alixunderplatz !
Running on two parsers could be feasible. We'd have to change a bit the backend though.
I'm worried about edge cases when we receive data from two different providers, as all of our calculations, pipeline etc.. have assumptions that are often tied to one specific provider.
I prefer having one specific source running at all time (but we can always keep other parsers in case we want to revert).
I haven't gotten through the sources, but if they are better than ENTSOE, they it could make sense!

[systemcatch]

Is it possible to apply new calculations retrospectively to generation data collected? I'm thinking you could grab the data now and work on it later if some interesting use cases emerge.

[corradio]

Yes indeed. The problem is that we don't currently distinguish between multiple data providers. For practical purposes, if we agree that the real-time map should just take the last valid parser data, we can start adding arrays of parsers in zones.json. On our end, we would need to state (in the zones.json), which one is the main parser to be used when going through historical data (but that's in our backend). cc https://github.com/tmrowco/electricitymap-project/issues/369

…

On Sun, Jan 21, 2018 at 1:58 PM, Chris ***@***.***> wrote: Is it possible to apply new calculations retrospectively to generation data collected? I'm thinking you could grab the data now and work on it later if some interesting use cases emerge. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <electricitymaps/electricitymaps#1015 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABlEKFurGUUe8E1D7R_kaNkuEwDLiZQrks5tMzR3gaJpZM4RkpmE> .

[systemcatch]

Sounds good, we can make a start on some of these then. I suggest we name the parsers after the country with the operator included, e.g. PL_PSE.py for Poland.

I suppose using the 'source' key in the data returned is the easiest way to distinguish providers.

[alixunderplatz]

I think it will be totally necessary to collect the data for quite a while (before ever applying it to the map or sth. else, of course) and watch the ocurring differences in cross-border exchanges. There are obviously differences between the published data of neighbouring TSOs - their values, the timestamps, intervals and delay. We'd also have to think about using either the momentaneous value or the 5-minute-average of some high-frequency data, for example.
We'd have to apply the real-time values to the according entso-e generation data from the past with quite enough precision.

Recently, the sources in the list belong to a coherent area. Would be reallly interesting to see a comparison of the real-time values of the TSOs - by adding another arrow for the second source right next to the existing one or textboxes for the values or something like that.

BTW: I'll continue the work on filling the list above over the next days! ;)

[corradio]

@alixunderplatz that issue is a gold mine of information! I don't want it to go unnoticed.
Maybe we could add at the top a list of checkboxes that would serve as TODOs that we can spin out to individual issues?

[systemcatch]

With the current situation at ENTSOE these secondary sources become more important. It's also a good case for running multiple parsers per zone.

[systemcatch]

Maybe we could create new zones for each of these sources and have the data be collected in the backend only and not be displayed on the map. That way we'd avoid having to change the backend or deal with conflicting data but still be able to collect everything.

For example we could create a new zone called RO-TRAN for Romania.

[scriptator]

I created a parser for fetching AT exchange data from transparency.apg.at - see PR #2026 . I implemented it in order to find out

• how much faster the data is available compared to the ENTSOE data source
• whether the data is consistent with ENTSOE and with the data published by neighboring countries

I will now start to collect data from this parser so that I can evaluate these questions later. Should I open a separate issue?

[systemcatch]

@scriptator No it's fine, let's just use your PR for any discussion.

[jarek]

Is Selenium really needed for the Polish map? Doesn't https://www.pse.pl/transmissionMapService (which that page fetches with XHR) have everything that the web page does?

[ArturLange]

@jarek No, it's not. You're right, I deleted my comment when I realised that, but I guess someone did read it by then.

[corradio]

Hi @alixunderplatz!
What is the status here?
Should we try to split the information in this issue in individual issues for each country?

[ajoga]

I found historical data for switzerland, they look very reliable : https://www.swissgrid.ch/fr/home/operation/grid-data.html . On may 10, the latest update of the file was 31st marsh.. So not safe for live dataviz though

[VIKTORVAV99]

Moving this to a discussion as it's not really something we can "fix".