Time delay

[Bohne13]

Since several weeks, I have alway a time delay from two to three hours.
It doesn't matter which country or on which device I'm opening the app or website.

I also checked the raw data if there is a time delay but there is data provided from 15 mins ago
On the Screenshot it shows clearly the time of the desktop clock (11:38 o'clock) and the provided data (9:00 o'clock)

is there a fix to get the latest data shown in the app / website?

.

[PHPGangsta]

I also have this problem. Since 2 weeks or so the time in the graphs is 2-3 hours in the past.

I'm living in Germany with a German browser. Maybe a problem with the current timezone of the browser which is currently GMT+2 here?

[jarek]

I don't think it's related to user timezone since I'm in UTC-4, it is currently 20:56 local time, and the latest data I see is timestamped 18:00 local time (22:00Z) - almost 3 hours delayed

[Bohne13]

Yeah it's not a timezone issue. It was the first thing I had in mind but sometimes there are three hours in delay, sometimes only two.
I think it's something with the parser or more general in the backand.
Don't know how to find the root cause!?

[Kongkille]

Hello, sorry for the late answer on this issue, the time delay stems from the new pipeline we launched a few months ago. The new pipeline was (and is) a huge enabler on anything related to the backend and any upcoming features.

Specifically the time delay happens because the new pipeline awaits for all data to come in before performing the flow-tracing. Previously, it would sometimes calculate the origin without having received all the data it needs, which causes all kinds of issues.

We are looking into some different ways to reduce the delay, and I'll be sure to post an update here when we know more.

[Bohne13]

Thanks for your information, I hope there is a fix in the near future!
It's way more helpful with "realtime" data

[corradio]

@Kongkille @madsnedergaard @tonypls , now that we have estimations, couldn’t we attempt to show data from the current hour? This should give us the ability to minimize delay while still showing “real” data when available

EDIT: I made a quick test, and these are the zones that show up when we return the current hour.
Estimation pipeline currently only runs to H-1, but if we made run to H or H+1, then we'd show data for the gray zones.

[Kongkille]

I don't see any technical reasons to why we wouldn't be able to do that, other than a few side-effects on the api (?). I think it's more a question of whether we believe that to be a better experience?

If we always show the latest hour, most of the data shown on the map will be estimated. While you can of course drag the timeslider back to see measured data, my hunch tells me that this is not something most users do in a typical session. At least not for all zones visited.

[tonypls]

I think it would be great to move closer to a real time view in the app. We could look at the delta in # of estimated zones for the current H-2 display in the app and H

[corradio]

If we always show the latest hour, most of the data shown on the map will be estimated.

Note there would actually be quite a substantial amount of zones with measured data (see screenshot above). I find it unfortunate that we're not showing these data by introducing the artificial delay.

While you can of course drag the timeslider back to see measured data, my hunch tells me that this is not something most users do in a typical session. At least not for all zones visited.

Maybe users wouldn't have to drag back if the estimated data is quite close to the last measured data (i.e. the estimation model is good enough)?

[madsnedergaard]

FYI this would also be solved by the hackday project Felix (and I) did with data stream :)
Since I don't think that will/should be prioritised right now, I'd be happy to discuss if we can do an interim solution though.

[VIKTORVAV99]

Bringing some life back into this issue/discussion with one major blocker (and potential solutions).

• Exchanges, they are currently not estimated and plays a massive role in the map, this is being worked on however.

Estimation of exchanges and the future changes of the European, Nordic and Baltic grid to 15 min resolutions should bring down the ENTSO-E delay to 30 min so it should be possible for us to cut down the delay to at least 1 hour in the near future.

I think we should revisit this to see what our current data coverage with a 1 hour delay would be and what exchanges/zones need to be estimated to bring this up to the current shown zones and prioritize those areas for estimation if needed.

[jarek]

Specifically the time delay happens because the new pipeline awaits for all data to come in before performing the flow-tracing. Previously, it would sometimes calculate the origin without having received all the data it needs, which causes all kinds of issues.

Is it possible to split the pipelines into continents or other subdivisions? Current time in Ontario is 7:56pm and Electricity Map has latest data for 5:00pm. Ontario production and all exchanges come from IESO. The production parser returns data for 7pm and all exchanges return data for 6:55pm. That's almost 2 hours more timely data that Electricity Maps could be showing (assuming exchanging zones also have data on generation intensity, or can be estimated).

[VIKTORVAV99]

Specifically the time delay happens because the new pipeline awaits for all data to come in before performing the flow-tracing. Previously, it would sometimes calculate the origin without having received all the data it needs, which causes all kinds of issues.

Is it possible to split the pipelines into continents or other subdivisions? Current time in Ontario is 7:56pm and Electricity Map has latest data for 5:00pm. Ontario production and all exchanges come from IESO. The production parser returns data for 7pm and all exchanges return data for 6:55pm. That's almost 2 hours more timely data that Electricity Maps could be showing (assuming exchanging zones also have data on generation intensity, or can be estimated).

I've been thinking along the same lines, at least when it comes to calculating the data on the backend but I think we should keep showing the same "datetime" on the app to avoid confusion.

My idea however was to compute zones into grids based on their exchanges, in this case we would have one major American grid, one with Europe, Russia + China and maybe some others. All islands without interconnections to the mainland would become their own grids (australia and japan) and all single zone (islands) without exchanges of any kind could be bypassed in the flow tracing (nothing to flow trace).

This should allow us to run several flow tracing pipelines in parallel in async hopefully cutting down the time it takes to compute the values for all zones.

If we can cut down on the compute time and therefore latency from when we get the data to once it's processed, we could show the data faster which would become more relevant as our data sources increase their own resolution.

Note this is just my own ideas and nothing that has been discussed with the team yet.

[Killerbert]

Unfortunately this issue is still open
Are there no improvements possible in 3 years?

I would at least suggest to remove the "LIVE" icon and replace it with the latest available timestamp of the data

[VIKTORVAV99]

Unfortunately this issue is still open Are there no improvements possible in 3 years?

I would at least suggest to remove the "LIVE" icon and replace it with the latest available timestamp of the data

We have several updates planned (removing live is one of them) but the core issue is that the source data is also delayed by 1-2 hours, and then there is processing time on top of that on our side.

We have some potential solutions in mind like the one I've mentioned above but these are fundamental changes in our core infrastructure and takes time to plan out and implement.

[VIKTORVAV99]

For transparency sake I would just like to add that we have been able to reduce the delay internally and are currently evaluating the quality of it but hopefully we can remove the delay or reduce it to just 1 hour soon!