Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up the creation of the in-memory activities table #56

Open
KasiaKoz opened this issue Mar 5, 2024 · 0 comments
Open

Speed up the creation of the in-memory activities table #56

KasiaKoz opened this issue Mar 5, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@KasiaKoz
Copy link
Contributor

KasiaKoz commented Mar 5, 2024 •
edited by mfitz
Loading

In #54 we added the use of an in-memory "activities" Tablesaw table which is required to generate some KPIs. This activities table is generated from the trips table and is relatively slow to populate. For example, it takes nearly 9 minutes to create and populate the table on Michael's laptop for the Paris East baseline model:

2024-03-05T15:53:05,963  INFO TablesawKpiCalculator:990 Creating Activities Table
2024-03-05T15:53:06,106  INFO TablesawKpiCalculator:1005 About to iterate over 396900 unique persons in a trips table with 1444196 rows
2024-03-05T15:53:58,412  INFO TablesawKpiCalculator:1058 Created activities for 39690 persons so far
2024-03-05T15:54:49,894  INFO TablesawKpiCalculator:1058 Created activities for 79380 persons so far
2024-03-05T15:55:41,081  INFO TablesawKpiCalculator:1058 Created activities for 119070 persons so far
2024-03-05T15:56:34,626  INFO TablesawKpiCalculator:1058 Created activities for 158760 persons so far
2024-03-05T15:57:26,074  INFO TablesawKpiCalculator:1058 Created activities for 198450 persons so far
2024-03-05T15:58:18,169  INFO TablesawKpiCalculator:1058 Created activities for 238140 persons so far
2024-03-05T15:59:09,631  INFO TablesawKpiCalculator:1058 Created activities for 277830 persons so far
2024-03-05T16:00:01,579  INFO TablesawKpiCalculator:1058 Created activities for 317520 persons so far
2024-03-05T16:00:53,384  INFO TablesawKpiCalculator:1058 Created activities for 357210 persons so far
2024-03-05T16:01:45,991  INFO TablesawKpiCalculator:1058 Created activities for 396900 persons so far
2024-03-05T16:01:45,991  INFO TablesawKpiCalculator:1061 Finished creating Activities Table

We should look to make this quicker.
@KasiaKoz KasiaKoz mentioned this issue Mar 5, 2024
Merged
@KasiaKoz KasiaKoz added the enhancement New feature or request label Mar 5, 2024
@mfitz mfitz changed the title Improve activities table Speed up the creation of the in-memory activities table Mar 5, 2024
@mfitz mfitz mentioned this issue Mar 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KasiaKoz