Add cutter doc and files to support cutting TE #1

syhwawa · 2024-09-16T15:04:37Z

As discussed in the previous PR in the Magellan repo here, I have moved all the relevant files to this repo for easier maintenance in the future.

This PR includes two key documentation files, some Python scripts used in the cutting process, and example MATSim config files to help users track the workflow:

cutting-scenarios-Transport-East.md: This is a general guide on how to run the cutter. It includes instructions on setting up and executing the cutter for the Transport East (TE) simulations.
matsim-cutter-debug-notes.md: This file contains detailed notes on how we debugged the cutter, including two remaining questions. It provides a reference for others to follow or troubleshoot as needed.

Additionally, I added some other files:

The Python scripts included were used during the debugging process.
The MATSim config examples are attached to demonstrate different stages of the cutting process.

All relevant files from the Magellan repo have been copied here. Feel free to delete the original PR, and I assume we can continue working on this one.

KasiaKoz

Some comments and questions. I think you can probably ignore the ones in docs/cutting-scenarios-CML/matsim-cutter-debug-notes.md as those are just running notes that don't need to be polished? Up to you tbh

KasiaKoz · 2024-10-02T10:00:47Z

core/src/main/java/org/eqasim/core/scenario/cutter/population/PopulationCutterModule.java

@@ -136,7 +138,10 @@ public ModeAwareTripProcessor provideModeAwareTripProcessor(RoutingConfigGroup r

 		if (transitConfig.isUseTransit()) {
 			// TODO: This may not only be "pt"!
-			tripProcessor.setProcessor(TransportMode.pt, transitTripProcessor);
+			List<String> transitModes = Arrays.asList("pt", "bus", "rail", "subway", "ferry", "taxi", "tram");


I think "pt" should be TransportMode.pt and the rest of the pt modes should probably come from the mode mapping part of the SBB config

KasiaKoz · 2024-10-02T10:05:42Z

docs/cutting-scenarios-CML/cutting-scenarios-Transport-East.md

+Here is an example of 10% populations statisitcs before cutting:
+```
+Population Stats:
+--------------+--------+
+|         stat | total  |
+--------------+--------+
+|          hhs | 398691 |
+|      persons | 398691 |
+--------------+--------+
+Modes:


I would have combined the tables so you can see the number next to each other, so:

'Here is an example of 10% populations statistics before and after cutting:'

Population Stats: +--------------+---------------+--------------+ | stat | total before | total after | +--------------+---------------+--------------+ | hhs | 398,691 | 15,367 | | persons | 398,691 | 15,367 | +--------------+---------------+--------------+

etc.

KasiaKoz · 2024-10-02T10:22:02Z

docs/cutting-scenarios-CML/cutting-scenarios-Transport-East.md

+Before starting the cutting procedure, the following files need to be prepared:
+
+- MATSim configurations and simulation inputs
+- Network car geometry for snapping facilities to activities


you can add a note here that this is available in genet's standard outputs

I can't remember anymore, but what projection should this file be in?

KasiaKoz · 2024-10-02T10:45:09Z

docs/cutting-scenarios-CML/cutting-scenarios-Transport-East.md

+For cutting 0.01% sims: `/mnt/efs/analysis/ys/matsim_cutter/input_files_locations_facilities/matsim_config_cutter.xml`
+For cutting the 10% sims: `/mnt/efs/analysis/ys/matsim_cutter/input_files_locations_facilities_10pc_20240909/matsim_config_cutter_no_intermodal_bike.xml`
+
+The example running command is saved in a bash script: `/mnt/efs/analysis/ys/matsim_cutter/run_cutter.sh`


a bash script counts as code, right? :D why is it on efs, can we have this script included in this repo and point to that instead?

KasiaKoz · 2024-10-02T10:47:51Z

docs/cutting-scenarios-CML/cutting-scenarios-Transport-East.md

+
+For cutting 0.01% sims:
+```sh
+java -Xmx12G -cp "core-1.5.0.jar:libs/matsim-2024.0/matsim-2024.0.jar:libs/matsim-2024.0/libs/*" \
+ org.eqasim.core.scenario.cutter.RunScenarioCutter \
+ --config-path /mnt/efs/analysis/ys/matsim_cutter/input_files_locations_facilities/matsim_config_cutter.xml \
+ --output-path output_20240729_001pct \
+ --extent-path /mnt/efs/analysis/ys/matsim_cutter/cut_shape_file/te_bury_st_edmund_27700.shp \
+ --config:plans.inputPlansFile /mnt/efs/analysis/ys/matsim_cutter/input_files_locations_facilities/output_plans.xml\
+ --prefix TE_cutter_ \
+ --threads 4 
+```


ah, are those the contents of that bash script above? might not be worth referencing the path on efs if the command is here, or make it clear those are the contents, in case someone decides to spend time looking for it..(I was tempted) 😅

KasiaKoz · 2024-10-02T11:01:04Z

docs/cutting-scenarios-CML/matsim-cutter-debug-notes.md

+- My local runtime params (running in IntelliJ):
+
+```
+--config-path /Users/mickyfitz/matsim-cutter/cutter-inputs/te/matsim_config_cutter_intermodal_bike.xml


aha, these are args for intelij, but would be good to include the class I think it's using org.eqasim.core.scenario.cutter.RunScenarioCutter ?

KasiaKoz · 2024-10-02T11:01:23Z

docs/cutting-scenarios-CML/matsim-cutter-debug-notes.md

+- Some model metadata:
+  - network nodes: 447,610
+  - network links: 985,170
+  - number of agents: 


missing info on agents

KasiaKoz · 2024-10-02T12:40:17Z

docs/cutting-scenarios-CML/python_scripts/network_connectivity_validation.py

+is_strongly_connected = nx.is_strongly_connected(G_directed)
+print(f"Is the directed graph strongly connected? {is_strongly_connected}")


I wish you had spoken to me about before diving into this. We have code in genet to test these things. Also, this is not enough - the network has to be strongly connected for modal subgraphs of the graph (so you have to test for just car links (and walk and bike if needed to be routed but not applicable here). In a network links geojson you will have all links, including PT, which will not be strongly connected, which is expected and ok.

KasiaKoz · 2024-10-02T12:41:41Z

docs/cutting-scenarios-CML/python_scripts/network_connectivity_validation.py

+print(f"Is the directed graph strongly connected? {is_strongly_connected}")
+
+# Find connected components
+connected_components = list(nx.connected_components(G_undirected))


you shouldn't really be analysing undirected network graphs, because that's not fit for our purpose. To get connected components of a directed graph you need to use the method: nx.strongly_connected_components

KasiaKoz · 2024-10-02T12:47:36Z

docs/cutting-scenarios-CML/python_scripts/network_connectivity_validation.py

+isolated_nodes = [node for node, degree in G_directed.degree() if degree == 0]
+print(f"Number of isolated nodes: {len(isolated_nodes)}")
+
+dead_end_nodes = [node for node, degree in G_directed.degree() if degree == 1]


in a directed graph you have to distinguish between in and out degrees. A dead end (sink node) can have any number for in degree (loads of links going in) but none coming out, so you're only looking for nodes with out degree of 0. Here's a genet method for that and unreachable nodes (source nodes) https://github.com/arup-group/genet/blob/afa6e0f2219f8e1069fe56667e4d6517210780ab/src/genet/validate/network.py#L13

syhwawa added 8 commits July 25, 2024 16:12

Add the SBB module

0af743a

Add more modes for the transit config

6c58953

Remove the unnecessary changes to SBB module

dcddbe6

Add the cutter instruction doc and debug notes

9aa8085

Add the supporting python scripts

041ba8d

Add matsim exmaple configs for running cutter

8f9b2bb

Update the docs

09a1249

Remove unnecessary changes

d3ec872

syhwawa requested review from mfitz and KasiaKoz September 16, 2024 15:04

KasiaKoz reviewed Oct 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cutter doc and files to support cutting TE #1

Add cutter doc and files to support cutting TE #1

syhwawa commented Sep 16, 2024 •

edited

Loading

KasiaKoz left a comment •

edited

Loading

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

KasiaKoz Oct 2, 2024

		is_strongly_connected = nx.is_strongly_connected(G_directed)
		print(f"Is the directed graph strongly connected? {is_strongly_connected}")

Add cutter doc and files to support cutting TE #1

Are you sure you want to change the base?

Add cutter doc and files to support cutting TE #1

Conversation

syhwawa commented Sep 16, 2024 • edited Loading

KasiaKoz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

syhwawa commented Sep 16, 2024 •

edited

Loading

KasiaKoz left a comment •

edited

Loading