spanner: slow initial session creation

[danielnorberg]

Creation of google_cloud_spanner::client::Client can block for several seconds on the initial session pool population.

$ RUST_LOG=google_cloud_spanner=debug ...

2024-11-13T13:33:33.623060Z DEBUG google_cloud_spanner::session: spawn session creation request : session_count = 2
2024-11-13T13:33:36.207973Z DEBUG google_cloud_spanner::session: spawn session creation request : session_count = 2
2024-11-13T13:33:37.437913Z DEBUG google_cloud_spanner::session: spawn session creation request : session_count = 2
2024-11-13T13:33:39.892527Z DEBUG google_cloud_spanner::session: spawn session creation request : session_count = 2
2024-11-13T13:33:42.392172Z DEBUG google_cloud_spanner::session: initial session created count = 8

Would it make sense to defer the session pool initialization entirely to the "session creation task"?

[yoshidan]

I think the best approach to reduce initialization time would be to run the following in parallel,

google-cloud-rust/spanner/src/session.rs

Line 238 in 2f9219b

for _ in 0..channel_num {

[danielnorberg]

Sounds reasonable. Even individual session creation requests seem slow however. I wonder why that is. I will try to reproduce using one of the offical spanner clients.

[danielnorberg]

Indeed official spanner clients are also slow at creating sessions. The golang client compensates for this by performing the initial session pool population both in parallel over each gRPC channel and asynchronously using goroutines.

[danielnorberg]

That said now when I'm testing the golang client seems to be able to create 25 sessions in a single BatchCreateSessions rpc in 1.3s, whereas the rust client takes 5~10s for a single rpc creating 2 sessions. I'll investigate some more.

[danielnorberg]

The culprit seems to be the lack of the google-cloud-resource-prefix metadata in the BatchCreateSessions rpc. Setting this same as the golang client does speeds up the rpc so it takes ~1s, equivalent to the golang client.