You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey team, I am testing sql to fluvio prototype for Klarian (written by Bryony), but I can't
write more then 14000000 records into fluvio topic, I have a local fluvio cluster on mac
book m3 max, it shall have enough capacity. It's strange behaviour: python script just hangs,
adding batching or multithreading didn't help: code is https://gist.github.com/AlexMikhalev/8b5ee7f284f99b75f80f7b912117cfb9
I have modification which just dumps all records into parquet file locally (in batches) and it works fine. Any hints how to debug?
I am currently on a "smaller" sql dump which is ~5GB, my total intake is sqlite 90GB file.
A Troubleshoot entry should be added as part of new docs to tackle this issue.
The text was updated successfully, but these errors were encountered:
From Alexanders Mikhalev issue https://infinyon.slack.com/archives/CN1D204C9/p1720005955323709
Context:
Hey team, I am testing sql to fluvio prototype for Klarian (written by Bryony), but I can't
write more then 14000000 records into fluvio topic, I have a local fluvio cluster on mac
book m3 max, it shall have enough capacity. It's strange behaviour: python script just hangs,
adding batching or multithreading didn't help: code is https://gist.github.com/AlexMikhalev/8b5ee7f284f99b75f80f7b912117cfb9
I have modification which just dumps all records into parquet file locally (in batches) and it works fine. Any hints how to debug?
I am currently on a "smaller" sql dump which is ~5GB, my total intake is sqlite 90GB file.
A Troubleshoot entry should be added as part of new docs to tackle this issue.
The text was updated successfully, but these errors were encountered: