Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide Troubleshooting Entry for Excedeed Storage #83

Open
EstebanBorai opened this issue Jul 3, 2024 · 2 comments
Open

Provide Troubleshooting Entry for Excedeed Storage #83

EstebanBorai opened this issue Jul 3, 2024 · 2 comments

Comments

@EstebanBorai
Copy link
Collaborator

From Alexanders Mikhalev issue https://infinyon.slack.com/archives/CN1D204C9/p1720005955323709

Context:

Hey team, I am testing sql to fluvio prototype for Klarian (written by Bryony), but I can't
write more then 14000000 records into fluvio topic, I have a local fluvio cluster on mac
book m3 max, it shall have enough capacity. It's strange behaviour: python script just hangs,
adding batching or multithreading didn't help: code is https://gist.github.com/AlexMikhalev/8b5ee7f284f99b75f80f7b912117cfb9
I have modification which just dumps all records into parquet file locally (in batches) and it works fine. Any hints how to debug?

I am currently on a "smaller" sql dump which is ~5GB, my total intake is sqlite 90GB file.


A Troubleshoot entry should be added as part of new docs to tackle this issue.

@sehz
Copy link
Contributor

sehz commented Jul 4, 2024

@EstebanBorai need to clean up this as this will made public in the future

@EstebanBorai
Copy link
Collaborator Author

@EstebanBorai need to clean up this as this will made public in the future

Agree! This is just a reminder but of course it needs more elaboration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants