-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to distinguish an object is "large" #12
Comments
String cannot be large because no one holds book level string in a variable. Vector is the only concern because it can appear from numeric computations. The more difficult case is actually dictionaries because it can hold large items. |
Let me see what's the limitation in my 8GB ram laptop. 😄 |
Yes, that is something we should do, namely stress testing the magic and see when it breaks. Basically you can generate larger and larger arrays and see how well they can be passed around. |
After about 10 mins, memory is exhausted.
Also about 10 mins, memory is exhausted. The memory used by python is not that much, but R used >8GB after 10 mins. (There is only 8GB memory on my laptop) |
I also noticed that when I shut down the kernel, process |
And even though I shut down jupyter, those |
As we discussed before, some large objects should be transferred by I/O on disk instead of using memory. However, how can we distinguish the threshold?
object
? Or size of aobject
saving as file?data.frame
,list
,matrix
etc. are transferred by I/O on disk (i.e. usingfeather
). I suppose the only large objects not usingfeather
to be transferred arevector
orstring
?The text was updated successfully, but these errors were encountered: