Editing large JSON takes a lot of memory via sjson in JJ

[khuntia]

I was trying to edit (ie add or change a field of json) with jj. I see the memory is shooting up atleast 10 times before it can process the change. So I need for editing a 1GB json atleast 11GB memory allocated. Though the whole process is fast (for a i3, 15GB RAM machine, takes ~18 secs to edit the last index), it crashes many times because of out of memory.

I have seen reference about jj caching large amounts of memory. Please let me know if this can be optimized.

FYI, Fetching data is fast and fine and only takes the same amount of memory as the size of the json for a short period while doing the GETs.

-------------------Reference issue mentioned----------

Hi pkoppstein, I'm looking into these issues. I'm believe jj is buffering too much data prior to processing and low memory systems suffer when dealing with large json files. I'll look asap and keep you posted. Thanks!

Originally posted by @tidwall in #9 (comment)

[tidwall]

Oh no. 10x increase in ram for setting a value? 😲 Yikes!
What is the path that you're using to update the value?

[khuntia]

Hi Josh,

I used the below command to replace id in an element in a large 1gb json file. I tested in different system, the same result of 10x RAM usage is happening.

JJ command to replace element id=600 with 2500

jj -i 1gb.json -r -v '{"id":2500, "next":2501, "_id":"625664e3ea574290b931f172","index":0,"guid":"e300c649-6f2c-4a60-9b51-bc1be08d0a14","isActive":false,"balance":",764.44","picture":"http://placehold.it/32x32","age":38,"eyeColor":"brown","name":"Hart Kline","gender":"male","company":"LUNCHPAD","email":"[email protected]","phone":"+1 (840) 496-2259","address":"643 Clara Street, Groveville, North Carolina, 4785","registered":"2015-11-02T04:02:38 -06:-30","latitude":82.284556,"longitude":-53.359112,"tags":["ex","duis","commodo","et","ad","voluptate","cupidatat"],"friends":[{"id":0,"name":"Bradford Shaffer"},{"id":1,"name":"Monroe Kent"},{"id":2,"name":"John Carey"}],"greeting":"Hello, Hart Kline! You have 9 unread messages.","favoriteFruit":"strawberry","level-0":{"tags":[1,2,3],"value":"0","level-1":{"tags":[1,2,3],"value":"1","level-2":{"tags":[1,2,3],"value":"2","level-3":{"tags":[1,2,3],"value":"3","level-4":{"tags":[1,2,3],"value":"4","level-5":{"tags":[1,2,3],"value":"5","level-6":{"tags":[1,2,3],"value":"6","level-7":{"tags":[1,2,3],"value":"7","level-8":{"tags":[1,2,3],"value":"8","level-9":{"tags":[1,2,3],"value":"9","level-10":{"tags":[1,2,3],"value":"10","level-11":{"tags":[1,2,3],"value":"11","level-12":{"tags":[1,2,3],"value":"12","level-13":{"tags":[1,2,3],"value":"13","level-14":{"tags":[1,2,3],"value":"14","level-15":{"tags":[1,2,3],"value":"15","level-16":{"tags":[1,2,3],"value":"16","level-17":{"tags":[1,2,3],"value":"17","level-18":{"tags":[1,2,3],"value":"18","level-19":{"tags":[1,2,3],"value":"19","level-20":{"tags":[1,2,3],"value":"20","level-21":{"tags":[1,2,3],"value":"21","level-22":{"tags":[1,2,3],"value":"22","level-23":{"tags":[1,2,3],"value":"23","level-24":{"tags":[1,2,3],"value":"24","level-25":{"tags":[1,2,3],"value":"25"}}}}}}}}}}}}}}}}}}}}}}}}}}}' 'arr.#(id==600)#' -o 1gb_out.json

To generate 1gb file

./generatelargejson.sh > 1gb.json

#generatelargejson.sh

#!/bin/bash
#generateBigJson.sh

MAX=600000
echo "{"arr":["
i=1
while [ $i -le $MAX ]; do
echo -ne "{"id":$i, "next":$((i+1)), "_id":"625664e3ea574290b931f172","index":0,"guid":"e300c649-6f2c-4a60-9b51-bc1be08d0a14","isActive":false,"balance":"$2,764.44","picture":"http://placehold.it/32x32","age":38,"eyeColor":"brown","name":"Hart Kline","gender":"male","company":"LUNCHPAD","email":"[email protected]","phone":"+1 (840) 496-2259","address":"643 Clara Street, Groveville, North Carolina, 4785","registered":"2015-11-02T04:02:38 -06:-30","latitude":82.284556,"longitude":-53.359112,"tags":["ex","duis","commodo","et","ad","voluptate","cupidatat"],"friends":[{"id":0,"name":"Bradford Shaffer"},{"id":1,"name":"Monroe Kent"},{"id":2,"name":"John Carey"}],"greeting":"Hello, Hart Kline! You have 9 unread messages.","favoriteFruit":"strawberry","level-0":{"tags":["1","2","3"],"value":"0","level-1":{"tags":["1","2","3"],"value":"1","level-2":{"tags":["1","2","3"],"value":"2","level-3":{"tags":["1","2","3"],"value":"3","level-4":{"tags":["1","2","3"],"value":"4","level-5":{"tags":["1","2","3"],"value":"5","level-6":{"tags":["1","2","3"],"value":"6","level-7":{"tags":["1","2","3"],"value":"7","level-8":{"tags":["1","2","3"],"value":"8","level-9":{"tags":["1","2","3"],"value":"9","level-10":{"tags":["1","2","3"],"value":"10","level-11":{"tags":["1","2","3"],"value":"11","level-12":{"tags":["1","2","3"],"value":"12","level-13":{"tags":["1","2","3"],"value":"13","level-14":{"tags":["1","2","3"],"value":"14","level-15":{"tags":["1","2","3"],"value":"15","level-16":{"tags":["1","2","3"],"value":"16","level-17":{"tags":["1","2","3"],"value":"17","level-18":{"tags":["1","2","3"],"value":"18","level-19":{"tags":["1","2","3"],"value":"19","level-20":{"tags":["1","2","3"],"value":"20","level-21":{"tags":["1","2","3"],"value":"21","level-22":{"tags":["1","2","3"],"value":"22","level-23":{"tags":["1","2","3"],"value":"23","level-24":{"tags":["1","2","3"],"value":"24","level-25":{"tags":["1","2","3"],"value":"25"}}}}}}}}}}}}}}}}}}}}}}}}}}}"
if [ "$i" -ne "$MAX" ]; then
echo ","
fi
let i=i+1
done
echo "]}"