Copiyng and decompressing huge files on the fly

[almartinez123]

I am trying to copy an OS image file, image.raw.gz from web server (http://server/image.raw.gz) to an unzipped destination in s3 image.raw (s3://bucket/folder/image.raw)

The process seems to work but it consumes a lot of memory.

How can I make it use less memory to perform the copy "on the fly"?

i tried to do it by declaring a buffer at 1024 but it didn't work

the function is like:

def copy_files_to_bucket(file_urls, bucket, prefix):
    for file_url in file_urls:
        file=os.path.basename(file_url)
        print ("copy " + file_url + " to " + bucket + prefix + file)
        with open(file_url, 'rb', transport_params=dict(buffer_size=1024)) as fin:
            with open("s3://" + bucket +"/" + prefix + "image.raw", 'bw', encoding=None, transport_params=dict(buffer_size=1024)) as fout:
                for line in fin
                    fout.write(line)

[mpenkov]

The process seems to work but it consumes a lot of memory.

Can you run a profiler to see where the memory consumption is?

[almartinez123]

profile.txt

the process was using 7gb of rss before i canceled it

[mpenkov]

That's not the kind of profiling that we need here. You're profiling CPU time, we need to profile memory usage.

See e.g. https://pypi.org/project/memory-profiler/ in particular the "line-by-line memory usage" section.