iter_chunks functionality rather than reading S3 file by line

[pm-src]

I'm looking to replace the below method of chunking/multi-part copy file from S3 to S3 (multi-region) using smart_open:s3.

I have combination of files, data, txt, pdf, binary etc etc. The requirement is to not read/write the file by line, instead based on the given chunk size. How can I achieve this by chunk read/write in S3 using smart_open:s3?

objectKey = s3Client.get_object(Bucket=bucketName, Key=key)
chunk_size = (5 * 1024 * 1024)
s3Client.create_multipart_upload(...)
for chunk in objectKey['Body'].iter_chunks(chunk_size=chunk_size):
-----s3Client.upload_part(chunk, ...)
s3Client.complete_multipart_upload(parts)

[mpenkov]

If you open the read and write destinations in binary mode, you'll be able to do this:

while True:
   buf = fin.read(chunk_size)
   if not buf:
     break
   fout.write(chunk_size)

[pm-src]

This above code worked out, here is working code:

`
chunk_size = (8 * 1024 * 1024)
client_args = {'chunk_size': chunk_size}
read_arguments = {'mode': 'rb', 'bucket_id': sourceBucketName, 'key_id': sourceKey, 'defer_seek': True,
'buffer_size': chunk_size, 'client': s3Client, 'client_kwargs': client_args}

transport_params = {'client': s3Client, 'buffer_size': chunk_size, 'multipart_upload': True}
write_arguments = {'mode': 'wb', 'uri': targetUri, 'transport_params': transport_params}

with s3.open(**read_arguments) as read_file:
with open(**write_arguments) as write_file:
while True:
buffer = read_file.read(chunk_size)
if not buffer:
break
write_file.write(buffer)
iteration += 1
log.info('Stream copy finished in iterations :', iteration)
`