Storage.writer(...) breaks the idea of 'generation' #2476
Labels
api: storage
Issues related to the googleapis/java-storage API.
type: feature request
‘Nice-to-have’ improvement, new feature or different behavior or design.
Is your feature request related to a problem? Please describe.
I have an app that uploads a chunk of data to GCS, load that into BigQuery, and then delete this file from GCS. It does so many, many times over a period of time. Pseudo code below:
I noticed that, some delete calls seem to sporadically fail with 503 service unavailable. The error message suggests those errors are transitory and I should retry that. In looking at the GCS storage library code, I noticed that there's a built-in retry mechanism to transparently handle this kind of situation, it treats delete as idempotent operation if the generation match requirement is given (see
HttpRetryAlgorithmManager.getForObjectsDelete()
. That makes sense!Except, there's no way to actually capture the generation of my write. I can see that internally, the writer returned is
BlobWriteChannel
and itsstorageObject
property represents theBlobInfo
of a newly written blob, including the generation. That is how a method likeStorage.create()
can reliably return theBlob
object that represents the state at the point of creation. But there seems to be no way to access the same information with the writer(...) methods. I consider this a library design problem.Describe the solution you'd like
Storage.writer(...)
should return a subtype ofWriteChannel
that can returnBlob
after itsclose
method is invoked.Describe alternatives you've considered
Call
Storage.getBlob(BlobId)
after the write is done to obtain a freshBlob
from GCS separately. This risks the race condition.Additional context
#691 appears to be somewhat related, in the sense that it also wants additional information beyond
WriteChannel
The text was updated successfully, but these errors were encountered: