Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#396] Extend timeout for incremental single stream transfers #397

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

korydraughn
Copy link
Contributor

@korydraughn korydraughn commented Dec 18, 2024

This is a WIP.

The changes in this PR improve the server by bumping the timeout for read/write operations on data objects.

Read operations work very well. Write operations still need some improvements. To be specific, I want the server to be able to read/parse incomplete HTTP messages. v0.5.0 requires the full HTTP message to be sent to the server before processing it. Instead, it needs to be able to detect a write operation ASAP and immediately start pushing bytes to the iRODS server.

A few ideas I've been investigating (in no particular order):

  • (A) Use the Expect header as a signal to process incomplete HTTP messages
    • If the server sees the Expect header, the operation info would be included in the body
      • The file data would not be included by the client
    • The server would enter a state where it waits for another request containing the file data
    • The server would respond to the client with the HTTP status code of 100, along with a unique key
    • The client sends another request with the unique key and file data
  • (B) Use the Content-Type header as a signal to process incomplete HTTP messages
    • If multipart/form-data is set, handle incomplete HTTP messages (this would apply to all operations, but make the most sense for write operations)
  • (C) Change the API to be header-based
    • This makes it so that the operations are known up front
    • This could be provided as an alternative to the normal way of invoking the API
  • (D) Restructure the implementation to never require a full HTTP message
    • This means all messages are parsed on the fly
    • This affects the internals only and should not break existing user code
  • (E) Move operation parameter into the HTTP headers and branch differently based on its value
    • This allows the server to know what operation is getting ready to happen at the front door
    • The server would take a different path which is optimized for write operations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant