Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add blobKeepAliveTimeout, queueKeepAliveTimeout, tableKeepAliveTimeout #2454

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Slach
Copy link

@Slach Slach commented Aug 26, 2024

second try after #2443

fix #2053 and workaround ClickHouse/ClickHouse#60447

Slach added a commit to Altinity/clickhouse-backup that referenced this pull request Aug 26, 2024
@blueww
Copy link
Member

blueww commented Aug 27, 2024

@Slach

Would you please share more background of this change?

  1. Does it target for a temp fix of CPP SDK issue
  2. Does product Azure server has same behavior after the change?

If it target a temp fix, or will make Azurite behavior not aligned with product Azure, as I have indicated in the before PR #2443 (comment), we will take the change to public Azurite, you can build your own private Azurite as a workaround.

@Slach
Copy link
Author

Slach commented Aug 27, 2024

  1. This change allows properly using azurite in e2e test scenarios with old version of azure cpp-sdk
    current version of cpp sdk still contains issue when client dont respect Keep-alive header from server

  2. Yes, by default azurite will have the same behavior as with the previous version, when send Keep-alive: timeout=5

@Slach
Copy link
Author

Slach commented Aug 27, 2024

You can't upgrade azure cpp sdk version in already released products

too much work for backports even when new cpp sdk will release

@blueww
Copy link
Member

blueww commented Aug 27, 2024

  1. This change allows properly using azurite in e2e test scenarios with old version of azure cpp-sdk
    current version of cpp sdk still contains issue when client dont respect Keep-alive header from server
  2. Yes, by default azurite will have the same behavior as with the previous version, when send Keep-alive: timeout=5

for #2, if user set none default value, will Azurite behavior different from product Azure?

If this is only for a temp SDK issue fix, Azurite won't take it. Especially when the change could make Azurite behavior different from product Azure.

We would suggest building your own private Azurite package.

@Slach
Copy link
Author

Slach commented Aug 27, 2024

If this is only for a temp SDK issue fix

this is not temp SDK issue fix.

You can't guarantee same behavior from other SDKs

issue become to your product from standard nodejs nodejs/node#34560

@blueww
Copy link
Member

blueww commented Aug 27, 2024

If this is only for a temp SDK issue fix

this is not temp SDK issue fix.

You can't guarantee same behavior from other SDKs

issue become to your product from standard nodejs nodejs/node#34560

Does the client SDK with this issue works with product Azure Server?
And if user set none default value, will Azurite behavior different from product Azure?

@Slach
Copy link
Author

Slach commented Aug 28, 2024

for second point, if user set none default value, will Azurite behavior different from product Azure?

when you pass --blobKeepAliveTimeout=0 or --blobKeepAliveTimeout=""
then this parameter will just ignore and nodejs httpServer will initialize with default 5000ms value

Actually, Azurite is different with Azure in this part, right now, without this PR

Azurite already send Keep-alive: timeout=5 header in server response
but Azure doesn't

check how exactly nodejs handle Keep-Alive

Azure example request+response

   PUT https://xxxbackups.blob.core.windows.net/xxxbackups/altinity-cloud-managed-clickhouse/backup-test/full_backup_8596655552399465575/shadow/test_partitions_TestIntegrationAzure/t2/default_435ef8e8df48e77cefdcc330dc8b8d8c_3_3_0.tar?blockid=lPNYkFCYSziFsOQF4bwpJQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%3D%3D&comp=block&timeout=14401
   Authorization: REDACTED
   Content-Length: [12288]
   User-Agent: [Azure-Storage/0.15 (go1.22.6; linux)]
   X-Ms-Client-Request-Id: [6cfaab65-8c46-4f48-68b5-a992ec7524ee]
   X-Ms-Version: [2020-10-02]
   x-ms-date: [Wed, 28 Aug 2024 12:31:05 GMT]
   --------------------------------------------------------------------------------
   RESPONSE Status: 201 Created
   Content-Length: [0]
   Date: [Wed, 28 Aug 2024 12:29:30 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [6cfaab65-8c46-4f48-68b5-a992ec7524ee]
   X-Ms-Content-Crc64: [ppWapnj/W7U=]
   X-Ms-Request-Id: [9b01ff55-d01e-006f-0445-f946cc000000]
   X-Ms-Request-Server-Encrypted: [true]
   X-Ms-Version: [2020-10-02]

Azurite for the same code

PUT http://devstoreaccount1.blob.azure:10000/container1/backup/full_backup_2226074798298314988/shadow/test_partitions_TestIntegrationAzure/t1/default_0%252D20220102_2_2_0.tar?blockid=sCJz0hyuTlWyUK7qdIfDHgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%3D%3D&comp=block&timeout=14401
   Authorization: REDACTED
   Content-Length: [12288]
   User-Agent: [Azure-Storage/0.15 (go1.22.6; linux)]
   X-Ms-Client-Request-Id: [161470f4-dc6f-41e4-6ea7-6f2e39b7ad9d]
   X-Ms-Version: [2020-10-02]
   x-ms-date: [Wed, 28 Aug 2024 12:45:10 GMT]
   --------------------------------------------------------------------------------
   RESPONSE Status: 201 Created
   Connection: [keep-alive]
   Content-Length: [0]
   Date: [Wed, 28 Aug 2024 12:45:10 GMT]
   Keep-Alive: [timeout=5]
   Server: [Azurite-Blob/3.32.0]
   X-Ms-Client-Request-Id: [161470f4-dc6f-41e4-6ea7-6f2e39b7ad9d]
   X-Ms-Request-Id: [bd712af4-70f8-4c94-aecd-af01199f6399]
   X-Ms-Request-Server-Encrypted: [true]
   X-Ms-Version: [2024-11-04]

Connection: [keep-alive]
Keep-Alive: [timeout=5]

present in Azurite response headers.

@blueww
Copy link
Member

blueww commented Aug 29, 2024

@Slach

Thanks for the details!

I see Azurite will return 2 more headers than Azure server :

Connection: keep-alive
Keep-Alive: timeout=5

In this case, it's better we make the default Azurite behavior aligned with Azure server as not return the 2 headers. Would you please help to update the PR for this?
And could we just remove the 2 headers instead adding the customized keeplive time?

We will review/test the PR and update you later.

src/queue/QueueServer.ts Outdated Show resolved Hide resolved
@Slach
Copy link
Author

Slach commented Aug 29, 2024

In this case, it's better we make the default Azurite behavior aligned with Azure server as not return the 2 headers. Would you please help to update the PR for this?

could you read details in nodejs/node#34560 ?

this is nodejs standard library behavior for any nodejs application which use httpServer, unfortunately i not so familiar with nodejs to make PR to nodejs

need time to think how to do it, current PR just allow us avoid conflicts with cpp-sdk

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@blueww
Copy link
Member

blueww commented Sep 2, 2024

In this case, it's better we make the default Azurite behavior aligned with Azure server as not return the 2 headers. Would you please help to update the PR for this?

could you read details in nodejs/node#34560 ?

this is nodejs standard library behavior for any nodejs application which use httpServer, unfortunately i not so familiar with nodejs to make PR to nodejs

need time to think how to do it, current PR just allow us avoid conflicts with cpp-sdk

Hope we could make the default behavior works like Azure server which won't return the headers.
If really can't find a way to get so, keep the default as current behavior should be the choice.

@blueww blueww closed this Sep 2, 2024
@blueww blueww reopened this Sep 2, 2024
@Slach
Copy link
Author

Slach commented Sep 2, 2024

If really can't find a way to get so, keep the default as current behavior should be the choice.

This is my working plan

@Slach
Copy link
Author

Slach commented Sep 3, 2024

Unfortunately, I didn't find how to completely disable Keep-Alive: timeout=XX header, will try to continue

@Slach
Copy link
Author

Slach commented Sep 3, 2024

@blueww could you trigger github actions?

README.md Outdated
@@ -186,10 +186,13 @@ Following extension configurations are supported:

- `azurite.blobHost` Blob service listening endpoint, by default 127.0.0.1
- `azurite.blobPort` Blob service listening port, by default 10000
- `azurite.blobKeepAliveTimeout` Blob service keep alive timeout, by default 5000
Copy link
Member

@blueww blueww Sep 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could more description add here? Like what's the unit of the time, is it ms or s? And the impact by adding this parameter.

After discuss with my team member, a question is raised:
When set blobKeepAliveTimeout = 5001, will the responds still be Keep-Alive: [timeout=5] or Keep-Alive: [timeout=5.001] . If the before one, we might should only allow user to set timeout with unit as second here, the default value should be 5.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figure out and switched from ms to seconds, look 11704a3

@@ -4,6 +4,7 @@ import SqlBlobConfiguration from "../src/blob/SqlBlobConfiguration";
import SqlBlobServer from "../src/blob/SqlBlobServer";
import { StoreDestinationArray } from "../src/common/persistence/IExtentStore";
import { DEFAULT_SQL_OPTIONS } from "../src/common/utils/constants";
import { DEFAULT_BLOB_KEEP_ALIVE_TIMEOUT } from "../src/blob/utils/constants";
Copy link
Member

@blueww blueww Sep 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add test case for B/Q/T service to check adding this parameter works., to avoid any regression in the future break the function?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for late response, could you suggest how to make this TestCase? How to check response headers to Keepalive-Timeout header exists and contains properly value?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EmmaZhu

Could you please share how to get a header in raw server responds with JS SDK?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EmmaZhu any news from your side?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Slach ,

You can use code like following to get keep-alive header:

    const properties = await newBlobClient.getProperties();
    const keepAliveHeader = properties._response.headers.get("keep-alive");

@blueww
Copy link
Member

blueww commented Sep 9, 2024

@Slach
I believe the PR is ready to review after you update it.
Just add 2 comments for the parameter description and test cases.

README.mcr.md Show resolved Hide resolved
@blueww
Copy link
Member

blueww commented Sep 18, 2024

@Slach

Could you please look at my comments above, and see if can fix them.
Feel free to let me know if you have different opinion.

@Slach
Copy link
Author

Slach commented Sep 18, 2024

sorry i'm on vacation will fix when return

@Slach
Copy link
Author

Slach commented Oct 14, 2024

Hi @blueww, i prepared some changes need your advice about tests
#2454 (comment)

current test code base is too complicated for my understanding
i tested changes internally manually and it works

@@ -77,6 +77,7 @@ export default class VSCServerManagerBlob extends VSCServerManagerBase {
const config = new QueueConfiguration(
env.queueHost(),
env.queuePort(),
env.blobKeepAliveTimeout(),
Copy link
Member

@blueww blueww Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be env.queueKeepAliveTimeout()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

unexptected TCP RST from azurite side
3 participants