ADLSgen2 implementation guidance #1

blueww · 2023-07-10T08:36:18Z

Add a doc for ADLSgen2 implementation guidance , to better coorporate with community on implement ADLSGen2 in Azurite.

schoag-msft · 2023-07-10T15:01:45Z

doc/ADLSgen2 implementation guidance.md

@@ -0,0 +1,134 @@
+### Background
+Azurite is an open-source Azure Storage API compatible server (emulator). It currently supports the Blob, Queue, and Table services. We have received many customer asks on ADLSGen2 support in Azurite from many channels, include but not limited to [github issues](https://github.com/Azure/Azurite/issues/553), email, and requests from interested teams at Microsoft.


Correct "ADLSGen2" to "ADLS Gen2" or "Azure Data Lake Storage Gen2"

will update.

schoag-msft · 2023-07-10T15:03:13Z

doc/ADLSgen2 implementation guidance.md

+### Background
+Azurite is an open-source Azure Storage API compatible server (emulator). It currently supports the Blob, Queue, and Table services. We have received many customer asks on ADLSGen2 support in Azurite from many channels, include but not limited to [github issues](https://github.com/Azure/Azurite/issues/553), email, and requests from interested teams at Microsoft.
+
+We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community , try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectation and merge bar.


submitted by the community to implement ADLSgen2 in Azurite. However, we are unable to merge them at this time since they do not meet our expectation and merge bar.

Will update.

schoag-msft · 2023-07-10T15:04:31Z

doc/ADLSgen2 implementation guidance.md

+
+We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community , try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectation and merge bar.
+
+Azurite welcome contribution. To better coorporate with community on implement ADLSGen2 in Azurite, this document gives the details of the plan we suggest to implement ADLS Gen 2 in Azurite, and our expectations for community submissions that we can accept as PRs. 


Azurite welcomes contributions. To better cooperate with the community on an implementation of ADLS Gen2 in Azurite

Will update

schoag-msft · 2023-07-10T15:09:00Z

doc/ADLSgen2 implementation guidance.md

+### ADLSGen2 Introduction
+It's very important to understand ADLSgen2 feature before implementing it in Azurite.
+
+[Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) (aka: AdlsGen2) is a set of capabilities dedicated to big data analytics and built on Azure Blob Storage. 


Remove en-us/ culture from all URLs

schoag-msft · 2023-07-10T15:10:09Z

doc/ADLSgen2 implementation guidance.md

+
+[Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) (aka: AdlsGen2) is a set of capabilities dedicated to big data analytics and built on Azure Blob Storage. 
+#### FNS vs. HNS
+A normal Azure storage account is with Flat namespace (FNS). 


Storage accounts can configured with a flat namespace (FNS) or a hierarchical namespace (HNS). By default, storage accounts are FNS.

will update all links.

schoag-msft · 2023-07-10T15:11:05Z

doc/ADLSgen2 implementation guidance.md

+        1. Don’t need to change data store structure.
+        2. Don’t need Azurite user to differ HNS/FNS account.
+
+2. The change will add all dfs API interface to Azurite, which can help to support phase II.


capitalize DFS

Will update all DFS.

schoag-msft · 2023-07-10T15:11:20Z

doc/ADLSgen2 implementation guidance.md

+
+2. The change will add all dfs API interface to Azurite, which can help to support phase II.
+
+4. Code change should be split into several small PRs as following:


as follows:

will update

XiaoningLiu · 2023-07-11T03:35:44Z

doc/ADLSgen2 implementation guidance.md

+### Background
+Azurite is an open-source Azure Storage API compatible server (emulator). It currently supports the Blob, Queue, and Table services. We have received many customer asks on ADLSGen2 support in Azurite from many channels, include but not limited to [github issues](https://github.com/Azure/Azurite/issues/553), email, and requests from interested teams at Microsoft.
+
+We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community , try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectation and merge bar.


[minor]

Suggested change

We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community , try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectation and merge bar.

We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community, try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectations and merge bar.

Will update.

XiaoningLiu · 2023-07-11T03:38:33Z

doc/ADLSgen2 implementation guidance.md

+    2. 1 PR to add DFS endpoint 
+    3. Several PRs to implement each dfs API (with credential handler), include testing
+
+5. Need make sure each API behavior is aligned on rest API doc , also aligned with real Azure Server. See more in validation criteria.


Suggested change

5. Need make sure each API behavior is aligned on rest API doc , also aligned with real Azure Server. See more in validation criteria.

5. Need make sure each API behavior is aligned on rest API doc, also aligned with real Azure Storage services. See more in validation criteria.

Will update.

XiaoningLiu · 2023-07-11T03:40:09Z

doc/ADLSgen2 implementation guidance.md

+    1. E.g. .Net SDK need changes the blob/dfs Uri convert function in [this file](https://github.com/Azure/azure-sdk-for-net/blob/e8c40cc204b8cf750fcc820eab90d11f80612c3a/sdk/storage/Azure.Storage.Files.DataLake/src/DataLakeUriBuilder.cs#L275)
+
+##### Phase II: implementation Blob/DFS on HNS account
+1. Azurite user need configure each Azurite Account type as HNS/FNS when Azurite starts up. 


Suggested change

1. Azurite user need configure each Azurite Account type as HNS/FNS when Azurite starts up.

1. Azurite users need to configure each Azurite Account type as HNS/FNS when Azurite starts up.

Will update.

XiaoningLiu · 2023-07-11T03:41:54Z

doc/ADLSgen2 implementation guidance.md

+##### Phase II: implementation Blob/DFS on HNS account
+1. Azurite user need configure each Azurite Account type as HNS/FNS when Azurite starts up. 
+    1. Need design how to input the config (default should be FNS)
+    2.How to handle it when user start Azurite with change account type? (Report error? )


I'll suggest Azurite doesn't support FNS/HNS migration and account type change. The storage account type is finalized after creation. A wrong configuration (like wrong HNS/FNS type) should get error reported.

Suggested change

2.How to handle it when user start Azurite with change account type? (Report error? )

2. How to handle it when user start Azurite with change account type? (Report error? )

Add to Phase II 1.iii.

blueww · 2023-07-11T10:34:41Z

@schoag-msft , @XiaoningLiu

Thanks for the review!
I have updated the PR per your review comments.
Would you please help to see if any further comments?

Besides that, I have added a draft wiki page for it: https://github.com/blueww/Azurite/wiki/ADLS-Gen2-Implementation-Guidance.md

blueww force-pushed the dfsdoc branch from 2a3c679 to 3d5c5bb Compare July 10, 2023 08:40

Add doc for ADLSgen2 implementation

ac58b7a

blueww force-pushed the dfsdoc branch from 3d5c5bb to ac58b7a Compare July 10, 2023 08:41

schoag-msft reviewed Jul 10, 2023

View reviewed changes

XiaoningLiu reviewed Jul 11, 2023

View reviewed changes

Update ADLSgen2 implementation guidance.md

c6bb9e0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADLSgen2 implementation guidance #1

ADLSgen2 implementation guidance #1

blueww commented Jul 10, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

schoag-msft Jul 10, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

schoag-msft Jul 10, 2023

blueww Jul 11, 2023

XiaoningLiu Jul 11, 2023

blueww Jul 11, 2023

XiaoningLiu Jul 11, 2023 •

edited

Loading

blueww Jul 11, 2023

XiaoningLiu Jul 11, 2023

blueww Jul 11, 2023

XiaoningLiu Jul 11, 2023

blueww Jul 11, 2023

blueww commented Jul 11, 2023 •

edited

Loading

		@@ -0,0 +1,134 @@
		### Background
		Azurite is an open-source Azure Storage API compatible server (emulator). It currently supports the Blob, Queue, and Table services. We have received many customer asks on ADLSGen2 support in Azurite from many channels, include but not limited to [github issues](https://github.com/Azure/Azurite/issues/553), email, and requests from interested teams at Microsoft.


		We have get 2 PRs ([PR1](https://github.com/Azure/Azurite/pull/1933), [PR2](https://github.com/Azure/Azurite/pull/1934)) submitted by the community , try to implement ADLSgen2 in Azurite. However, we can't merge them now since they might not meet our expectation and merge bar.

		Azurite welcome contribution. To better coorporate with community on implement ADLSGen2 in Azurite, this document gives the details of the plan we suggest to implement ADLS Gen 2 in Azurite, and our expectations for community submissions that we can accept as PRs.


		2. The change will add all dfs API interface to Azurite, which can help to support phase II.

		4. Code change should be split into several small PRs as following:

	5. Need make sure each API behavior is aligned on rest API doc , also aligned with real Azure Server. See more in validation criteria.
	5. Need make sure each API behavior is aligned on rest API doc, also aligned with real Azure Storage services. See more in validation criteria.

	1. Azurite user need configure each Azurite Account type as HNS/FNS when Azurite starts up.
	1. Azurite users need to configure each Azurite Account type as HNS/FNS when Azurite starts up.

	2.How to handle it when user start Azurite with change account type? (Report error? )
	2. How to handle it when user start Azurite with change account type? (Report error? )

ADLSgen2 implementation guidance #1

Are you sure you want to change the base?

ADLSgen2 implementation guidance #1

Conversation

blueww commented Jul 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaoningLiu Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueww commented Jul 11, 2023 • edited Loading

XiaoningLiu Jul 11, 2023 •

edited

Loading

blueww commented Jul 11, 2023 •

edited

Loading