Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No custom options for env parameter in ingestions #11865

Open
SergeyMaslikhov opened this issue Nov 15, 2024 · 1 comment
Open

No custom options for env parameter in ingestions #11865

SergeyMaslikhov opened this issue Nov 15, 2024 · 1 comment
Labels
bug Bug report

Comments

@SergeyMaslikhov
Copy link

SergeyMaslikhov commented Nov 15, 2024

Describe the bug
I have several instances of Kafka running in different geographic regions. Some of them have the same naming of datasets, so without using env parameter only one of those datasets can be fetched. However env has only these options : {'STG', 'NON_PROD', 'RVW', 'UAT', 'PROD', 'TEST', 'PRE', 'CORP', 'QA', 'DEV', 'EI'} and trying to set another value in config leads to value_error. The issue is that I need to separate Kafka(and any other platform) environments by other value such as "GER", "FRA", etc., otherwise choosing from the listed values misleads users as all of the Kafka instances are actually 'PROD' but in its own region.
To Reproduce
Steps to reproduce the behavior:

  1. Go to Ingestion
  2. Click on Create new Kafka source
  3. In connection configuration choose YAML and set env : GER
  4. See error

Expected behavior
Setting custom env parameter without type_error

Screenshots
Снимок экрана 2024-11-15 в 13 48 37

Снимок экрана 2024-11-15 в 13 45 46
@SergeyMaslikhov SergeyMaslikhov added the bug Bug report label Nov 15, 2024
@deepgarg-visa
Copy link
Contributor

deepgarg-visa commented Nov 16, 2024

@SergeyMaslikhov, As you mention that they all are PROD but differ in regions. You can actually leverage "platform_instance" key for this.

source:
    type: kafka
    config:
        platform_instance: GER
        env: PROD
        connection:
            bootstrap: ' '
            schema_registry_url: ''
source:
    type: kafka
    config:
        platform_instance: FRA
        env: PROD
        connection:
            bootstrap: ' '
            schema_registry_url: ''

Then the expected urn for the dataset becomes:

urn:li:dataset:(urn:li:dataPlatform:kafka,GER.my-dataset, PROD)

urn:li:dataset:(urn:li:dataPlatform:kafka,FRA.my-dataset, PROD)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug report
Projects
None yet
Development

No branches or pull requests

2 participants