Fix ingest api 0.12.0 #143

xiphl · 2024-01-25T07:14:17Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Co-authored-by: Harshal Sheth <[email protected]>

…8760) Co-authored-by: Tamas Nemeth <[email protected]>

…atahub-project#8357) Co-authored-by: cccs-Dustin <[email protected]> Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]>

Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: RyanHolstien <[email protected]>

…t#8764)

…8745)

…project#8456)

)

…efault (datahub-project#8791)

…tgres docker setup (datahub-project#8762)

)

…ub-project#8672) Co-authored-by: Aseem Bansal <[email protected]> Co-authored-by: Indy Prentice <[email protected]>

…atahub-project#8154) Co-authored-by: Dominik Schüssele <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]>

…nnotations (datahub-project#8779) Co-authored-by: Harshal Sheth <[email protected]>

…atahub-project#8778)

Co-authored-by: Tamas Nemeth <[email protected]>

…-project#9058)

…b-project#9109)

Co-authored-by: John Joyce <[email protected]>

…e conflicts

…-impl:checkRestModel'

github-actions · 2024-01-25T07:14:43Z

.github/workflows/airflow-plugin.yml

+          token: ${{ secrets.CODECOV_TOKEN }}
+          directory: .
+          fail_ci_if_error: false
+          flags: airflow-${{ matrix.python-version }}-${{ matrix.extraPythonRequirement }}


[actionlint] _{reported by reviewdog 🐶}
property "extrapythonrequirement" is not defined in object type {extra_pip_extras: string; extra_pip_requirements: string; python-version: number} [expression]

github-actions · 2024-01-25T07:14:43Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute DataHub Ingestion (Base) Tag
+        id: tag
+        run: echo "tag=${{ steps.filter.outputs.datahub-ingestion-base == 'true' && needs.setup.outputs.unique_tag || 'head' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:120: Double quote to prevent globbing and word splitting [shellcheck]

github-actions · 2024-01-25T07:14:43Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute DataHub Ingestion (Base-Slim) Tag
+        id: tag
+        run: echo "tag=${{ steps.filter.outputs.datahub-ingestion-base == 'true' && needs.setup.outputs.unique_slim_tag || 'head-slim' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:130: Double quote to prevent globbing and word splitting [shellcheck]

github-actions · 2024-01-25T07:14:43Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute DataHub Ingestion (Base-Full) Tag
+        id: tag
+        run: echo "tag=${{ steps.filter.outputs.datahub-ingestion-base == 'true' && needs.setup.outputs.unique_full_tag || 'head' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:125: Double quote to prevent globbing and word splitting [shellcheck]

github-actions · 2024-01-25T07:14:43Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute Tag
+        id: tag
+        run: echo "tag=${{ (steps.filter.outputs.datahub-ingestion-base == 'true' || steps.filter.outputs.datahub-ingestion == 'true') && needs.setup.outputs.unique_slim_tag || 'head-slim' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:184: Double quote to prevent globbing and word splitting [shellcheck]

github-actions · 2024-01-25T07:14:43Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute Tag (Full)
+        id: tag
+        run: echo "tag=${{ (steps.filter.outputs.datahub-ingestion-base == 'true' || steps.filter.outputs.datahub-ingestion == 'true') && needs.setup.outputs.unique_tag || 'head' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:174: Double quote to prevent globbing and word splitting [shellcheck]

xiphl · 2024-01-25T07:32:04Z

tried to cheapskate and just pip install acryl-datahub, cos mostly we just need to call the existing datahub types, but it doesnt work cos of the UnknownClassType.

xiphl · 2024-01-25T09:48:12Z

hmm. ingestion image have the unknowntype class. now to figure out how to make ingest-api do the same.

github-actions · 2024-01-29T03:07:06Z

.github/workflows/docker-unified.yml

    outputs:
-      image_tag: ${{ steps.docker_meta.outputs.tags }}
-      image_name: ${{ env.INGEST_API_IMAGE }}
+      tag: ${{ steps.tag.outputs.tag }}


[actionlint] _{reported by reviewdog 🐶}
property "tag" is not defined in object type {filter: {conclusion: string; outcome: string; outputs: object}} [expression]

github-actions · 2024-01-29T05:19:37Z

.github/workflows/docker-unified.yml

+          platforms: linux/amd64,linux/arm64/v8
+      - name: Compute Tag
+        id: tag
+        run: echo "tag=${{ (steps.filter.outputs.datahub-ingestion-base == 'true' || steps.filter.outputs.datahub-ingestion == 'true' || steps.filter.outputs.ingest-api == 'true' ) && needs.setup.outputs.unique_slim_tag || 'head-slim' }}" >> $GITHUB_OUTPUT


[actionlint] _{reported by reviewdog 🐶}
shellcheck reported issue in this script: SC2086:info:1:230: Double quote to prevent globbing and word splitting [shellcheck]

neojunjie · 2024-02-13T08:31:43Z

added the code in pr142

mayurinehate and others added 30 commits August 30, 2023 14:08

ci: separate airflow build and test (datahub-project#8688)

e867dbc

Co-authored-by: Harshal Sheth <[email protected]>

fix(ingest/athena): fix container linting (datahub-project#8761)

4c69f9a

fix(datahub-frontend) Give permission for start.sh so it can run (dat…

e7d140f

…ahub-project#8594)

feat(sql-parser): schema-aware output column casing (datahub-project#…

21b2851

…8760) Co-authored-by: Tamas Nemeth <[email protected]>

fix(ingest/bigquery): Filter out fine grained lineage with no upstrea…

a4e7268

…ms (datahub-project#8758)

feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (d…

6fe60a2

…atahub-project#8357) Co-authored-by: cccs-Dustin <[email protected]> Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]>

Allow frontend to use http proxy (datahub-project#8691)

d78a790

Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: RyanHolstien <[email protected]>

docs(observe): Dataset Profile volume assertion guide (datahub-projec…

ee06a65

…t#8764)

docs:fix broken img links under managed-datahub (datahub-project#8769)

5458c6b

fix:small typo on graphql tutorial (datahub-project#8741)

31fc687

refactor(build): upgrade to gradle 7 & guava update (datahub-project#…

7dd6e09

…8745)

fix(siblings): space icons out (datahub-project#8767)

23f24f4

chore(build): upgrade gradle wrapper (datahub-project#8776)

59b59c2

feat(EntityService): batched transactions and ebean updates (datahub-…

1b79142

…project#8456)

fix(frontend): Fix"Logout with OIDC not working" (datahub-project#8773)

d308986

docs:upgrade docusaurus version (datahub-project#8770)

801208e

fix:change global graph url to static-assets (datahub-project#8742)

065a290

doc(tests): fix endpoint param to push results (datahub-project#8783)

9f223c0

fix(elastic): improve error handling for profiling (datahub-project#8785

c38bb91

)

chore(analytics): bump version (datahub-project#8786)

7b443e5

docs(session): add documentation for session token duration and fix d…

5b7487d

…efault (datahub-project#8791)

fix(ingest/datahub): Support postgres; build(postgres): Modernize pos…

ac025e5

…tgres docker setup (datahub-project#8762)

feat(airflow-plugin): add package type information (datahub-project#8795

f649d6f

)

feat(systemMetadata): Adding a lastRunId field system metadata (datah…

94fa514

…ub-project#8672) Co-authored-by: Aseem Bansal <[email protected]> Co-authored-by: Indy Prentice <[email protected]>

refactor(ingest): Add support for group-owners in dataflow entities (d…

25148f4

…atahub-project#8154) Co-authored-by: Dominik Schüssele <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]>

fix(ingest/tableau): fix tableau native CLL for snowflake, add type a…

8bf28bf

…nnotations (datahub-project#8779) Co-authored-by: Harshal Sheth <[email protected]>

fix(ingest/bigquery): fix partition and median queries for profiling (d…

e680a97

…atahub-project#8778)

docs: add datahub source to integrations page (datahub-project#8787)

f353bc7

chore(ingest): upgrade sqlglot fork (datahub-project#8775)

4ffad4d

Co-authored-by: Tamas Nemeth <[email protected]>

docs: minor fix on versioning navbar and dropdown (datahub-project#8790)

6a1a1dd

siddiquebagwan-gslab and others added 10 commits October 25, 2023 11:18

fix(ingestion/redshift): fix schema field data type mappings

b9508e6

fix(datahub-protobuf): add check if nested field is reserved (datahub…

1c77bca

…-project#9058)

fix(ingest): better handling around sink errors (datahub-project#9003)

32f5dcb

feat(ingest/bigquery): Attempt to support raw dataset pattern (datahu…

6c932e8

…b-project#9109)

docs(observability): Column Assertion user guide (datahub-project#9106)

2ebf33e

Co-authored-by: John Joyce <[email protected]>

resolved conflicts

ce54d2d

not sure why the build.gradle content was removed after i resolved th…

1dc459a

…e conflicts

removed certain things from yarn.lock

51cebf7

regen snapshot to solve failed task ':metadata-service:restli-servlet…

a3435e7

…-impl:checkRestModel'

update ingest-api build step

beac758

github-actions bot reviewed Jan 25, 2024

View reviewed changes

github-actions bot added product devops ingestion docs smoke_test labels Jan 25, 2024

verbose?

a736218

xiphl added 3 commits January 25, 2024 18:11

.

513d329

remove -vvv

f6478c3

change docker and ci build to mimic ingestion

81d704f

github-actions bot reviewed Jan 29, 2024

View reviewed changes

xiphl added 4 commits January 29, 2024 11:09

missed scan step edit

ef1a6e3

changes

78acb6b

base image change

d1d1566

update entrypoint

7d89dab

github-actions bot reviewed Jan 29, 2024

View reviewed changes

neojunjie closed this Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ingest api 0.12.0 #143

Fix ingest api 0.12.0 #143

xiphl commented Jan 25, 2024

github-actions bot Jan 25, 2024

github-actions bot Jan 25, 2024

github-actions bot Jan 25, 2024

github-actions bot Jan 25, 2024

github-actions bot Jan 25, 2024

github-actions bot Jan 25, 2024

xiphl commented Jan 25, 2024

xiphl commented Jan 25, 2024

github-actions bot Jan 29, 2024

github-actions bot Jan 29, 2024

neojunjie commented Feb 13, 2024

Fix ingest api 0.12.0 #143

Fix ingest api 0.12.0 #143

Conversation

xiphl commented Jan 25, 2024

Checklist

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

github-actions bot Jan 25, 2024

Choose a reason for hiding this comment

xiphl commented Jan 25, 2024

xiphl commented Jan 25, 2024

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

neojunjie commented Feb 13, 2024