DataHub v0.8.11

Release Highlights

• Business Glossary: Phase 1 is feature complete. Full support for UI viewing and API-based edits, no support for UI edits.
• Users and Groups: Just-in-time User and Group provisioning on login (SSO/OIDC), basic Group pages with membership information
• New Integrations: Redash

Notable Changes

• GraphQL and REST API-s are now both served by datahub-metadata-service (new name for gms). Frontend is now a proxy. Container names are not changed.
• Kafka source will no longer tokenize on . in the topic name. This will result in a flat browse experience in UI.
• Airflow lineage emission will only populate specific properties of Tasks and DAGs to limit bloat and avoid leaking environment variables.
• Schema history feature turned off in UI based on feedback from the community. Will re-emerge in a future release!
• Mongodb collections with extremely wide schemas will have schema fields sampled to keep UI responsive.
• Full changelog below.

ChangeLog

• #3156 @swaroopjagadish fix(frontend): replacing broken link for default avatar
• #3154 @swaroopjagadish fix(frontend): fixing broken link to default avatar
• #3153 @swaroopjagadish feat(ingest): adding maxSchemaSize to mongodb source
• #3150 @saxo-lalrishav fix(business-glossary): business glossary visual changes
• #3142 @greysond fix(metadata-service): actually load keys from keystore for elastic connections
• #3110 @frsann feat(ingestion): bring your own SQL parser
• #3146 @jjoyce0510 fix(react): refactoring hasKeySchema computation
• #3145 @swaroopjagadish deps(ingest): upgrade to pick up acryl-pyhive changes
• #3144 @sgomezvillamor fix(profiles): prevent NoneType exception when profiling empty datasets
• #3140 @swaroopjagadish fix(glossary): Make terms searchable and browseable
• #3139 @swaroopjagadish fix(deps): Adding min version to python-dateutil to guard against isoparse failures
• #3135 @dexter-mh-lee fix(kafka): Change consumer id of mae/mce processor
• #3137 @swaroopjagadish fix(airflow): only emit specific keys for airflow lineage properties
• #3131 @jjoyce0510 feat(graphql): migrating GraphQL API to metadata-service (nee GMS)
• #3082 @jjoyce0510 feat(sso): Just-In-Time User & Group Provisioning on SSO Login (oidc)
• #3129 @saxo-lalrishav feat(business-glossary): Business glossary relationship UI
• #3113 @dexter-mh-lee feat(ingest): Add custom browse paths for kafka sources and remove browse lowercase filter
• #2918 @taufiqibrahim feat(ingest): adding redash source
• #3103 @saxo-lalrishav feat(business-glossary): glossary term relationship graphql changes
• #3015 @jjoyce0510 refactor: remove unused gms code, frontend endpoints part 2/4
• #3094 @jjoyce0510 feat(group ui): Basic group search membership in UI
• #3012 @Shikha-Trivedi-Saxo feat(business-glossary): Glossary term relationship backend
• #3049 @neojunjie feat(frontend): logout with oidc
• #3099 @gabe-lyons fix(schema-version): temporarily hide schema version tab
• #3048 @saxo-lalrishav feat(business-glossary): added field level glossary terms
• #3095 @shirshanka fix(ingest): increasing default ingestion REST timeout to 30 seconds
• #3096 @dexter-mh-lee fix(upgrade): Fix MAE consumer and upgrade's dependency issue
• #3092 @jensenity fix(postgres): fix postgres setup to handle existing database